| Field | Value |
|---|---|
| Date | 2026-09-10 |
| Harness version | → 2026-09-10-code-routing-2 |
| Models | qwen2.5:1.5b-instruct, qwen2.5-coder:7b |
| Evidence grade | A (narrow: 1 scenario × 3 repeats × 2 models) + C |
| Pre-registered | no (backfilled 2026-09-13) |
| Source record | PYTHON-FOLLOWUPS.md, python-followup-1.5b.json, python-followup-7b.json |
Observation
A transcript showed two Python Snake requests creating tasks, a kilobyte
line of repeated terminal escapes entering draft memory, "run that code" being
treated as generation, and continue failing with an irrelevant data-query
error. Switching to the 7B model did not fix the routing.
Hypotheses (reconstructed)
- H1 — The misroute is a harness rule gap (language + short subject not recognised as a code request), so fixing the rule fixes both models equally.
- H2 — Computing reply room from the exact prompt prevents truncated continuations without enlarging context.
Intervention
Code detection accepts python snake game; execution requests resolved before
loading draft context (truthful "no runtime connected"); you generate it uses
the active goal; runaway-escape rejection; availableReplyTokens for code
requests with refusal before inference when a replacement cannot fit; tkinter
preference for Windows Python GUIs.
Method
npm run sim -- python-followup 3 on each model; LOCAL_LLM_MODEL set per
process. Checks: fenced Python, no task writes, truthful runtime capability,
acknowledgement handling.
Results
| Model | Runs passed |
|---|---|
| qwen2.5:1.5b-instruct | 3/3 |
| qwen2.5-coder:7b | 3/3 |
Unit tests 385/385. Journal under this version: 14 turns, 0 failed (all 7B).
Threats to validity
The scenario does not test whether Snake is playable. Journal n is small and uncontrolled.
Conclusion
H1 supported: the same rule fix brought both model sizes to 3/3 — the failure was not model capacity. H2 supported by harness tests (C).
Follow-ups
Functional verification of generated code (→ RL-010).
Reproduce
npm run sim -- python-followup 3