| Field | Value |
|---|---|
| Date | 2026-09-13 |
| Harness version | 2026-09-13-short-form-draft-implementation-9 → 2026-09-13-bounded-edit-reasoning-10 |
| Models | qwen3:4b-thinking, qwen3:8b; test: MockLlm |
| Evidence grade | B + C |
| Pre-registered | yes |
Observation
The generic task weigher classifies every request containing implement as
heavy. A complete active draft is edited through a bounded exact-replacement
protocol with a host-owned plan and verification step, but the request still
inherits high reasoning effort. This spends latency on deciding a workflow the
harness has already decided.
Hypotheses
- H1: A complete-draft patch explicitly supplies normal reasoning weight.
- H2: The edit still uses exact replacements and completes the host-owned draft/check checklist.
- H3: Full builds and debugging retain the existing automatic task weight.
Intervention
Pass weight: normal only for patchMode code calls. Do not disable
reasoning globally and do not change generation, repair, or verification.
Method
Capture the completion options in the existing exact-edit regression test and assert normal weight, successful draft mutation, and completed checklist. Run the targeted suite and typecheck. Live time-to-first-code remains a follow-up.
Results
- H1 supported (C): the active-draft implementation replay supplied
weight: normalto the model client. - H2 supported (C): the exact replacement changed the draft and both
$draftand$checkcompleted. - H3 supported by inspection and regression coverage (C): the override is
confined to
patchMode; full generation retains automatic weighting, and the reasoning-capability suite still passes. - Targeted suites: 98 tests passed, 0 failed (17 code-conversation tests and 81 harness tests). Typecheck passed.
- Live latency and model-quality effects remain unmeasured.
Live follow-up
Ten distinct active-draft edits were split evenly across two local reasoning models at temperature 0 and an 8,192-token context. Output capability was bounded to 512 tokens for the benchmark; the always-thinking profile still reserved another 1,024 tokens on four 4B edit turns.
qwen3:4b-thinking: 0/5 usable edits; median 209.3 s, mean 180.7 s.qwen3:8b: 1/5 harness-accepted edits; median 112.9 s, mean 112.1 s.- Overall: 1/10 accepted, mean 146.4 s, total wall time 24.4 minutes.
- Eight turns exhausted the entire completion allowance on reasoning and returned no answer. One returned an invalid exact edit. The sole accepted edit was not browser-verified for semantic compliance in this instrument.
- The cumulative
reasoningCharsobserver field inlive-models.jsonis not a token count: the UI callback reports a growing snapshot repeatedly. Server usage is authoritative; the ten turns used 9,027 completion tokens.
The live result refutes the practical optimization hypothesis for Ollama-native
reasoning models. The harness supplied normal weight, but the native transport
only carries a boolean think control, so the effort distinction did not
constrain deliberation.
Threats to validity
The ten prompts were split between models rather than repeated on both, so this is not a paired quality comparison. The reply cap differs from the normal model maximum and was chosen to keep the already long benchmark finite. The accepted edit was structurally accepted, not browser-verified for the requested effect.
Conclusion
The harness correctly labels bounded edits as normal effort, but that label is not an effective optimization on Ollama-native reasoning models. For this use case, bounded draft edits should be handed to a non-reasoning coder after the harness creates the plan; reasoning models should be reserved for ambiguous planning and review rather than writing exact replacements.
Follow-ups
Run the same ten prompts through the installed non-reasoning coder and compare accepted edits, browser checks and latency. If supported, route only bounded patch generation to that coder while retaining the selected model for genuinely ambiguous planning or repair.
Reproduce
node --import tsx --test tests/codeConversation.test.ts
npm run typecheck