Research log / RL-020

RL-020 Bounded edits use bounded reasoning

Evidence grade BC

Field Value
Date 2026-09-13
Harness version 2026-09-13-short-form-draft-implementation-9 → 2026-09-13-bounded-edit-reasoning-10
Models qwen3:4b-thinking, qwen3:8b; test: MockLlm
Evidence grade B + C
Pre-registered yes

Observation

The generic task weigher classifies every request containing implement as heavy. A complete active draft is edited through a bounded exact-replacement protocol with a host-owned plan and verification step, but the request still inherits high reasoning effort. This spends latency on deciding a workflow the harness has already decided.

Hypotheses

Intervention

Pass weight: normal only for patchMode code calls. Do not disable reasoning globally and do not change generation, repair, or verification.

Method

Capture the completion options in the existing exact-edit regression test and assert normal weight, successful draft mutation, and completed checklist. Run the targeted suite and typecheck. Live time-to-first-code remains a follow-up.

Results

Live follow-up

Ten distinct active-draft edits were split evenly across two local reasoning models at temperature 0 and an 8,192-token context. Output capability was bounded to 512 tokens for the benchmark; the always-thinking profile still reserved another 1,024 tokens on four 4B edit turns.

The live result refutes the practical optimization hypothesis for Ollama-native reasoning models. The harness supplied normal weight, but the native transport only carries a boolean think control, so the effort distinction did not constrain deliberation.

Threats to validity

The ten prompts were split between models rather than repeated on both, so this is not a paired quality comparison. The reply cap differs from the normal model maximum and was chosen to keep the already long benchmark finite. The accepted edit was structurally accepted, not browser-verified for the requested effect.

Conclusion

The harness correctly labels bounded edits as normal effort, but that label is not an effective optimization on Ollama-native reasoning models. For this use case, bounded draft edits should be handed to a non-reasoning coder after the harness creates the plan; reasoning models should be reserved for ambiguous planning and review rather than writing exact replacements.

Follow-ups

Run the same ten prompts through the installed non-reasoning coder and compare accepted edits, browser checks and latency. If supported, route only bounded patch generation to that coder while retaining the selected model for genuinely ambiguous planning or repair.

Reproduce

node --import tsx --test tests/codeConversation.test.ts
npm run typecheck