| Field | Value |
|---|---|
| Date | 2026-09-13 |
| Harness version | 2026-09-13-intent-awareness-1 → 2026-09-13-tail-continuation-2 |
| Models | observed: qwen3:4b-thinking, desktop, 4,096-token context (.env default). Tests: MockLlm |
| Evidence grade | D (journal) + C (replay, tests, prompt measurement on the real draft). No live model run after the change. |
| Pre-registered | yes: observation, baseline and hypotheses recorded before implementation |
| Source record | journal turns 117–119, runs 386–388; RL-013 post-deployment |
Observation
Journal, 2026-09-13, intent-awareness-1, qwen3:4b-thinking:
| Turn | Time (UTC) | Message | Outcome |
|---|---|---|---|
| 117 | 00:17:34 | lets create a web app, about dogs, with little write ups on each dog breed, use 10 dog breeds, 10 different pages, use css and styling | 125.0 s; 5,609-char reply; draft 5,532 chars, incomplete (ends inside <h2>Dach) |
| 118 | 00:20:31 | continue | refused in 12 ms: code_context_capacity, est. prompt 2,875 tok, reply room 1,157 tok |
| 119 | 00:21:24 | continue | identical refusal; UI showed Plan · 0/2 with both steps still pending |
Three distinct defects:
- The multi-page guard (RL-009) did not match "10 different pages", so a request one draft cannot satisfy went to generation.
- Continuation is a whole-file replacement. The draft needs ≈ 2,588 tokens of reply room (draft + 128); 1,157 were available. No number of "continue" messages could succeed at this context size — the refusal was correct but a dead end.
- The code plan was emitted before the capacity check and never updated, so it displayed unstarted work after the run had stopped.
Baseline (before the change)
Multi-page guard, 11 phrasings (7 multi-page, 4 single-page controls):
| Class | Guard fired |
|---|---|
| Multi-page requests | 2/7 (only "multiple pages", "multi-page") |
| Single-page controls | 0/4 |
Continuation at capacity, real stored draft, 4,096 tokens: 0/2 attempts could proceed.
Plan state after refusal: $draft pending.
Hypotheses
- H1 — Matching page counts (digits and number words), "separate/different pages", "N-page" and "a page for each", with "web app" as a site noun, raises guard recall to 7/7 with 0/4 false positives. Refuted if any listed case is missed or any control fires.
- H2 — Sending only the draft's tail (≤ 1,200 chars, line-aligned) and joining the model's remainder in code gives an unfinished draft reply room that is independent of draft length, so the observed draft becomes continuable at 4,096 tokens. Refuted if the tail prompt for the real draft leaves < 256 tokens, or the join corrupts the draft in tests.
- H3 — Marking
$draftfailed on a capacity refusal removes the pending plan display. Refuted if the last plan event after a refusal still shows$draftpending.
Intervention
CodeConversation.ts—isMultiPageRequest(replaces an inline regex inAgentRunner; its reply no longer hard-codes "dogs homepage").CodeConversation.ts—continuationTail(last ≤ 1,200 chars from a line start) andappendContinuation: extracts the fenced fragment; rejects a restart (<!doctype/<html) or an empty addition; removes the longest overlap (≥ 12 chars) between the draft's end and the fragment's start; joins with a newline only when no overlap was found and the draft ends on>,;,{or}; HTML completion means</html>present, trailing text after it dropped.AgentPrompt.ts—buildContinuationMessages: 5-line system prompt, original request (≤ 600 chars), tail.AgentRunner.ts—tailModewhen the follow-up iscontinue, the draft is incomplete and a whole replacement cannot fit; the draft is then built byappendContinuation; live partial chunks are suppressed in tail mode (a fragment would briefly replace the editor buffer); existing no-progress, stall-counter, runaway-escape and 12,000-char guards still apply. Step log recordsoperation: continue_tailandtailModeon refusals.AgentRunner.ts— capacity refusal marks$draftfailed and re-emits the plan.- Unchanged: context size, whole-file edits for complete drafts (still refused when they cannot fit), model selection.
Method
- Replay of the 11 phrasings through the old and new guard.
- Prompt measurement on the actual stored draft from turn 117 (
conversationsstate, 5,532 chars) with the harness estimator, 4,096-token window. - Unit tests with synthetic model output:
tests/continuation.test.ts(7 tests: guard recall and controls, tail bounds, seam overlap, mid-token join, restart/empty rejection, end-to-end tail continuation, plan failure on refusal); existing continuation-stall tests unchanged. - Full suite, typecheck, build.
Results
H1 — guard recall
| Class | Before | After |
|---|---|---|
| Multi-page requests (incl. the reported message) | 2/7 | 7/7 (+ "build a 10 page website": 8/8 in tests) |
| Single-page controls (+ "page 2 of my notes") | 0/4 | 0/4 |
H2 — reply room on the real draft (estimated tokens, 4,096 window)
| Mode | Prompt | Reply room | Needed |
|---|---|---|---|
| Whole-file rewrite (logged at the time) | 2,875 | 1,157 | 2,588 |
| Tail continuation | 641 (1,570 chars; tail 1,112 chars) | 3,391 | — remainder only |
The tail prompt is bounded by construction (tail ≤ 1,200 chars + request ≤ 600 + fixed system text), so room no longer shrinks as the draft grows.
Mock end-to-end: a 10.6k-char unfinished draft that cannot be rewritten continued in 1 inference; the prompt did not contain the draft's head; the result begins with the original draft and ends with </html>; complete true. A restart reply left draft and revision unchanged.
H3 — after "make the header blue" on a complete draft too large to edit: 0 inferences, last plan event $draft = failed.
Suite — 668 → 675 tests, 0 failures; typecheck clean; build passes.
Threats to validity
- H2 is verified for room, not for quality. Whether
qwen3:4b-thinking(which also spends reply tokens on reasoning, +1,024 requested) continues seamlessly from a tail it cannot see the head of — consistent class names, closing the right tags — is untested. The model sees ≈ 1,100 chars, not the page's CSS or structure. - Seam repair is heuristic: overlap detection handles repetition of the draft's end, not a model that paraphrases it, and a join without overlap can split a token.
</html>present is a delimiter check; verified-website checks still run afterwards in the desktop (RL-011).- Guard phrasings were written by the same author as the lexicon; real recall on unseen phrasing is unknown.
- Token counts are estimates (RL-002).
Conclusion
- H1 supported (C).
- H2 supported for reply room (C, measured on the real draft); continuation quality not tested.
- H3 supported (C).
Follow-ups
- Live: replay turn 117's draft with "continue" on
qwen3:4b-thinkingandqwen2.5-coder:7b, n = 5 each; score seam validity,</html>reached, and verifier pass counts after joining. - If seams break, add the draft's
<style>block and section IDs (bounded) to the tail prompt and re-measure. - Context size: the manifest lists 8,192 for this model and the native transport sends
num_ctx, soLOCAL_LLM_CONTEXT_TOKENS=8192would have allowed a whole rewrite here (≈ 5,200 room). A user configuration decision (VRAM), to be measured rather than defaulted. - Repeated identical refusals: consider answering the second with alternatives rather than the same text.
Reproduce
npm test
npm run typecheck
npm run research:snapshot