Research log / RL-014

RL-014 Tail continuation, multi-page recall, abandoned plan state

Evidence grade DC

Field Value
Date 2026-09-13
Harness version 2026-09-13-intent-awareness-1 → 2026-09-13-tail-continuation-2
Models observed: qwen3:4b-thinking, desktop, 4,096-token context (.env default). Tests: MockLlm
Evidence grade D (journal) + C (replay, tests, prompt measurement on the real draft). No live model run after the change.
Pre-registered yes: observation, baseline and hypotheses recorded before implementation
Source record journal turns 117–119, runs 386–388; RL-013 post-deployment

Observation

Journal, 2026-09-13, intent-awareness-1, qwen3:4b-thinking:

Turn Time (UTC) Message Outcome
117 00:17:34 lets create a web app, about dogs, with little write ups on each dog breed, use 10 dog breeds, 10 different pages, use css and styling 125.0 s; 5,609-char reply; draft 5,532 chars, incomplete (ends inside <h2>Dach)
118 00:20:31 continue refused in 12 ms: code_context_capacity, est. prompt 2,875 tok, reply room 1,157 tok
119 00:21:24 continue identical refusal; UI showed Plan · 0/2 with both steps still pending

Three distinct defects:

  1. The multi-page guard (RL-009) did not match "10 different pages", so a request one draft cannot satisfy went to generation.
  2. Continuation is a whole-file replacement. The draft needs ≈ 2,588 tokens of reply room (draft + 128); 1,157 were available. No number of "continue" messages could succeed at this context size — the refusal was correct but a dead end.
  3. The code plan was emitted before the capacity check and never updated, so it displayed unstarted work after the run had stopped.

Baseline (before the change)

Multi-page guard, 11 phrasings (7 multi-page, 4 single-page controls):

Class Guard fired
Multi-page requests 2/7 (only "multiple pages", "multi-page")
Single-page controls 0/4

Continuation at capacity, real stored draft, 4,096 tokens: 0/2 attempts could proceed.

Plan state after refusal: $draft pending.

Hypotheses

Intervention

Method

  1. Replay of the 11 phrasings through the old and new guard.
  2. Prompt measurement on the actual stored draft from turn 117 (conversations state, 5,532 chars) with the harness estimator, 4,096-token window.
  3. Unit tests with synthetic model output: tests/continuation.test.ts (7 tests: guard recall and controls, tail bounds, seam overlap, mid-token join, restart/empty rejection, end-to-end tail continuation, plan failure on refusal); existing continuation-stall tests unchanged.
  4. Full suite, typecheck, build.

Results

H1 — guard recall

Class Before After
Multi-page requests (incl. the reported message) 2/7 7/7 (+ "build a 10 page website": 8/8 in tests)
Single-page controls (+ "page 2 of my notes") 0/4 0/4

H2 — reply room on the real draft (estimated tokens, 4,096 window)

Mode Prompt Reply room Needed
Whole-file rewrite (logged at the time) 2,875 1,157 2,588
Tail continuation 641 (1,570 chars; tail 1,112 chars) 3,391 — remainder only

The tail prompt is bounded by construction (tail ≤ 1,200 chars + request ≤ 600 + fixed system text), so room no longer shrinks as the draft grows.

Mock end-to-end: a 10.6k-char unfinished draft that cannot be rewritten continued in 1 inference; the prompt did not contain the draft's head; the result begins with the original draft and ends with </html>; complete true. A restart reply left draft and revision unchanged.

H3 — after "make the header blue" on a complete draft too large to edit: 0 inferences, last plan event $draft = failed.

Suite — 668 → 675 tests, 0 failures; typecheck clean; build passes.

Threats to validity

Conclusion

Follow-ups

  1. Live: replay turn 117's draft with "continue" on qwen3:4b-thinking and qwen2.5-coder:7b, n = 5 each; score seam validity, </html> reached, and verifier pass counts after joining.
  2. If seams break, add the draft's <style> block and section IDs (bounded) to the tail prompt and re-measure.
  3. Context size: the manifest lists 8,192 for this model and the native transport sends num_ctx, so LOCAL_LLM_CONTEXT_TOKENS=8192 would have allowed a whole rewrite here (≈ 5,200 room). A user configuration decision (VRAM), to be measured rather than defaulted.
  4. Repeated identical refusals: consider answering the second with alternatives rather than the same text.

Reproduce

npm test
npm run typecheck
npm run research:snapshot