RL-013 Draft-edit routing, registry-derived capabilities, previous-turn route

Observation

Journal, 2026-09-12 23:29–23:53, app-workflows-15, qwen3:4b-thinking, n = 8 turns (2 failed, 2 fast path):

The first and third were addressed by the unrecorded changes (RL-012a). The fourth shows the mechanism that remained: the chat model had no record of how the previous turn was routed, so it denied the misroute.

Baseline replay on the code as found (2026-09-13, before this change), classifyIntent with code mode on and a complete HTML draft:

Self-description baseline: "what can you do?" returned a fixed string listing notes, tasks, reminders and status. Registered but unlisted: code drafts, web search, workspace file read/edit, terminal read/run. The chat system prompt stated only that no tools exist in this conversation; "can you edit files?" and "can you run commands?" went to inference with no capability information.

Hypotheses

• H1 — With a draft open, an edit verb plus a named page part ("header", "section", "background", …) routes to the code path, while harness nouns and named files keep precedence. Refuted if any of the 2 misrouted edits still routes to task, or any of the precedence controls stops routing to task.
• H2 — Deriving the capability answer from the tool registry makes it list every registered capability class and nothing unregistered, with approval described from tool permission. Refuted if the description diverges from the registry in tests.
• H3 — Supplying a host-recorded PREVIOUS TURN line plus an instruction not to deny it makes the chat model acknowledge a misroute rather than deny it. Refuted if a live model given the misroute context still denies. Not yet tested live.

Intervention

• src/agent/CodeConversation.ts — isDraftEdit (edit verb × page-part lexicon), consulted in code mode only; Intent.ts code-mode rule and AgentRunner writingCode/wasAsked.
• src/agent/Capabilities.ts — summarizeCapabilities / describeCapabilities / capabilityBrief from registered tools; Intent.ts isCapabilityQuestion (general forms plus anchored "can you edit files / run commands / search the web", matched before politeness stripping).
• src/agent/Conversation.ts — recordRoute, persisted in state, cleared by reset(), rendered as PREVIOUS TURN: handled as <a fixed reply | a saved-data lookup | an action | conversation> [using <tool>], status <status>; recorded in AgentRunner.respond and the desktop repair-existing path.
• src/agent/AgentPrompt.ts — chat system prompt gains APP ABILITIES and the previous-turn instruction.
• Unchanged: task-loop prompt, retrieval prompt, code prompt, tool permissions.

Prompt cost (harness estimateTokens, not server-counted), chat system prompt:

That is roughly +290 estimated tokens (+97%) per chat inference on the full desktop, against a 4,096–8,192 window. This conflicts with the AGENTS.md preference for code fixes over prompt text and must be justified by H3's live result or trimmed.

Method

1. Deterministic replay of 9 messages through classifyIntent with code mode and a complete draft, before and after.
2. Unit tests with synthetic model output: tests/capabilities.test.ts (5 tests), one new case in tests/codeConversation.test.ts.
3. Full regression suite, typecheck, build.

Results

• Misrouted draft edits: 2/2 → 0/2. Precedence controls: 4/4 unchanged.
• "make the header blue" through AgentRunner with a mock model: draft replaced, code prompt used, 0 tool calls, 0 tasks.
• Capability answer: 0 inferences; approval text changes with tool permission (tested both ways).
• Chat prompt contains APP ABILITIES and the PREVIOUS TURN line after a prior turn; survives restoreState; cleared by reset.
• Tests 662 → 668, 0 failures; typecheck and build pass.

Threats to validity

• H3 is untested on any real model. Presence of text in a prompt does not establish that qwen3:4b-thinking or a 1.5B model will use it.
• The page-part lexicon is hand-selected from observed phrasings; recall on unseen edit phrasings ("make it pop", "bigger please") is unmeasured and likely low. False positives in long-lived code mode (e.g. "change the background of my desktop") are possible.
• Replay cases were chosen after seeing the failures (selection bias).
• Token figures are estimates; the estimator's error varies by content (RL-002).
• Journal baseline is uncontrolled: 8 turns, one model.

Conclusion

• H1 supported (C) on the replay set.
• H2 supported (C).
• H3 not tested — mechanism in place, effect unknown, prompt cost significant.

Post-deployment observations (grade D)

Journal under intent-awareness-1, desktop, qwen3:4b-thinking, n = 2:

• Neither turn exercises H1–H3 (no draft edit, no capability question, no misroute to acknowledge).
• New finding, unrelated to this intervention: the RL-009 multi-page guard matches only multiple|several|multi- pages; "10 different pages" passed it, spent 125 s and produced an incomplete draft. A recall gap in an earlier intervention's lexicon — the same risk named for isDraftEdit above.

Evidence record

Backfilled 2026-09-13 under protocol step 10, after the entry was written. Raw journal rows (turns 108–116, runs 377–385, with steps and stored model output): evidence/RL-013/journal-turns-runs-steps.json. The replay and prompt-size measurements were run as one-off scripts whose output was not saved; their results are the tables above.

User prompts (verbatim)

Workspace, file and session names in replies are withheld here; the raw rows are kept only in the private evidence file.

In Tiny Agent (journal, app-workflows-15, qwen3:4b-thinking):

To Claude Code, which made the change:

1. "check were we are up to with the local ai agent harness, how we can improve output and further allow the models to understand user intent and the harness and controls they have around them"
2. "okay get them done, fast and efficient, clean and proper"

Model prompts and responses (verbatim excerpts)

Turn 109: the model, given the broad task catalogue, emitted

{"type":"tool","tool":"get_workspaces"}

for a request to style the open draft — a wrong tool on a misrouted turn.

Turn 115 (qwen3:4b-thinking, chat prompt without a route record):

Nope, I didn't get lost! 😄 Just need a quick "what's your main goal for the day?" to help you plan. What's on your mind?

The previous turn (114) had answered a request about building apps with a list of coding-agent sessions. The model denied it.

Turns 108 and 111, visible in the stored steps: all three browser-verified repair attempts in each run were rejected with

Last repair rejected: the model spent its reply budget on reasoning and returned no answer

After the change, no live model was run (see Threats); the only responses are from MockLlm in tests.

Issues encountered, in order

Assumptions and what turned out to be true

What was correct and what was not

• Correct: the routing diagnosis and fix on the replay set; the registry-derived capability answer.
• Untested: whether any model uses the route record; whether the prompt cost is justified; recall on unseen phrasings.
• Incorrect and corrected: the estimated prompt-size row in the first draft of this entry.
• Not addressed here: the reasoning-budget failures in turns 108 and 111 (later RL-015).

Follow-ups

0. Widen the multi-page guard to numeric and "different/separate" page counts; log as a new entry with this turn as its observation.
1. Live A/B for H3: replay the 23:53 two-turn sequence on qwen3:4b-thinking, qwen2.5-coder:7b and qwen2.5:1.5b-instruct, n = 5 each, with and without APP ABILITIES/PREVIOUS TURN; score "acknowledges misroute" vs "denies"; record server prompt tokens.
2. If H3's effect is small, cut APP ABILITIES to the Can: line or include it only when the user asks about abilities or signals confusion.
3. Measure draft-edit recall with a held-out set of ≥ 30 edit phrasings written before looking at the lexicon.
4. Check the journal after a week on intent-awareness-1 for task turns while in code mode (grade D).

Reproduce

npm test
npm run typecheck
npm run research:snapshot -- --out docs/research-log/metrics/2026-09-13.json