| Field | Value |
|---|---|
| Date | 2026-09-14 |
| Harness version | phone-chat-21 → sketch-scope-alignment-22 |
| Models | qwen2.5:1.5b-instruct and qwen3.5:2b via local Ollama, 4,096 context, temperature 0, thinking off; exploratory probe also qwen2.5-coder:7b |
| Evidence grade | B for the real-model scope benchmark (14 distinct scripted phrases per model, one run each); C for 14 unit tests and 14 desktop fixture checks |
| Pre-registered | yes — hypotheses, instrument and cases written before implementation and before the baseline run |
| Source record | LIVE-SKETCH.md; evidence/RL-032/ |
Observation
User, 2026-09-14: “the voice mode works well, but the idea of it building the app in real time needs humanizing, the agent needs to be able to understand scope and alignment”.
- Live sketch sent each finished phrase to a 1.5B model as a stateless JSON patch (
sketchPromptinsrc/agent/LiveSketch.ts). No memory of the goal, audience or agreed scope; no acknowledgement; no way to ask. - The only saved real sketch (
data/live-sketch/f74985b6…/sketch.json, 2026-09-13 13:30) has title “Happy Dogs” under the placeholder subtitle “Describe a place, a feeling, or something you want to make.” and placeholder sections: half-applied intent (n = 1, D). - No live sketch runs are in the journal (
agent_runstask “Live sketch visual change”: 0 rows). RL-030 measured fixtures and three synthetic phrases and already recorded unrequested style changes and unchanged content. - The Spark voice canvas the user says works well keeps kept/proposed cards and asks scope questions (saved transcript: “To keep scope tight, which primary direction should we ship first…”).
- Exploratory probe before design (
evidence/RL-032/exploratory-scope-probe.json; 5 phrases × 3 models, n = 1 each, not controlled): correct scope relation qwen2.5:1.5b-instruct 2/5 (1.1 s warm), qwen3.5:2b 4/5 (1.6 s), qwen2.5-coder:7b 2/5 plus one invalid reply (~3.1 s). No model classified “Hmm, I'm not sure yet, let me think about the colours.” as thinking aloud.
Hypotheses
- H1 — Thinking-aloud and deferral phrases change nothing on the page, and deferrals are recorded as parked scope. Refuted if any such benchmark phrase changes a document field.
- H2 — A phrase that brings a parked idea into the page, or replaces an established goal, leaves the page unchanged and produces one question; a later “yes” applies it and “no” leaves it. Refuted if the page changes before the answer, or no question is produced.
- H3 — Obvious visual phrases change style fields only and stay on the fast path (no understanding call). Refuted if content fields change or a model is called.
- H4 — After the first goal phrase, title, subtitle and sections are not the blank template. Refuted if any placeholder remains.
- H5 (cost) — Obvious visual edits stay ≤ 1 s warm; careful edits ≤ 3 s warm on this laptop.
Intervention
src/agent/SketchScope.ts(new): host-owned scope record (goal, audience, included items, parked items, pending question), persisted per conversation beside the sketch.routePhrasedecides in code, before any inference: answer to a pending question (short yes/no only), thinking aloud, deferral (park with the subject extracted), a parked idea returning (ask), obvious look words (style route), a goal switch once a goal exists (ask), otherwise understanding.styleFromWordsresolves colour, warmth, calm/minimal, spacing, type and layout words without a model.src/agent/LiveSketch.ts:proposereturns either a page candidate or a reply-only outcome (listen, park, ask, decline, noop) that saves scope without a page revision. The understanding call returns relation, reply, question, goal, audience, included items and page words under strict schemas; style keys it adds are stripped. Conflict or unclear → ask, page unchanged; a question attached to a clear edit is ignored. Confirmed parked ideas gain a section if the model omits one; a confirmed goal switch starts from a clean page. With a goal agreed, placeholder copy is replaced from the scope record. Queued thoughts (sent together while one is in flight) are understood line by line; a question stops the rest.src/integrations/sketchModel.ts: per-call format schema and output cap (96 tokens for look-only).src/desktop/agentHost.ts: scope returned with status and start; questions spoken through the existing local Windows synthesiser only for microphone sessions (recognition is muted while it speaks);sketch_scopejournal status for reply-only outcomes.- Renderer: scope strip (Making / For / Includes / Later), the open question in the accent colour, a short reply line, and “Asking you a question · microphone paused” while speaking.
- Not changed: the fixed page template, the render check, commit/undo, model choice default (qwen2.5:1.5b-instruct), no cloud or paid inference, one sketch model on the GPU.
- Prompt cost: the understanding prompt adds the scope record (≈ 60–100 tokens) and asks for more output; look-only and rule-routed phrases now make no model call.
Method
Instrument: scripts/benchmark-sketch-scope.ts, written before implementation. Two distinct scripted conversations (dog rescue; beginner pottery classes), seven phrases each, sent in order through the real LiveSketch.propose/commit path with the real local Ollama model; deterministic checks per phrase (unchanged, style-only, content-only, no placeholders, asks, applies pending). Baseline on the unchanged harness for both models, then after with the same script and models, each run alone on the GPU with no test suite running. Temperature 0 makes repeats identical, so distinct phrases are the cases: n = 14 phrases, 17 checks per model, one run. Full suite baseline 789 passing.
Results
| Condition | qwen2.5:1.5b-instruct before → after | qwen3.5:2b before → after | n |
|---|---|---|---|
| Phrases passing all checks | 3/14 → 13/14 | 2/14 → 13/14 | 14 phrases |
| Checks passing | 3/17 → 16/17 | 4/17 → 16/17 | 17 checks |
| H1 thinking aloud / deferral unchanged | 1/4 → 4/4 | 2/4 → 4/4 | 4 phrases |
| H2 asks before parked idea or goal switch, page unchanged | 0/3 → 3/3 | 0/3 → 3/3 | 3 phrases |
| H2 “yes” applies, “no” leaves | 2/2 → 2/2 (baseline passes accidental) | 1/2 → 2/2 | 2 phrases |
| H3 look phrases change style only | 0/2 → 2/2, 12–17 ms, no model call | 0/2 → 2/2, 14–21 ms, no model call | 2 phrases |
| H4 no placeholders after first goal phrase | 0/2 → 2/2 | 0/2 → 2/2 | 2 phrases |
| Clear audience statement updates the page | fail → fail (needless question) | fail → fail (needless question) | 1 phrase |
| Understanding-call phrase time, warm | 0.5–3.3 s (one incomplete reply) → 1.5–2.3 s | 1.0–1.8 s → 3.2–3.6 s | 4 phrases |
| Full deterministic suite | 789 → 803 passing | 1 run | |
| Desktop fixture checks | 11 → 14 passing | 1 run after final display fix |
Baseline failures included “Um, give me a second.” restyling palette, layout, typeface, density and motif (1.5B), and “Hmm, I'm not sure yet, let me think about the colours.” changing palette, layout and motif. The remaining after failure is the same on both models: “It's mainly for families with young kids.” was read as a conflict with the audience the model itself had written in the first turn.
Evidence record
Raw files under evidence/RL-032/: exploratory-scope-probe.py and .json; scope-benchmark-baseline-*.json and scope-benchmark-after-*.json (every phrase with exact prompts, raw replies, usage, changed fields, scope and verdicts); live-sketch-electron.json, live-sketch-question.png, live-sketch-page.png, live-sketch-mood.png, live-sketch-narrow.png. Not captured: any real user speech or audio, a live spoken question through the Windows synthesiser, or a journal from real use (none exists yet).
User prompts (verbatim)
| # | Where / when | Prompt, exactly as typed | What followed |
|---|---|---|---|
| 1 | Chat, 2026-09-14 | “the voice mode works well, but the idea of it building the app in real time needs humanizing, the agent needs to be able to understand scope and alignment” | Code and journal reading, exploratory probe, three design questions |
| 2 | AskUserQuestion answers | Talk back: “Spoken only when it asks”; timing: “Instant for obvious, careful for rest”; unclear or conflicting: “Ask, don't change (Recommended)” | Design fixed: rules first, look words instant, understanding call for the rest, ask and speak only on questions |
Model prompts and responses (verbatim excerpts)
Prompts are built by understandingPrompt / stylePrompt (src/agent/SketchScope.ts, harness 22) and, for baseline, sketchPrompt (harness 21). Responses below are raw model output before harness processing.
Baseline failure, qwen2.5:1.5b-instruct, Instruction: Hmm, I'm not sure yet, let me think about the colours.:
{ "palette": "ocean", "layout": "centered", "typeface": "serif", "density": "airy", "motif": "dogs",
"sections": [{ "title": "Adopt a Dog", "body": "Find your new furry friend at our local dog rescue. Visit us today and help save a life." }] }
After success, qwen3.5:2b, They said: Hello, I'd like to make a website for my local dog rescue so people can adopt dogs. (excerpt):
{ "relation": "new_goal", "reply": "I understand! You want to create a website specifically for your local dog rescue organization to help people find and adopt new pets.",
"question": "", "goal": "local dog rescue adoption platform", "audience": "potential dog owners looking for their next pet", "title": "Find Your Best Friend", ... }
After success with an ignored stray question, qwen2.5:1.5b-instruct, same phrase (excerpt): "relation": "add", ... "question": "What is the main goal of the website?" — page updated; question discarded by the rule that only conflict/unclear may ask.
After failure (both models), They said: It's mainly for families with young kids.:
{"relation":"conflict","reply":"The website is primarily for people interested in adopting dogs from a local rescue, not families with young kids.","question":"Is the website primarily for families with young kids? ", ...}
{ "relation": "conflict", "reply": "I understand you want to emphasize that this site is primarily for families with young children, but the current goal specifies it is for potential dog owners looking for their next pet. This creates a conflict because the target audience (young kids) and the website's purpose (dog adoption) are not aligned.", "question": "Should we adjust the subtitle or sections to better refle…" }
Rule-routed outcomes (no model call), both models, e.g. Actually let's put the merch shop on the home page right now. → question “Earlier you said sell merch could wait. Put it on the page now?”; Actually, make it about a candle making workshop instead. → “That changes what we're making, from … Start this new idea instead?”
Issues encountered, in order
| # | Issue | How it surfaced | Outcome |
|---|---|---|---|
| 1 | Shell heredocs for new source files failed to parse; nothing was written | bash “unexpected EOF while looking for matching quote” | Switched to direct file writes and edit scripts; verified each edit matched exactly once |
| 2 | Existing liveSketch.test.ts no longer typechecked: propose now returns a union |
tsc TS2339 on .id / .document |
Wrapped calls in asCandidate, which also asserts a candidate was returned. Instrument change: typing plus one added assertion per call; behavioural assertions unchanged |
| 3 | Desktop check “newest combined intent wins” failed: queued thoughts “slow dogs\nuse forest” routed as one look phrase and dropped the content thought | Timeout waiting for “Working version updated”; status “Page unchanged. Keep talking.” | Real regression. Fixed with line-by-line interpretation; new unit test covers queued content + look, and a question stopping later lines |
| 4 | Desktop fixture model replied in the old patch shape; unknown keys are now stripped, not rejected | Fixture review before running | Fixture replies follow the content-route shape; the “invalid” reply is now an invalid motif; undo expectation changed from ocean to sand because the content route no longer applies a model palette. Instrument change |
| 5 | The desktop check wrote into evidence/RL-030/, so re-running would overwrite RL-030's record |
Script review | Output directory moved to evidence/RL-032/; RL-030 files untouched |
| 6 | Screenshot showed the question twice, and “On the page: Nothing yet” contradicting a visible page | Visual review of live-sketch-question.png |
Reply line hidden while a question is open; label renamed “Includes”; desktop check re-run |
| 7 | Some baseline passes were accidental (“No, leave it for later.” changed nothing because the model returned the same values) | Reading baseline replies | Baseline pass counts reported as measured, flagged here |
| 8 | Both models still ask on a clear audience statement | After benchmark | Not fixed; recorded as the open failure |
Assumptions and what turned out to be true
| # | Assumption | Verdict | Evidence |
|---|---|---|---|
| 1 | Thinking aloud, deferral and yes/no answers follow fixed wording that code can recognise | Correct for the benchmark (8/8 rule-routed phrases per model) and 16 held-out unit phrases; untested on the user's own speech | After benchmark; tests/sketchScope.test.ts |
| 2 | Small models cannot be trusted to recognise thinking aloud | Correct | Exploratory probe 0/3 models; baseline changed the page on “Um, give me a second.” |
| 3 | qwen3.5:2b would handle scope better than qwen2.5:1.5b-instruct | Not supported once rules route the conversational moves: both 13/14, and 2B is slower (3.2–3.6 s vs 1.5–2.3 s) | After benchmark |
| 4 | Only the person's words enter scope | Incorrect as implemented: goal and audience fields accept the model's paraphrase on the understanding route, and that paraphrase caused the remaining false conflict | After failure replies above |
| 5 | Obvious look words can be resolved without a model | Correct for 2 benchmark and 4 unit phrases; a single sentence mixing content and look is treated as content (look part ignored) — untested in the benchmark | Unit tests; design limitation |
| 6 | Speaking questions through the existing Windows synthesiser works inside live sketch | Untested: host wiring only; no automated or live check of the spoken question | Not captured |
| 7 | Temperature 0 repeats are identical, so distinct phrases are the right unit | Carried from RL-016, not re-measured | — |
What was correct and what was not
- Correct: routing conversational moves in code before inference; asking instead of editing; look words without a model; placeholder replacement; line-by-line queued thoughts (after the fix).
- Incorrect and corrected: queued thoughts handled as one phrase (issue 3); duplicated question and misleading “On the page” label (issue 6).
- Incorrect and still open: model-paraphrased goal/audience in scope and the resulting needless audience question (assumption 4, issue 8); mixed content + look sentences; spoken question untested.
- Incorrect in the measurement: accidental baseline passes (issue 7); the desktop check's evidence directory (issue 5).
Threats to validity
Scripted phrases were written by the implementer, and the routing rules were written knowing those phrases; the 16 held-out unit phrases reduce but do not remove that overfitting. The user's own speech, accent, pauses and phrasing are not in any measurement. n = 1 per phrase at temperature 0. Timings are warm, on this laptop, with nothing else on the GPU. Desktop checks use a fixture model (C) and typed input, not the microphone. Undo restores page versions, not the scope record.
Conclusion
- H1 supported at grade B for the benchmark phrases on both models (4/4), C for held-out unit phrases.
- H2 supported at grade B: 3/3 questions with the page unchanged, yes/no handled 2/2 on both models.
- H3 supported at grade B: 2/2 look phrases changed style only in ≤ 21 ms with no model call.
- H4 supported at grade B: 2/2 on both models.
- H5 partly refuted: look edits ≤ 21 ms (supported); understanding calls 1.5–2.3 s on qwen2.5:1.5b-instruct (supported) but 3.2–3.6 s on qwen3.5:2b (refuted). The default stays qwen2.5:1.5b-instruct.
- Not established: behaviour on the user's own speech, the spoken question, or understanding beyond the fixed template.
Follow-ups
- Keep the person's own words for goal and audience in scope (quote, don't paraphrase), then rerun the benchmark to see whether the needless audience question disappears.
- Try the live sketch by voice and log which phrases route wrongly; add those as distinct benchmark phrases.
- Handle a single sentence that mixes content and look.
- Check the spoken question end to end with the microphone on.
Reproduce
npx tsx scripts/benchmark-sketch-scope.ts --model qwen2.5:1.5b-instruct --label after
npx tsx scripts/benchmark-sketch-scope.ts --model qwen3.5:2b --label after
npm test
npm run typecheck
npm run check:live-sketch
node --import tsx --test tests/sketchScope.test.ts tests/liveSketch.test.ts
The benchmark requires the two local models in Ollama and overwrites the matching result file; run each model alone on the GPU.