Research log
How much can a small local model really do?
Every change meant to make small models more capable through the harness is recorded here: what we saw, what we predicted, what we changed, how we measured it, and what happened, including negative and refuted results.
Method
Every change that can alter model-facing behaviour (routing, prompts, context, budgets, repair, verification, tool catalogue) gets an entry.
- Observe first. Cite the triggering evidence: journal rows
(
agent_turns,agent_runs), an eval report, or a reproduction. Quote the user message and reply verbatim where short. - Write the hypothesis before the change. State a falsifiable prediction
("with a draft open,
make the header blueroutes tochat"). Backfilled entries say plainly that their hypotheses were reconstructed afterwards. - Record a baseline on the unchanged harness: replay, test count, eval
pass rate or journal metrics (
npm run research:snapshot). - Bump
HARNESS_VERSION(src/agent/HarnessVersion.ts) so turns logged before and after can be separated. - Measure after with the same instrument. If the instrument itself changed (a stricter verifier, a new test), say so — results across instruments are not comparable.
- Report n, model, context size and configuration for every number. One run is a reproduction, not a reliability rate.
- Keep negative and null results. A fix that did not work, a hypothesis refuted, a measurement that was confounded — these are findings.
- Separate what was verified from what was assumed. Unit tests with a mock model establish harness behaviour, not model behaviour.
- Check the public claims. The website states limitations and
capabilities in plain language. Go through
website/CLAIMS.md: mark any claim this entry supports, weakens or overturns, update the page, and moveLast reviewed throughto this entry. The website build warns while it lags. - Keep the evidence record — every entry, no exceptions. It is the most
important proof of improvement this project has. Record, verbatim: the user
prompts that led to and shaped the change; the model prompts and responses
(at least one success and every distinct failure, marked raw or
post-processing); every issue met, including tooling and measurement
problems; every assumption made, with a verdict — correct, incorrect, not
supported or untested — and its evidence; and a summary of what was correct
and what was not. Save raw outputs under
evidence/RL-NNN/and state what was not captured. An entry without an evidence record is not finished. Entries are published on the website, so withhold anything from the user's own machine or life: document, file, folder, workspace, project and session names, note and reminder contents, contacts, device, account or network identifiers, and secrets. Replace them with a bracketed description such as(list of workspaces; names withheld); raw data stays inevidence/RL-NNN/, which is never published.
| Grade | What it means |
|---|---|
| A | Controlled and repeated: fixed model, version and settings; at least 3 runs per condition; measured before and after with the same instrument. |
| B | Live reproduction: a real model run once or twice, or a single before-and-after pair. |
| C | Harness only: deterministic replay or tests with simulated model output. Proves routing and state, not model quality. |
| D | Observational: patterns in real use, uncontrolled. Motivates work; does not confirm it. |
An entry can carry several grades. Each one names the claim it supports.
Findings so far
- Most real failures were routing and harness state, not model reasoning. Switching 1.5B → 7B did not fix code requests becoming tasks (RL-004); tool selection errors disappeared when the harness narrowed the catalogue.
- Evaluate through the real host. CLI evals passed while the desktop silently dropped search results (RL-006).
- Measurement instruments drift. A strengthened verifier turned a recorded 6/6 into a failure (RL-012). Re-check saved artifacts with the current instrument before comparing.
- Configuration fixes are necessary, not sufficient. Correcting Ollama's 4,096 vs 8,192 context mismatch did not change behavioural failures (RL-010).
- Reasoning models trade answer budget for deliberation. Budget exhaustion
is the most frequent recorded failure message for
qwen3:4b-thinking(RL-009, RL-013, RL-015). It has its own failure category from RL-015. - The configured value is not the served value. The harness resolved 8,192
tokens while Ollama served 4,096, because a background ping used a route
that cannot carry the setting (RL-015). Check
/api/ps, not configuration. - Split work by what each model is good at, and give the GPU to one at a time. On an 8 GB card the coder model writes no faster than the 8B thinking model, but it spends no tokens reasoning: writing the same pages, the thinking model took ≈ 2.4× as long per page and exhausted its budget on the longest one (RL-016). Load each model as its own step; a load inside a request is charged to its first-token clock.
- Temperature 0 makes repeats identical. Repeating a request measures timing variance and reproducibility, not reliability. Use distinct requests (RL-016).
- Benchmark the hardware state as well as the model. A competing 14B Ollama load caused four 45-second empty starts; the same styling scenario passed 5/5 once the intended 1.5B model held the GPU (RL-017).
- Decide conversational moves in code before asking a small model. Thinking aloud, deferral and short yes/no answers were misread by every model probed, but follow fixed wording. Routing them first took scripted scope handling from 3/14 and 2/14 to 13/14 on both small models and removed model calls from those turns; the larger model then gained nothing (RL-032).
Log index
| Entry | Date | Intervention | Grade | Outcome |
|---|---|---|---|---|
| RL-001 | 2026-09-09 | Evidence packet format for answer turnspre-journal |
B | Readable fields 4/4 vs compact rows 3/4; mixed default kept |
| RL-002 | 2026-09-09 | Deterministic totals, request normalisation, token-estimate calibrationpre-journal |
B | Counts answered with 0 inference; estimator bias is workload-dependent |
| RL-003 | 2026-09-10 | Whole draft in context; explain/continue as separate operationsartifact-context-1 |
A narrow | 15/15 scripted turns; game-edit scenario still failing |
| RL-004 | 2026-09-10 | Code-request routing, execution requests, reply-room accountingcode-routing-2 |
A narrow | 3/3 repeats on 1.5B and 7B; a bigger model did not fix routing |
| RL-005 | 2026-09-10 | Identity corrections, catalogue-grounded descriptions, typed search outcomesgrounded-followups-3 |
AB narrow | 24/24 scripted turns; a reviewer's diagnosis was refuted |
| RL-006 | 2026-09-10 | Desktop source reader for web evidencedesktop-search-4 |
BC | Desktop-only defect invisible to CLI evals |
| RL-007 | 2026-09-12 | Model capability manifest, reasoning plan, timeouts, output contract, failure taxonomy≈ -5 (unrecorded) |
CD | Fixed the observed 400s/timeouts by construction; live effect confounded |
| RL-008 | 2026-09-12 | Host-owned multi-step checklistschecklists-9 |
BC | 1 combined note → 2 distinct notes (n=1) |
| RL-009 | 2026-09-12 | Continuation guards, stall stop, reasoning allowancecode-progress-11 |
BD | 73 → 759 chars in one live run; reasoning-budget errors recur later |
| RL-010 | 2026-09-12 | Element-scoped verified repair; native Ollama context≈ -12 |
B | 7B 10/12 → 12/12; 1.5B no gain; context fix alone did not help |
| RL-011 | 2026-09-12 | Automatic browser checks and bounded repairs in chatverified-code-13 |
B | 10/10 checks on one live 7B run |
| RL-012 | 2026-09-12 | Calculator/task probes, repair evidence, rollbackapp-workflows-15 |
B | Reactive filters 5/6 after 3 rejected repairs (negative) |
| RL-012a | 2026-09-13 | Unrecorded changes, 00:43–00:55app-workflows-15 |
— | Reconstructed; not attributable |
| RL-013 | 2026-09-13 | Draft-edit routing, registry-derived capabilities, previous-turn routeintent-awareness-1 |
CD | 2/2 misrouted edits fixed in replay; model effect unmeasured |
| RL-014 | 2026-09-13 | Tail continuation, multi-page guard recall, failed plan statetail-continuation-2 |
CD | Guard 2/7 → 7/7; real draft reply room 1,157 → 3,391 tok; seam quality untested live |
| unlogged | 2026-09-13 | Per-model context default, patch edits for complete drafts (concurrent session)profiled-context-patch-edit-1 |
— | Not logged |
| RL-015 | 2026-09-13 | Native keep-warm load, tail continuation for reasoning models, draft outline, failure category and attributionreasoning-continuation-3 |
BCD | Keep-warm ping reverted served context 8,192 → 4,096 (n=2); reasoning models now continue from the tail |
| RL-016 | 2026-09-13 | Planner model plans, coder model writes each page; one model on the GPU at a time; host-only save after approvaldelegated-pages-4 |
BC | 16/16 pages passed structural checks on 4 distinct requests; the thinking model writing the same pages stopped at 2/4 (budget exhausted); checks missed styling and content defects |
| RL-017 | 2026-09-13 | Native timing telemetry, direct named tasks, casual task recall, CSS-sized generic styling editscss-sized-edits-6 |
AC | 40 → 42 of 44 scenarios; tasks and styling fixed; 237/237 final calls measured; GPU contention recorded separately |
| RL-018 | 2026-09-13 | Bounded structural workspace map before recent file rowsworkspace-map-7 |
C | Landmarks and omission count survived the 600-character observation cap in a 65-file fixture; live model navigation untested |
| RL-019 | 2026-09-13 | Agreement-prefixed implementation requests edit the active draftexplicit-draft-implementation-8 |
CD | Reported request now reaches exact editing and completes the host checklist; live model rerun pending |
| RL-020 | 2026-09-13 | Bounded active-draft edits use normal rather than heavy reasoning effortbounded-edit-reasoning-10 |
BC | Live reasoning models accepted 1/10 edits and averaged 146.4 s; an effort hint is ineffective over Ollama native |
| RL-021 | 2026-09-13 | Conversation-scoped reuse of observed repair outcomes, with fresh verifier checksverified-repair-experience-11 |
C | 713/713 tests; identical repeat used 0 model calls and 2 verifier calls; stale experience evicted on concurrent edits |
| RL-022 | 2026-09-13 | Typed verification records linking observations to exact source revisionsrevision-linked-verification-12 |
C | 717/717 tests; one repair records 6 linked entries with no extra inference; concurrent changes fail before publication |
| RL-023 | 2026-09-13 | Bounded rendered preview review for vision-capable modelsvision-preview-verification-13 |
C | Image transport, gating and real PNG capture pass; live visual quality unmeasured |
| RL-024 | 2026-09-13 | Draft-backed success, research continuation, grounded status and cited health-copy guardgrounded-page-build-completion-14 |
CD | Exact prompts route 2/2; 6/6 new regressions and 734/734 full tests pass; live Qwen rerun pending |
| RL-025 | 2026-09-13 | Voice conversation, connected idea canvas and revision-checked local build handoffvoice-idea-canvas-15 |
C | Covered transitions pass; live audio, model quality and fluidity unmeasured |
| RL-026 | 2026-09-13 | Optional Spark (Codex ChatGPT sign-in) planning with local Windows speechspark-local-voice-16 |
BC | One live Spark text reply in 4,762 ms; 757/757 tests; real microphone quality unmeasured |
| RL-027 | 2026-09-13 | Remembered microphone selection, local level and dictation test, bounded native PCMmicrophone-selection-17 |
C | Synthetic native PCM recognition passes after two stream defects fixed; user microphone accuracy unmeasured |
| RL-028 | 2026-09-13 | Local Whisper recognition and voice checklocal-whisper-18 |
BCD | Two distinct synthetic sentences matched on local GPU; user accent accuracy untested |
| RL-029 | 2026-09-13 | Measured microphone feedback and persistent uncertainty reviewlive-voice-input-19 |
BCD | Overwritten-review defect fixed in fixtures; original no-response report not causally explained |
| RL-030 | 2026-09-13 | Local voice to bounded live visual sketch patcheslive-sketch-20 |
BC | Three distinct synthetic speech examples; neither tested model reliably followed every instruction; accent and sustained latency unmeasured |
| RL-031 | 2026-09-13 | Bounded private HTTPS phone Chat, one-use QR, PC approval and revocationphone-chat-21 |
C | 789/789 tests; six actual HTTPS/UI checks with synthetic model output, iOS untested |
| RL-032 | 2026-09-14 | Host-owned scope record; thinking-aloud, deferral, parked-idea and goal-switch rules before inference; understanding route asks on conflict; look words without inferencesketch-scope-alignment-22 |
BC | Scripted phrases 3/14 → 13/14 (qwen2.5:1.5b-instruct) and 2/14 → 13/14 (qwen3.5:2b); both still ask a needless question on a clear audience statement |
Real-use journal snapshot
Turns recorded per harness version on the developer's machine, up to 2026-09-13. D Observational only. Versions saw different requests, models and phrasings, so these rates are not a comparison between versions, and small n makes them volatile.
| Harness version | Turns (n) | Failed | Fast path | Failure rate |
|---|---|---|---|---|
2026-09-10-artifact-context-1 | 11 | 3 | 1 | 27% |
2026-09-10-code-routing-2 | 14 | 0 | 4 | 0% |
2026-09-10-grounded-followups-3 | 8 | 0 | 2 | 0% |
2026-09-10-desktop-search-4 | 63 | 10 | 14 | 16% |
2026-09-12-modular-workspace-6 | 2 | 0 | 1 | 0% |
2026-09-12-live-editing-8 | 3 | 1 | 0 | 33% |
2026-09-12-checklists-9 | 1 | 1 | 0 | 100% |
2026-09-12-studio-10 | 2 | 1 | 0 | 50% |
2026-09-12-code-progress-11 | 1 | 0 | 0 | 0% |
2026-09-12-verified-code-13 | 2 | 1 | 1 | 50% |
2026-09-12-app-workflows-15 | 8 | 2 | 2 | 25% |
2026-09-13-intent-awareness-1 | 4 | 2 | 0 | 50% |
2026-09-13-profiled-context-patch-edit-1 | 1 | 1 | 0 | 100% |
Failed runs by recorded category: unrecorded 21, unknown 6, context_overflow 1. unrecorded means the run predates failure categories. Closing that gap is an open follow-up.