Research log
How much can a small local model really do?
Every change meant to make small models more capable through the harness is recorded here: what we saw, what we predicted, what we changed, how we measured it, and what happened, including negative and refuted results.
Method
Every change that can alter model-facing behaviour (routing, prompts, context, budgets, repair, verification, tool catalogue) gets an entry.
- Observe first. Cite the triggering evidence: journal rows
(
agent_turns,agent_runs), an eval report, or a reproduction. Quote the user message and reply verbatim where short. - Write the hypothesis before the change. State a falsifiable prediction
("with a draft open,
make the header blueroutes tochat"). Backfilled entries say plainly that their hypotheses were reconstructed afterwards. - Record a baseline on the unchanged harness: replay, test count, eval
pass rate or journal metrics (
npm run research:snapshot). - Bump
HARNESS_VERSION(src/agent/HarnessVersion.ts) so turns logged before and after can be separated. - Measure after with the same instrument. If the instrument itself changed (a stricter verifier, a new test), say so — results across instruments are not comparable.
- Report n, model, context size and configuration for every number. One run is a reproduction, not a reliability rate.
- Keep negative and null results. A fix that did not work, a hypothesis refuted, a measurement that was confounded — these are findings.
- Separate what was verified from what was assumed. Unit tests with a mock model establish harness behaviour, not model behaviour.
- Check the public claims. The website states limitations and
capabilities in plain language. Go through
website/CLAIMS.md: mark any claim this entry supports, weakens or overturns, update the page, and moveLast reviewed throughto this entry. The website build warns while it lags.
| Grade | What it means |
|---|---|
| A | Controlled and repeated: fixed model, version and settings; at least 3 runs per condition; measured before and after with the same instrument. |
| B | Live reproduction: a real model run once or twice, or a single before-and-after pair. |
| C | Harness only: deterministic replay or tests with simulated model output. Proves routing and state, not model quality. |
| D | Observational: patterns in real use, uncontrolled. Motivates work; does not confirm it. |
An entry can carry several grades. Each one names the claim it supports.
Findings so far
- Most real failures were routing and harness state, not model reasoning. Switching 1.5B → 7B did not fix code requests becoming tasks (RL-004); tool selection errors disappeared when the harness narrowed the catalogue.
- Evaluate through the real host. CLI evals passed while the desktop silently dropped search results (RL-006).
- Measurement instruments drift. A strengthened verifier turned a recorded 6/6 into a failure (RL-012). Re-check saved artifacts with the current instrument before comparing.
- Configuration fixes are necessary, not sufficient. Correcting Ollama's 4,096 vs 8,192 context mismatch did not change behavioural failures (RL-010).
- Reasoning models trade answer budget for deliberation. Budget exhaustion
is the most frequent recorded failure message for
qwen3:4b-thinking(RL-009, RL-013, RL-015). It has its own failure category from RL-015. - The configured value is not the served value. The harness resolved 8,192
tokens while Ollama served 4,096, because a background ping used a route
that cannot carry the setting (RL-015). Check
/api/ps, not configuration.
Log index
| Entry | Date | Intervention | Grade | Outcome |
|---|---|---|---|---|
| RL-001 | 2026-09-09 | Evidence packet format for answer turnspre-journal |
B | Readable fields 4/4 vs compact rows 3/4; mixed default kept |
| RL-002 | 2026-09-09 | Deterministic totals, request normalisation, token-estimate calibrationpre-journal |
B | Counts answered with 0 inference; estimator bias is workload-dependent |
| RL-003 | 2026-09-10 | Whole draft in context; explain/continue as separate operationsartifact-context-1 |
A narrow | 15/15 scripted turns; game-edit scenario still failing |
| RL-004 | 2026-09-10 | Code-request routing, execution requests, reply-room accountingcode-routing-2 |
A narrow | 3/3 repeats on 1.5B and 7B; a bigger model did not fix routing |
| RL-005 | 2026-09-10 | Identity corrections, catalogue-grounded descriptions, typed search outcomesgrounded-followups-3 |
AB narrow | 24/24 scripted turns; a reviewer's diagnosis was refuted |
| RL-006 | 2026-09-10 | Desktop source reader for web evidencedesktop-search-4 |
BC | Desktop-only defect invisible to CLI evals |
| RL-007 | 2026-09-12 | Model capability manifest, reasoning plan, timeouts, output contract, failure taxonomy≈ -5 (unrecorded) |
CD | Fixed the observed 400s/timeouts by construction; live effect confounded |
| RL-008 | 2026-09-12 | Host-owned multi-step checklistschecklists-9 |
BC | 1 combined note → 2 distinct notes (n=1) |
| RL-009 | 2026-09-12 | Continuation guards, stall stop, reasoning allowancecode-progress-11 |
BD | 73 → 759 chars in one live run; reasoning-budget errors recur later |
| RL-010 | 2026-09-12 | Element-scoped verified repair; native Ollama context≈ -12 |
B | 7B 10/12 → 12/12; 1.5B no gain; context fix alone did not help |
| RL-011 | 2026-09-12 | Automatic browser checks and bounded repairs in chatverified-code-13 |
B | 10/10 checks on one live 7B run |
| RL-012 | 2026-09-12 | Calculator/task probes, repair evidence, rollbackapp-workflows-15 |
B | Reactive filters 5/6 after 3 rejected repairs (negative) |
| RL-012a | 2026-09-13 | Unrecorded changes, 00:43–00:55app-workflows-15 |
— | Reconstructed; not attributable |
| RL-013 | 2026-09-13 | Draft-edit routing, registry-derived capabilities, previous-turn routeintent-awareness-1 |
CD | 2/2 misrouted edits fixed in replay; model effect unmeasured |
| RL-014 | 2026-09-13 | Tail continuation, multi-page guard recall, failed plan statetail-continuation-2 |
CD | Guard 2/7 → 7/7; real draft reply room 1,157 → 3,391 tok; seam quality untested live |
| unlogged | 2026-09-13 | Per-model context default, patch edits for complete drafts (concurrent session)profiled-context-patch-edit-1 |
— | Not logged |
| RL-015 | 2026-09-13 | Native keep-warm load, tail continuation for reasoning models, draft outline, failure category and attributionreasoning-continuation-3 |
BCD | Keep-warm ping reverted served context 8,192 → 4,096 (n=2); reasoning models now continue from the tail |
| RL-016 | 2026-09-13 | Delegated page builds: planner and coder models taking turns on one GPU2026-09-13-reasoning-continuation-3 → 2026-09-13-delegated-pages-4 |
BC | Not yet summarised in the log index. |
Real-use journal snapshot
Turns recorded per harness version on the developer's machine, up to 2026-09-13. D Observational only. Versions saw different requests, models and phrasings, so these rates are not a comparison between versions, and small n makes them volatile.
| Harness version | Turns (n) | Failed | Fast path | Failure rate |
|---|---|---|---|---|
2026-09-10-artifact-context-1 | 11 | 3 | 1 | 27% |
2026-09-10-code-routing-2 | 14 | 0 | 4 | 0% |
2026-09-10-grounded-followups-3 | 8 | 0 | 2 | 0% |
2026-09-10-desktop-search-4 | 63 | 10 | 14 | 16% |
2026-09-12-modular-workspace-6 | 2 | 0 | 1 | 0% |
2026-09-12-live-editing-8 | 3 | 1 | 0 | 33% |
2026-09-12-checklists-9 | 1 | 1 | 0 | 100% |
2026-09-12-studio-10 | 2 | 1 | 0 | 50% |
2026-09-12-code-progress-11 | 1 | 0 | 0 | 0% |
2026-09-12-verified-code-13 | 2 | 1 | 1 | 50% |
2026-09-12-app-workflows-15 | 8 | 2 | 2 | 25% |
2026-09-13-intent-awareness-1 | 4 | 2 | 0 | 50% |
2026-09-13-profiled-context-patch-edit-1 | 1 | 1 | 0 | 100% |
Failed runs by recorded category: unrecorded 21, unknown 6, context_overflow 1. unrecorded means the run predates failure categories. Closing that gap is an open follow-up.