RL-017 Local simulation benchmark and failure-driven harness improvement

Observation

The existing simulator records scenario pass/fail and wall time, but not the server token counts and durations needed to compare useful work per second. RL-016 measured isolated writing speed for six models. It did not measure speed and task success together across the conversational harness.

Hypotheses

• H1 — native measurements are complete. Ollama's final response frames provide prompt tokens, completion tokens, prompt-evaluation duration, generation duration and load duration for at least 95% of model calls in the simulation. Refuted if more than 5% of calls lack any of those fields.
• H2 — repetition exposes a real 1.5B reliability gap. Three repeats of every current distinct scenario produce at least one failed or intermittent non-known scenario on qwen2.5:1.5b-instruct. Refuted if every non-known scenario passes all three repeats.
• H3 — direct task routing fixes the selected cluster. The largest repeatable actionable cluster was explicit task naming: add a task to review the game and stick a reminder on to check the game server were both routed to the broad action loop, which chose write_note in 3/3 repeats for each request (0/6 correct). A narrow extractor will send those no-date forms directly to create_task. Prediction: each affected scenario passes 5/5, the stored titles omit the instruction prefix, no note is written, and the full simulator and test suite gain no new failure.
• H4 — casual task-list routing fixes the exposed lookup error. In the H3 focused run, the direct write was correct 5/5, but wats on my list selected read_notes 5/5 and therefore missed the new task. Adding that exact casual family to the existing deterministic task-list route will make the lookup select list_tasks 5/5. Prediction: the task creation and lookup pair is correct 5/5 without changing note-list routing; a whole-scenario pass remains contingent on independent chat-model availability.
• H5 — a CSS-sized contract fixes generic styling follow-ups. In both full three-repeat runs, the styling scenario passed 0/3 because the 1.5B model's long exact-replacement JSON was unterminated or structurally invalid. For a generic styling request, ask for CSS rules only, reject markup/imports, and insert the bounded CSS into the preserved HTML in code. Prediction: the styling scenario passes 5/5, each accepted draft contains the model's CSS, and specific content edits continue to use exact replacements.

Intervention

The baseline ran before any routing change. Measurement-only changes add optional timing fields to the existing usage observer and aggregate them in npm run sim; they do not alter prompts, routing, budgets or model requests. The interventions were: direct creation for explicitly named undated tasks; one deterministic fast path for casual task-list wording; and a CSS-only model contract for generic visual restyling, with bounded CSS validation and host-side insertion into the preserved HTML. Specific code changes still use exact replacements.

Method

1. Run all simulator scenarios three times on the configured 1.5B model. Each scenario uses a fresh disposable database and synthetic content.
2. Record every inference's server token counts and nanosecond durations, aggregate prompt and generation throughput, and read the served context and GPU allocation from Ollama's local /api/ps endpoint.
3. Select the largest non-known repeatable failure cluster. State H3's concrete prediction and baseline before changing model-facing code.
4. Add the narrow deterministic guard and regression case, bump HARNESS_VERSION, then rerun the cluster five times and the full simulator.
5. Run unit tests, typecheck and build. Review ../website/CLAIMS.md.

Results

Baseline — all scenarios, three repeats

• 44 scenarios, 414 turns, 250 model calls.
• 250/250 calls carried all native timing fields (100% coverage).
• 76,775 prompt tokens at 9,885.8 tokens/s and 14,677 generated tokens at 90.5 tokens/s; 200,033 ms end-to-end wall time.
• 40/44 scenarios passed, one failure was marked known, and three unmarked scenarios failed reproducibly.
• Explicit task/reminder wording selected write_note in 6/6 affected turns.

H3 focused run — casual conversation, five repeats

• Direct task creation selected create_task, stored check the game server, and wrote no note in 5/5 repeats.
• The full scenario passed 0/5 because wats on my list then selected read_notes in 5/5 repeats. One independent greeting call also returned no output before the 45-second stall limit.
• H3's mutation prediction is supported; its stronger whole-scenario 5/5 prediction is refuted. H4 was registered above before the next routing edit.

H4 full comparison — all scenarios, three repeats

• 41/44 scenarios passed, with one flaky free-form chat case, one known 4K-context code-edit gap, and the generic styling failure.
• 236/236 calls carried native timings; 71,013 prompt tokens ran at 10,279.7 tokens/s and 17,040 generated tokens at 89.9 tokens/s; wall time was 260,024 ms.
• The task/reminder creation forms and casual task-list lookup were correct in every repeat. The focused casual scenario then passed 5/5 (25 turns).
• H4 is supported. H5 was registered above before its model-facing change.

H5 focused and final results

• The first H5 attempt was confounded when another process loaded deepseek-r1:14b onto the 8 GB GPU. Four initial page generations hit the 45-second no-output limit. These results were retained as a scheduling finding and excluded from the clean speed comparison.
• Once qwen2.5:1.5b-instruct again had the GPU, the styling scenario passed 5/5 (15 turns, 15/15 measured model calls). H5 is supported.
• The final full run passed 42/44 scenarios: 42 passed 3/3, one known 4K-context edit gap failed 0/3, and one unmarked vague code follow-up failed 0/3. The harness kept the prior draft in both failures.
• The final run made 237 model calls: 237/237 had full timing fields, 71,642 prompt tokens ran at 11,730.7 tokens/s, and 19,702 generated tokens ran at 95.1 tokens/s. Server time was 221,191 ms and wall time was 226,602 ms.
• Against baseline, scenario passes rose 40 → 42 of 44. Model calls fell 250 → 237 despite three successful styling inferences being added; prompt tokens fell 6.7%, and clean wall time rose 13.3% because successful code paths generated 34.2% more output. Generation speed stayed similar (90.5 → 95.1 tokens/s).

Threats to validity

• Simulated conversations are distinct cases but do not represent a random sample of users.
• Repeats at temperature 0 can be identical; they measure reproducibility and timing variance, not independent reliability.
• Ollama duration fields measure the server. End-to-end wall time also includes harness work and local database access.
• Other local Ollama clients can replace the loaded model. The confounded H5 run shows that hardware recommendations and benchmarks need an idle-device check or a benchmark reservation; model identity alone is insufficient.

Conclusion

H1, H2, H4 and H5 were supported. H3's direct-mutation claim was supported, but its initial whole-scenario prediction was refuted because the repeat exposed a second independent lookup error. Repeated local simulation improved the harness from 40 to 42 passing scenarios without changing the model. The useful pattern was to reduce inference where the operation is explicit and reduce the response shape where creative model output is still needed.

Follow-ups

• Treat vague code follow-ups as the next measured cluster; the final run's a snake game replacement was incomplete in 3/3 repeats.
• Add an idle-GPU preflight and model-reservation state to the benchmark and future hardware recommendation UI.
• Keep structural task success separate from visual/browser quality; RL-016 showed that structural page checks missed obvious content and styling defects.

Reproduce

npm run sim -- 3
npm test
npm run typecheck