| Field | Value |
|---|---|
| Date | 2026-09-13 |
| Harness version | live-voice-input-19 → live-sketch-20 |
| Pre-registered | yes |
| Evidence grade | B: three distinct real synthetic speech examples and real local model runs; C: harness/UI regressions |
Observation
User: “okay it picks up my voice pretty well, in real time we want to use the smallest model we have to real time transcribe my audio and have a slightly larger model be building a moodboard or visual implimentations like html being rendered and edited, but only showing in the working states, so a change is made it updates the visuals on the page. does this make sense? it must be cheap and fast, doesn't have to be high level output, the user is changing the files in basic real time, we need low latency, instant feedback on screen.”
Authorization: “okay, get it done, assume this is full implimentation, latency research backed, fast, clean, next generation”. Baseline 772 tests; existing speech uses distil-large-v3, 1.2 s end silence and 1.4 s UI debounce, no live HTML sketch.
Hypotheses
H1: tiny.en int8 CPU speech can provide bounded provisional transcripts without competing with the visual model for GPU memory. Accent preservation must be tested, not assumed from size. H2: a small installed local model emitting a bounded visual patch, not regenerated HTML, reduces output latency. Compare measured installed 1.5B candidates; no new cloud provider or model download for visual generation. H3: stage → hidden render check → revision-checked commit and swap keeps the last working view intact; undo, manual changes, cancellation and chat switches cannot apply stale work.
Research basis
- Whisper Streaming research and implementation: incremental recognition needs an explicit stability/latency policy; offline Whisper is not intrinsically a token-streaming audio model. Here partial hypotheses are provisional; only completed clauses drive edits.
- faster-whisper: supports quantized CPU inference; published hardware benchmarks are not this laptop's performance.
- Whisper sizes: tiny is 39M parameters. We preserve the existing accurate recognizer as an explicit alternative.
- Ollama structured outputs: constrain JSON format and still validate it in the host.
- Ollama generation: bounded generation and keep-alive plus load/prompt/evaluation timing telemetry.
- OpenAI latency guidance: smaller outputs/models, fewer requests and deterministic work inform the architecture, not a paid API dependency.
Intervention
Implemented: opt-in local live-sketch workspace, tiny.en CPU int8 speech with provisional text, local bounded visual patches, predefined safe HTML/CSS component rendering, hidden candidate checks, revision-checked commits, undo/redo, persisted app-owned working files and explicit keep-to-editor. No network assets, arbitrary model JavaScript, unreviewed real-project writes or automatic publishing. Separate matching tiny-speech check makes no page edits. Existing Spark/Realtime workflows remain independent.
Model comparison exposed two removal failures, so exact removal commands now bypass inference. An end-to-end run exposed the model leaving the initial hero unchanged for a simple website topic; that narrow pattern now compiles a visible hero directly. Both failures were reproduced by failing tests before the guard was added. Complex instructions still go to the model; these are not broad language-parser guarantees.
Method
Three visual requests on each of two real 1.5B models, n=1 per request/model. Three distinct synthetic English phrases through real CPU ASR; three through the real Electron/host/speech/model/commit/render path. Fixed Windows System.Speech synthesizer, 16 kHz mono PCM sent in 100 ms frames, no user microphone. No expected sentence is supplied to recognition. Last visible working revision is sampled every ~100 ms, so timing includes that observation granularity. Audio includes synthetic pauses; end-of-fixture is not the last phoneme.
Hardware: Windows, i7-12650H, RTX 4060 Laptop 8 GB. Node 20.16.0, Electron 33.4.11, Ollama 0.34.0, Python 3.12.5, faster-whisper 1.2.1 / CTranslate2 4.8.2. Tiny speech revision 0d3d19a32d3338f10357c0889762bd8d64bbdeba, CPU int8, four threads, beam 1, max 128 tokens. Visual: qwen2.5:1.5b-instruct and qwen2.5-coder:1.5b, installed Q4_K_M builds; temperature 0, context 4096, max 384 generated tokens, schema-constrained JSON, host input estimate capped at 3000 tokens. Exact prompts, replies and available server-reported usage are in the raw JSON evidence.
Results
| Check | Result |
|---|---|
| Existing test instrument | 772 baseline → 780 passing tests, 0 failures |
| Python segmentation/decode checks | 5/5 |
| New real desktop fixture checks | 11/11: working preview, typed patch, overlap/session guard, direct edits/history, rejection, cancellation, microphone, matching voice check, stale editor review, keep, narrow layout |
| Existing desktop fixture checks | Spark 8/8; Realtime 9/9; microphone 8/8 |
| Typecheck / build | Pass |
| Real model-only comparison | All six replies schema-valid; 2/3 semantic checks passed for each 1.5B model. Both failed removal. Cold initial responses: 6219 ms instruct, 4162 ms coder. Warm style-only: 372 / 452 ms. Not a general model ranking |
| CPU speech, corrected instrument | 3/3 synthetic phrases match ignoring case/punctuation; 5 finalized clauses. First partials 1314 / 1546 / 1528 ms. Final decode 485–781 ms across later independent runs; specific ASR-only run 516–781 ms. No invented text in 2.5 seconds of silence |
| Final real pipeline | Startup 3492 ms; fixture-end to last visible working revision 333 / 874 / 1535 ms for dogs / reading-room / children-no-shop |
| Text-to-working-view in final pipeline | Simple topic shortcut 26 ms; four model edits 499–592 ms; hidden render checks 8–15 ms |
| Provider usage | No Spark or paid API inference in these new checks/benchmarks; local CPU/GPU work only |
The initial real pipeline measured 1206 / 1540 / 1756 ms with 3955 ms startup, but the dogs request only changed below-fold text. Its file is retained. Some unit tests ran concurrently with that first measurement, so it is not a controlled speed comparison. The later run was after the full suite finished, with only a short targeted check near startup. Three distinct phrases, not three repeats of each condition: grade B, no p95/reliability claim.
Negative semantic results remain: the small model can add a reading-room section without replacing the dogs hero, return unchanged content for constraints, or alter styles without being asked. Render and schema checks do not validate intent. The narrow topic/removal regressions are fixed; general nuanced semantic fidelity is not established. Stronger-model review is not invoked in this fast loop.
Threats to validity
No original accent recording. Synthetic success cannot establish accent accuracy. Warm/cold and CPU/GPU contention must remain distinct. A template sketch is not arbitrary functional web-app generation; render checks don't prove aesthetic or semantic correctness.
Conclusion
H1 is supported for these synthetic phrases, not the user's accent. H2 supports short patches and deterministic work as practical latency controls; cold load is material and neither tested model reliably followed every instruction. H3 is supported by fixture/state tests for cancellation, stale sessions, revision checks, working-only swaps and explicit editor review. SQLite owns state; three scratch-file replacements are not a single crash-atomic filesystem transaction.
The workflow is implemented end to end as a constrained live visual sketch, with a clear functional-app boundary. Primary-source latency guidance informed smaller outputs, preload and avoiding inference for exact operations; no paid speech API was added. Public claims reviewed through RL-030; no universal instant, accent accuracy or arbitrary-app guarantee added.
Follow-ups
User Northern British accent check with the tiny recognizer, long sessions, repeated idle/cold/load tests, and broader instruction-fidelity evaluation.
Reproduce
npm test; npm run typecheck; npm run build;
data/speech/venv/Scripts/python.exe tests/test_whisper_worker.py;
npm run check:live-sketch; npm run check:voice:spark; npm run check:voice;
npm run check:microphone; npm run check:speech:fast;
npm run benchmark:live-sketch; npm run benchmark:live-pipeline.
Benchmarks require the explicitly installed local models/runtime and overwrite
their current result artifact. Baseline files were copied before rerunning.
See evidence record, raw model comparison,
fast-speech.json, live-pipeline-first-pass.json, live-pipeline.json,
live-sketch-electron.json, and desktop screenshots in evidence/RL-030/.