RL-028 Local Whisper recognition and voice check

Observation

User reports saying “Hello i would like to make a website about dogs” and receiving “But I would like to make an excellent idea of how”. User: “I have a northern british accent”. Probe finds only en-US Windows recognition installed. User accepted replacing dictation with local Whisper: “okay go ahed”. No original audio is available, so the exact acoustic failure cannot be replayed.

Hypotheses

H1: A locally installed Whisper model can replace Windows recognition on the selected microphone path, without cloud transcription or a silent fallback. H2: Bounded voice-activity segmentation, pause handling and conservative uncertainty review can preserve complete requests while avoiding transcription of silence and stale results after interruption. H3: A short explicit voice-check exercise can show the user's actual transcript versus expected words and elapsed transcription time, without sending calibration sentences to Spark or retaining audio.

Intervention

Implemented an explicit isolated runtime/model installer, default local Whisper, bounded offline worker protocol, retained local Windows TTS, microphone test and three-sentence voice-check UI. Windows recognition is an explicit alternative; missing Whisper refuses startup without cloud/provider fallback. Setup downloads fixed packages and model data, not audio. No Spark prompt changes or Spark/API inference during setup/tests. The voice check uses the actual local speech model.

The worker keeps pre-roll and short pauses, bounds clips/queues/output/lifetime, flags forced splits and uncertain decoding for whole-thought review, and discards stale results after mute/interruption. Renderer submission waits for ongoing speech and decoding. Mute and spoken playback reset those waits too. Successful voice-check sentences pause capture; the next sentence reuses the loaded model.

Method

Baseline: previous 765 passing tests; Windows-only local recognition. Add regression before implementation, verify deterministic tests, native generated-audio transport and one real local-model transcription without microphone capture. Keep observed user recording unavailable distinct from synthetic speech evidence.

Results

• 769/769 JavaScript tests pass (baseline 765), plus 4/4 new Python endpoint and decode tests. Typecheck and build pass. These establish behavior, not accent accuracy.
• Eight microphone UI/host checks, seven Spark fixture checks and nine Realtime fixture checks pass. No real Spark or API requests in this experiment.
• Real local model: Systran/faster-distil-whisper-large-v3, revision c3058b475261292e64a0412df1d2681c06260fab, faster-whisper 1.2.1 / CTranslate2 4.8.2, English, beam 5, temperature 0, no previous-text conditioning or prompt, max 256 new tokens, CUDA int8_float16 on RTX 4060 Laptop 8 GB.
• n=2 distinct locally synthesized inputs, each run twice (not four independent accuracy cases). Both outputs matched all words each time. “Hello, I would like to make a website about dogs.” and “A quiet reading room. No accounts. No tracking.” Silence produced no text. Original user audio is unavailable.
• First startup 27,480 ms; second 17,777 ms. Dogs decoding 719 then 3,312 ms; reading-room decoding 562 then 594 ms. Concurrent verification/installation varied between runs; no controlled speed comparison or latency promise.
• Real installer rerun over the installed environment completed its speech, optional GPU, model-download and ready stages. pip check found no broken requirements. Clean-machine setup, missing-Python recovery and real installer cancellation were not exercised end to end.
• Negative results retained: an asynchronous dialog-close race in the microphone test, a Spark fixture timeout under concurrent work, and a newly reproduced mute/ASR wait regression. See the evidence record.

Threats to validity

Synthetic speech cannot establish northern British accent accuracy. Accuracy, latency and GPU sharing require measurement; a locale label alone is not an accent solution. Whisper confidence is not a calibrated correctness probability.

Conclusion

H1 supported for installed local GPU recognition and fixture microphone wiring. H2 supported by deterministic guards, synthetic silence and real sequential transcriptions; noisy speech, sustained narration and user interruptions remain to be measured acoustically. H3 supported for UI comparison and reference isolation. Improved northern British accent recognition remains untested.

Follow-ups

Run the explicit voice check with the user speaking normally; retain only results they choose to retain.

Reproduce

After explicit local speech setup: npm run build, npm run typecheck, npm test, npm run check:microphone, npm run check:voice:spark, npm run check:voice, npm run check:whisper, and data/speech/venv/Scripts/python.exe tests/test_whisper_worker.py. The first three voice UI checks use fixtures; only check:whisper runs real ASR, on generated audio. Evidence is under evidence/RL-028/; no user audio retained.