| Field | Value |
|---|---|
| Date | 2026-09-12 |
| Harness version | between 2026-09-10-desktop-search-4 and 2026-09-12-modular-workspace-6 (no turns logged under it) |
| Models | observed failures: qwen2.5:1.5b-instruct, qwen2.5-coder:1.5b, deepseek-r1:8b, deepseek-r1:14b, qwen3:4b-thinking |
| Evidence grade | D (motivating journal rows) + C (tests); live effect not isolated |
| Pre-registered | no (backfilled 2026-09-13) |
| Source record | HARNESS.md §2–§8 |
Observation
Journal, 2026-09-12, version desktop-search-4, after a thinking toggle was
added to the UI:
| Time (UTC) | Model | Message | Reply |
|---|---|---|---|
| 06:02 | deepseek-r1:14b | you try | model request timed out after 60000ms |
| 06:03 | qwen2.5:1.5b-instruct | create a website about dogs | HTTP 400 "does not support thinking" |
| 06:26 | qwen2.5:1.5b-instruct | what do you think of those? | HTTP 400 "does not support thinking" |
| 07:25 ×2 | qwen2.5-coder:1.5b | yo | HTTP 400 "does not support thinking" |
| 06:42 | deepseek-r1:8b | okay, create a html page describing dogs | timed out after 60000ms |
| 06:01 | qwen3:4b-thinking | what was the issue | I'm not sure what to say to that. |
| 06:39 | qwen3:4b-thinking | what coding languages do you understand | I don't understand coding languages… |
| 06:41 | deepseek-r1:8b | the new website | Opening the new website project file. (no tool ran) |
Hypotheses (reconstructed)
- H1 — Sending only parameters a model supports (from a capability manifest) eliminates capability 400s.
- H2 — Separate connect / first-token / stall / total clocks, re-armed on reasoning tokens, stop healthy reasoning models being killed by a 60 s wall clock.
- H3 — Failure explanations answered from the journal replace invented ones.
- H4 — A prompt that separates knowledge from available actions, plus reply guards, stops ability denials and false action claims.
Intervention
src/llm/capabilities.ts (four-state thinking), planReasoning with
task-aware effort and drop-and-retry on rejection; src/llm/timeouts.ts;
src/agent/outputContract.ts deterministic repair before model repair;
src/failures.ts taxonomy persisted to agent_runs.failure; isFailureQuestion
fast path; deniesCodeAbility and claimsAnAction guards.
Method
Unit and regression tests only at the time (tests/harness.test.ts). No
controlled before/after live run was recorded.
Results
- No
does not support thinking400s appear in the journal after this version (19 later turns). Confounded: every later desktop turn usedqwen3:4b-thinking, which supports thinking, so the absence is not evidence. failurecategories now persist, but of 25 failed runs in the journal (error,invalid_output,max_steps,repeated_tool_call, all versions) 20 have no category recorded, 4 areunknownand 1 iscontext_overflow. Budget exhaustion appears under bothunknownandcontext_overflow(metrics/2026-09-13.json). Most unrecorded runs predate the taxonomy, so this measures coverage over history, not the current miss rate.
Threats to validity
No live replication on the non-thinking models; version string not captured in turns; several interventions shipped together, so effects are not separable.
Conclusion
H1–H4 supported by construction and tests (C); not measured live. The taxonomy is incompletely applied.
Follow-ups
- Controlled replay of the table above against each installed model with thinking on.
- Add
reasoning_budget_exhaustedand ensure every non-ok run records a category.
Reproduce
npm test
npm run research:snapshot