| Field | Value |
|---|---|
| Date | 2026-09-12 |
| Harness version | 2026-09-12-live-editing-8 → 2026-09-12-code-progress-11 |
| Models | qwen3:4b-thinking (observed); local model for the live check |
| Evidence grade | D (journal) + B (one live reproduction) + C (9 regression cases) |
| Pre-registered | no (backfilled 2026-09-13) |
| Source record | CODE-PROGRESS.md |
Observation
Journal, 2026-09-12, qwen3:4b-thinking:
| Time | Message | Outcome |
|---|---|---|
| 10:20 | create a website about dogs, multiple pages, tailwind | incomplete draft: <!DOCTYPE html><html lang="en"> |
| 10:23 | continue | incomplete draft |
| 10:25 | keep going | error: spent its reply budget on reasoning |
| 10:51 | continue | error: spent its reply budget on reasoning |
| 11:09 | continue | incomplete draft shorter than before |
| 11:14 | continue | error: spent its reply budget on reasoning |
4 of 6 continuation attempts failed; one "success" shrank the draft.
Hypotheses (reconstructed)
- H1 — Refusing drafts that shrink or repeat keeps a longer draft from being replaced by a header.
- H2 — Stopping after two no-progress attempts on the same revision and model avoids futile inference loops.
- H3 — Extra output allowance (+1024 tokens) for always-reasoning models leaves room for code.
Intervention
Multi-page briefs get the single-file limitation before generation; continuation
cannot shrink or repeat; incomplete edits cannot replace complete drafts;
two-stall stop per revision/model; +1024 output tokens for thinking: always,
bounded by available context.
Method
Nine regression cases replaying the reported failures. One disposable live run.
Results
- Live: a dogs-page fragment grew 73 → 759 characters with three breed cards via "keep going" (n = 1).
- Later journal (
verified-code-13, 20:35): "now add css to the html]" → reasoning-budget error again.
Threats to validity
n = 1 live; length and delimiter checks are not semantic correctness; the stall counter does not survive restart.
Conclusion
H1, H2 supported (C). H3 not supported in real use: budget exhaustion recurred after the change (D).
Follow-ups
Retry once with low reasoning effort on continuations; classify the failure explicitly; recommend a non-reasoning coder for drafts.
Reproduce
npm test