| Field | Value |
|---|---|
| Date | 2026-09-12 |
| Harness version | → 2026-09-12-app-workflows-15 |
| Models | qwen2.5-coder:7b, 8,192-token context |
| Evidence grade | B (one live sequence per scenario) + C (25 browser fixtures) |
| Pre-registered | partially: scenarios of increasing complexity fixed in advance |
| Source record | APP-WORKFLOWS.md, docs/app-workflows/* |
Observation
Verification covered card websites only. Calculators and task apps, and follow-up edits through the same conversation, were untested.
Hypotheses
- H1 — The verified loop generalises to a calculator and a task app with host-owned behaviour probes.
- H2 — Supplying a bounded state table and the candidate's actual failing observation lets the model repair reactive filter logic within three attempts.
Intervention
Narrow routing fixes (retained search ≠ research; pasted HTML is data; tasks inside an HTML app request stay in the draft); calculator and task-lifecycle probes; stateful probes reload between checks; stronger filter transitions; 4,000-char repair elements; verifier helper excluded; actual browser error in retry evidence; restoring a previous draft reports the edit as stopped.
Method
npx electron scripts/check-verified-chat.mjs --live --scenario=… for edit,
calculator, todos, todo-repair. Saved stages rechecked with the current
verifier via scripts/recheck-workflow.mjs.
Results
| Scenario | Stages | Checks |
|---|---|---|
| Dog site edit (seeded) | create → heading → remove FAQ | 10/10 → 10/10 → 9/9 |
| Tip calculator | create → add Reset | 6/6 → 7/7 |
| Task app | lifecycle | 5/5 |
| Task app filters | first edit | 6/6 on original verifier; fails strengthened verifier |
| Filter repair | 3 rejected repairs | 5/6 — completing a task under Active does not hide it |
Tests 653; 25 browser fixtures pass. No failing candidate was committed.
Threats to validity
n = 1 per scenario; the dog sequence was seeded, not a fresh uninterrupted creation; the instrument changed mid-study — the strengthened filter probe invalidates the earlier 6/6.
Conclusion
- H1 supported for calculator and task lifecycle (B).
- H2 refuted for
qwen2.5-coder:7b: reactive filters not repaired in 3 attempts (negative result).
Follow-ups
Try qwen2.5-coder:14b on the saved failing seed; consider repair-by-rewrite
of the single filter function.
Reproduce
npx electron scripts/check-verified-chat.mjs --live --scenario=todo-repair --seed=docs/app-workflows/todo-repair/stage-0.html