| Field | Value |
|---|---|
| Date | 2026-09-10 |
| Harness version | → 2026-09-10-artifact-context-1 |
| Models | qwen2.5:1.5b-instruct |
| Evidence grade | A (narrow: 2 scenarios × 3 repeats) + C |
| Pre-registered | no (backfilled 2026-09-13) |
| Source record | ARTIFACT-FOLLOWUPS.md, artifact-followup-evaluation.json, artifact-code-evaluation.json |
Observation
User requested an HTML dogs page "using information from the internet". The
app produced an unverified image URL, then answered explain and continue by
generating different pages. Root causes found in code: the code prompt accepted
a draft argument without including it; sticky code mode sent explanations to
the whole-file writer; code routing bypassed web retrieval.
Hypotheses (reconstructed)
- H1 — If the exact current draft is in the prompt and explanation is a separate, non-writing operation, follow-ups stop replacing the page.
- H2 — Gating researched pages on a successful search prevents invented sourcing.
Intervention
One whole draft with request, revision and completion flag; explain cannot
update the draft; bare continue on complete output asks what to change;
research-before-generation with bounded evidence; invented remote img src
stripped; 12,000-char draft cap. Added the agent_turns journal and harness
versioning — the basis for all later grade-D evidence.
Method
npm run sim -- artifact-followup 3 and npm run sim -- code 2, disposable
databases, synthetic search (no live provider).
Results
| Suite | Passed | Turns |
|---|---|---|
| artifact-followup (2 scenarios × 3) | 6/6 runs | 15 |
| code subset (3 scenarios × 2) | 2/3 scenarios | — |
The game-editing scenario stayed red: in one repeat the full draft exceeded the protected prompt budget; in another the model offered prose instead of an edit. Unit tests 379/379.
Journal under this version (real use, n = 11 turns): 3 failed.
Threats to validity
Scripted phrasing; synthetic search; one small model; completion is a delimiter check, not functional verification.
Conclusion
H1 supported for the scripted dog-page interaction (A, narrow); not supported for game edits under a 4k window. H2 supported in harness tests (C).
Follow-ups
Reply-room accounting for large drafts (→ RL-004, RL-010).
Reproduce
npm run sim -- artifact-followup 3
npm run sim -- code 2