| Field | Value |
|---|---|
| Date | 2026-09-12 |
| Harness version | ≈ 2026-09-12-…-12 (evaluation scripts; not logged in turns) |
| Models | qwen2.5-coder:1.5b, qwen2.5:1.5b-instruct, qwen2.5-coder:7b; temperature 0, thinking off, 8,192-token native context |
| Evidence grade | B (one run per model/workflow) |
| Pre-registered | partially: the 12-check brief and checks were fixed before generation |
| Source record | WEBSITE-EVALUATION.md, docs/website-eval/* |
Observation
Generated pages "looked complete" but no functional evidence existed. Early runs exposed routing ("complete single-file HTML website" → task; "search input" → internet research), a draft discarded by a character estimate, and Ollama serving 4,096 tokens when 8,192 was requested.
Hypotheses
- H1 — Host-owned browser checks plus bounded repair raise functional pass counts.
- H2 — Sending the model only the failing element and its observation works better than whole-source repair.
- H3 — Fixing the context mismatch alone fixes the behavioural failures.
Intervention
scripts/check-website.mjs (12 checks, isolated renderer, restricted network);
src/agent/CodeRepair.ts exact edits with re-verification, regression
rejection, repeat detection, ≤ 8 attempts; element-scoped repair surface;
native Ollama transport sending num_ctx/num_predict.
Method
Fixed brief (scripts/website-challenge.ts); one generation per model; repair
runs; final artifact replayed through the full suite.
Results
| Model / workflow | Initial | Final |
|---|---|---|
| qwen2.5-coder:1.5b | 4/12 | 4/12 |
| qwen2.5:1.5b-instruct | 8/12 | 8/12 |
| qwen2.5-coder:7b | 10/12 | 12/12 (2 accepted element repairs) |
| 1.5B coder repairing 7B draft | 10/12 | 10/12 (ineffective repair rejected) |
Whole-source repair edited unrelated repeated classes; element-scoped repair fixed grid and search label in two calls. Tests 630.
Threats to validity
n = 1 per condition; CDN variability; mixed transports across retained attempts.
Conclusion
- H1 supported for 7B, not for 1.5B (B).
- H2 supported (B).
- H3 refuted: correcting the context mismatch did not change behaviour.
Follow-ups
Attach verification to chat (→ RL-011); repeated runs for a rate.
Reproduce
node --import tsx scripts/website-challenge.ts qwen2.5-coder:7b docs/website-eval/new-run
npx electron scripts/check-website.mjs docs/website-eval/new-run/page.html