| Field | Value |
|---|---|
| Date | 2026-09-10 |
| Harness version | → 2026-09-10-grounded-followups-3 |
| Models | qwen2.5-coder:7b (live use); deterministic paths in simulation |
| Evidence grade | A (narrow: 2 scenarios × 3 repeats) for routing/state; B for the search diagnosis |
| Pre-registered | no (backfilled 2026-09-13) |
| Source record | FOLLOWUP-REVIEW.md, grounded-followup-evaluation.json |
Observation
im bored was treated as a name introduction; describing a workspace invented
gameplay details ("guards, inmates") for prisonidle; empty searches were
indistinguishable from failed ones. A pasted review asserted the empty searches
must have been transport or parser failures.
Hypotheses (reconstructed)
- H1 — Answering project descriptions only from catalogue metadata and saved notes eliminates invented project details.
- H2 — The reviewer's claim: previous empty searches were transport/parser failures.
Intervention
Mood-statement guard; name corrections as conversation-local facts with
provenance; catalogue-ID-grounded descriptions that say when only metadata is
available; deterministic search queries; typed error categories (HTTP_403,
HTTP_429, INVALID_JSON, TIMEOUT, …); failed searches clear cached results.
Method
npm run sim -- grounded-followups 3 with a synthetic workspace catalogue.
Direct live check of the search integration with the query fun games to play.
Results
- Simulation: 2 scenarios × 3 repeats, 24/24 turns passed, no routing drift.
- Live diagnosis: DuckDuckGo and Wikipedia both returned HTTP 200; six results.
- Unit tests 391.
Threats to validity
Simulated paths are deterministic, so they establish routing and state, not improved model reasoning. The diagnosis ran on the current implementation, not the one that produced the original empties.
Conclusion
- H1 supported at harness level (A, narrow).
- H2 not supported: the current integration works for that query, so the claim is unproven. Recorded as a refuted external diagnosis.
Follow-ups
Acceptance-criteria checking of generated content (→ RL-010, RL-011).
Reproduce
npm run sim -- grounded-followups 3