| Field | Value |
|---|---|
| Date | 2026-09-09 |
| Harness version | before the turn journal existed |
| Models | qwen2.5:1.5b-instruct |
| Evidence grade | B (live reproductions, calibration n = 12 workloads) |
| Pre-registered | no (backfilled 2026-09-13) |
| Source record | REVIEW-FIXES.md, budget-calibration.json |
Observation
An external review found the model reporting a capped retrieval page as the database total, polite phrasings ("could you please…") missing the task path, and a character-based token estimator of unknown bias.
Hypotheses (reconstructed)
- H1 — Counting questions can be answered exactly with zero inference by reading a real total in the same transaction as the rows.
- H2 — Repeated anchored prefix stripping recognises polite imperatives without unanchored action matching (which had caused false mutations).
- H3 — A single correction ratio could safely tighten the token estimator.
Intervention
True total alongside capped rows from one SQLite read transaction; count
questions rendered in code; normalizeRequest strips anchored wrappers up to
four times; labelled FOCUS:/LAST: lines instead of JSON state.
Method
Live reproduction with 25 matching notes and 25 open tasks. Calibration:
scripts/check-budget.mjs, twelve synthetic workloads, estimated vs
server-reported prompt tokens.
Results
| Question | Reply | Inferences |
|---|---|---|
| how many notes do I have about inventory? | 25 matching notes. | 0 |
| what did I write about inventory? | 25 matching notes; showing 10. | 1 |
| how many open tasks do I have? | 25 open tasks. | 0 |
| Workload | Estimated / actual tokens |
|---|---|
| Ten 900-char notes, full | 3217 / 1845 |
| Timestamp packet | 1351 / 1331 |
| Identifier packet | 1624 / 1671 |
| Chinese packet | 2128 / 485 |
| Arabic packet | 2195 / 530 |
Automated tests: 241 passing.
Threats to validity
Single reproductions; synthetic calibration text; one tokenizer.
Conclusion
- H1 supported (B). H2 supported (C, regression tests).
- H3 refuted: the estimator over-counts prose and non-Latin scripts but under-counts identifiers, so no single ratio is safe.
Follow-ups
Exact tokenizer/chat-template measurement per model.
Reproduce
node --import tsx scripts/check-review.mjs
node --import tsx scripts/check-budget.mjs