The phrase was inside _build_user_prompt. Four words: “a current sourced figure.” That exact string turned out to be the structural cause of every hallucination in the redraft loop. Not a flaky model. Not a temperature setting. A prompt pattern that asked a closed-book model to open a book it didn’t have.
That was X.8.1. X.8.2 closed it.
What the pipeline was doing
The ukcalculators editorial pipeline runs a draft-and-redraft loop. First pass produces a raw draft from a cluster of related content. If the draft fails a quality gate, it routes through a redraft path. The loop has bounded retries.
The drafter (draft.py) was building its user prompt from a cluster’s titles and URLs. Then asking Haiku to fill in the sourced data itself. Standard enough that it barely registers as a design choice: hand the model context, ask for prose, get prose.
The problem: the “sourced figure” Haiku returned came from Haiku’s training data, not from the URLs. Haiku has no live web access. The titles and URLs were metadata, not content. The model did what models do under that constraint: it produced something plausible. Bank of England rate figures. Percentages. Dates. All coherent, all calibrated against the cluster topic, none of them fetched from anywhere real.
The redraft loop ran. The gate caught the hallucinations. Haiku tried again. Same failure. Each re-ask inherited the prior framing and produced the same class of error.
What SC-23 flagged
SC-23 is a lint rule on the prompt pattern itself. It flags the structural shape: titles + URLs → ask for current sourced figure. When X.8.2 (T4a) dropped, the commit was explicit: _build_user_prompt no longer takes a cluster’s titles and URLs and demands “a current sourced figure.” That exact phrase pattern, SC-23’s structural flag, was the cause.
The pattern was the bug. Not the content of any individual prompt. The shape of every prompt.
The envelope
The fix reframes the problem. Instead of asking Haiku to produce data, you fetch the data first, package it into a JSON envelope, and tell Haiku to compose prose from the blocks you hand it. The model’s job shifts from “find a figure” to “build a sentence around this figure.”
The envelope composer lives in X.8.2. It pre-fetches the data blocks for a cluster before the drafter runs. The drafter’s prompt then contains the actual figures (not titles and URLs) as structured JSON. Haiku reads from the envelope, not from memory.
This is the change that looks obvious in hindsight. The prior approach (trust the model to source its own context from metadata) is the default pattern for LLM-assisted drafting. It works for topics where the model’s training data is dense and stable. It fails for live economic figures, where “current” means this month and the model’s knowledge ends somewhere behind it.
Wiring it in
X.8.2 wired the envelope composer into both draft.py and redraft.py, with bounded retries and stateless re-asks. The stateless part matters: each retry is a fresh compose-from-blocks, not a continuation of the prior attempt. The prior attempt’s failure is irrelevant to the next one. You’re not asking the model to fix its own output; you’re asking it to draft from the same clean input again.
The gate was untouched. The quality gate that catches bad output is a separate concern from the drafting path. Keeping them separate was explicit: the gate catches the symptom, the envelope path fixes the cause. Conflating them would produce a component that was harder to test and harder to reason about.
The snapshot-mismatch detour
Two Bank of England iter2 runs were showing snapshot-mismatch verdicts at the gate. Separate from the hallucination issue: a calibration problem, not a sourcing one. Root cause: Haiku had tagged a cluster differently across two runs, producing a structural change the snapshot comparator read as a regression.
PR #40 (scoped snapshot check, merged at c18e11a) fixed this by narrowing the comparison scope. The prior check compared full outputs. The scoped version compares only the sections that should be stable across retries. Narrower claim, more useful signal.
The BoE MMC March data was the first redraft pair to complete via the envelope path on the merged calibration. S-2 passed on every pre-registered criterion.
Pre-registering the pass criterion
One discipline enforced during X.8.2: define S-2’s pass criterion before running it. Not after.
When you define success after the result is in front of you, you drift toward fitting the criterion to the output. The result looks acceptable, so the criterion relaxes. Pre-registration closes that drift. You commit to what “pass” means before you know whether the run passes.
The directive was explicit: “Pre-register S-2’s pass criterion before running it. Keep paragraph calls bounded.” The subsequent commit: S-2 PASS on every pre-registered criterion. Criterion first, run second, verdict against the locked definition.
Post-wiring housekeeping
After the envelope path was live and S-2 had passed, PR #43 cleaned up. One of Cj’s four notes on the PR: remove test_verify_py_not_modified_by_wiring_pr. That test had a shelf life. The moment PR #43 squash-merged, the diff it was guarding against was empty by definition. The test would pass forever, checking nothing, while occupying a slot in the suite. A test that can’t fail isn’t a test; it’s noise.
The other three notes: add replay hashes, surface B-044 and data-model.md as named rows. Replay hashes give a stable handle on a specific run’s input state. Named rows make the two outstanding items visible rather than buried in a comment.
The #39 lesson as doctrine
The directive in the wiring commit: “Make the #39 lesson doctrine, not just a new test row.”
The #39 lesson was that retries must be stateless. The envelope path implements this structurally: the JSON envelope is assembled fresh each time, the model’s prior output is not passed back in. The lesson moved from a note in a PR to a constraint in the code.
Doctrine means the constraint survives the next person who touches the file without knowing the history. A comment saying “retries must be stateless (see #39)” is documentation. A composing function that takes no prior-state argument is enforcement.
What changed
Before X.8.2: Haiku asked to source figures it couldn’t reach, produced plausible ones, failed the gate, tried again the same way.
After X.8.2: data fetched before the drafter runs, handed to Haiku as a JSON envelope, Haiku composes prose from blocks it has in front of it. Retries start fresh. The gate sees clean output.
The prompt pattern (titles + URLs → ask for current sourced figure) was the root cause. The envelope pattern is the structural fix. SC-23 flags any re-emergence of the original shape. The redraft loop doesn’t hallucinate on BoE figures. S-2 passed.