The test was called test_verify_py_not_modified_by_wiring_pr. Its job was to confirm that verify.py had not been touched by the wiring pull request. When PR #43 squash-merged on 12 June, the diff became empty by definition, and every subsequent run returned green. The test was checking nothing, and because it had a file guard baked into its logic, it was also blocking legitimate changes to verify.py. A green result that reported nothing true and prevented something useful. That is the shape of a shelf-life test.
The post-wiring housekeeping note on PR #43 called it out directly: the test had a shelf life, the shelf life had expired, and it needed to go. Remove the test. Add replay hashes. Surface B-044 and data-model.md as named rows. Four notes, two landed in the same session. Removing the test was a fix; Phase X.8 was asking whether the whole verification approach had the same structural problem.
What a shelf-life test looks like
A shelf-life test has an expiry condition baked into its premise. Before that condition fires, it catches real problems. After it, the test passes unconditionally, because its underlying assertion became trivially true. The CI output stays green. No alarm fires. The system drifts underneath it.
The wiring diff check is the clean example. Pre-merge, confirming that the wiring PR hadn’t touched verify.py was a meaningful constraint. Post-merge, the diff was empty and the assertion was assertTrue(True). The test didn’t fail; it couldn’t. That inability to fail is the liability.
What X.8 replaced them with
Three mechanisms, each targeting a different failure mode.
Replay hashes. A diff check collapses to nothing post-merge. A replay hash records the output of a deterministic pipeline run against fixed input. A gate that compares the hash of a redraft output against a pre-registered value bites on any change to the pipeline’s behaviour: altered prompt, changed model parameters, modified retry logic. The hash either matches or it doesn’t. There is no post-merge condition that makes the comparison unconditionally true.
Pre-registered pass criteria. The directive for the S-2 gate: “Pre-register S-2’s pass criterion before running it.” That sequencing is load-bearing. A criterion registered after you have seen the result is a description. A criterion registered before is a constraint. S-2 probed the envelope-path redraft latency against a bound that existed before the first run. The BoE MMC March pair was the first through that path. The test did not know in advance what the latency would be; the criterion had already decided what was acceptable.
Hermetic sabotage cycles. S-3.1 and S-3.2 are adversarial by design. S-3.1 is defer-ungrounded: it forces the system into a state where a deferral has no grounding citation, then confirms the gate surfaces the condition rather than passing. S-3.2 is stale-rate BLOCK: a stale rate is injected into the pipeline, and the gate must fire before the draft proceeds. Both are hermetic. They construct the failure condition explicitly rather than waiting for it to appear naturally. If the code path that catches a defer-ungrounded state is removed, S-3.1 fails. If the stale-rate check is weakened, S-3.2 fails. S-3.3 (clean control) was deferred until after S-3.1 and S-3.2 had been observed in CI.
The doctrine, not the test row
X.8.2 wired the envelope composer into draft.py and redraft.py. Bounded retries. Stateless re-asks. The greenlight directive was explicit: “Make the #39 lesson doctrine, not just a new test row. Retries must be stateless.”
The stateless requirement is the doctrine. A retry that carries state from the previous attempt can mask a systematic failure: the system appears to recover because it remembered what it tried last time. Stateless retries force each attempt to succeed independently. If three independent attempts all fail, the signal is unambiguous.
The envelope path existed before X.8.2. What X.8.2 did was make it the canonical route in both draft.py and redraft.py, not a fallback. The gate remains untouched by retry logic; the retry logic remains unaware of what the gate will decide. That separation is what gives the gate authority. A gate whose retry path can influence its result is not a gate. It is a negotiation.
S-1 on every gate-touching commit
S-1 runs on every gate-touching commit. test_s1_pr21.py and test_s1_integrated.py are the enforcement layer. Green on every gate-touching commit is the bar; it is not optional per sprint.
A gate that is only checked at the milestone run is not a gate. It is a checkpoint. Checkpoints catch problems after the fact; gates catch them during development. By the time H-1, H-2, and H-3 lifted on 12 June, every gate-touching commit in the branch had an S-1 result. PR #46 had landed operator-path call_trail persistence. The milestone was not a moment where everything was checked at once; it was a moment where a series of incremental checks converged.
What this costs
Hermetic sabotage cycles take longer to write than happy-path tests. S-3.1 and S-3.2 required constructing the failure condition explicitly. Building a defer-ungrounded state. Injecting a stale rate. That construction work is real, and it is the only way to know whether a gate fires when it should. A gate whose sabotage cycle has never run has not been tested; it has been installed and assumed.
Pre-registering criteria before running tests adds friction. Deciding what acceptable looks like before you have data is harder than registering the criterion from the result. It is also the only sequencing that produces a real constraint. A criterion derived from the result describes what happened; it does not constrain what is allowed to happen next time.
The shelf-life test was cheap to write and cheaper to maintain. It cost nothing until it became a liability: blocking changes to verify.py while reporting green. The Phase X.8 replacements are more expensive to author. They are also the kind of tests that can fail.
Where this lands
Phase X.8 shipped with S-1 green on every gate-touching commit, S-2 PASS against its pre-registered latency criterion, S-3.1 and S-3.2 hermetic sabotage cycles in place, and H-1 through H-3 lifted. The cadence clock primed at Day 0.
B-044 and data-model.md are now named rows in the verification fixture list. Out of scope here: the full scope of B-044. What the devlog records is that naming them as rows makes them first-class check targets. A change to the data model that does not update the named row will surface in the check. Same logic as the replay hash: make the check depend on the artefact, not on a condition that becomes trivially true.
A test suite where every test can only pass is CI theatre. Phase X.8 was about replacing theatre with constraint. The measure is not whether the gates are green; it is whether they can go red.
