The three-second trace
The 18:00 BST verification run on 14 June produced this in the audit trail:
id=51 → id=53 → id=54 → id=55 rejected → id=58 → breach
Three seconds. It looked concurrent: a burst of slot IDs evaluated in quick succession before the breach fired. It wasn’t. Evaluation was serial. One candidate rejected, the next tried, then the next rejected before the breach. The slot state machine exhausted the pool in order, then alarmed. The Sunday 09:00 BST run before it had produced the same shape.
Two verification runs, same pattern, same outcome.
Seventy-two hours of the same miss
The pattern started before the verification runs. Between 12 and 13 June, three cancellations in 72 hours: queue rows id=35, id=36, and id=39. The 08:00 UTC cancellation on 13 June was the third. Autonomous success across the last five scheduled slots: 20%.
One in five is worth sitting with. The pipeline was scheduling work correctly. LaunchD fired on time, the dispatcher picked candidates, and the queue populated. None of that was broken. What was broken was the step after: recovery when a selected slot couldn’t publish. Four of those five expirations came from the same upstream failure mode, the one B-016 had been built to address. The fix existed. The recovery path around it didn’t work reliably enough to matter.
The already-published piece on the substitute pool examined the editorial policy problem. The failure here was different. It wasn’t about what candidates the pool was generating. It was about whether the pipeline could execute a recovery sequence at all when the primary slot fell over.
Three mechanisms, one requirement
The SLA recovery work that landed on 13 June combined three things:
- Sibling cancel. When a slot is cancelled, immediately cancel any in-flight sibling that shares the same source, before the substitute sweep begins. Not after.
- Substitute fallback. When the primary slot fails, pull from the substitute pool. The same day’s separate fix (substitute pool steals from future-scheduled rows and recurses naturally) gave the pool a self-replenishment path: if the immediate pool is empty, pull from rows scheduled further ahead and recurse until a viable candidate is found or the pool runs dry.
- Missed-slot alarm. A
schedule_sweepdetection layer that fires when a slot expires without resolution, distinguishing “in progress” from “silently failed.”
Independently, each is insufficient. Sibling cancel without substitute fallback clears the failure but leaves the schedule gap. Substitute fallback without sibling cancel means the sibling and the substitute both run simultaneously: substitution doesn’t replace the failed run, it competes with it. Missed-slot alarm without the other two produces accurate noise. It names the failure without changing what follows.
The 12:00 BST walkthrough on 13 June showed the gap in practice. Queue id=37: schedule_sweep fired the publisher at 11:00:08Z. The publisher cancelled the row, substitute fallback kicked in, and the missed-slot alarm fired. The sibling was still running, because sibling cancel hadn’t landed yet. The substitute and the sibling competed. The alarm was accurate. The recovery wasn’t clean.
Why serial matters
The three-second trace from 14 June looked like a race condition. It wasn’t. The slot state machine (first-class slot entity with explicit transitions, landed in the same sprint) made the evaluation sequence legible: each state change had an ID and a timestamp. The rapid succession of slot IDs wasn’t parallelism. It was serial evaluation of a pool that couldn’t produce a viable candidate.
Misreading the trace would have pointed the next sprint in the wrong direction. The fix for a race condition is different from the fix for pool exhaustion. Before explicit transitions, slot evaluation was implicitly ordered by table scan, so the trace read as noise. The state machine made the diagnosis possible, which meant the correct fix (pool depth and replenishment, not concurrency guards) could be identified without a false start.
The alias-map fix that surfaced mid-sprint
The Friday integration test on 13 June was supposed to confirm the substitute pool mechanics. It exposed something else. G-L3 was over-aggregating on platform-tag alias matches: platform tags were collapsing distinct articles into the same topic bucket, triggering sibling-detection and blocking articles from the queue before they reached slot evaluation at all.
Three articles blocked or blockable in one morning. Queue id=36 (“Scheduling articles through Telegram taught me cron is the wrong model”) was among them.
The fix: G-L3 ignores platform-tag matches when deriving topic_key. Platform tags are broad categorical labels; they’re not a signal that two articles are siblings. An article blocked by false sibling-detection never reaches slot evaluation. With no candidate present, sibling cancel has nothing to cancel and substitute fallback has nothing to substitute. The missed-slot alarm fires on an empty sequence.
Out of scope for the SLA recovery spec. A dependency for it in practice.
What changed
After the three mechanisms and the alias-map fix:
- Every slot transition is auditable: candidate ID, timestamp, rejection reason if applicable.
- Sibling cancel fires before the substitute sweep begins.
- The substitute pool can steal from future-scheduled rows and recurse when the immediate pool is empty.
- The missed-slot alarm connects to the retry path, not only to notification.
The 14 June 18:00 BST trace, with slot IDs id=51 through id=58 in three seconds, was a controlled regression test confirming B-020 in the patched system. Same shape as the live failures. Different context: this time, the expected outcome was breach.
What four cancellations proved
Each of the four cancelled queue rows passed the individual tests for whatever mechanism was live at the time. Sibling cancel shipped and its tests passed, then the next slot expired anyway. Substitute fallback shipped, its tests passed, and the slot after that still expired. The pipeline failed under exactly the conditions the spec was written to prevent.
SLA recovery requirements don’t decompose the way implementation tends to hope. You can ship each mechanism in a separate PR, watch each one pass in isolation, and still have a pipeline that can’t hold its schedule. The acceptance test isn’t whether each mechanism has passing unit tests. It’s whether four consecutive slot failures all recover autonomously without manual intervention.
Until that test passes, you have the components of SLA recovery. Not SLA recovery itself.
