The skipped counter was at 7 on 13 July. Two weeks prior it had been 2. Between those two points, ten cron cycles had run, each on schedule, each exiting clean, none producing more than one ready draft, most producing zero.
The pipeline was running. The pipeline was failing. Nothing in the exit codes distinguished between those two states.
The three-weekday gap
Between 7 and 9 July, the UK Calculators content pipeline published nothing. The cron fired each morning on schedule. Gap anatomy from the 10 July review: 7 July produced eight clusters triaged to draft and no publish. Two more zero-publish days followed.
On 10 July, two articles went out by operator intervention: stamp duty and a Lifetime ISA pillar. The 14-day cadence clock restarted from that date. The direct cost was a fortnight of spacing compressed into a single morning, the cadence arithmetic reset, two publishes required in one day just to close the gap.
The gap analysis didn’t immediately surface why the preceding days had been empty. The cron had reached the drafting stage on 7 July. Clusters were being triaged. The failure was somewhere between “cluster identified” and “ready draft produced,” and the log wasn’t naming it.
What draft.py was doing
Root cause, found on 13 July: draft.py treated any entry in the index, including failed-* entries, as permanently consumed. A cluster that had previously failed the drafting step was filtered out on the next cycle, and every cycle after. Failed clusters accumulated. Each one narrowed the eligible pool. Across ten cycles from 30 June to 13 July, skipped climbed from 2 to 7 as the dead zone grew. ONE ready draft emerged from the entire period. Zero auto-publishes.
The filter was silent. It produced no log output for the clusters it excluded. The skipped count climbed; nothing flagged it. Exit code: 0, throughout.
The IHT zombie as parallel evidence
On 6 July, a long-slug IHT file made it into its third PR before its regeneration source was root-caused. It had slipped through PR #171 (created and deleted within a squash) and PR #176 (merged live) before anyone traced it back to a stale pending-publish entry.
Different bug, same failure mode: an earlier failure leaving residue the pipeline kept acting on, with no visibility into why it kept appearing. The IHT zombie was visible because it surfaced in PRs. The failed-* clusters in draft.py’s dead zone were not. They were absent from the ready-draft count, which was not being tracked.
What exit codes measure
On 2 July the cron fired at 08:22 BST. 150 clusters ranked. Top by calculator score: mortgage at 20. Zero drafts produced. Exit code: 0.
On 7 July: eight clusters triaged, no publish. Exit code: 0.
A monitoring system watching exit codes calls both cycles healthy. The cadence dashboard stays green. Nothing triggers a review.
The actual health signal was the ready-draft count and the skipped counter. Both tracked something exit codes miss: not “did the job run” but “is the pipeline producing output at the expected rate.” Rate is not a boolean. Exit codes are booleans. The signal you need to monitor cadence health does not fit in a pass/fail field.
The fix and what it adds
Improving on the prior documentation of this failure: the root cause (draft.py’s filter treating failed-* entries as consumed) and its cost were established when the gap closed on 10 July. The 13 July fix addressed what that failure proved about monitoring.
Two changes landed together. The dead zone was removed: failed-* index entries are now re-eligible on subsequent cycles. Ready-draft heartbeat visibility was added alongside the filter fix, surfacing the state of the pipeline in a form that lets drift become visible before it becomes a three-weekday gap.
Those two changes are not the same kind of thing. A bug fix prevents recurrence of this specific failure. A monitoring primitive catches whatever the next failure turns out to be.
Heartbeat visibility in practice
The ready-draft heartbeat surfaces discrete pipeline-state counters after each cycle: ready drafts queued, clusters skipped, last publish date, gap since last publish. Not aggregated into a single health status. They stay as discrete values that can be trended over time.
A skipped counter climbing from 2 to 7 over two weeks is a signal worth acting on, even if every cycle exits clean. Ten consecutive cycles producing 0 or 1 ready drafts is worth flagging too. So is a last-publish date drifting past the cadence target, before the gap shows up in what actually went live.
None of that requires the pipeline to fail in a detectable way. It requires the pipeline to report.
The July gap could have surfaced on cycle two or three if the heartbeat had been in place. It surfaced on cycle ten by human inspection of the gap itself, not by any pipeline signal. The difference is two weeks of cadence debt, a forced clock reset, and operator time spent on two manual publishes that the cron was supposed to handle.
Out of scope from the 13 July fix: automated alerting on heartbeat signals, historical trend storage, a pipeline health dashboard. The heartbeat visibility lands as an observable output; instrumentation built on top of it is a separate sprint.
What this proves
The pipeline ran correctly, in the sense that no process crashed, no exception was thrown, no exit code was non-zero, for ten cycles while failing to produce the output it was built to produce. Those facts coexisted because “ran correctly” and “produced the expected output at the expected rate” are different measurements.
Cadence health is a rate. Exit codes are booleans. If your only monitoring signal is “did the job complete,” you have no visibility into whether the pipeline is doing its job. You have visibility into whether it ran.
The skipped counter climbing from 2 to 7 was the signal. It was available the entire time. Nobody was watching it because it wasn’t being surfaced. The heartbeat fix makes it visible. Whether the system alerts on it, trends it, or just exposes it for manual inspection is a later question. The first requirement is that it exists somewhere you can see it.
