The morning of 7 July, the cron fired on time. Eight clusters triaged to draft, six source packs assembled. The pipeline completed without error.
Zero articles published.
Same on 8 July. Same on 9 July.
What the pipeline was supposed to do
The UK Calculators content pipeline runs once a day. It ranks topic clusters by calculator score, assembles source packs for the top candidates, and passes them to a draft generator. If a cluster clears the quality gates, it queues a publish. The whole chain is automated.
Between 30 June and 13 July, across ten cron cycles, it produced exactly one ready draft and zero auto-publishes. Operator publishes on 1, 2, and 10 July kept the site moving, but those were by hand. The automation was running but not producing.
The symptom
The error log was clean. Each morning the cron completed successfully, processed candidates, and exited without error. What the logs didn’t surface: the pipeline was reaching draft.py and silently skipping every cluster it touched.
The skipped counter in the triage log was the only visible signal. It climbed from 2 to 7 across those ten cycles, one or two increments per run. On its own, a rising skipped count looks like a content quality problem: the candidates aren’t passing the gate. That was where I looked first.
It wasn’t a quality problem.
The root cause
draft.py maintains a consumption index. When a topic cluster is successfully drafted, its identifier is written to the index. Subsequent runs check the index before any draft attempt: identifier present, skip. That logic is correct for the success case. You don’t re-draft a shipped article.
The bug: any index entry was treated as consumed. Including failed-* entries.
When a cluster failed a draft attempt (API timeout, quality gate rejection, malformed output), draft.py was writing a failed-<identifier> marker to the same index. On the next run, the existence check found the marker, logged a skip, and moved on. Permanently.
A topic that had ever failed to produce an acceptable draft was locked out of the pipeline. Not temporarily, with a retry window. Gone. The consumption index had no concept of failure as a distinct state from success.
What ten cycles looked like
The 7–9 July gap anatomy, reconstructed from a 10 July analysis:
- 7 Jul: 8 clusters triaged, 6 packs assembled, zero publishes
- 8 Jul: similar throughput, same outcome
- 9 Jul: same
The cron wasn’t idle. Fetching, ranking, and assembling source packs is real work. All of it terminated at the consumption check before draft.py ever attempted a generation call.
The failed-cluster accumulation started no later than 30 June. The mortgage rate cluster was present in the top candidates on 1 July, producing one draft that morning from BBC, Zoopla, Moneyfacts, and Bank of England sources. It had likely been marked on an earlier failed run. By 7 July, enough high-scoring clusters had accumulated failed markers that the pipeline was triaging into a near-empty eligible set. The skipped counter rising 2→7 was the slow exposure of that dead zone.
A second state management failure in the same window
This wasn’t the only pipeline state problem active in July.
On 6 July, a long-slug IHT draft file (uk-inheritance-tax-rate-2026-40-charge-7-75-interest-rate.mdx) reached its third PR in a fortnight. Create-and-delete within a squash via PR #171. Merged live via PR #176, before I root-caused the regeneration source. The file existed simultaneously as merged-live content and as the target of a stale pending-publish regeneration cycle.
Different bug, same class. The pipeline’s record of completed work didn’t match actual state. For IHT, the conflict sat between git history and the pending-publish queue. For draft.py, between the failure log and the consumption index. Both produced phantom work: real processing cycles spent on inputs that should have been inert.
The fix
Improving on what “the cron ran clean and drafted nothing” covered: those pieces documented the observable pattern (zero auto-publishes across clean cron runs, operator publishes bridging the gap) without the root cause, which wasn’t yet identifiable. On 13 July, I found it.
draft.py now separates failed markers from consumed markers. A successfully drafted cluster is consumed: skip permanently. A cluster that failed a draft attempt is failed: eligible for retry on the next cycle.
The skipped counter now reflects a usable signal. Failed clusters re-enter the candidate pool. If the source pack is still viable and the API is available, they can produce a ready draft.
The ready_draft heartbeat counter is legible again. Non-zero means something cleared the full pipeline. Zero means a real problem to investigate, not a phantom accumulation of past failures presenting as a clean run.
What the architecture exposed
The consumption index was doing two jobs: recording successful work and recording failed work. That conflation was invisible at the interface level. Nothing in the contract between the triage layer and draft.py distinguished the two states. The index was append-only; anything written to it was evidence of completion.
Content automation pipelines have a state asymmetry worth naming explicitly. Successful work is permanent: a published article shouldn’t be re-drafted. Failed work is transient; an API timeout doesn’t mean the topic is permanently unwriteable. Treating both through a shared append-only index works at small scale, where failed markers are rare. At the scale where clusters accumulate over weeks, failures compound and the pipeline progressively starves itself.
The three-weekday gap was when that accumulation crossed a threshold visible in the publish record. Ten cycles of silent degradation before the signal appeared in output.
What the instrumentation missed
The skipped counter was the right signal. It was present in the logs throughout. The lag between it climbing and the root cause being identified was a reading problem, not an instrumentation gap.
Two changes that would shorten the diagnosis next time.
Separate counters per skip reason. skipped_consumed, skipped_failed, and skipped_quality are three different failure modes. A single skipped count obscures which population is growing. If skipped_failed had climbed from 2 to 7, the consumption index would have been the first place to look.
A failed-cluster audit command. The index is queryable. What’s missing is a CLI surface that lists clusters currently marked failed-* alongside their last-attempt timestamp. That’s a one-minute diagnostic that currently requires a raw SQL query.
Neither was in scope for the July 13 fix. The pipeline produced no publishes for three weekdays on a bug fixable in one pass once found. The instrumentation gap is what made it three days.
