Scheduling and SLA recovery: shipping on cadence

Chapter 7 of 7. Prerequisite: Chapter 3. You know the queue states and the gates an article clears before it ships. This closing chapter is the clock: how slots fire on a fixed cadence, and what the system does in the minutes after a slot fails to fire.

Every previous chapter assumed an article publishes when its turn comes. In practice the turn is the fragile part. The pipeline runs on a laptop that sleeps. Its publisher can cancel a fire, and its duplicate gate can block one at the last second. A missed slot that vanishes silently kills the cadence, and a cadence that dies quietly is how a publishing habit holds for three weeks and then stops. This chapter documents the slot model, the past-due grace ladder, and the SLA recovery stack rebuilt after 72 hours in which five scheduled slots produced one published article, and that one only because a human intervened before the alarm TTL.

Build in public: turning notes into posts on a schedule

This layer exists for a failure mode older than the pipeline. One Indie Hackers post, "After 8 months of drafts, I completed my first blog post", puts a number on the backlog: "During these 8 months, I have accumulated a handful of partial drafts and 30 or so ideas for posts." The notes exist. The workflow from notes to posts exists in the author's head. What never existed was a date on which any specific post had to ship.

Against that failure mode the pipeline sets an editorial SLA, one sentence in the devlog: "one article publishes at 09:00 BST every day, no exceptions." Everything in this chapter is machinery in service of that sentence. The devlog supplies the notes. The gated queue turns notes into drafted posts. The scheduling layer makes a specific drafted post due at a specific time, and the recovery stack makes a missed time an alarm instead of a shrug.

Writing the SLA down matters more than the number in it. A cadence in your head degrades one reasonable-feeling skip at a time. A cadence written as a rule gives the machine something to enforce, and the rest of this chapter is what enforcement looks like when the enforcer is a daemon on a laptop.

Slots are data, not cron lines

Cadence lives in one config row, a JSON value shaped like this:

{
  "mode": "once_daily",
  "times": ["09:00"],
  "tz": "Europe/London"
}

Slot times are local wall-clock times, resolved through the timezone database so the cadence survives DST. The 09:00 slot means 09:00 on the reader's clock in June and in January; the UTC instant shifts underneath it.

Slot assignment is a pure function: given a start time and the set of taken slots, return the next free one. No database reads inside the function, a 30-day lookahead ceiling as a defensive bound. Purity here is a testing decision: a function with no clock and no database can be driven with synthetic times, so DST shifts and window boundaries get exercised in tests before they cost a morning slot.

Execution is a sweep: a loop in the pipeline daemon ticks every 30 seconds, fires at most one due row per pass, and hands it to the publisher, where the full gate stack from Chapter 3 still runs. The sweep decides when. The gates still decide whether. The batch limit of one serialises the irreversible end of the pipeline, so a bad fire can be stopped before the next one starts.

A missed slot must not silently vanish

One constraint underlies everything in this section, stated plainly in the article it draws on: the sweep runs on a MacBook, not a server. Laptops sleep. The sweep pauses with the lid.

Its founding incident was small and clarifying. The daemon woke at 09:07 and found a row scheduled for 09:00 that morning. It had no policy to apply: fire late, or sit in scheduled indefinitely. Neither behaviour was defined, so both were wrong. An article seven minutes late is fine. An article six days late was written for a context that has expired. The fix is a past-due grace ladder, classifying every due row by how late it is:

Of the four tiers, expire is the load-bearing one. Expiry is a decision, recorded as a state transition with a reason, recoverable by rescheduling if the article still deserves a slot. A silently dropped row is none of those things. The resulting invariant: every article either publishes, expires with a reason, or surfaces as a chat decision. "There is no fourth outcome."

20% in 72 hours: when recovery consumes the future

Grace handles a sweep that was asleep. It does nothing for a slot that fires and then fails, and that is where the scheduler was actually bleeding. The numbers come from the rewrite post-mortem: across 72 hours, five consecutive scheduled slots produced one published article, a 20% success rate in the outcome log. Even that one required a human intervening before the alarm TTL, so it was not autonomous success. The other four expired.

All the while, the substitute pool was running, promoting the next available candidate exactly as designed. The design was the bug: nothing distinguished unscheduled rows from rows already committed to a future window, so "next available" drained tomorrow's inventory to patch today.

Here is the concrete cascade, from the audit trail of 13 June: queue id=37's slot came up, the sweep fired the publisher at 11:00:08Z, and the publisher cancelled the row. The pool promoted a candidate from the 12:00 window. The 09:00 slot was covered. At 12:00 the pool had nothing, and the missed-slot alarm fired twice that morning. One primary failure, two cascade events. The pool had not malfunctioned; its definition of "available" had.

Three fixes shipped the same day, each closing a distinct path:

1. Sibling cancel. When an article ships, every other queue row converging on the same published slug and still in a cancellable state is cancelled immediately. Before this, sibling variants of an article that had just shipped stayed live in the queue, eligible for promotion as substitutes, which is how a rehash of something already published gets handed a slot.
2. Substitute ordering. The pool now draws from unscheduled rows first and reaches into future-committed windows only as a last resort. A same-afternoon follow-up fixed the walk's recursion so it terminates at the last viable candidate.
3. Alarm rewrite. The missed-slot alarm moved out of a post-hoc report and into the dispatch path: every sweep pass checks for slots between 2 and 15 minutes past their fire time with no shipped article and no recorded substitution, and raises a chat alarm, idempotent per slot for 24 hours. Each payload carries the queue id, the slot time, and the upstream cancellation reason, so the alarm tells you what to do inside the response window, not just that a slot died. The stance behind it, recorded in the devlog: "Better an alarm than a sibling-rehash publish." When no substitute clears the citation-overlap gate, publishing a near-duplicate to save the slot would satisfy the SLA and rot the archive; the alarm hands the slot to a human instead.

After the rewrite the stated target was autonomous success above 80%, four of five slots resolving without human input, framed in the post-mortem as a testable claim. Below that number, the fallback ordering or the sibling-cancel scope is still wrong.

Make the slot a first-class row

Underneath the three fixes sat a structural repair. Before the rewrite, a slot existed only by convention: slot identity was implicit in each queue row's scheduled time. When the pool promoted one row to cover another, no record linked them. Reconstructing a substitution chain meant querying queue-row events by timestamp and inferring the lineage, and inference under incident pressure is where audit trails go to die.

So the slot became its own table row with an explicit state machine: pending → firing → shipped | substituting | breach. Every transition writes a paired update, the state change plus an audit event, in one transaction, with the event carrying from-state, to-state, actor, and attempt number. One transaction because of the property SQLite's documentation states exactly: "Atomic commit means that either all database changes within a single transaction occur or none of them occur." A crash between the state change and the event would leave a trail that lies. Atomicity makes that gap unrepresentable.

Each rule the state machine carries was earned by an incident:

• Forbidden self-transitions. firing → firing and substituting → substituting are illegal; a re-fire must go through substituting and back. This kills the bug class where a retry loop relabels the same attempt forever.
• Idempotent re-calls. The same transition with the same queue row is a no-op; with a different queue row it raises. Terminal states accept repeats silently, so a daemon that restarts mid-operation can replay its last intent safely.
• A per-slot rate limit of five seconds between transitions. The motivating record predates the state machine: an 18:00 audit trail once showed 11 events land in three seconds with no rate limiting, a blur of interleaved actions no human could replay under incident pressure. The limit forces distinct decisions to occupy distinct moments.
• Orphan recovery. Any slot stuck in firing or substituting for over 15 minutes, or still pending 15 minutes past its fire time, transitions to breach with the reason recorded as recovery after restart. A laptop-hosted daemon will die mid-operation eventually; the recovery pass turns "eventually" into a known state instead of a wedged one.

A verification run on 14 June shows what the machine records now: a full promotion chain, id=51 to id=53 to id=54 to id=55 (rejected) to id=58 and then breach, every transition carrying a reason. During the 20% week, reconstructing that lineage would have meant inference over timestamps. Now it reads as complete information straight out of the audit trail.

One recorded trade-off is worth internalising: the editorial SLA was prioritised over schedule determinism. A slot now fires something from the ready pool; "the assigned row is a preference, not a binding." If you need slot N to publish article X and nothing else, this design is wrong for you. If you need 09:00 to publish, it is right.

A substituted article clears every gate the primary would have, from the duplicate fingerprint check through to the deterministic voice budget. Voice has its own course; if your substitutes can ship prose that reads generated, the schedule is saving slots and spending trust, and the companion course's pre-publish style audit is the gate to bolt on before anything publishes unattended.

The pool that was empty by construction

One assumption survived the rewrite and fell later. Substitute ordering preferred unscheduled drafted rows, and that preference could never trigger: the auto-driver pre-schedules every drafted row the moment it exists, so the free pool was empty by construction. Sound design, wrong steady state.

Steal-from-later replaced it: the walk may take a future-scheduled drafted row, preferring a slot-of-day match, rewrite its scheduled time, and record a reassignment event carrying the before and after fire times. The vacated future slot refills on later ticks. Both walks are capped, three attempts on the publisher-block path and fifty candidates on the citation-rejection path, so a pathological pool cannot spin the dispatcher.

Adopt the shape, not the constants

1. Write the SLA as one enforceable sentence

   "One article publishes at 09:00 every day" is enforceable. "Publish regularly" is not. The sentence defines what counts as a breach; everything else keys off that definition.

2. Store slots as rows with explicit states

   A slot implicit in a timestamp column cannot record a substitution chain. Give it a table, named transitions, and a paired audit event in the same transaction as every state change.

3. Classify lateness on a ladder

   Silent tolerance for jitter, automatic fire for recoverable lateness, a nudge when something real held the publish up, hard expiry past the point where context is stale. Expiry is a recorded decision, never a silent drop.

4. Recover with an ordered substitute walk

   Unscheduled inventory first, stolen future slots as a capped last resort, every reassignment recorded. Check the ordering against your steady state: a preference for a pool that is always empty is a no-op wearing a design document.

5. Alarm inside the response window, idempotently

   Alarm minutes after the miss, with the queue id, the slot time, and the upstream reason in the payload, deduplicated per slot per day. An alarm a human can act on beats a recovery that publishes something worse.

Where the course lands

Seven chapters, one system: a devlog you append to mechanically, a gate that forces the thesis before the prose, a state machine in a local SQLite file that remembers every decision, and now a clock that makes the whole thing ship at 09:00 whether or not you felt like publishing.

None of this final layer was designed up front. The grace ladder exists because a laptop woke at 09:07 with no policy. The slot state machine exists because a 20% week demanded an audit trail that did not require inference. Steal-from-later exists because the first fix preferred an inventory that could not exist. Each mechanism is scar tissue over a specific failure, recorded in a devlog entry, and those entries became the published articles this chapter cites. That loop is the course's last argument for itself: the pipeline that publishes the notes is also generating them.