Posts on DevOps.
89 posts tagged "DevOps".
The publisher cancelled the 12:00 slot. My recovery logic called the publisher again.
Queue id=37 was scheduled for 12:00 BST on 13 June. schedule_sweep fired publisher at 11:00:08Z, publisher cancelled the row, and Fix B (the SLA recovery layer) called publisher again. Recovery logic without a model of who holds the lock cannot distinguish a concluded failure from a recoverable one.
Auto-mode opened the PR. It didn't merge it. That was gap one.
Phase 9's 'self-driving overnight' was overstated by two wiring gaps. A 09:00 BST live run on 11 June found both. The afternoon of fixes taught more about auto-mode's real edges than any pre-flight design session.
The publishing queue ran at 20% success rate. Fixing it took three changes that only work together.
Four autonomous publishing slots expired before the pattern became clear. Sibling cancellation, slot fallback, and a missed-slot alarm are a set — without all three, the queue degrades silently.
20% in 72 hours: rewriting the slot scheduler's SLA recovery
Twenty percent autonomous success across five scheduled slots in 72 hours. The substitute pool was running but consuming the wrong inventory — draining future windows to cover present failures. Three mechanisms and a first-class slot state machine fixed it.
I treated publication slots as timer events. Three days of SLA breaches say otherwise.
Between 11 and 13 June, the 09:00 BST publication slot fired and produced nothing three times in a row. Each incident had a different surface cause. All of them trace back to the same modelling mistake.