Posts on DevOps.
63 posts tagged "DevOps".
The sourcing step was always the bottleneck. Day 6 moved it into the cron.
For five days, every ukcalculators claim block was hand-sourced by the operator from individual gov.uk pages. On Day 6, the cron polled the gov.uk Search API live for the first time. The sourcing method changed. The claim-block discipline didn't.
The cron kept drafting mortgage because gov.uk wasn't in the source set yet
For five days, the cron produced news-cluster drafts and operators shipped gov.uk regulatory pillars by hand. Day 6 changed one thing: gov.uk Search API sources entered the polling loop. Two data-pegged pillars from a single cycle followed immediately.
The privacy work isn't on the /privacy/ page
The /privacy/ page went live in June and covers what it covers. The actual privacy work on this site happened in a data-locality choice, a brand-separation rule, and a username leak in published MDX. Only one of the three announced itself.
10 of 13 sentences made it. The verifier caught the rest before they reached a pillar page.
The June 30th cron produced a mortgage rate-spike draft grounded in April 2026 data. Ten of thirteen sentences passed. The three that didn't never reached a pillar page — and the next two days showed why that distinction matters at publishing scale.
14 days of daily publishing: the three failure modes I hadn't designed for
A 14-day publish cadence on UK Calculators surfaced three failure modes: title-mangling produces permanent ugly slugs, zero-draft mornings expose the absence of a fallback queue, and triage without memory republishes the same cluster it shipped the day before.
The E2E suite billed 1,130 minutes to test a page that never existed
A Playwright suite on a private repo ran 69 times in three weeks, failed 61 of them, and billed roughly a thousand minutes doing it. The root cause was three mistakes stacked on top of each other, and the deepest one was mine: I did not know GitHub Actions had a bill.
The backlog called it a content gap. It was navigation.
Completing an eight-chapter course on the learning hub felt like progress. Then I looked at what had actually moved the needle — track sections, prerequisite chips, and a JSON-LD breadcrumb. The backlog had been mis-filing the bottleneck for months.
The cron produced zero drafts. That was the correct answer.
Four mornings into a 14-day operator-publish sprint on UK Calculators, the pipeline ranked 150 clusters, fetched candidates, and produced nothing. Not a failure. A signal that the ground-truth layer, not the LLM, is the binding constraint in a pipeline built on live gov.uk sources.
Fifteen claim blocks and a zero-draft cron: the cost of grounding a content pipeline in gov.uk
Hand-curated gov.uk claim blocks stopped the ukcalculators drafter fabricating figures. Three days after the first clean pillar shipped, the cron fired on time, ranked 150 clusters, put mortgage at the top — and produced nothing. That was the gate working correctly.
The eight-pillar map closed in three days. ga-telemetry and gsc-telemetry are why.
Phase 1.6 shipped two observability skills on 20 June: ga-telemetry and gsc-telemetry. Six days later the eight-pillar topic-authority map was complete. The link is an operator loop built on next-morning indexation signal.
`ignoreBuildErrors: true` kept the type checker quiet. Turning it off found twelve product bugs.
UK Calculators had `strict: true` in tsconfig and `ignoreBuildErrors: true` in next.config.mjs. The two settings coexisted for months. One afternoon removing the flag surfaced twelve masked type errors on a YMYL money site where wrong numbers carry real consequences.
Scope the MySQL user first: the newsletter DDL as a security boundary on cPanel
On shared hosting, you don't control the firewall or the infrastructure. What you do control is which privileges the MySQL user holds. Scoping it to the newsletter schema alone made a compromised endpoint's blast radius structurally finite — not as a mitigation layered on top, but as a property of the design.
Ten cycles, one ready draft: the pipeline was consuming its failures
Between 30 June and 13 July, ten cron cycles produced one ready draft and zero auto-publishes while the skipped counter climbed from 2 to 7. The drafter was not stalling. It was treating failed attempts as permanent completions, and only heartbeat visibility on ready-draft count made the dead zone visible.
The cron exited clean. There were no drafts.
Two zero-draft mornings out of four cron firings across 29 June to 2 July, both exits clean. The failure was Haiku's claim extraction underfiring one stage before draft: structurally invisible, no error, no warning, cron returns success.
The cron fired at 08:22 and drafted nothing — scheduling is the easy part
On 2 July, the pipeline ran on time, ranked 150 clusters, fetched candidates, and produced zero drafts. The scheduler worked. The grounding step didn't. Automating gov.uk pillar articles forces a distinction that took a zero-drafts morning to make explicit: cron reliability and content freshness are different problems, and only one of them is solved.
Three weekdays, zero publishes: draft.py was consuming its own failures
Between 7 and 9 July the UK Calculators cron fired on time, triaged candidates, and assembled source packs. It published nothing. Across ten cycles, skipped climbed from 2 to 7 before the cause was found: draft.py was writing failed-* markers to the same index it used for successful consumption, permanently locking out any topic that had ever failed a draft attempt.
Five days, ten operator publishes, two cron drafts: where the ukcalculators pipeline actually breaks
Running a 14-day pillar schedule with a parallel operator-publish path made the bottleneck legible: the cron fires on schedule, triage completes, 150 clusters rank — then the ground-and-extract stage drops everything silently. The schedule is not the problem.
Ten of thirteen: why verification without correction is just expensive triage
The morning cron produced a draft with 10 of 13 sentences passing the verifier. Without a corrective step, that's a triage ticket, not a publish. Four days of cron output showed the rescue pattern fires once in four — and what covers the other three.
150 clusters ranked, zero drafted: ground attrition is the load-bearing constraint
Five days of autonomous publishing logs made the pattern clear. The cron fired on schedule every morning. Ground attrition killed the drafts before the drafter ran. One usable cron draft across five days; ten articles published via a different route entirely.
Six PHP classes, no Composer: a cPanel newsletter backend with enumeration-resistant auth
The captainrandom.co.uk newsletter backend runs on a cPanel shared host with zero framework and no Composer. Six small PHP classes, around 600 lines, and an eight-stage subscribe pipeline that tells the caller nothing about whether an email address already exists.
I hand-wrote the newsletter backend in PHP. The threat model is the code.
The captainrandom newsletter subscription backend is 600 lines of PHP, six classes, no framework, no Composer. Every security decision — rate limits, honeypots, time-traps, token hygiene — lives in the endpoint files, readable end-to-end in one sitting. That's the trade: no ecosystem, but a threat model you can hold in your head.
GA4 went dark for four days. Both fixes were right. One was in the wrong place.
Two PRs, each diagnosing a real fault, and GA4 still flat four days later. The problem wasn’t whether the consent signal was present — it was where in the document it lived relative to gtag.js.
Four cron zeroes in five mornings. Eight articles shipped by hand. The pipeline problem isn't the scheduler.
Five days into a fourteen-day publishing sprint on UK tax calculators, the scheduler fires within a minute of its window every morning. Automated drafts produced: two, one a duplicate cluster. Gov.uk cooperates. The bottleneck is claim extraction and cluster survival.
The voice-check agents passed two chapters. A three-line recount proved they couldn’t count.
An ultracode workflow’s voice-check agents passed two course chapters as within the em-dash budget. A three-line Python recount found both over. A model can’t reliably count its own tells.
The process changed. The docs didn’t. The fix was a folder.
Three times in the captainrandom repo, documentation drifted from actual practice. Each time the fix was the same: a dedicated folder with its own PRD, roadmap entry, and devlog trail — not a patch, but enclosure.
Four lines in a published article leaked my macOS username to the live site
PR #17 was a four-line diff across two files. One of those lines was in the body of a published article, rendering a real macOS username in the browser. The fix was substitution — but the pattern it revealed was wider: three surfaces in the same audit window, the same failure.
I patched the same privacy leak four times. The lesson isn't to be more careful.
Private repo names in a language stats page. A macOS username in a published plist example. A missing privacy policy when the newsletter went live. Four separate fixes, each reactive, each after deployment. Four is enough to name the pattern: the publish boundary has no scrubber.
145 words shipped on inheritance tax. 949 words fixed it. The gate measured the wrong thing.
The operator-publish gate cleared 145 words on UK Inheritance Tax because word count was above minimum and the verifier passed. IHT is YMYL content. A same-day ultracode rescue to 949 words proved the gate needs a depth signal, not just a length floor.
The substitute pool stole from future slots because slots didn't exist as rows
Phase B's autonomous publishing pipeline sat at twenty percent success. The substitute pool was recursing through future-scheduled rows to breach — and nothing in the data model could stop it because slots had no identity. Making the slot a first-class entity with explicit state transitions turned six log events into six queryable rows.
Correct answers, wrong questions: four DVLAW scope errors in 72 hours
Between 2026-06-10 and 2026-06-13, four DVLAW bugs caused cascading slot failures, auto-published style violations, and a near-duplicate article. None threw an exception. All of them were answering the wrong question correctly.
The identifiers I didn't know I'd published
Five fixes across /learning, /now, a published article, the privacy policy page, and a global pre-commit hook — all tracing back to the same root cause. Not tracking, not data harvesting. Just private names in places I'd stopped looking.
The privacy policy was correct. The article body wasn't.
A production audit found the macOS username in a published plist example and private project names on two public pages. Neither would have been caught by a policy document. The fix was a pre-commit hook and an aggregation change — pipeline enforcement that runs whether the author remembers or not.
Five Phase B fixes in 48 hours. They were all patching the same missing type.
Twenty percent autonomous success across five scheduled slots. The same breach shape surfacing twice from different audit trails. Five fixes shipped in 48 hours — and when the sixth investigation opened, the pattern became unmistakable: the slot had never been an entity.
The sabotage tests passed. The live cycle still found the bug.
Phase X.8 shipped with two hermetic sabotage cycles green. The first live run produced no article — not because the pipeline crashed, but because calculator_scores in ground.py was wrong in a way the fixtures had never exercised. Hermetic tests prove the logic path. Live harnesses prove the integration.
The publisher cancelled the 12:00 slot. My recovery logic called the publisher again.
Queue id=37 was scheduled for 12:00 BST on 13 June. schedule_sweep fired publisher at 11:00:08Z, publisher cancelled the row, and Fix B (the SLA recovery layer) called publisher again. Recovery logic without a model of who holds the lock cannot distinguish a concluded failure from a recoverable one.
Auto-mode opened the PR. It didn't merge it. That was gap one.
Phase 9's 'self-driving overnight' was overstated by two wiring gaps. A 09:00 BST live run on 11 June found both. The afternoon of fixes taught more about auto-mode's real edges than any pre-flight design session.
The publishing queue ran at 20% success rate. Fixing it took three changes that only work together.
Four autonomous publishing slots expired before the pattern became clear. Sibling cancellation, slot fallback, and a missed-slot alarm are a set — without all three, the queue degrades silently.
20% in 72 hours: rewriting the slot scheduler's SLA recovery
Twenty percent autonomous success across five scheduled slots in 72 hours. The substitute pool was running but consuming the wrong inventory — draining future windows to cover present failures. Three mechanisms and a first-class slot state machine fixed it.
I treated publication slots as timer events. Three days of SLA breaches say otherwise.
Between 11 and 13 June, the 09:00 BST publication slot fired and produced nothing three times in a row. Each incident had a different surface cause. All of them trace back to the same modelling mistake.
Four expired slots and the SLA recovery rule we learned in sequence
Between 12 and 14 June, four scheduled DVLAW slots expired without autonomous recovery. Each failure confirmed the same thing: sibling cancel, substitute fallback, and a missed-slot alarm don't work in isolation. They're one system.
The S-4 cohort made two editorial pipeline decisions. I shipped them.
Two weeks of UK Calculators editorial pipeline sprints ended with two decisions that came from the cohort data, not preference: live Bank of England rate fetches and a constrained LLM composer. The pipeline silently stalled for four days after shipping.
Five slots, twenty percent: the substitute pool had no editorial policy
Five scheduled publishing slots, four expirations, and a breach chain that exhausted six candidates in three seconds. The scheduler fired on time every run. The substitute pool was offering articles the drafter’s citation rules were going to refuse.
Two correct cancellations, two SLA breaches, four manual ships
On 13 June, the 09:00 and 12:00 BST editorial slots both failed in the same day. G-L3 cancelled the queued articles correctly each time. Manual recovery covered every slot. The experience made the gap between row-level correctness and slot-level reliability impossible to ignore.
Four sprints shipped before the installer resolved to the right directory
The idea park soak cron installer used a directory convention copied from the article pipeline, correct in the article pipeline's layout and wrong in the idea park-soak's co-located layout. Two more post-merge fixes followed the same shape: a sanitiser wired into one code path but not the other, and a caller convention documented but never enforced.
Three newsletter hotfixes. The events table found all of them.
The newsletter form went live on 9 June. I subscribed from my own address and got nothing. No row, no email, no error. Three consecutive bugs followed, all found the same way: reading the events log, not the code.
No Composer, no autoloader: the newsletter backend cPanel rewarded
The deployment target was a cPanel shared host with FTP access. Composer wants SSH or a committed vendor directory. Neither fits cleanly on shared hosting. The alternative was require_once, six PHP classes, and a 190-line schema that doubles as documentation. Six sprints in, the backend is completely understood.
Scheduling articles through Telegram taught me that cron is the wrong model
The article pipeline pipeline could draft and voice-fix articles but had no scheduling layer. Adding one took six sprints, not because the cron was hard but because scheduling through Telegram needs to capture intent, not timestamps. The grammar parser, the 2×2 keyboard, and the 48-hour grace policy are the same design decision expressed three different ways.
The inbox filter model drops strangers silently. I built a queue instead.
Inbox filters drop what they don't recognise. I wanted a record instead: every unknown sender parked, every blocked arrival logged, nothing silently gone. Five tables, four sync branches, and a historyId cursor later, the editorial pipeline's Inbox Hub does exactly that.
Archon was burning quota. The fix was a hook, not a scheduler.
Three Archon runs in flight simultaneously, two of them fired without a deliberate decision. The instinct was to build a smarter scheduler. What worked was a global PreToolUse hook that intercepts archon workflow run and forces a pause before anything spawns.
Making Gmail tractable in a day. Four branches and one partial index.
The Inbox Hub shipped in five sprints, all on the same day. The classifier is four comparisons against a rules table. The speed comes from a partial index on pending senders, not from the sync logic. Neither changed how I read email.
I built six sprints of article scheduling. For four of them, nothing fired.
Four sprints of Telegram button wiring, queue state machines, and free-text time parsing — all recording intent into `article_queue` without dispatching a single article. Sprint 5 was the first one that actually fired. Separating mechanism-to-record from mechanism-to-execute isn't a new idea; applying it deliberately at the sprint level is.
The schedule queue took four sprints to remember. One to act.
After four sprints of B-schedule, a scheduled article existed as a database row with a correct timestamp. No article had published. Sprint 5 added the sweep daemon and publisher_lock. That is when scheduling became real.
A missed slot at 09:00 should not silently vanish: building past-due grace into the article pipeline publisher
The `schedule_sweep` runs on a laptop. Laptops sleep. Without a grace policy, an article scheduled for 09:00 silently strands when the machine wakes at 09:07. Sprint 6 of B-schedule — past-due grace plus 48-hour hard-expire — turns that silent drop into a recoverable state with a reason, an audit trail, and a path back to publish.
The classifier is the product: how four branches replaced an inbox
Inbox Hub is five SQLite tables, twelve route handlers, four Shadow DOM components, and a launchd cron. Strip those away and what remains is a four-branch decision tree that runs on every message. Everything else is plumbing around that core.
monkeypatch lies to you when your tests run in subprocesses
The fixture looked correct. The patch was applied. The subprocess ignored all of it and wrote to the real registry. Three days of roadmap work taught me that monkeypatch only rebinds the parent process — env-var overrides are the only isolation that survives a fork.
One input to rule the dashboard: how the editorial pipeline's chat surface replaced three UIs
The editorial pipeline's chat interface started as a conversational wrapper. Two days of commits later it had absorbed the Archon SPA, the dashboard nav, and a 16-command palette. The unifying mechanism is simpler than it looks.
Ten AI-prose tells, one shell script, no more guessing
The em-dash density in my own published articles was running at four times the normal rate for edited prose. That was a measurable problem. So I made it measurable: codified ten AI-prose tells into STYLE-GUIDE.md and a pre-publish shell audit that catches them mechanically.
The backtick that could run anything: hardening AppleScript shell escaping
A Sprint 2.11 review flagged a shell command-substitution gap in a single Python helper. Backticks and dollar signs were passing through unescaped into a double-quoted `do script` string, meaning a crafted issue title could execute arbitrary code. The fix was four lines. Understanding why it mattered took longer.
Two assumptions the pipeline held until it shipped
B8b's first end-to-end Telegram ship-tap exposed two bugs that had been invisible in every prior test. Neither was logic. Both were assumptions the pipeline had been making silently since the day the code was written.
Telegram 409 Conflict: two bots, one token, infinite errors — why a bot registry is infrastructure
Retrofitting the editorial pipeline's Telegram modules revealed that the tunnel bot and the brain server had been polling the same token for weeks, throwing 409 conflicts that neither log named clearly. The fix wasn't a config tweak. It was accepting that one-bot-per-purpose needs to be enforced at registration time, not discovered at incident time.
Why the editorial pipeline's rewrite layer runs on a project-owned harness, not Archon
The Opus rewrite layer inside the editorial pipeline's de-AI-ish stack runs through a project-owned osascript Terminal harness, not a general orchestration framework. That choice was deliberate and narrow. Here's what it cost and what it bought.
macOS won't let your cron read your work: three days inside TCC
A daily launchd job that reads from /Volumes/b/ looked trivial. Three days, four wrong approaches, and one AppleScript .app bundle later, I'd learned why granting Full Disk Access to a shell script is a UI fiction.
Static Next.js on cPanel in 2025
GitHub Actions to FTP to cPanel. Simpler than it sounds, with a few real gotchas worth documenting.