Incidents and recovery

Ten chapters have described a system that mostly works: gates that catch mistakes, pipelines that deploy them, dispatches that run on a clock. This chapter is about the times it didn't. Every incident below actually happened on this site's own pipeline, and every one of them ended with an entry in docs/devlog.md before the next piece of work started. That last habit matters as much as any of the recoveries. An incident you don't write down is one you get to have again.

The push rejection: gitignore before staging, not after

The most mechanical incident in the repertoire is also the easiest to prevent and the easiest to walk straight into: a build artefact, tens of megabytes, staged and committed before anyone checked its size. GitHub's own documentation is specific about where the wall is. It warns "if you attempt to add or update a file that is larger than 50 MiB," and it hard-blocks with "GitHub blocks files larger than 100 MiB" (the browser upload path is capped lower still, at 25 MiB). A 60 MB dist/-style bundle sits squarely inside that warned-then-blocked band: small enough that git add -A doesn't complain, big enough that the push to GitHub does, with a rejection that lands well after the commit already exists in local history.

The fix that actually holds is not "delete the big file and recommit." Git's own docs say plainly that a gitignore file specifies intentionally untracked files that Git should ignore, and that "files already tracked by Git are not affected." Once a build artefact has been committed, adding it to .gitignore does nothing, it's still tracked, still in the object database, still going to get pushed on the next commit that touches it. The correct sequence is git rm --cached <path> to untrack it, then add the ignore pattern, then commit both. Skipping the rm --cached step is the single most common way this recovery gets half-done: the file stops changing in future diffs but the original oversized blob is still sitting in history, still costing push bandwidth, still there if anyone runs git gc and wonders why the repo hasn't shrunk.

The orphaned PR and the lock that outlived its process

The sharpest incident in this site's history wasn't a bad commit, it was a process that died holding something. A scheduled publish opened a pull request, then the process driving it died before the PR merged and before it released a single-publisher lock it held internally. Because that lock lived inside a long-running daemon's own memory rather than anywhere git or GitHub could see, it never auto-released when the process that acquired it disappeared. Every subsequent publish attempt, roughly every 30 seconds, failed against the held lock. For 25 hours. Around 3,000 failed attempts. No alarm fired, because nothing was watching for "the lock has been held for an implausibly long time," only for "did the last attempt succeed."

Nobody noticed a system-level failure. What got noticed was an absence: two days went by with nothing new on the site, and a human asked why. That's the shape worth sitting with. A crashed process that leaves no trace looks identical, from the outside, to a system with nothing scheduled to do.

The recovery had three parts, in order, and the order was the point. First, merge the orphaned PR by hand, the actual content it carried was good, it just never got the chance to land. Second, reconcile the internal state that thought the publish had failed, marking it as manually recovered rather than leaving a record that contradicted what actually happened on GitHub. Third, and only after the first two, restart the daemon. A new process acquired the lock cleanly within seconds and fired the stuck slot 24 seconds later. Restarting first would have "fixed" the symptom while leaving state and GitHub disagreeing about what had shipped.

Rerun once before you diagnose

Not every red run is a regression. A CI failure on a pull request touching completely unrelated code once traced back to the workflow file layer itself, no source change nearby, no reason for a real fault. The correct first move in that situation is not a deep dive, it's a single rerun of exactly the failed jobs:

gh run rerun <run-id> --failed

The GitHub CLI documents this precisely: it reruns "only failed jobs, including dependencies," not the whole workflow. On that occasion the rerun passed clean, which is the evidence that the failure was transient (a runner hiccup, a momentary GitHub-side blip) rather than something in the diff. That one rerun is cheap enough to always try first and informative enough to actually change what you do next.

It's also worth what the same incident surfaced downstream. The tooling watching for the PR to go green had been polling the wrong signal, a checks summary endpoint that returned empty during the exact window the failure happened, which meant the watcher itself stalled waiting for information that never arrived. The fix wasn't a better rerun policy, it was rewriting the watcher to poll the run directly instead of a derived summary that can be empty precisely when you need it most. A transient CI failure and a fragile piece of automation had been coexisting the whole time, and the transient failure is what exposed the second problem.

If a rerun does not come back clean, that's your signal to stop treating it as noise. Look at the diff, look at the runner logs, and if the run has aged past its rerun window, the fallback lever is a fresh manual dispatch:

gh workflow run <workflow> --ref <branch-or-tag>

Quota-blocked Actions is its own incident class

Some incidents aren't a bug in your workflow at all, they're a billing ceiling you didn't know you were near. This site's Actions minutes tripped a billing block that ran from 2026-06-18 to 2026-06-29, well over a week, and for that whole window every deploy that would normally be git push and a green run instead became a manual tar | ssh ceremony run by hand. Nothing was broken in the YAML. The platform simply stopped running jobs until the account cleared.

The honest line from the article that came out of that stretch is the retro for the whole incident class: the billing model is documented, public, and completely invisible until you go looking. Chapter 10 covers the mechanics of watching usage before you hit the wall. What belongs here is the recovery posture: know your manual deploy path cold, before you need it, because the moment Actions stops running is the worst possible moment to be improvising an FTP push by hand for the first time. A pipeline with no offline fallback doesn't have a recovery option, it has a total outage with the platform's name on it.

The retro is the recovery: write the entry before you move on

None of the recoveries above would be worth as much without the discipline underneath all of them: a plain, append-only file at docs/devlog.md, no tooling, no automation, just the practice of writing the entry before starting the next piece of work. It didn't always exist. It was created deliberately to mirror a convention used elsewhere, and on the day it was created it was immediately backfilled with eleven entries covering an entire prior audit phase that had shipped with zero written record. The backfill wasn't nostalgia, it was closing a gap: work that isn't written down is work nobody, including the person who did it, can reliably reconstruct later.

Every incident in this chapter has a devlog entry that names the chain of failure and the exact recovery steps, in order. That's not incidental to how they got fixed, it's part of why they stayed fixed. When a comparable failure shows up months later, the fastest diagnosis is not first-principles reasoning from scratch, it's grep. A retro is only worth writing if it's specific enough that "grep this" actually works: name the symptom, name the root cause, name the recovery recipe in the order it has to happen, and say what changed afterwards so the same shape of failure doesn't repeat unnoticed.

Building your own repertoire

You will not hit these exact four incidents in this exact order. You will hit your own versions: a build tool that changed its output path and started committing something huge, a background job that dies holding a resource nothing else can see, a CI failure that turns out to be nothing, a quota you didn't know existed until it stopped you cold. The specific recoveries above matter less than the pattern behind all of them: check before you stage, know what a stuck lock looks like and fix state before you restart anything, rerun once before you dig deeper, know your manual fallback before the automated path disappears, and write down what happened while you still remember the order it happened in.