The E2E suite billed 1,130 minutes to test a page that never existed

This month I asked a simple question: why is GitHub charging me? The answer took three commands and turned into the best CI lesson I have had in years. A Playwright end-to-end suite on the UK Calculators repo had run 69 times in three weeks, failed 61 of those runs, and billed about 1,130 minutes doing it. Roughly ninety per cent of the repo’s entire Actions consumption was a red test suite reporting the same failures on every push.

If you have never looked at your Actions bill, this article is for you. I had never looked either.

The part nobody tells you: Actions has a bill

Here is the model, because it is genuinely buried. GitHub Actions is free for public repositories on standard runners. For private repositories, every plan includes a monthly pot of minutes: 2,000 on Free, 3,000 on Pro and Team. Past that, you pay per minute. And the rates are not flat: Linux runners bill at $0.006/min, Windows at $0.010, and macOS at $0.062 — roughly ten times Linux. A test suite that quietly runs on a macOS runner costs you ten Linux suites.

Two more things I learned the hard way. First, if there is no payment method on file, GitHub does not bill you; it blocks Actions entirely once the pot is empty. If your deploys ever mysteriously stopped mid-month and came back later, that was probably the quota, not the pipeline. Ours did, in June. I diagnosed everything except the billing page. Second, the billing page itself lives under your account settings in the usage section for metered products, which is not somewhere a developer passingly visits. You have to go looking.

The three stacked mistakes

The bill was not one mistake. It was three, each hiding the one beneath it.

Mistake one: the suite was testing a 404. The mortgage page-object navigated to /mortgage. That route never existed; the real page is /mortgage-calculator. So the entire interaction and visual estate ran against the error page, and the visual-regression baselines the suite created were literally screenshots of a 404 labelled as calculator states. The suite was green-ish and worthless. I only caught it by eyeballing a regenerated baseline and recognising the error page in the thumbnail.

Mistake two: the honest repair made the suite red, and red became normal. Fixing the page-objects meant the suite tested reality for the first time, and reality had debt: assertion drift against a redesign, validation tests correctly catching a production no-op, a contrast case on decorative kanji. All real, all tracked in the backlog. But a suite that is red for known, catalogued reasons stops being a gate. Nobody reads a report that always says the same thing.

Mistake three: the red suite ran on every push, on a private repo, for sixteen minutes a time. This is where the first two mistakes started costing money. During an iteration loop that pushed twenty-plus commits, each push spent a quarter of an hour of billed compute to confirm what the backlog already recorded. The maths is brutal in its smallness: 61 failed runs at an average of 16 minutes is roughly 980 minutes. Half a Free plan’s monthly pot, spent on the word “failure”.

Diagnose it in three commands

You do not need the billing page to find this pattern; the API tells you everything. Count the runs and their durations per workflow:

gh api "repos/OWNER/REPO/actions/runs?per_page=100" \
  --jq '[.workflow_runs[] | {name, conclusion,
        mins: (((.updated_at | fromdateiso8601) - (.created_at | fromdateiso8601)) / 60)}]
        | group_by(.name)
        | map({workflow: .[0].name, runs: length,
               total_mins: (map(.mins) | add | round),
               failures: (map(select(.conclusion=="failure")) | length)})'

That one query produced the whole story: 69 runs, 1,130 minutes, 61 failures for Playwright; 31 runs, 102 minutes, zero failures for the fast quality gate. If a workflow’s failure count is most of its run count, you are paying for a broken alarm.

Then check what triggers it (on: push plus pull_request is the expensive default), whether concurrency with cancel-in-progress is set so stacked pushes cancel their predecessors, and which runner it uses. Ours had concurrency configured correctly and still burned, because cancellation only helps when runs overlap. Sixteen sequential minutes per push cancels nothing.

What to actually do

The fix is not deleting the suite. An honestly red suite is worth more than a green one that tests a 404; the repair that made ours truthful was the most valuable testing work on the project. The fix is matching spend to information value:

1. Keep the fast gate per-push. Typecheck, lint, unit tests, build: three minutes, always green, protects every commit.
2. Move the expensive suite to pull requests, a nightly schedule, and manual dispatch while it is known-red. You lose nothing: the failures are already catalogued. You regain the pot.
3. Cut the timeout to just above a healthy run. A twenty-minute timeout on a suite that passes in eight means every hang bills you twelve minutes of nothing.
4. Check the runner label. If runs-on: macos-latest appears in a workflow that would be equally happy on Ubuntu, you are paying the ten-times rate for no reason.
5. Restore per-push only when the suite is green and a failure would once again be news.

The rules I am keeping

A permanently red gate is not a gate. It is a meter. Either fix what it reports, or stop paying it to repeat itself.

And the meta-rule, the one that embarrasses me slightly: know which of your tools has a bill. I had run private repos with CI for months without knowing the pot existed, found out only when it emptied, and paid the tuition in blocked deploys and a pile of minutes spent screenshotting an error page. The billing model is documented, public, and completely invisible until you go looking. Go looking.