Posts on Testing.
5 posts tagged "Testing".
The E2E suite billed 1,130 minutes to test a page that never existed
A Playwright suite on a private repo ran 69 times in three weeks, failed 61 of them, and billed roughly a thousand minutes doing it. The root cause was three mistakes stacked on top of each other, and the deepest one was mine: I did not know GitHub Actions had a bill.
The first visible change to /learning/ arrived after the tests, the breadcrumb, and the JSON-LD
P2 of the learning-hub navigation was the first thing a visitor could see. By the time it shipped, the repo already had 37 passing fixture tests, a semantic breadcrumb, and a data model — none of it visible. The spec changed what shipped first, which changed what was easy to skip.
The diff check passed forever. That was the bug.
A test that checked whether verify.py was modified by a wiring PR became trivially true the moment PR #43 squash-merged. Phase X.8 replaced it with replay hashes, pre-registered gates, and hermetic sabotage cycles — tests designed to fail when they should.
The sabotage tests passed. The live cycle still found the bug.
Phase X.8 shipped with two hermetic sabotage cycles green. The first live run produced no article — not because the pipeline crashed, but because calculator_scores in ground.py was wrong in a way the fixtures had never exercised. Hermetic tests prove the logic path. Live harnesses prove the integration.
monkeypatch lies to you when your tests run in subprocesses
The fixture looked correct. The patch was applied. The subprocess ignored all of it and wrote to the real registry. Three days of roadmap work taught me that monkeypatch only rebinds the parent process — env-var overrides are the only isolation that survives a fork.