Posts on Testing.
6 posts tagged "Testing".
post_today opens two connections. The test fixture only controlled one.
Three tests in test_post.py broke when auto_mode was turned back on in production. No code had regressed. post_today also opens its own connection to read the auto_mode flag, independent of the connection the fixture supplies, and the test hadn't controlled it.
The E2E suite billed 1,130 minutes to test a page that never existed
A Playwright suite on a private repo ran 69 times in three weeks, failed 61 of them, and billed roughly a thousand minutes doing it. The root cause was three mistakes stacked on top of each other, and the deepest one was mine: I did not know GitHub Actions had a bill.
The first visible change to /learning/ arrived after the tests, the breadcrumb, and the JSON-LD
P2 of the learning-hub navigation was the first thing a visitor could see. By the time it shipped, the repo already had 37 passing fixture tests, a semantic breadcrumb, and a data model — none of it visible. The spec changed what shipped first, which changed what was easy to skip.
The diff check passed forever. That was the bug.
A test that checked whether verify.py was modified by a wiring PR became trivially true the moment PR #43 squash-merged. Phase X.8 replaced it with replay hashes, pre-registered gates, and hermetic sabotage cycles — tests designed to fail when they should.
The sabotage tests passed. The live cycle still found the bug.
Phase X.8 shipped with two hermetic sabotage cycles green. The first live run produced no article — not because the pipeline crashed, but because calculator_scores in ground.py was wrong in a way the fixtures had never exercised. Hermetic tests prove the logic path. Live harnesses prove the integration.