On 2026-09-17 I turned auto_mode back on in production. The next morning, three tests in tests/dvlaw/test_post.py failed with assert 0 == 2. No PR had merged overnight. No code had touched those paths. Just a flag, now on, and three red test cases.
Finding the cause took longer than it should have.
What the tests were doing
post_today is the function under test. It runs the article pipeline posting pipeline for the current day. Three test cases called it with different inputs, asserted on the output, and passed consistently, right up until the 17th.
What none of those tests documented, and what I hadn’t spotted in the implementation: post_today opens its own database connection internally. Not the connection the test passes in as a fixture. A separate one, pointing at dvlaw.db on disk, which it opens to read the auto_mode flag.
The function takes one path when auto_mode is off in production. It takes another when the flag is on. The three tests were written, reviewed, and merged while auto_mode was off. They expected the off-path behaviour. When I flipped the flag on the 17th, the function’s behaviour changed. The tests caught the change. None of them were written to test auto_mode.
The category of failure
This isn’t a code regression. No logic changed. No refactor introduced a new edge case. The code does exactly what it did the day before; the environment it’s running in changed, and the tests had silently coupled to that environment.
The tests weren’t hermetic. A hermetic test carries everything it needs to make an assertion within its own fixture: inputs, dependencies, state. It doesn’t inherit anything from the machine it runs on. post_today’s second connection broke that. The test controlled the first connection, the function reached past it and grabbed the second, and the second pointed at whatever state the operator had last set.
The result is a test that passes under normal operating conditions and fails whenever the operator changes the ambient state. That’s an inversion. A test that fails when production changes but no code changed is not a guard against regressions; it’s a monitor of operator behaviour.
The distinction matters because the two failure modes ask for different responses. A regression failure tells you to look at recent code changes. An environment failure tells you to look at what the operator last did, which is a different investigation entirely. When the test doesn’t distinguish between them, you spend the first stretch of diagnosis looking in the wrong place.
Why the coupling is invisible
The test looks isolated. Call post_today(conn, ...), assert on the return value, see green. The fixture covers the arguments; it covers the connection it explicitly hands in. What it doesn’t cover is a secondary connect() call buried in the implementation. That hidden read reaches past the fixture to the file on disk.
You’d catch it by reading post_today line by line and noticing the second connection. The alternative is running the suite in a clean environment with a fresh dvlaw.db and watching things break differently. Most reviewers don’t do either, not because they’re careless, but because the test surface looks clean. The argument list is typed. The fixture is wired up. The assertion is specific. Nothing in that view shows you the hidden read.
The pattern generalises. Any function that reads state it fetches itself is a candidate for this coupling: state from a file, from an environment variable, from a database connection it opens internally rather than receives. The fixture covers the arguments. It doesn’t cover the implementation’s private I/O.
Tripwires
A tripwire doesn’t detect the thing you care about; it detects whatever crosses it. These three tests didn’t detect a logic error in post_today. They detected that I’d touched a production flag. If I’d introduced a genuine bug in post_today on the 17th but not touched auto_mode, those tests might still have passed, depending on which code path the bug was on. The tests weren’t asserting on the logic; they were asserting on the conjunction of the logic and the ambient state.
The danger isn’t just the false positive. It’s the false negative. A test suite that fires on innocent config changes desensitises you to failures. You flip a flag, three tests go red, you assume it’s the flag, and now the suite’s signal-to-noise ratio is degraded. You stop treating red as significant, because you’ve been trained to expect red from causes that aren’t bugs.
The fix isn’t subtle: give each test its own database with an explicit auto_mode value. Not reading from dvlaw.db on disk. Not inheriting whatever the operator set last. Each test fixture is explicit about the database state it runs against. The assertion is then about the code, isolated from the machine. The operator can toggle auto_mode freely. The tests don’t know it exists.
That’s the 2026-09-18 fix. Three tests, now hermetic.
The useful question
When a test fails on a config change, the useful question isn’t “why did the config change break the test?” The useful question is “how did the test know about the config?” If the config isn’t in the fixture, it shouldn’t be visible to the test. The fact that it was means there’s a hidden read somewhere between the assertion and the production environment.
Finding the read is the diagnosis. Sealing it is the fix: bring the relevant state inside the fixture and make the value explicit. The config can then live wherever the running system needs it. The test doesn’t care.
Out of scope
I’m not describing a general pattern for mocking database connections or dependency injection. The fix here is specific: post_today’s hidden connection gets replaced or intercepted so tests can supply the state it reads. The broader point is that wherever a function opens a connection it wasn’t given, you have a leak in the fixture boundary.
Whether that leak matters depends on what the connection reads. If it reads something that never changes across environments, the test passes everywhere and you never notice. If it reads something an operator might reasonably toggle, such as a feature flag, a mode setting, or a threshold, the test is a tripwire. You find out when someone touches the flag.
I found out when I touched the flag.
What this costs
Three red test cases don’t sound expensive. The real cost was the diagnosis. Looking at assert 0 == 2 and knowing the code hadn’t changed. Working backwards to the function. Reading the implementation to find the second connection. Recognising the pattern. If the test had failed for a clear reason, such as a wrong argument, a missing fixture, or a logic error, the stack trace would have pointed there. A hermetic failure is self-documenting. A fixture-boundary leak is not.
The three tests now document what state they run against. They’re more verbose for it. That’s the right trade: explicitness in the fixture is cheaper than surprise in the failure.
