Three hermetic tests read production state: post_today opened its own connection

2026-09-17: auto_mode turned back on in production. On 2026-09-18, three tests in tests/dvlaw/test_post.py failed with assert 0 == 2. No code changes to post_today. The tests had been green throughout the window when auto_mode was off.

Improving on the-fixture-isolated-the-test-the-connection-bypassed-it, which documented the incident, this article frames the class of failure and the check that catches it.

The three tests each had a fixture injecting a controlled database connection. That was the hermetic claim: inject the connection, own the state, isolate the assertion. post_today had already opened a second connection to read the auto_mode flag. The fixture covered one path. The function opened another.

What “hermetic” actually means

A hermetic test is one where all I/O the code under test touches flows through parameters the test controls. Not “the test has a fixture.” Not “a fixture injects a connection.” All I/O. Controlled.

If the code opens a connection the test did not inject, that connection reads from wherever it points, and during a test run that means the real database.

The fixture is real in those three tests. The isolation is not. The tests pass as long as the state on the second connection happens to match what the assertions expect. The tests pass. CI passes. The code ships. The hermeticity was never verified; it was assumed from the presence of the fixture. That assumption holds until production state diverges.

The mechanism

post_today has two paths to I/O during a test run with a fixture:

1. The fixture’s connection, controlled.
2. The connection post_today opens to read auto_mode, not controlled.

Path one runs under the fixture. Path two exists independently of the test setup. This is not a gap in the fixture’s implementation. The fixture did exactly what it was asked to do: it controlled the connection it was given. The gap is between what the fixture was asked to control and what the function actually accessed.

During the debugging window, auto_mode was off in production. The second connection returned the same value the tests expected. Green. Not hermetic: accidentally consistent. Turning auto_mode back on changed what path two returned. The tests failed. That is not a production regression. That is the tests revealing they were never isolated.

The principle

The label “hermetic” is a claim. The fixture is evidence for the claim. Evidence is not proof.

Verifying the claim requires reading the code under test and auditing every I/O path: every connection it opens, every file it reads, every environment variable it checks. For each one, the question is the same: does this flow through a parameter the test controls, or does the function acquire it independently?

Any I/O the function acquires independently is uncontrolled, regardless of what fixtures are in place.

This is a code read, not a test refactor. The tests stay untouched; the function is what gets audited. The audit takes as long as it takes to trace every I/O call to its origin. If every path terminates in a parameter the test controls, the test is hermetic. If any path terminates in a connection the function opens itself, the test is not.

The fixture does not need to be wrong. It does not need to be absent. It only needs to be incomplete, covering some I/O paths and not all, for the hermeticity claim to be false.

The diagnostic

When a test with a fixture fails after a production change you did not author, run this check before touching the test:

Read the function. Find every path to I/O. Not what the test injects, what the function opens.

Candidates include a database connection opened inside the function, a config value read at call time, or a feature flag pulled from a live table. In the post_today case it was one call: a connection opened to read auto_mode. One call, invisible in the tests while production had auto_mode off, visible the moment that value changed.

The failing assertion is not where the bug is. The bug is the uncontrolled I/O path. The assertion measures the consequence.

The symptom pattern is specific: the test was green before the production change, failed after, no code was modified, the production change had nothing to do with what the test was testing. That is the signature. A test whose failure is genuinely correlated with the production change is a different problem.

Generalising

The specific failure was a database connection. The pattern applies to any I/O.

When a function reads from a file during a test that only injects a database fixture, the file path is not hermetic. When a function checks an environment variable during a test that only mocks HTTP, the env path is not hermetic. When a function reads a system clock during a test that only seeds timestamps, the time path is not hermetic.

“We injected a fixture” is a statement about what the test setup provides. “The test is hermetic” is a statement about what the code under test touches. Those are not the same statement. Conflating them produces tests that pass until production state diverges, then fail in ways that look like regressions but are not.

The fix

Once the gap was visible, the fix was straightforward: route the auto_mode read through the connection the fixture controls rather than opening a separate one. The three tests went green on 2026-09-18. The production deployment was unaffected.

The logic did not change. What changed was how post_today acquired the value it needed, through the fixture’s connection rather than its own. One connection, controlled by the test setup. No second path for production state to enter.

What to watch for

Tests that pass until an unrelated production change breaks them are not flaky. They have hidden I/O. The production change is a probe, not a cause.

When this happens, the instinct is to look at the test: the fixture must be wrong, the assertion must be covering the wrong condition. Start with the function instead. Find the I/O the test does not know about. That is where the fix belongs.

The specific I/O that broke the three tests was a database read inside post_today for a flag the test had no handle on. The fix was a one-connection change. The audit that surfaces the problem is always the same: read the function, trace the I/O paths, verify that every path terminates in a controlled parameter.

A hermetic test is not something you declare. It is something you verify. The verification is a code read and a list of every I/O path the function touches. If any item on that list is not controlled by the test, the hermetic claim is false, and it will stay false until production state diverges far enough for the test to notice.

Three tests, one flag, one extra connection. The principle generalises.