The form went live on 9 June. Phase 8-10 closed: real signups accepted, four static pages at their canonical URLs, privacy policy live at /privacy/. I subscribed from my own address to smoke-test the golden path. Expected the styled HTML confirmation email. Got nothing. No row in the database.
That was bug one.
What the events table showed
The events table told the story. Two consecutive bot_detected rows, both for the same submission, both carrying the same reason. The bot-trap had fired. I’d used a +tag address (the kind you use to track which services sell your data) and the validation layer had read it as synthetic.
This is the kind of bug that doesn’t announce itself. The form returned successfully. No error page. No user-facing feedback. The rejection was silent, which made it a lie: I’d just been told, implicitly, that my address was accepted, when it had been quietly dropped.
Without the events table, this is an invisible failure. The user leaves thinking they’ve subscribed. You never find out. No error rate to observe, no exception logged, just a growing gap between form submissions and actual subscriber rows.
The timezone that ate the token
The bot-trap fix wasn’t the day’s first hotfix. P8-9 had already shipped. After the load-path fix landed, POST /api/subscribe/ returned a clean 302 to /newsletter/check-your-inbox/. The events table showed the right shape: signup followed by confirm_sent. But confirmation emails weren’t completing their flow. The confirm-token TTL was expiring before users could click.
Root cause: PHP and MySQL were operating on different timezone offsets. The TTL window looked correct to the application layer. By the time MySQL compared the stored timestamp against the current time, the offset mismatch had already eaten the margin. Token appeared stale before it was.
The specific failure shape (signup and confirm_sent both present, correct sequence, nothing visibly wrong) is exactly the kind of bug that hides well. The code path executed. The email dispatched. The token was written. Nothing errored. The TTL quietly expired.
Again: events table. Without those rows, you’re staring at a subscription flow that looks functional until a user complains their confirmation link is dead.
The hydration problem
The third fix came in the same hour as the bot-trap fix. After the silent-reject patch deployed and the MIN-1s + please-try-again flow went live, I re-tested. Hit /please-try-again/ again.
The diagnostic instrumentation from the first hotfix is what surfaced it. rendered_at (a timestamp used to determine page state) was being set during server-side render, then overwritten during React hydration on the client. Server and client were producing different values for the same moment. React was reconciling them in a way that put the component into the wrong state. The fix: move rendered_at into a useEffect, so it only runs client-side after hydration, where there’s no server-rendered value to conflict with.
The causal chain matters. The first hotfix added diagnostic instrumentation. That instrumentation made the hydration bug findable. One fix created the conditions under which the next became visible. You don’t normally think of your diagnostic layer as a dependency, but this sequence made it concrete.
What actually got fixed
Three separate root causes, all shipped on launch day:
- Timezone TTL: PHP/MySQL offset mismatch corrected; confirmation window now closes when it should.
- Bot-trap validation:
+tagaddresses pass correctly; the detection rule tightened to not read tagged addresses as synthetic. - React hydration:
rendered_atmoved touseEffect, eliminating the server/client reconciliation conflict.
Each fix is small. None is interesting in isolation.
The part that actually matters
The point isn’t what got fixed. It’s how it got found.
All three bugs were invisible without instrumentation. The timezone mismatch looked like a working confirmation flow until you read the token age from the events table and noticed the gap. The bot-trap looked like a successful form submission until you checked whether the row had landed. The hydration bug looked like an incorrect redirect until the timestamp discrepancy in the event data pointed at the reconciliation conflict.
This is the difference between a debugging session that takes an hour and one that takes three days. The events log isn’t a sophisticated observability stack. It’s one table, one row per action, each row with a reason field. But it changes the work from “read the code and reason about what might have gone wrong” to “read the log and identify where execution diverged from expectation.” Those are different jobs. The second is faster.
The confirmation email shipped in the same window: multipart MIME, brand styling, the Captain Random icon embedded inline via Content-ID. That’s the deliverable. The deliverable was never going to be the lesson.
What you’d tell past-you
Write the events log before you write the feature. The confirmation flow had logging because I’d built it in from the start. The bot-trap didn’t, which is why it was invisible until the first hotfix added it. Logging is not a debugging convenience you reach for when something breaks. It’s the thing that tells you something broke.
Silent failures are worse than noisy ones. The bot-trap returned success to the user. The timezone mismatch dispatched an email. Both looked like working systems at the surface. There’s a failure category that looks like success: if your system can fail silently, it will, and you’ll find out when a confused user reports it. A hard error is preferable.
Hydration bugs present as logic bugs. The rendered_at issue looked, from the symptom, like the redirect was firing incorrectly. It would have been easy to spend an hour in the redirect logic. Reading the event data instead surfaced the timestamp discrepancy immediately. When a React component behaves differently in SSR versus the client, check what gets set during server render before chasing the logic that uses it.
Diagnostic instrumentation compounds. The first hotfix made the third bug findable. Each row you add to the events table is leverage for the next incident. That’s not a side effect of good logging; it’s the point of it.
Where this sits
Phase 8 is shipped. The form accepts real subscribers. The confirmation email is multipart/alternative with inline brand assets. Three hotfixes landed on launch day, all logged at docs/devlog.md.
The events table is the one artefact I’d salvage if I were starting again. Everything else in Phase 8 is rewritable. The record of what the system actually did (in order, with reasons) is what makes the next debugging session shorter than the last one.