My HSM project had a 978-test pytest suite (At time of discovery, I think we are up to 3k now…), written over months, maintained alongside every feature, cited in PR descriptions, and generally treated as the safety net under a service whose entire product is cryptographic operations. In June I went to check why a CI run felt faster than anticipated, what I found was that the suite had NEVER run in CI. It was not flaky, it was not intermittently skipped, it had not quietly regressed at some point, it had never once executed, across the entire life of the repository.
The first full run surfaced 94 failures, and not one of them was new, they had all been sitting there for as long as the code they were written against, waiting on a job that did not exist.
How a suite never runs without anyone noticing
There was no pytest job in the workflow. That is the whole mechanism, and it is not much of one. The repo grew up around Terraform pipelines, plan and apply jobs, and security scanners, and somewhere in there everybody involved, me and every AI session that ever worked the repo, started reading “CI is green” and “the tests pass” as the same sentence. The tests DID pass locally whenever somebody ran them against the slice they were touching, which kept the suite honest and propped up the illusion.
Nothing warns you about a job that does not exist. Every green checkmark was truthfully reporting that the jobs which existed had passed. A failing suite sends back a red X, a suite with no job attached to it sends back nothing at all, and from where the merge button sits that second case renders as a column of green with nothing visibly missing from it.
The 94, triaged
Adding the gate itself took a few lines of workflow, and then triaging what came back out of it took considerably longer than that. The failures broke into three classes:
- Real hardening gaps, the smallest class and the one that made the finding matter. The standout was an error-metrics hook that ran on every 4xx response and could throw on its own account, turning a well-formed client error into a 500. A client sends garbage to a crypto API and gets a server error handed back. The unit tests had caught that behavior correctly, in a suite nobody was executing. The fix made the hook best-effort, because metrics emission must NEVER change the response class.
- Test rot. Tests written against earlier versions of handlers, seed data that no longer matched its own validators, and assertions pinned to error messages somebody had reworded at some point without ever going back to the test that quoted them. Each one was a small fix on its own, and the pile of them was what months of a suite drifting with nothing checking it turned out to cost once the bill arrived all at once.
- Infra mismatches between local and CI. Import-mode quirks, and a mocking library that wanted a different JWT dependency in the runner image than the one that happened to be installed on my machine. Any repo’s first CI run turns up this class, and this was that repo’s first CI run, arriving at whatever point in its life I happened to go looking at a fast build.
All 94 were fixed in the same push, three PRs.
The rule that came out of it
CI confirms what you already believe about the code, and when the first meeting between code and test suite happens inside the pipeline, the pipeline has taken over local development’s job, long after you have moved on to something else. I had been careful about that one for years. A suite that only ever runs on somebody’s laptop hands enforcement to whoever last remembered to run it, against whichever slice of the tree they cared about that morning, and the green it reports drifts down toward whatever the least diligent contributor in the repo, human or model, decides to let slide.
The check costs nothing. Open your workflows directory and grep for your test runner, not the badge, not the checkmarks, the actual job definition. I would have bet money mine was in there. It was not, and 94 latent failures is what “I was sure it was there” turned out to weigh, which embarrassed me more than any single bug in the pile, and I would not assume this repo was the only one sitting in that state.
One more habit changed since. When a PR description says “tests pass,” it now has to say WHERE they passed, because “978 green in CI” and “978 green on my machine” are two different facts, and the whole life this repo spent treating them as one is the gap the 4xx-to-500 bug lived in.