A gate that was never red is no gate

01 — The opinion in one sentence

A check only becomes one once you have seen it red.

For that pinup has a mutation suite: a table of named, targeted breakages in its own code. Every entry names the file, the spot, the broken version and the tests that must find the break. The suite copies the code, plants one break, runs the tests and expects red.

At the end of September 2026 there were 65 breaks across nine areas, every area with at least two. On top comes a 66th entry that must not be noticed: a changed comment. If it turns red, the suite cannot tell red from green.

02 — The alternative: coverage

The usual yardstick is coverage: the share of lines that tests execute. Many projects keep it very high. It is cheap to measure and reliably finds code no test touches.

Coverage measures what was executed, though, not what was checked. A test can run through every line and make no statement at all about the result. So the mutation suite asks differently: if this line were wrong, would anyone notice?

03 — The price

Slow. Every break means a test run. So the suite runs only in a CI job of its own, not with every local test.

Every new area costs at least two breaks. An area counts as done only when two mutations in it are red. That makes some small features twice the work.

Breaks go stale. An entry looks for an exact spot in the code. After a refactoring it no longer fits. A test catches that, but it means maintaining the table alongside.

Golden files without a shortcut. Expected results are read, never rewritten. There is no switch that updates them. A new recording is a new directory, reviewed as a diff.

04 — The occasion

On 23 September 2026 the suite found a break that was red on my machine and green in CI. The reason: eight tests demanded a repository that was only checked out locally, although seven did not need it at all. In CI they were silently skipped. Six of them were older than that day and had never checked in CI what they were meant to check.

So the suite reported the break in CI as not found, which was exactly its purpose. Since then seven of these tests run in CI too. The eighth really reads files from the repository and skips rightly. That a run in which all tests merely skipped counts as unchecked rather than passed was already the rule before.

Green is not clean

The same lesson holds outside pinup. A scan gate with a grace period for new findings stays green meanwhile. Whoever only looks at the status merges an update with a new HIGH. Since then I read the scan report before every merge, not just its status.

05 — The enforcement

• The table checks itself. On every test run: consecutive numbers, every area at least twice, exactly one sentinel, and every spot searched for occurs exactly once in the code.
• Skipped is not green. If all tests of a break merely skipped, the break counts as not found.
• Lint rules with a counter-check. The in-house lint rules have tests that prove they can turn red.
• No update switch. A lint forbids a switch that rewrites golden files and a test that writes into test data.

Conclusion

A check you have never seen red reassures without protecting. Deliberate breaks are slow and make work. In return I know for every check in pinup that it can fire.

Next week: versions nothing updates, and which of them may stay that way. The overview of all parts and the terms that recur are in the opening of the series.