The build said fine. It shipped a game with no game in it.

No. 09

Our release build completed successfully and produced an app with the entire economy missing. Then the check we wrote to catch that skipped, in exactly the state it existed to catch. Both were found the same way.

Three posts ago I wrote about checks that return an honest green about something adjacent to the thing you cared about. I did not expect the series to continue this well.

The build

Spoolbound's simulation — every price, every contract, every machine — lives in a native library compiled separately from the game and loaded at runtime. That is a normal arrangement and it works.

Producing a release build of the app ran to completion. Exit code zero. One warning in the log.

The app it produced did not contain the library. No economy at all: no contracts, no printers, no money. A shop of zeroes.

The warning said so, in the way warnings do, in the middle of a few hundred lines nobody reads when the thing succeeded. The exit code said everything was fine, and the exit code is what anything automated would have believed.

Why it stayed invisible

Every mechanism we had was pointed slightly to one side of it.

The debug build was fine. Everyone develops against debug. It loaded the library correctly, so every day of work confirmed a pipeline that was only half-working.

The release path had never been exercised. There was no reason to build one until there was a reason to ship one, and by then it had been quietly broken for as long as it had existed.

And success was reported. This is the part worth sitting with: nothing failed. There was no red anywhere to investigate. A pipeline that returns zero and emits an artefact is a pipeline that every reasonable observer — human or automated — reads as working.

The failure was not that a build broke. It was that a build could be completely broken and still say fine, and nothing in the system was positioned to notice the difference.

The fix, and the rule it needed

The fix itself is small — a release target, a declaration pointing at the compiled library, and the wiring that registers it.

One detail is worth extracting, because it is a trap rather than a fix. Declaring the library without having built it is worse than the state we were in: it converts a loud warning into a silent, empty ship. So the declaration and the binary have to land in the same change, always. A half-applied fix here is a downgrade.

Then the actual protection: a check that runs a real release build and asserts the library is present in the finished bundle, by name and by plausible size.

It deliberately trusts neither the exit code nor the absence of warnings, because both were measured lying about this exact bug. The only thing it believes is the artefact.

And then the check did it too

Mutation-test the gate: remove the library, confirm it screams, restore.

It did not scream. It skipped, and reported a cheerful green.

The first version of the check skipped whenever the release library was absent — reasonably enough, on the theory that a machine with no build toolchain should not fail somebody's test run. Except the library being absent is the entire state the check exists to catch. It was written to detect exactly one condition and it had been taught to look away from it.

That is the same bug as the original, one level up, written by someone who had just spent hours thinking about the original. Which is the strongest argument for mutation-testing that I know: reasoning about a check cannot find this. Only breaking the thing it protects and watching what the check does.

It now skips only where there is no toolchain at all. Toolchain present and library missing is a failure. Verified by removing the library and watching it go red, then restoring it and watching it go green.

The pattern, four times now

  • An exit code read from the last command in a pipe, not the one being tested.
  • A noindex flag that filtered a sitemap and emitted no meta, so it controlled nothing.
  • A test asserting a delete button was greyed out, passing on code that would delete an account anyway.
  • A build reporting success while producing an empty artefact — and the check for it skipping on the one state it was written to detect.

None of these was a wrong answer. Every one was a correct answer to a question nobody had asked, and each looked exactly like working software right up until somebody broke something on purpose.

The rule we keep arriving at, from a different direction each time:

> Before you trust a green, break the thing it protects and confirm it screams.

We have written that down three times now and it has caught something new on each of them. I have stopped treating that as a coincidence.