Three green lights that were not wired to anything

There is a rule on this project that gets quoted more than any other: a check that cannot come back negative is decorative. Before you trust a gate, break the thing it protects and confirm it screams.

It is a good rule. It has caught a test suite that ran zero tests and exited zero, a quality gate that passed a deliberately blurred image, and a colour check that sorted its input before testing whether it ascended, so it could never fail.

This week it caught me three times, on the website rather than the game, and all three were the same bug wearing different clothes. Writing them down because the shape is the useful part — I did not recognise the second one as the same mistake as the first, and I had made the first one four days earlier.

One: the exit code that belonged to the wrong process

I was mutation-testing a gate. Break the input, confirm it fails, restore. The command was roughly this:

node scripts/check-thing.mjs | tail -3
echo "exit=$?"

It printed exit=0 every time. Every mutation reported success. I nearly concluded the gate was broken.

$? in a pipeline is the exit status of the last command in it. I was reading whether tail had succeeded, and tail always succeeds. The gate underneath was failing correctly and shouting about it; I had built a machine to discard the answer and report on the messenger instead.

The fix is to stop piping while measuring:

node scripts/check-thing.mjs > /dev/null 2>&1
echo "exit=$?"

I hit the same thing later with a grep in the middle of an && chain — the grep found the failure, so it exited zero, so the && fired and the next command ran as though everything had passed. A search that succeeds at finding bad news is not the same as bad news being absent, and shell status codes do not distinguish those.

Two: the flag that only did half its job

Pages that should not be indexed carry a noindex flag in our route table. I added a page, set the flag, and moved on.

The flag removed the page from the sitemap. That is all it did. It emitted no <meta name="robots"> at all, so a crawler that found the URL any other way — a link, a redirect, someone pasting it — would have indexed it happily.

Worse, the sitemap and the page were now telling a crawler two different things, which is its own class of warning. The flag was not merely incomplete; it was decorative in the precise sense of the rule. It looked like a control. Nothing tested that it controlled anything, because the only observable effect was in a file nobody diffs.

It now emits the meta and filters the sitemap, and the build prints which routes it omitted, so a route silently dropping out is visible instead of mysterious.

Three: the one that actually mattered

We publish a document that the app and the site both have to agree with. It has a marker partway down: everything below it is internal working notes and only the part above it is meant to be public.

Our generator did not know about the marker. It published the whole file.

That is a plain bug and it would be a short entry, except for what I did next. I checked. I searched the live page for a string I knew was in the private half, got no match, and concluded the marker was being honoured.

The string I searched for had been added to the document that same day. It had never been in the version the live site was built from. My probe could not have found anything, and I read that as proof of absence. A single-string check, treated as evidence of a general property, on the one occasion the general property was false.

It had been wrong for days and my check had been incapable of telling me.

What the three have in common

None of them was a wrong answer. All three were a correct answer to a question I had not asked.

• Did tail run? Yes. That was never in doubt.
• Is the page in the sitemap? No. Also not the question.
• Is this one specific string on the page? No. Still not the question.

Every one of those returned an honest green, and every green was about something adjacent to the thing I cared about. The failure mode is not a broken check — a broken check gets noticed. It is a working check pointed slightly to the left of the problem, which is invisible precisely because it works.

The corollary we already had was: mutation-test the gate. The one I would add is narrower and I would have found all three faster with it:

> Before trusting a green, write down the exact question you think it answered — then read the code and check that is the question it asked.

The tail did not answer "did the gate fail". The sitemap did not answer "will this be indexed". The grep did not answer "is the private half leaking".

Three greens. None of them wired to the thing I was watching.