The smoke detector nobody ever tested.

There is a smoke detector on the ceiling. Its light is green. Does it work?

You have no idea. You know it has power. There is exactly one way to find out whether it works: hold a flame near it and listen.

A green test suite is the same little light. It tells you it did not fail. It does not tell you it could have.

Two different questions

Most teams measure code coverage: how much of the code the tests pass through. It is useful, and it answers the question "is there a detector in every room?".

That is not the question that matters. You can pass through 100% of a codebase while checking nothing at all: write tests that execute everything and assert nothing. The coverage report will be perfect. Every detector installed, not one of them wired to a speaker.

The question that matters is the second one: if something breaks, does anyone notice? And to answer it, you have to light the match.

Breaking your own code, methodically

The method is three gestures. Take one line of source code. Break it deliberately, in a precise and tiny way: a comparison flipped, a condition loosened, an "and" turned into an "or". Then run the whole test suite.

Two outcomes. Either a test reddens, and it has just proved it earns its keep. Or nothing moves, and you have learned that this piece of code is guarded by nobody. Repair the line, do it again elsewhere.

The idea is not new in computing, and it is in fact far older than the cousin that made it famous: Netflix killing instances at random inside its own production system. Mind the common misreading: it was not to check that the service held up. It was to force its engineers to write only services that can absorb the loss of a part. Rather than wait for the failure, manufacture it, and manufacture it permanently.

The difference between the two is more interesting than it looks. Netflix breaks the system to discover where it gives: that is an experiment, you are looking for what you had not anticipated. Here we break the code to verify a known property, and the answer is binary: this test reddens, or it does not.

What they share sits elsewhere, and it matters more. In both cases the lasting effect is not the report produced at the end. It is the standing constraint: when you know a machine will kill an instance at random, you design differently. When you know someone will break that line to see, you test differently.

It is also one notch more uncomfortable, because the answer speaks badly of your own verification work.

What it gave us

We did it across seven core modules of runward. Forty-two breakages, one at a time.

Thirty-six were stopped. Twenty-seven by a test that reddened, nine by the compiler refusing to build the program at all. Six went through with nothing moving.

Three of those were real holes, and two deserve naming because they are so ordinary. First: the check that stops a piece of evidence pointing at a file outside the project. A test carried exactly that name, on exactly that path, and it guarded nothing: you could weaken the protection four different ways without it flinching. Second: nothing verified that the tool works when you run it from a subdirectory, which is how everybody actually uses it.

The part nobody tells you

Not every breakage that survives is a hole. Some change nothing observable: the program behaves exactly the same. These are called equivalents, and you have to set them aside, or you end up writing decorative tests to push a score up.

So we sorted our six survivors. Three real holes, three harmless. We wrote it up, published it, and argued the case.

We were wrong on two of the three.

We had reasoned. The reasoning was solid, clean, and false. On one, we analysed line by line a mechanism that carries memory from one line to the next: a single hash character inside a web address, in an ordinary configuration file, made an important rule about background-job guardrails disappear. We had written that this case "does not occur". On the other, we had classified the defect by its output channel: "it is only an informational message". That message could state the opposite of what the program had just established, or point at the wrong file.

These are not slips of attention. They demonstrate something more useful: reasoning is not enough to sort survivors. You have to experiment. We eventually did, by turning verifiers loose with the instruction to prove us wrong and the right to execute the code to do it. They built the file that breaks.

Why it matters for runward

runward exists for a reason that fits in one sentence: a claim is not proof. Writing "this decision is validated" is worth nothing; what is worth something is a pointer to a file, a line, a test, that a machine can go and check.

A green test is exactly the same kind of claim. It asserts that it protects you. Until you have watched it redden in front of a real defect, that is an assertion, not proof.

A tool that demands proof from others and does not apply it to itself is a scale merchant who never weighs himself. It is that simple. And there is something worse than a missing test: a test that carries the name of a guarantee and does not hold it. That one does not leave a visible hole, it puts a lid on it. That is precisely what we found on our own path check.

Why it matters for you

Two reasons, one passive and one active.

The passive one. When your gate is green, that green is worth whatever the checks behind it are worth. You now have a number: of forty-two defects injected into the core, six went through, three were real, and they are closed. That is not a guarantee of perfection. It is a measurement where there used to be an intention.

The active one, and it is the more useful. None of this is specific to runward. Your code has tests. Those tests probably have holes, and you will only learn which ones by breaking things on purpose. Getting started needs no particular tooling: take the function you depend on most, flip a condition, run your tests. If nothing reddens, you have just learned something about your net.

That is what a team shipping with runward did on the previous version, on their own initiative. They did not wait for us.

Three precautions are worth knowing before you start. Check that your breakage actually modified the file: a change that fails to apply always "survives", and we nearly reported three imaginary holes for that reason. Do not turn the result into a percentage to push up, or you will write tests for the score. And do not sort survivors in your head: we did, and we were wrong two times out of three.

The green light proves nothing. The match does.

← All news