We mutated our own code. Six tests noticed nothing..

A green test suite proves one thing: it did not fail. It does not prove it could have. A test whose assertions are true by construction passes forever, and guards nothing.

The only way to find out is to break the code on purpose. Mutate a line of source, re-run the tests, watch whether they redden. A test that stays green in front of an injected defect will not see a real one either.

We did it across seven core modules of runward. 42 valid mutations, 36 stopped, 6 survivors. Those 36 deserve their breakdown: 27 killed by a test that reddens, 9 by the compiler refusing to build. Nine never met a test at all.

Three real holes

The traversal guard had a test that did not guard it. Every piece of evidence in a mission passes through a function refusing a pointer that escapes the project. A test carried the name of the thing, on that exact path. It only exercised an absolute path, which two redundant protections catch: you could remove either one separately, or collapse the containment check, without reddening the full suite. A test that carries the name of a guarantee and does not pin it is more dangerous than no test, because it reads as proof.

Nothing checked that runward works from a subdirectory. The command climbs the tree until it finds the mission root: that is what lets you run runward check from anywhere inside a repository, and it is how the tool is actually used. Stopping that climb on its first iteration went unnoticed.

A block comment that ate its line. In the deployment-manifest reader, making a comment start on any slash went unnoticed, because the existing test only asserted on keys sitting on other lines. Everything following a /* on the same line vanished in silence.

A fourth defect was found outside this bench, by a mutation played by hand: * crossing path separators in the pattern language. A declared territory would silently become wider than what the rule declares. It is fixed, but it is not one of the six survivors, and saying so keeps the bench's result honest.

Three that were not

This is the section we wrote first, and the one verification demolished. We had declared three survivors harmless. Two were real defects. We leave the faulty reasoning visible, because it is instructive.

We wrote that a mutation in the TOML manifest reader was near-equivalent, that the triggering case "does not occur", and that building a fixture to redden it would be theatre. All three are false. The error: we reasoned line by line about an automaton that carries state across lines. Under the mutation, a hash two characters past an opening quote becomes a comment, the line truncates, an array loses its closing bracket, and the balance loop swallows the rest of the file. The hash need not even sit on a line runward reads:

[vars]
DOC_LINKS = ["https://acme.dev/docs#install"]

[triggers]
crons = ["0 3 * * *"]

That array, in a table runward never consults, destroyed the [triggers] below it. A HIGH rule stopped surfacing, silently. A documentation URL with an anchor is not a contrived fixture, it is an ordinary configuration file.

The second mutation sat in a derivation note. We filed it as "an informational message, never a verdict". That is accurate about exit codes, and it is the wrong criterion. Under the mutation the note can claim a manifest declares no scheduled work in the same run that just derived a cron from it, silence a real absence, or name the wrong file. Classifying a defect by its output channel rather than its nature, in a product whose thesis is "no step without proof", was paying ourselves in words.

The third really is equivalent, across 3,267 comparisons. But by accident: it holds only while every shipped template ends with a newline. Remove one, and a deliverable flips from "in progress" to "filled", which opens a phase. Nothing guarded that invariant. The test we added does not kill the mutation: it guards the reason the mutation cannot hurt.

107 candidate mutations were not tried. The bench is bounded per file. This is a sample, not a proof of coverage, and saying so is part of the result.

What the same pass found on our own site

The published catalog advertised "the 64 craft rules the gate can require, each mapped to one or more phases". Both halves were false: the gate only requires a rule with CRITICAL or HIGH impact mapped to a gated phase, and 33 rules carry no phase at all.

Our first correction was wrong too. We published 27. It is 31. The computation compared rules against a hand-written list of phases missing handover, which is a gated phase. Four rules vanished from the count, one of them CRITICAL. Our own figures contradicted each other and we did not see it: 27 requirable plus 33 phaseless makes 60, not 64.

And we wrote that the number was "computed from the machine contract, so it can no longer drift". Impact and phases did come from the contract. The set of gated phases was restated by hand, and had already drifted by the time we published the sentence. The correction of an overclaim reproduced the exact failure mode it claimed to remove.

The machine contract now publishes the gated phases, and the documentation reads them instead of guessing. If the field is missing, the build fails rather than falling back on a silent default, since a silent default is what let the first drift through.

The thread running through all of it

One clarification in passing, since we invoked a test above that refuses the phrase "audit-grade" outside negation: that test reads a single doctrine file. It covers neither the rest of the repository nor the site where the fault occurred. Invoking a discipline that was not watching the place of the fault was rhetoric.

Four things found in two days, one shape behind them: a mechanism that is correct and unreachable is not a mechanism.

A territory map written under a level-one heading instead of level two was silently voided. The diagnostic existed, said the right thing, and sat at line 16 of an output nobody scrolls. The instrument measuring that map's decay lived in the one command a governed mission has no reason to run. The charter handed to the agent ordered it to confront the rules "at the point of action" without ever saying how to learn which ones. And six tests could not fail.

None of those four is a reasoning error. Each is a sound mechanism, placed where it reaches nobody. It is a class of bug you cannot see by reading code, because the code is correct. You see it by measuring whether anyone can actually use it.

What 0.30.0 changes for you

One visible thing: the agent charter written at init now carries the gesture its own obligation demanded. It already said to confront the rules at the point of action, then named only a command that reads a rule whose name you already know. runward rules --for <paths> finally answers "which ones", showing the pattern that retained each.

Nothing else moved in 0.30.0: no change to gate behaviour, no exit code touched. An existing mission does not receive this sentence: runward update never rewrites your agent charter, because it is a deliverable you own, and that is exactly the boundary the update protects. The line has to be copied by hand. We wrote the opposite in the first version of this article.

Version 0.31.0 fixes both defects verification found, and publishes the gated phases in the machine contract.

What this article cost, and why it stays online

Its first version carried seven errors. A wrong number in the very paragraph correcting an overclaim. A delivery promised that does not happen. Eight modules for seven. A missing test asserted where a test existed. Thirty-six mutations "caught", nine of them only by the compiler. A survivor borrowed from another bench. And, above all, two real defects declared harmless.

We did not find these by re-reading. We found them by turning verifiers loose on our own text with the instruction to prove us wrong, and the right to execute the code to do it. They built the TOML file that breaks, and they measured that our traversal guard had a test incapable of watching it fall.

We do not claim the suite is complete. We no longer claim to know which survivors are harmless either: we believed that three times and were wrong twice.

← All news