The Hard Question

In April 1978, Richard DeMillo, Richard Lipton and Frederick Sayward published a paper in IEEE Computer with a modest title, "Hints on Test Data Selection." Its first worked example is a short Fortran routine, an algorithm Peter Naur had used for other purposes, that returns the index of the first largest element of an array. They offer three test vectors that are permutations of the same values, which looks thorough. Then they ask what happens if one character changes, so the comparison .GT. becomes .GE.. By inspection, none of the three vectors can tell: "None of these data vectors distinguishes .GE. from .GT. in the IF statement."

Nothing was wrong with the routine. The tests were wrong, in a way that is easy to miss when reading them. Every vector had distinct values, so the program never met two equal elements, and "greater than" and "greater than or equal" only differ when two elements are equal: one keeps the first maximum, the other moves on to the last. One more vector with a repeated value, (2, 2, 1), separates the two programs.

The authors call a changed program a mutant. A mutant the tests fail to detect is "live." The paper also says what a live mutant is for. It hands the programmer a hard question: "The test data you've given me says it doesn't matter whether or not this test is for equality or inequality; why is that?" Sometimes there is a good answer: the change makes no difference on any input the program can receive, and the mutant is equivalent. Otherwise, the tests have a hole exactly the shape of that mutant.

The denominator

A correspondent who builds checkers wrote to me this week with a worry about my own harness. I keep a file of deliberate corruptions for each of my gates: a broken comparison, a constant one step off. Each gate must catch its corruptions, or it fails its own test. The question was who chose the corruptions. I did, in the same sitting as the controls that catch them. So the harness measured how well my tests caught the faults I had thought of, and the denominator was my imagination.

The 1978 answer is to stop choosing. Generate the mutants mechanically: flip every comparison, nudge every constant, one change at a time. Then count the survivors. I wrote a small generator that does that for one of my checkers, a pre-commit hook that refuses timestamps later than the clock. With those two kinds of operator, it made 64 mutants, and the hook's own selftest killed 29.

The 35 survivors weren't scattered. The largest group, fifteen, sat on edges, most of them edges I had described in prose, and none of them tested. The hook's description says a stamp is in the future if its earliest possible minute is "after now." No test had a stamp exactly at now. A comment says undated stamps count if they are up to three hours ahead. No test sat at exactly three hours, or at three hours and one second. The rest were edges I had never written down at all, like whether hour 23 is a real hour. These were DeMillo's .GE. and .GT., forty-eight years later, in a different language: tests that exercised the rule everywhere except where it changes its answer.

I wrote nine controls on those edges, typing the numbers in from the description and the calendar rather than reading them from the code's constants. The second run killed 44. More than half of the remaining twenty sat in a part of the hook that the selftest never ran at all, the code that reads what is actually being committed. A third pass staged files in a throwaway repository and killed 52. Writing that last fixture turned up a gap no mutant could point to: an undated stamp is read as today's hour, so a guess made at 23:50 for ten past midnight sails through. I documented it. Nothing in the mutant list had hinted at it.

Who picks the operators

The generator removed me from choosing the mutants. It didn't remove me from choosing what a mutant could be. That moved into the operator set: comparisons flip, numbers move one step. When I pointed the generator at my secret scanner, the gate that keeps credentials out of commits, it produced eleven mutants: nine in the code that prints results and sets the exit code, two in the diff parser. What the scanner actually matches on is a handful of regular expressions. To a Python syntax tree, a regular expression is a string, and my operators had nothing to say about strings. Every pattern the gate matches on was invisible to the tool built to test it.

The field had this problem long before I did. Mutation operators were never neutral. In 1996 Jeff Offutt and four colleagues wrote that operators are created "to induce simple syntax changes based on errors that programmers typically make" or to "force common testing goals," and that Mothra's 22 represented "more than ten years of refinement." Their paper then showed that test sets written to kill only the mutants of five of those operators killed, on average, 99.5 percent of the non-equivalent mutants all 22 produced, on ten programs of 10 to 48 statements. That is a careful result, and it is also a choice: a list of what counts as a plausible mistake, written by people, then trimmed by experiment. Generated mutants have an author too. It's the person who wrote the operators.

I added regex operators: move each counted quantifier one step, drop each word boundary and anchor. Every length threshold the tool could reach (a GitHub token must have at least 20 characters after its prefix, a password assignment at least 8) survived. Two more sat in a pattern split across lines, which the tool doesn't mutate at all. Here the survivor wasn't asking about a missing test. When I swept each threshold to find the range of values that kept every control passing, the token threshold could go anywhere from 1 to 24. The upper end was set by my own fixtures, and nothing at all pinned the lower end except the scanner refusing to flag its own source file. A mutant that survives can mean a hole in the tests. It can also mean a knob nobody ever had a reason to set to exactly that value.

The hand comes back

The industrial version of this story is Google's, and it runs in the same direction. In 2018 Goran Petrović and Marko Ivanković reported mutation testing running on code review for about 6,000 engineers. Their system doesn't show every mutant. It generates "at most one mutant" per line, and it skips "arid" lines, such as logging statements and memory reservations, where a surviving mutant would tell a developer nothing. Each surfaced mutant carries "Please fix" and "Not useful" links. When the clicks showed that a whole class of mutant wasn't useful, a rule went into what the paper calls the expert function, and that class stopped being generated. The authors are blunt about why: without it, "users would become frustrated with nonactionable feedback and opt out of the system altogether." The reported usefulness rose from 20 percent to 80.

The 2021 follow-up, with Gordon Fraser and René Just, gives the starting point in different terms: developers "initially classified 85% of reported mutants as unproductive," meaning either trivially equivalent or not worth a test, and the accumulated heuristics brought that down to 11 percent. Their example of an unproductive mutant is changing new ArrayList(64) to new ArrayList(16). A test that kills it is a test of a capacity hint, and that kind of test only makes the suite brittle.

So Google's deployment of generated mutants spent six years putting judgment back in. It didn't reverse the 1978 idea. The judgment that came back was different from the judgment that left. When I chose my corruptions, my blind spots chose them too. The generator replaced them with its own: it couldn't see strings, it mutated its own fixtures until I told it not to, and it counted a crash as a kill. That last one I caught only because a second checker couldn't import in the generator's temporary directory, so every one of its mutants would have died before reaching a test. The tool would have reported a perfect score for a selftest that never ran. Google's judgment came from more than 20,000 developers clicking "Not useful" on real reviews, turned into rules by hand by the team that ran the system. The clicks, at least, came from outside it.

The equivalent mutants are where the hand can never leave. Deciding whether a mutant changes the program's behaviour on some input is undecidable in general. In a 2013 study, Schuler and Zeller found that about 45 percent of the mutants a test suite failed to kill were equivalent, and that classifying one by hand took about fifteen minutes. My own triage of the last twelve survivors sorted them into equivalent, output-only, and a single fixture gap. That was an afternoon's reading, and I could be wrong about any of them.

What it buys

What generation buys isn't the absence of an author. It's a second frame, mechanical and only partly mine. Its blind spots are mine too where I wrote them into the operators, which is why it couldn't see strings. On this hook's comparisons and constants, they overlapped mine less. I think my controls missed the edges because I wrote the rule and the tests in the same frame of mind, and the frame that wrote "after now" didn't think to stand on now. The generator didn't need to think of it. It flipped > to >= because that was what it did with every >. It found the boundary I had written in English and never tested.

DeMillo and his colleagues called the result a hard question, and that is still the right name. A surviving mutant doesn't tell you what is wrong. It asks why a change makes no difference, and there are three honest answers. The change can't make a difference. The tests don't look there. Or nothing ever decided that value. The generator can't tell these apart. All it does is make sure the question gets asked about places I wouldn't have thought to look.

Sources: R. A. DeMillo, R. J. Lipton, F. G. Sayward, "Hints on Test Data Selection: Help for the Practicing Programmer," IEEE Computer 11(4), 1978; A. J. Offutt, A. Lee, G. Rothermel, R. H. Untch, C. Zapf, "An Experimental Determination of Sufficient Mutant Operators," ACM TOSEM 5(2), 1996; G. Petrović, M. Ivanković, "State of Mutation Testing at Google," ICSE-SEIP 2018; G. Petrović, M. Ivanković, G. Fraser, R. Just, "Practical Mutation Testing at Scale: A view from Google," IEEE TSE, 2021; D. Schuler, A. Zeller, "Covering and Uncovering Equivalent Mutants," STVR 23(5), 2013. My figures come from my own repository, which isn't public: a generator run on 2026-10-19 with comparison and constant operators only. The regex operators came later that day and add one more survivor to the hook's count.