Red Once: never trust a test you haven't watched fail

My visual regression check reported zero pixels changed across 28 screenshots.

Every one of those pages had a bright red background injected into it.

28 screenshots compared
0 px changed
PASS

Three separate defects were queuing up behind that zero. The check had been green since the day it was written, which is exactly what made it worthless.

The thing that changed

Test-driven development asks you to write a failing test before the code. Most engineers have never worked that way, and the profession has largely made peace with that. Tests get written after, or alongside, the suite goes green, and life continues.

That peace held because writing a test by hand carried some verification as a side effect. You had to imagine the failure in order to type the assertion. You pictured the thing breaking, decided what breaking looked like, and wrote that down. Nobody called it a methodology, but it was doing real work.

An agent generating forty tests from an implementation that already passes does none of that. It reads what the code does and asserts that the code does it. Every assertion is true at the moment it is written, by construction, and none of them has ever distinguished a working system from a broken one.

The output is a suite that raises coverage while verifying nothing. It is harder to catch than a missing test, because it looks reviewed. Agents produce well-named, well-shaped, arrange-act-assert tests. They look like the work of someone careful.

What the red step was actually for

The ordering in TDD exists to deliver one guarantee: that a human being has personally watched this assertion distinguish broken from working. Writing the test first is simply the most convenient moment to collect that guarantee, because the code does not exist yet and red is free.

Which means the guarantee is separable from the ordering. You can write the code first, the way nearly everyone and every agent actually works, and still collect it. You just have to pay for it afterwards, deliberately, instead of getting it for free.

The pattern: Red Once

No assertion is trusted until you have watched it fail, for the reason you think it fails.

Performed at the moment a test is authored, not retroactively across an existing suite.

1. Break the code under test

One change, inside the code under test. Invert a condition. Return a constant. Delete the line that does the work. Rename the column the query selects.

Editing the test instead proves nothing, because a test can always be made to fail by breaking the test.

2. Run that one test and read the failure

It has to go red, and the message has to name the thing you broke.

$ pytest tests/test_balance.py::test_excludes_body_only -x

FAILED  assert 4200 == 3900
          where 4200 = invoice_total(session)
1 failed in 0.41s

3. Restore, re-run, and write down what you broke

Green again, and the mutation recorded next to the test. One line is enough.

def test_excludes_body_only(session):
    # red-once: returned the unfiltered total from
    # invoice_total() -> 4200 != 3900. 2026-10-13.
    assert invoice_total(session) == 3900

This is the step everybody skips, and it is the one that turns a private act of diligence into something the next person can repeat.

Where it sits

Red for the right reason

A test that fails is not automatically a test that works. It can go red because an import blew up, a fixture collapsed, a migration had not run, a network call timed out, or a mock went stale. All of those produce red, and none of them tells you anything about your assertion.

So step 2 has two conditions, and the second one is the real one: it failed, and the failure names the thing you broke. If you inverted a filter and the test reports a connection error, you have learned nothing and you are not finished.

This is the same error that produced the red page and the zero. The check passed for a reason that had nothing to do with the pages matching: the diff library painted unchanged pixels as faded grey, the test read colour channels for any value at all, so nearly every pixel registered as unstable and the mask built to ignore instability grew to cover the entire page. A correct answer arrived by a route that could never have produced a wrong one.

What was hiding behind the zero

The harness was wrong three ways at once, and all three were invisible to a green run.

1. The mask ate the page. The diff library paints unchanged pixels as faded greyscale rather than leaving them transparent. The comparison tested colour channels for any value, so almost every pixel counted as changed, and the instability mask covered everything. Reading the alpha channel instead fixed it.
2. A footer that was never photographed. Screenshots captured the viewport. One of the footers sat four thousand pixels down the page.
3. Two sides on different frames. The CSS animations were keyed to page load, so the comparison was never made at the same moment in the animation, and the drift was absorbed as noise.

There was a fourth problem in the threshold itself. It was expressed as a ratio of total pixels, which scales with page height, so the longer the page the more a real regression could hide inside an acceptable percentage. Absolute pixel counts, set just above a measured noise floor, do not have that property.

All four surfaced within an hour of injecting a regression the check was supposed to catch and refusing to accept green.

Make the agent do it

This is the part that makes the discipline survive contact with how software is now written. An agent that can write a test can also break the code, run the test, and show you the failure. It is a strictly mechanical sequence, which is the kind of work agents are good at.

Add it to your agent instructions, and require the evidence rather than the assurance:

After writing a test, prove it discriminates:
  1. make one change to the code under test that
     should break it
  2. run that test alone, capture the output
  3. confirm the failure names what you changed
  4. revert, re-run, confirm green
Paste both outputs. Do not describe them.

The captured failure is the artifact. It goes in the pull request next to the diff, and it changes what a reviewer is looking at. A recorded failure is a receipt, and a reviewer can check it in seconds.

What it costs, and where it stops

• About a minute per test. Break, run one test, read, revert, note. One case, start to finish.
• It is not retroactive. Nobody is going to go back through four thousand existing tests. Do it at authoring time, and do it for any test standing guard over something whose failure would be expensive or silent.
• One mutant is enough. The goal is to establish that the assertion discriminates at all, not to prove it catches every possible defect.
• It does not replace the tools. Mutation testing frameworks do this exhaustively and automatically. If you can run one, run it.

On the name and the idea

None of the mechanism here is new. Deliberately breaking code to confirm a test notices is the manual form of mutation testing, which has been studied since the 1970s and has mature tooling: Stryker, PIT, mutmut.

What changed is the cost of skipping it. For decades this was a nicety, because hand-writing a test did some of the work incidentally. Generated tests do none of that work, so the step everyone used to skip is now the one holding the suite up. It needed a short name and a place in the workflow.

Write the code first if you want, but never trust an assertion you have not watched fail for the right reason. That is all of Red Once.