My tests were green. They caught 0 of 10 real bugs.
I had a small web shop with four end-to-end tests. All four passed, every time. I felt good about them. Then I broke the shop on purpose, ten different ways, and ran the same four tests against each broken version. They
I had a small web shop with four end-to-end tests. All four passed, every time. I felt good about them.
Then I broke the shop on purpose, ten different ways, and ran the same four tests against each broken version. They caught zero.
That's the problem I built Saboteur to measure: a green suite tells you nothing about what it misses. I built it for the KaneCLI Online Hackathon by TestMu AI.
Live report: https://softalpha0.github.io/saboteur/
Code: https://github.com/softalpha0/saboteur
The idea in one paragraph
Saboteur plants one small, realistic bug at a time, builds the app, and runs your test suite against it. If every test still passes, the bug "slipped". For each bug that slipped, Claude writes a new test, and the test is only kept if it passes on the correct app and fails on the bugged one. No test gets in on the model's word alone.
This is basically mutation testing, but for end-to-end browser tests instead of unit tests, with an AI closing the gaps and a real browser acting as the judge.
The pieces
The tests are Kane CLI _test.md files: plain English steps that Kane runs in a real Chrome browser. Here is one of the four starter tests:
---
mode: testing
url: http://localhost:4173
tags: [starter, cart]
max_steps: 30
---
# Cart shows the added item
## Add the mug
Open the home page and add the Ceramic Mug to the cart.
Verify the cart badge in the header shows 1.
## Open the cart
Open the cart page.
Verify a line item named "Ceramic Mug" is listed with a unit price of $14.50.
The bugs are git patches. Each one is the kind of mistake that ships for real. This one makes the "+" quantity button do nothing:
- onClick={() => onChange(value + 1)}
+ onClick={() => onChange(value)}
The ten bugs in the demo shop include a cart total that skips the last item, a discount code that takes 1% instead of 10%, a checkout that accepts the email a@, and a cart that empties when you refresh.
The loop for each bug:
- Copy the app to a temp folder,
git applythe patch, build it, serve it. - Run the whole suite with Kane CLI, with self-healing off (
--no-adaptive-heal). With healing on, Kane would try to work around a broken page, which is great for flaky selectors and terrible for catching bugs. - Score it: caught or slipped.
Being strict about what counts as "caught"
This took most of the work, and it's the part I'd tell anyone building something similar to care about most.
A failed test is not the same as a caught bug. While building this I saw tests fail because:
- Chrome crashed (
browser_disconnected) - my disk filled up mid-run and Chrome couldn't write screenshots
- Kane's own page check errored (
analyzer_failed) - a hover simply didn't land the first time
If you count those as catches, your score is a lie. So Saboteur:
- treats crashes, disk errors and Kane-side errors as errors, never catches, and retries them
- re-runs every catch on its own. If the test doesn't fail a second time, the first failure is treated as flaky and the whole suite runs again
That second rule saved me on a real run. On two bugs, a test failed with analyzer_failed, which had nothing to do with the bug. The confirm run passed, the suite re-ran clean, and both bugs were correctly scored as slipped instead of fake catches.
Closing the gaps with Claude
For every bug that slipped, Claude gets the bug's description, the app's routes and facts, and the existing tests, and writes a new _test.md. Then Kane runs it twice:
| Run | Build | Healing | Must |
|---|---|---|---|
| A | correct app | on | pass |
| B | bugged app | off | fail |
If run A fails, the test is wrong about the app. If run B passes, the test doesn't catch the bug. Either way, Kane's failure text goes back to Claude for another try, up to three attempts. Only tests that pass both checks join the suite.
Results (all from real Kane runs)
Demo shop: the 4 starter tests caught 0 of 10 bugs. With Claude's Kane-proven tests: 10 of 10.
Duel mode. I then pointed two Claude agents at each other. Red reads the current tests and invents new bugs they would miss. Blue writes tests to catch them. Kane referees, and a bug only counts if Kane can see it in the browser (the check passes on the bugged app and fails on the correct one).
Final score: Red 3, Blue 3. Red found three real holes:
- the cart badge counted products instead of items
- free shipping was checked before the discount was applied
- signing out left you signed in
Blue closed all three, and the suite grew from 18 to 21 tests.
An app I didn't write. A demo app I built myself is easy to game, so I also ran it on the official React TodoMVC with ten new bugs. Four typical starter tests (add, complete, delete, edit a todo) caught 2 of 10. Claude has written Kane-proven tests for 6 of the 8 misses so far. The final two were cut short when Kane lost its sign-in mid-run.
Things that went wrong
- The first full attack took hours. Every bug is a fresh build plus a full browser run. Running two tests at a time, stopping at the first failure on attacks, and saving progress so an interrupted run can resume brought a caught bug down to a few minutes.
-
A Kane run hung for over two hours. On Windows, kane-cli runs behind a
cmdlauncher, so my timeout killed the launcher and left the real process alive. The fix was killing the whole process tree, plus a watchdog that stops a run when no output arrives for ten minutes. -
A full disk looks like flaky tests. Kane keeps screenshots for every run. When my disk hit zero, tests started failing in odd ways. Saboteur's
doctorcommand now checks free space before it starts. - Claude got every test right on the first attempt in my real runs, so the "send Kane's failure back to Claude" loop has never had to fire for real. It's covered by automated tests, but I want to see it in a live run.
Try it
git clone https://github.com/softalpha0/saboteur
cd saboteur
pnpm i
kane-cli login
pnpm saboteur doctor --replay
pnpm saboteur run --replay
--replay needs no Claude key. It uses the tests Claude already wrote and Kane already proved. To point it at your own app, edit saboteur.config.json (build command, serve command, URL) and add your own bug patches.
The question I keep coming back to: how many of your tests would still pass if the feature they test was broken? I'd love to hear what you find if you try it.
Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.