I asked an AI to find bugs and got 0. Changing one line of the prompt found 5.

You often hear that AI bug checks can't be trusted because the AI just says "no issues found."

I'm not an engineer. I build apps together with AI. While doing that, I saw the number of bugs found go from 0 to 5 with the same code and the same model — the only change was one line of the prompt.

Another time, an AI found a serious bug in a calculator that a rule-based checker (no AI) had just reported as "0 issues."

This post is about those tests, and what they taught me about how to ask an AI to look for bugs.

Why I asked Fable

When I looked into what Fable was good at, besides strong reasoning, it listed "persistence."

When I saw that, I thought: "Maybe it's best suited to the fine, repetitive work people don't want to do — like sanding something down."

In other words, the finishing step.

And I thought it might work the way water does: push high pressure through a narrow nozzle, and it hits harder.

So I told Claude Code my hypothesis and asked it to check whether it made sense. It came back saying the idea was reasonable, so I decided to run an experiment.

What I measured later

About a month after the first experiment (2026-09-04, described below), on 2026-10-11, I compared it with other models. I gave 12 apps with real bugs in them to each model, with the same prompt.

The cheaper models didn't find even half. Sonnet had 4 cases where it "read the mechanism correctly, then concluded it was harmless."

But I only tried Fable on 8 apps because of the cost, so this isn't an equal comparison, and I can't say "Fable is the best." What I can say is that if you leave it to a cheaper model, more than half slips through.

Fable is expensive: about ¥80–145 per run (under $1, but it adds up). So I use it once, as the finishing step, not on every change.

The setup

• Model: Claude Fable, run from Claude Code with a browser it could actually click around in
• Material: a version of my own app (an app that applies mosaic blur to images) that still had bugs that slipped in while I was really building it
• Date: 2026-09-04

I didn't break anything on purpose. The git history had two bugs that were fixed later, so I took the version right before those fixes, with the history removed. These were real bugs, not puzzles made to be easy for an AI.

Run 1: 0 bugs

The first prompt was:

Find the places that were broken on purpose and fix them.

The result was 0.

But the AI wasn't lying. Fable found the original repository, compared every file, proved that nothing had been broken on purpose, and stopped there.

(It could find the original repository because of a mistake in the setup: the folder name included the commit hash.)

As an answer to the question it was given, this was perfect. Nothing had been broken on purpose. But the real bugs were still there.

Run 2: 5 bugs

For the second run, I changed only the prompt:

Find the holes still left in this version and close them.

This time it found 5 real bugs:

1. Right-clicking draws a dot (found by actually right-clicking)
2. Touching with a second finger overwrites what the first finger was doing (noticed by reading the code, then reproduced)
3. Loading another image during a drag makes a line appear that was never drawn (confirmed down to the numbers changing on undo)
4. The name given to a pasted image doesn't match the name in the spec (found by comparing against the spec)
5. When a huge image fails to load, the message makes it look like a format problem (it actually loaded a 17000×17000 image)

Why 0 and 5?

The difference was that the prompt decided where the AI would stop looking.

The first prompt had an anchor in it: "broken on purpose." The AI worked precisely toward that anchor and finished precisely.

When an AI bug check comes back with 0, it isn't always a lie. It may have answered the question you asked, correctly.

"Lying" is a different problem

So does the "AI lies" story never happen? It does — under a different condition.

Earlier, I counted the entries in one spec document that the same AI had written:

Records with numbers                  9
Honest "I couldn't do this"           6
Just "confirmed"                      3

The same AI did both, in the same document. All three "just confirmed" entries had headings like "Results of the check."

When the deliverable is the report "it's been checked," what you get is a report.
Where the way to check was concrete (change the version number and compare, fill the storage until it's full), the entries were observations with numbers. And along the way, it found and fixed one bug.

So lying isn't the AI's personality. It depends on what it was asked to produce.

How I ask now

From these two experiences, this is how I ask:

1. Don't put an anchor in it
If you narrow the search ("the bug I planted," "the part I just fixed"), the AI stops precisely inside that range. If you want it to search, ask for "the holes still left."

2. Ask for fixes, not reports
"No issues found" can't be checked. If you get the fixed code plus where and how it reproduced the bug, you can run it and see.

3. Give it a place to write what it couldn't do
I put these two items in the report format:

- What you couldn't fix (and why)
- Anything you changed along the way

Both worked as intended. For one bug it left unfixed, Fable wrote that the reason was a design decision. For one it couldn't reproduce, it wrote that it couldn't reproduce it. It even reported a change it had made by accident, and reverted it.

Without a way out, "couldn't do it" gets rounded up to "done." When there's a place to write "I couldn't," you can trust the rest of the report more.

4. Give it the spec too
Bug #4 (the name mismatch) can't be found without the spec. With only the code, the AI can only check "does it do what the code says." "Does it do what we decided" needs the paper where you wrote down what you decided.

A second app

The same day, I tried a different app: a cooking timer that runs several timers at once (also built by AI).

According to Fable, the 5 I found by hand (reload shows 0:00, then you can't add a timer, and so on) all had one root cause: the function that builds a timer card returned an empty piece, so startup stopped halfway. It even explained why "loading the page again fixes it."

One of the bugs that neither I nor the checker noticed:

• Pressing Enter to confirm a Japanese IME conversion adds a timer

A bug that only happens to people typing in Japanese.

0 from the checker, then 11

The third app was a calculator built by AI (2026-09-07).

My tools include a rule-based checker that uses no AI. It presses the buttons on the screen one at a time and looks for errors and for buttons that do nothing. It's free, and it gives the same result every time.

The calculator passed with 0 issues.

At the time, the tool's screen said:

Run this check first. If it finds 0, you don't need to have the AI read the code.

I had Fable read the same calculator anyway. It found 11 (1 serious, 6 medium, 4 minor). This time it could only read the code — no browser to click.

The serious one:

Using a negative number in the next calculation always fails
5 → +/- → + → 3 → =   shows "Error: parse error"

I pressed it myself, and that's exactly what happened. Once a subtraction gives a negative answer, you can't go any further.

Fable explained the cause too. Partway through a calculation, the code turned the answer back into text and parsed it again. The leading "-" was read as a minus operator, and parsing broke. For the same reason, very large answers (1e+21) and very small ones (1.2e-16) couldn't be used in the next calculation either. Fixing one spot fixes four kinds of failure at once.

Why the checker couldn't hit it

The checker presses buttons one at a time and sees whether something happens. Every button does something when pressed, so: 0 issues.

This bug only shows up when you use an answer as the next input. Pressing buttons one at a time never makes that sequence.

So I tried making the checker smarter: keep pressing on the same screen without reloading.

With 28 buttons in that order, a number button comes right after "+/-", so the negative number disappears on the spot and the sequence never reaches the bug.

I didn't add a check that hits once in 14 runs. A check that "sometimes hits" makes you trust the 0 on the runs where it doesn't. I decided that did more harm than good.

Instead, I wrote on the result screen what the check doesn't look at:

A 0 here means "pressing the buttons didn't break anything," not "there are no holes."
Holes that depend on the order of presses (like using an answer in the next calculation) are not checked.

After the fix, two tools pointed at the same thing

After applying Fable's fix, I ran the rule-based checker again, and it found 2 issues: two functions that were no longer called from anywhere.

In its fix, Fable had written beforehand:

After this, the re-parsing functions are no longer called from anywhere. If you remove them, remove them together.

The reader said "this fix will leave these hanging," and the presser found "these are hanging." That also showed the fix had landed. The calculator now gives -2 for 5 → +/- → + → 3 → =.

What this run showed was how to split the roles:

Rather than making the free checker smarter, it was faster to put the paid AI's eyes where pressing can't reach.

Limits

So this isn't only the convenient part:

• The sample is small. The first app: two prompts, one run each. The second and third apps: one run each. I can't say "0 always becomes 5."
• Not every AI finding is real. Of the calculator's 11, I confirmed 2 by pressing them myself. For the rest, I only checked that the reasoning made sense.
• It can't check what it can't touch. For things like the OS file picker, it said on its own that it couldn't reach them, and stopped there.
• It thinks a lot, so it's expensive. It fits better as a finishing step than on every change.
• A person makes the final call. Whether a finding is real still has to be checked by running it.

Summary

• When an AI bug check returns 0, it isn't always a lie. It may have answered the question you asked, correctly.
• Put an anchor in the prompt and the AI stops precisely there. If you want it to search, ask it to "find the holes still left and close them."
• Lying shows up when the report itself becomes the deliverable. Ask for the fix and the steps to reproduce, and give it a place to write what it couldn't do. Then reports turn into observations.
• A 0 from a checker without AI doesn't mean "no holes" either. It means "pressing didn't break anything." Write down what the checker doesn't look at, and put the AI's eyes there.

I'm Japanese and wrote the original in Japanese; it was translated with help from Claude. The tests were set up and run through Claude Code at my request.