When a Code Review Says "No Issues" but Found Bugs

Originally published on the DevReview blog.

DevReview caps every review at eight comments, ranked by severity. The point is to keep a pull request readable: a few findings that matter, not forty that don't. A cap like that only works if the reader can trust what is left out. This post is about a path where they couldn't.

We found it while checking our own claims against our own code, and we haven't measured whether it ever happened in production.

What the review did

After the model returns its findings, the review code does three things in this order:

1. Sort the findings by severity and keep the top eight.
2. Drop any finding whose file path is not in the pull request. The model sometimes cites a file that doesn't exist in the diff, and a comment on a file that isn't there can't be posted.
3. If nothing is left to post, write "No issues worth your attention" with a green check.

The problem is the order of steps 1 and 2. Suppose the model returns ten findings, and the top eight all cite file paths that aren't in the PR. Step 1 keeps those eight. Step 2 drops all of them. Step 3 sees nothing left and posts the green check. The two real findings that were ranked ninth and tenth were already cut by the cap.

So the pull request shows "no issues" while the model has reported eight bugs it got the location wrong on, and two it got right. The reader gets a pass they shouldn't have.

Two separate things were wrong

A discarded finding looked the same as no finding. Step 2 removed findings without counting them anywhere the reader could see. A green check means "the reviewer looked and found nothing". It should not also mean "the reviewer found things and we threw them away".

Fake paths used up the budget. Even when the result wasn't a false all-clear, findings with invalid paths took slots from valid ones. A valid warning could be suppressed to make room for a comment that was never going to be posted.

What we changed

• The path check now runs before the cap. Only findings that can actually be posted compete for the eight slots.
• Discarded findings are counted and disclosed. If any findings were dropped because their file wasn't in the PR, or because they were malformed, the review says so: "N findings could not be posted because they did not match this PR's files or the expected format."
• The green check is stricter. It appears only when the model reported zero findings. One discarded finding is enough to remove it.
• Suppressed comments show a breakdown. When the cap cuts comments, the note now says what was cut, for example "2 remarks suppressed to keep this review focused: 1 NIT, 1 PRAISE." We also removed the word "lower-priority" from that note, because the cap can also cut a bug when there are more than eight.

We also weakened some of our own marketing copy. We had said nothing is hidden silently. That is true of comments cut by the cap, which are disclosed, but not of malformed comments, which were dropped without a count. The copy now says what is true.

What we checked, and what we didn't

We wrote the tests first and confirmed they failed on the old behaviour. The review function is now tested end to end with faked responses from the model and from GitHub, including the case above: eight invented paths plus two valid findings now posts the two valid ones and discloses the eight. Removing the counting of malformed findings makes a test fail, so that test does something.

What we have not done:

• We haven't watched this on a real production review. The tests use faked responses. A real GitHub and Anthropic exchange is the next check.
• The dashboard can still mislead. A review where every finding was discarded shows as "completed, 0 comments", which looks like a pass. The discarded count appears in the pull request comment and in our logs, but we don't store it in the database yet. Showing it on the dashboard needs a new column.
• Related gaps exist. A green check can still appear when some files in a pull request were not reviewed at all, and a code fence inside a comment can make the whole review fail. Both are on our list as separate fixes.

The general rule

If a system filters, ranks and then reports, the order of those steps decides what the report means. Ask what the report says when the filter removes everything. For us, the answer was "all clear", which is the one thing it must never say after discarding findings.

DevReview is a GitHub App that reviews pull requests and caps every review at eight comments, ranked by severity. If you want to see how it works before trusting it with your repos, start with How DevReview Works.