I Made My AI Coding Agent Log Its Own Mistakes. 8 of 23 Were the Same Kind.

I'm building an options trading tool with Claude Code. Everything runs in a sandbox account. The agent writes most of the code, so I wanted a real number on how it fails, the same way a team tracks its bugs.

Every mistake it made on this project went into a log. At 23 entries I sorted them, and one pile came out clearly bigger than the rest.

Step 1: Log every miss in one place

Each entry goes into the project's task file under one fixed label, "AI Miss", with the date. It records what the agent claimed, what was actually true, and how the gap got caught. Here's a shape you can copy:

AI Miss (YYYY-MM-DD): what the agent said or did
Actually: what was true, with the file, number or output that shows it
Caught by: the test, rerun or person that caught it

One label matters more than the format. When every miss carries the same tag, you can pull the whole list with a single search.

Step 2: Sort it twice, the second time blind

The session that made the mistakes sorted all 23 first. Then a fresh session got only the entry text, with account numbers masked and no labels, and sorted them on its own.

The two agreed on 21 of 23. I settled the last 2 by asking which guard would have caught each one.

The blind pass is the step people skip. Without it, all you have is the agent's own reading of its own mistakes.

Step 3: Look at the piles

The pile that should worry you

Eight of the 23 are confident statements nobody checked.

A draft said a pattern appeared 50 times across four files. A scan of the actual code counted 57.

In another one, the agent wrote that a certain code path meant the position would get no stop-loss at all. Two later branches in the same function already caught that case. I had started planning work around a problem that didn't exist.

Nothing about either sentence looks off when you read it. That's exactly why this pile is the expensive one.

What happens next

This is round zero of a loop based on Jeff Dean's idea of a self-improving system. His point is that the loop can't spin infinitely fast, because the experiments still have to run and somebody still has to check them.

So round zero measured the starting line. One full run of this tool's 7,000+ checks takes about eleven and a half minutes on my machine. That's too slow for every edit, so there's a fast lane while I work and the full run before anything goes live.

From here on, every round logs whether the agent added to that pile of 8. Round one is running now.

Follow the loop

Each round goes up on my YouTube channel first, with the numbers and the loop diagram. If you want to see what happens to that pile of 8, subscribe here:

Subscribe to Dr_Delta on YouTube

I'll add each round to this series here on dev.to as well.

If you keep a mistake log for your own AI agent, tell me in the comments: what's your biggest pile?