I Tracked Every AI Coding Mistake for a Week — the 5 Patterns That Keep Coming Back

I spent a week building a small side project with an AI coding assistant doing most of the typing. Not to test whether it could code. I already knew it could. I wanted to know where it goes wrong, how often, and what each wrong turn actually costs.

So I kept a log. Every time the assistant produced something broken, misleading, or quietly wrong, I wrote it down: what it did, how long it took me to notice, and how long the fix took.

By Friday I had 47 entries. They fell into five patterns, and those five patterns covered 41 of the 47. Here they are, with the receipts.

1. The hallucinated API (11 times)

The assistant invented functions that do not exist. Not once in a while. Eleven times in five days.

The worst one: it called response.json(strict=False) on a Python requests response. That parameter does not exist. The code looked right. It read right. It ran fine until that exact line, then threw a TypeError, and I spent 25 minutes reading requests documentation before I realised the argument was made up.

The check that catches it: before I run anything the assistant wrote, I search the docs for every function name I do not recognise.

2. The confident wrong import (8 times)

Close cousin of number one, but sneakier. The function exists. The import is wrong.

It imported urlparse from Python 2 into a Python 3 project. It pulled datetime.utcnow and used it in code that then compared timezone-aware and naive datetimes, which blew up three functions away from the import.

The check: I run the file the moment it is written. Not after the whole feature. The moment.

3. Tests that pass for nothing (7 times)

This one scared me more than the others.

I asked for tests for a pricing function. The assistant wrote five tests. All green. I felt good for about ten seconds, then I read them. Two tests asserted the same thing twice. One test created an object, never called the function under test, and asserted that the object existed.

The suite passed because it tested almost nothing. And the worst part is the feeling it gives you: the green checkmarks feel like proof. They are not proof. They are a screensaver.

Total cost: roughly two hours, because I had to write the real tests from scratch after deleting the fake ones. Plus the embarrassment of almost shipping them.

The check: read every generated test and ask one question. If I delete the function being tested, does this test fail? If the answer is no, the test is decoration. Delete it.

4. The silent wrong default (8 times)

Some mistakes throw errors. These do not. They just quietly do the wrong thing.

The assistant set a pagination default of 100 items per page in an API that rate-limited at 50. It sorted a list of dates as strings, which worked perfectly until the 10th of a month showed up before the 2nd.

None of these crashed anything. All of them shipped wrong behaviour that a user would have found before I did. In my log I counted the debugging time only: about 40 minutes across the week.

The check: for every default, limit, or sort the assistant picks, I ask it to say out loud what happens at the boundary. What happens with zero items? With 101? With a date in December? The assistant is decent at answering that question. It is terrible at asking it unprompted.

5. The copy of my own bad code (7 times)

This one is my fault, partly.

The assistant copies patterns from my existing code. That is usually good. But my project had a helper function with a bug in it, a date formatter that dropped the timezone on weekends. Don't ask. The assistant used that helper in four new places, spreading the bug further.

It does not know which of my patterns are good and which are leftovers. It treats all of them as house style.

Cost: about an hour, including the time I spent fixing the original helper and then hunting its copies.

The check: when the assistant says "following the existing pattern in your codebase," I stop and read the pattern it is copying. Half the time it is fine. The other half, I am about to multiply a mistake.

The honest downside

A week of logging made me slower. Stopping to write down every mistake, running every file the moment it was written, reading every test: that added maybe 20 minutes a day. And for small, throwaway scripts, it is overkill.

But for anything I plan to keep, the log paid for itself. Forty-seven mistakes in one week, and I caught 41 of them with five checks that take minutes. The six that slipped past were all in category four, the silent defaults, and I only found them because I wrote the log and went back looking.

The assistant is not a bad programmer. It is a fast programmer who never doubts itself. The work is building the habit of checking the exact places where confident code goes wrong: invented APIs, wrong imports, empty tests, silent defaults, and my own bad patterns copied back at me.