Where Agents Fail: Part 1
Your agent says the bug is fixed, but it's not. AI models are smarter than ever, yet they still make obvious mistakes. I'm starting Where Agents Fail, a new series where I analyze thousands of real agent sessions, find
Your agent says the bug is fixed, but it's not. AI models are smarter than ever, yet they still make obvious mistakes.
I'm starting Where Agents Fail, a new series where I analyze thousands of real agent sessions, find common failure patterns, and try to fix them.
This first post is the intro. It covers a few patterns that stood out early on, and each new episode will dig into one specific failure pattern and test whether it's fixable with strategies you can use today.
Looking across real sessions
I started this project by analyzing 160 of my own recent sessions, pulling out every moment where the agent did something wrong and I had to step in and correct it. That gave me 231 specific agent mistakes.
I've also started digging into public datasets to see how far these patterns go beyond my own work, and 'replaying' those key failure moments on frontier models like Gemini 4 Carbon to see if they persist.
One early signal from the public SWE-chat dataset (more than 11,000 coding sessions): developer messages reporting a failure dropped by about a third between early 2026 and the summer.
With each frontier model release, certain categories of issues dissolve and new ones show up.
Three patterns that stood out so far
- Saying something is fixed when it isn't. The agent makes a change, sees the tests pass, and announces that everything works, while the actual app is still broken.
- Going further than you asked. You ask for a plan or quick thoughts on a problem, and the agent jumps straight into coding a feature or opening a pull request.
- Overcorrecting on feedback. You leave narrow feedback on one detail, and the agent turns it into a blanket global rule across the entire project.
Staying up to date on AI best practices means paying just as much attention to the limitations that stick around as to what each new generation unlocks. In the next post, I'll go deep on one of these patterns and share what happened when I tried to fix it.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.