The AI PR that "fixed" a flaky test by sleeping longer
I keep seeing the same move in AI-written PRs. Flaky test fails. Agent "fixes" it by bumping a timeout, adding sleep(2), or wrapping an assert in a retry loop. CI goes green. Everyone moves on. The race is still there.
I keep seeing the same move in AI-written PRs.
Flaky test fails. Agent "fixes" it by bumping a timeout, adding sleep(2), or wrapping an assert in a retry loop. CI goes green. Everyone moves on.
The race is still there. You just papered over it.
The 60-second check I run now
When the diff touches tests and the code claim is "flaky" or "intermittent":
- Look for new sleeps, longer timeouts, or retry wrappers around the same assertion.
- Ask: did the PR remove the shared mutable state / order dependency, or only wait harder?
- If it's only waiting harder, I bounce it. Green CI that still races will fail again under load.
This is one of the stops on a short bar I use before I merge agent PRs. Free one-pager if you want the rest: https://chopragunji.gumroad.com/l/zpnmdn
If you'd rather someone else do a hard pass on one scary AI PR, I still have founding slots at $49 (normally $99): https://chopragunji.gumroad.com/l/byoyi/FOUNDING
Curious what your team does when agents "fix" flakes — genuine timeouts, or the sleep-and-pray patch?
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.