My AI agent's 34 tests passed. Then I broke the code 31 times.

"All tests passing" is the most common line in an AI agent's final summary. I wanted to know what it is worth.

Disclosure first: this week I'm working with the team behind Lemon Code, a free coding agent. That's exactly why I didn't want to stop at "it passed".

The build

I asked it for a weather and travel planner: search a city, show the 7-day forecast, pick trip dates, rank the best days to go, and suggest a packing list. Plain Vite and vanilla JS, live data from Open-Meteo.

About 3 minutes later I had a working preview. On the way, 3 of its tests failed on the first run. It read the errors, fixed them, and reran. The final summary said 34 tests passing.

Check 1: run the tests outside the agent

I downloaded the zip and ran it on my own laptop.

 ✓ test/storage.test.js (8 tests)
 ✓ test/weather.test.js (11 tests)
 ✓ test/planner.test.js (15 tests)

 Tests  34 passed (34)

Good, but that only proves the tests pass. It says nothing about whether they would fail.

Check 2: delete each function and see who notices

There's a good post here on dev.to, The Test That Survives Its Own Implementation, about agent tests that keep passing after you delete the code they cover. So I tried it.

For each of the 21 exported functions in the logic files, I added return undefined; as the first line and reran the suite.

for name, body_start in exported_functions(file):
    write(file, original[:body_start] + "\n  return undefined;" + original[body_start:])
    failed = run("npx vitest run --reporter=json")["numFailedTests"]
    write(file, original)

Result: 21 out of 21 caught. Every gutted function broke at least one test, from 1 failure (geocodeUrl) up to 8 (parseISODate). No dead weight.

Check 3: small, realistic bugs

Deleting a function is a loud bug. Real bugs are quiet. So I made 10 one-line changes to the trip planner:

4 caught, 6 survived.

The pattern is clear. Dates, ranges and sort order are well tested. The day-scoring rules are not. scoreDay has one test, and it only checks that a clear, calm day scores higher than a storm. Any weight you tweak still passes that.

Also worth saying: the UI code (ui.js and main.js, about 480 lines) has no unit tests at all. The agent checked the UI by driving the app in a browser instead, searching two cities and saving a trip. That's a reasonable choice for a small app, but it's not something npm test will protect.

What I'd take from this

• "34 passing" was true, and the tests weren't fake. They caught every deleted function and the date bugs I'd worry about most.
• The gaps are where the logic is fuzzy. Scoring rules with magic numbers got one sanity test. That's the kind of code where I'd ask the agent for more: "write a test that fails if the rain penalty changes."
• The check is cheap. Breaking 31 things and rerunning took a couple of minutes with a small script. I'd do this on any agent-written suite before trusting it.

If you want to try the same thing, Lemon Code is free and doesn't need an API key: lemoncode.ai. Build something, download it, and break it. I'd like to hear what your tests let through.

Related reading: Every Agent Task Should End With a Command You Can Run and How to Check Your Test Can Catch the Bug.