Dev.to AI 🤖 Ai 👁 0 📖 3 min read

I broke an online shop 42 times to see if free AI models can find the root cause

When a website breaks, the error rarely shows up where the problem started. A checkout page times out because the payment service is slow, because a network link is dropping packets. Engineers dig through metrics, traces

I broke an online shop 42 times to see if free AI models can find the root cause


When a website breaks, the error rarely shows up where the problem started. A checkout page times out because the payment service is slow, because a network link is dropping packets. Engineers dig through metrics, traces and logs until they find the one thing that actually broke: the root cause.

Everyone wants AI to do this job. I wanted to know how well free AI models really do it, and what evidence they actually use. So I built Rootline, an open benchmark that costs $0 to run.

How it works

  1. Break it. I run the OpenTelemetry Demo, a real microservice web shop under steady load, and break it on purpose in 14 different ways: payment errors, memory leaks, lock contention, slow shipping, network latency, packet loss, connection resets and more. There are also normal runs with nothing broken, as a control.
  2. Freeze it. For each incident, I save ten minutes of metrics, traces and logs around the fault.
  3. Ask the AI. The same frozen evidence goes to three free models: gpt-oss-120b (on Groq's free tier), Qwen3 8B and Gemma 3 4B (on a free Kaggle GPU). Every line of evidence has an ID, and every answer has to cite the IDs it relied on.
  4. Score it. Each answer is checked for the right service, the right kind of fault, the right layer (network or application), and whether the cited evidence actually exists.

That's 42 incidents × 4 kinds of evidence × 3 models = 504 scored runs.

The hardware part

I wanted to know if emulated network faults are a fair stand-in for real ones. So two Arduino UNO Q boards became part of the shop: one carried real payment traffic over Wi-Fi, the other could flood the Wi-Fi channel. Overnight, fully unattended, the controller cut the radio for 30 seconds, crowded the channel and turned the transmit power down, then took snapshots.

What I found

The headline is negative, and I'm reporting it that way:

Model Exact root cause (combined evidence) Average confidence
gpt-oss-120b 9.5% 84%
Gemma 3 4B 4.8% 88%
Qwen3 8B 2.4% 95%


A few things stood out:

  • Very confident, mostly wrong. All three models said they were 84–95% sure on average. If you trusted the confidence number, you would be misled.
  • Nothing was ever "nothing". On the healthy runs, with no fault at all, every model still claimed something was broken, in every single answer.
  • A layer answer can be a habit. Gemma answered "application problem" on all 42 incidents, so its good score on application faults isn't a skill.
  • The caller gets the blame. On broken network links, the best model usually named the service making the call (checkout) instead of the service at the other end (payment), which is what I'd decided in advance to count as correct. Counting either end raises its score from 9.5% to 16.7%. One scoring rule changes the story a lot.
  • More evidence didn't help. Giving the model metrics, traces and logs together did not beat logs or metrics alone. With a fixed prompt size, more breadth means less depth.
  • Some faults barely leave a trace. I built a check that measures whether each fault actually shows up in its own data. The real Wi-Fi outages showed up clearly. Some emulated faults, like connection resets on a link that keeps one connection open, barely showed up at all, and no model can diagnose what the evidence doesn't show.

This is a pilot: 42 incidents gives wide error bars, and neither of my two pre-stated hypotheses reached significance. It's a starting point, not a verdict on AI for operations.

Everything is open

Clone the repo, run make data && make reproduce, and every table and chart rebuilds from the published runs.

What's next

For v0.2: stronger versions of the faults that left no trace, more healthy runs, asking models for both the symptom and the cause, and testing whether hiding service names changes their answers.

If you work on observability or AI for operations, I'd love feedback. Open an issue or tell me what you would test next.

Check out more details at: https://github.com/Flotrpy/RootLine

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.