I Broke My Own Payments Service on Purpose, Then Asked an AI to Snitch on It

The classic 2 AM page used to mean a human staring blankly at a dashboard, waiting for the third coffee to kick in. Today? It looks more like a panicked engineer desperately tossing a broken production environment at an AI agent with a silent plea: "please help me fix this, make no mistakes." We are officially in the era of handing zero context chaos to algorithms. I wanted to see if the reality lived up to the hype, minus the actual 2 AM wakeup call. So, I broke my own payment processor on purpose, wide awake, and watched three services fall over like dominoes. Then I asked an AI agent cold, zero hints, over an open protocol, to tell me why. Before I could even finish opening the logs, the agent had already parsed the entire blast radius. It named the guilty service, cited the exact trace, and did it faster than I could have mapped it by hand.

What I built (and why I immediately wanted to break it)

I didn't want to test this on five curl calls against a hello-world app, that's not a real incident, that's a toy. So I stood up three services that talk to each other the way production actually does, with all the fragility that implies:

• api-gateway (:8000) a public front door, a canary router: 80% of traffic to a stable orders-service v1, 20% gambled on an untested v2.
• orders-service (:8001 v1 / :8002 v2) that writes an order to SQLite, then synchronously calls payments to charge the card. If payments is slow, everything behind it waits.
• payments-service (:8003) a fake third-party payment processor, with a secret control endpoint I built to make it misbehave on demand.

All three are wired up with OpenTelemetry, the open standard for emitting traces (a timed, nested record of one request as it bounces between services), metrics, and logs. I got the whole HTTP layer reporting by wrapping each service in opentelemetry-instrument, zero code changes, shipping to a self-hosted SigNoz instance.

Standing SigNoz up was a truly simple affair, usually where afternoons go to die in a swamp of broken Docker configs and missing dependencies. Not this time. Within twenty minutes, I had a clean, open - source telemetry pipeline ready to ingest whatever chaos I could throw at it

curl -fsSL https://signoz.io/foundry.sh | bash
foundryctl cast -f casting.yaml   # deployment: {flavor: compose, mode: docker}

UI on localhost:8080, telemetry ingestion on 4317/4318, up in under five minutes from a cold clone.

Then I generated believable traffic with k6 ramping 1 to 15 virtual users over 11 minutes to mimic a real morning traffic curve, instead of flatlining it with a hammer:

• 5,814 requests, 245 failed (4.21%)
• http_req_duration: avg 418ms, median 103ms, p90 651ms
• p95: 3.26 seconds

That p95 is the whole story in one line. The median stays calm while a nasty subset of requests are having a very bad time.

The crime

Three minutes into sustained peak, I degraded the payments control endpoint: introducing a 2 to 4.5 second delay per request and dropping 40% of calls with an HTTP 503.

Think of it as a way simulate what happens when a dependency team deploys a minor change that wasn't properly load-tested. Because that service begins lagging, it consecutively triggers a massive slowdown across our main application. The system isn't dead, but it's suddenly limping and unreliable in a way that standard uptime checks completely miss until the blast radius has already spread.

Because api-gateway → orders-service → payments-service is a synchronous call chain, the pain shot straight back up. All three services' error rate and p99 latency spiked in the exact same 5 second buckets. That simultaneity is the tell, three services failing in lockstep isn't three problems, it's one problem wearing three costumes.

I pulled one trace that shows it end to end, trace 0050c21ff1f619a14f9bbbb7f7a5a182, 11 spans, 8 errored. A span is one timed unit of work inside the request. Stripped down to the guilty path:

That leaf span(the last span) is the confession. payments POST /charge doesn't call anything downstream, it just sits for 2.73 seconds and fails. Each span's duration is within ~40ms of its parent's: the proof this is a blocking cascade, not three independent slowdowns, each layer just waits on the one below it, then re-throws the failure on the way up (503 → 502 → 502). It took twenty minutes of reading raw telemetry by hand to be sure, exactly the twenty minutes I wanted to see if an AI could skip.

The actual experiment: can an AI reach the same verdict?

SigNoz's cloud product ships an AI teammate called Noz that investigates incidents in plain English, but it's cloud-only, no help on my self-hosted box. What is available is SigNoz's open-source MCP server: 40 tools spanning traces, logs, metrics, alerts, and dashboards. (MCP is just a standard way to hand an AI agent a toolbox it can call into.) Instead of taking the pitch on faith, I wired an ordinary AI coding agent to my own broken services and asked it, cold:

"My api-gateway service is having latency and error issues in the last 15 minutes. Investigate using SigNoz, check traces, errors, and top operations for api-gateway, orders-service, and payments-service and tell me what's actually wrong."

Seven MCP tool calls across two turns later, it came back with, close to verbatim:

"Root cause: payments-service is the failure point, everything else is a symptom. POST /charge in payments-service is failing with HTTP 503, taking 2.6 - 4.5s to fail (vs. its normal ~90ms p50)... All three services spiked simultaneously, in the same 5 second buckets, which rules out three independent problem, it's one upstream failure cascading through the call chain."

Correct, including the same simultaneous buckets reasoning I'd worked out by hand. I pushed further: "Is this a cascading failure from payments-service? Show me the specific trace." It pulled the exact trace ID above, unprompted, walked the span tree hop by hop, and landed on "each hop is just blocking on the one below it, not doing independent slow work of its own." Annoyingly, exactly what I'd concluded twenty minutes earlier.

Worth being precise about what that proves: I told it explicitly to investigate using SigNoz, not to read the source. Its tool call log shows seven calls to the SigNoz MCP server, not a single file read. The diagnosis came entirely from querying telemetry, not from opening main.py and spotting the exception I'd planted like a landmine.

It cross referenced operations, error rates, and trace timing across three services in about the time it took me to type the follow-up, because traces, metrics, and logs already lived in one queryable, OpenTelemetry native store, with no dashboard jumping to line up timestamps. The AI wasn't magic; it was another client on the same query surface I had.

While I was in there,the agent also caught a second bug I'd planted, unprompted: 5% of GET /orders/{order_id}/receipt calls on the v2 canary were quietly raising an unhandled RuntimeError. A plain uptime check sails right past this, 95% of requests still succeed, dashboard stays green. One Query Builder filter (service.name = 'orders-service' AND name = 'GET /orders/{order_id}/receipt' AND has_error = true) surfaced every occurrence, full stack trace attached, zero try/except written by me.

What did not go smoothly

• My first MCP connection came back with a flat 403. Cause of death: a service account with no role assigned. Fix: assign signoz-admin, problem gone.
• The role-picker under Settings → Service Accounts has a genuine bug: pick a role, click outside the panel, and the selection silently vanishes. The real Save/Cancel buttons only render via a direct deep-link URL.
• MCP tools only load when an agent session starts. I registered mid-session and had to start fresh before the new tools showed up.

None of that dents the core result. Once auth was correct, the agent's investigation was faster and more thorough than my own manual click-through, starting from the same raw telemetry, not a curated summary.

The Takeaway

AI isn't going to replace your senior engineers. But they’re right about one thing it can read a trace graph faster than you can.

The real plot twist here isn’t that the LLM is some kind of omniscient oracle; it’s that AI is only as useful as the telemetry foundation you feed it. The "AI teammate" pitch stopped being marketing fluff the second a generic agent, acting on raw data with zero hints, handed me a flawless incident report before I could even finish opening the logs.

When you strip away the hype, three practical realities made this work:

• One store, not four: The agent answered "what's wrong across three services" in one pass because traces, metrics, and logs live in the same backend. There was no tool-hopping or manual timestamp alignment just to figure out the blast radius.
• The AI tooling isn't locked behind a cloud tier: While Noz (SigNoz's builtin AI teammate) is cloud-only, the open-source Model Context Protocol (MCP) server that did the heavy lifting here is completely free. It runs locally and talks directly to a self-hosted instance without requiring an enterprise contract.
• Getting instrumented cost me nothing: opentelemetry instrument wrapped FastAPI and httpx out of the box with zero code changes. The classic friction of "telemetry is too hard to set up" simply wasn't there.

I haven't run this same chaos test against Datadog or Grafana, so I won't claim SigNoz uniquely wins the observability crown. But I will say this: the debate over whether an agent can handle real, zero-context production failures is officially over. It beat me to the punch on my own crime scene.

SigNoz: signoz.io · MCP server: github.com/SigNoz/signoz-mcp-server