Everything Was Green. The AI Was Still Wrong.

Three days before I wanted to call this done, I was staring at the SigNoz Logs Explorer trying to figure out why my OTLP logs weren't showing up. Traces had landed fine from the first run. Logs: nothing. No errors, no warnings, just silence — which is its own kind of bug, because "nothing happened" gives you nothing to grep for.

The agent is still wrong. And that's the thing standard observability cannot see, because it was never built to look for it.

The gap, and the rule we held ourselves to

Traditional observability answers three questions: is it up, is it fast, did it throw. An AI agent can pass all three and still hallucinate a fact, act on zero retrieved evidence, or run a plan that was broken from step one. None of that trips a 500. It shows up as a decision, and decisions don't have status codes.

Here's the design rule that mattered more than anything else we built: Veritas can never become a black box scoring another black box. If it can't say why it flagged a run, in plain English, it isn't accountability, it's a second opinion with no reasoning attached. So every signal it uses has to already sit on the trace, and every point of the score has to be arguable by a human.

That rule is why nothing in the risk engine below is machine-learned. It's a constraint we chose on purpose, and it's the single idea the rest of this post is built around.

What Veritas does

It reads real OpenTelemetry telemetry from a self-hosted SigNoz instance and computes a deterministic Semantic Risk score, 0–100, for every agent run. Then it draws a Replay Timeline so you can point at the exact step where things went sideways, instead of scrolling logs like an archaeology dig. Veritas stores none of the telemetry itself — SigNoz is the system of record, Veritas is a thin read-layer on top.

Our FastAPI agent runs five stages — planner → retriever → tool → memory → response — each its own nested OTel span under a root agent.run span, carrying domain attributes like tool.success, retrieval_score, response.confidence. Six independent heuristics turn those attributes into the score, each with a fixed point weight living in one file:

RETRIEVAL_ZERO_DOCS = 25
TOOL_FAILURE = 30
PLANNER_ZERO_TASKS = 25
CONFIDENCE_VERY_LOW_PENALTY = 30
LATENCY_HIGH_PENALTY = 10

We verified it against five scenarios, run through the real engine, not typed by hand:

Every weight in that file is a multiple of 5 — which meant an earlier planning doc's "illustrative" scores (48, 84) were never numbers the engine could actually produce. We kept the real ones instead of quietly rounding the engine to match a doc. A verified demo that was secretly tuned to look verified is exactly what this whole project exists to catch.

Here's the Replay Timeline on an actual tool-timeout run, score 75, HIGH — retriever thins to one document, the policy lookup times out at 330ms, confidence collapses to 0.41, each contribution shown inline instead of buried in a log line:

Click further and a flagged run expands into a full incident report — root cause, expected-vs-observed, a plain-English why-it-happened chain, business impact — generated straight from the same risk breakdown, no extra model call:

Why SigNoz, specifically

Traces, logs, and metrics as one correlated signal — not three tools duct-taped together (Prometheus + Grafana + Tempo + Loki) — is the only reason the Replay Timeline was buildable in a weekend. Here's the raw Logs Explorer, mid-debugging one of the failures below, structured risk_score and risk_level fields sitting right there:

What actually broke

Logs that vanished into nothing. Traces landed in SigNoz from the first run. Logs didn't. We wired the OTLP log handler, hit the endpoint, checked the Logs Explorer: nothing, no errors, no warnings. We suspected propagate = False somewhere blocking records before they reached the root logger's handler, grepped the whole logging setup for the word propagate, found zero hits — theory dead, no error message to chase. The fix that worked for us was attaching the handler directly to each veritas.agent.* logger by name instead of the root logger, which got logs flowing immediately, with the honest side effect that lines now duplicate, since both handlers fire on the same record.

One thing worth flagging here, per checking this against the actual OpenTelemetry Python docs rather than trusting our own memory of why it worked: the documented, standard setup attaches the logging handler once, to the root logger, and named child loggers reach it automatically through Python's default propagate=True — the opposite of what fixed it for us. The OTel contrib docs call out a much more common cause for exactly this symptom: if anything in your app calls logging.basicConfig() before the OTel logging integration is enabled, the integration's format silently never takes effect. If you hit this same "logs go nowhere, no errors" wall, check your init order first — that's the documented gotcha, and it's entirely possible that's what we were actually working around rather than a root-vs-named-logger issue at all. We're leaving our own account exactly as it happened, working handler-by-handler under time pressure, but pointing at the source so nobody copies our fix as gospel.

A route that was never created. The Replay Timeline threw 404s and every instinct said broken import. It wasn't — app/trace/[id]/page.tsx genuinely did not exist yet, components built, route never wired:

Once it did exist, the trace lookup only recognized two hardcoded keys, "high-risk" and "low-risk" — all five new demo fixtures sat on disk, fully generated, invisible to the UI.

SigNoz's version gap on percentiles. A P50/P90 latency panel threw Function with name 'histogramQuantile' does not exist — a real version mismatch in the exact SigNoz/ClickHouse build we self-hosted, not our config. We rebuilt every percentile panel as a two-query sum ÷ count average instead.

Zooming out

Beyond one run, there's a fleet-level view — risk distribution, latency by stage, top failing agents, rule-based recommendations keyed off whatever's actually failing most:

In the interest of not overstating what's live: the numbers on this page are simulated 24-hour fleet traffic across three agents, and the dashboard says so on its own face. The Refund Agent card is backed by a real, openable trace; Research and Support Agent aren't, yet. We'd rather say that here than let a reader find the caveat themselves.

What I'd tell past-me

"No errors" is not a clean bill of health — it's a symptom with the volume turned down, and it deserves the same suspicion as a stack trace, maybe more. The moment we caught our own planning doc describing scores our own code could never produce was the moment to delete the doc, not the truth. And before chasing a theory about broken imports, spend five minutes confirming the file you're debugging actually exists.

One line

We don't think AI agents need less autonomy. We think they need the same accountability every other system eventually earned before anyone trusted it with production traffic.

Veritas is open source:https://github.com/ishantgupta30/veritas-flight-recorder