Dev.to Security πŸ” Cybersecurity πŸ‘ 0 πŸ“– 8 min read

Catching Silent AI Agent Failures in Production: Metrics, Drift and Hallucinated Actions

In Part 1 we instrumented an AI agent with OpenTelemetry and learned to read a trace: one run, broken into spans, showing exactly where a refund agent skipped a failed policy check. But nobody reads every trace. In prod

In Part 1 we instrumented an AI agent with OpenTelemetry and learned to read a trace: one run, broken into spans, showing exactly where a refund agent skipped a failed policy check.

But nobody reads every trace. In production you need the system to tell you which traces to read. That's what this post is about:

  • The metrics and alerts that actually matter for agents
  • Agent drift: when behavior changes and nobody shipped anything
  • Hallucination detection, including a cheap check for hallucinated actions
  • How observability, evals and red teaming fit into one loop
  • Choosing a tool, and a get-started plan

Why bother? Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027, citing rising costs, unclear business value and weak risk controls. Cost and risk are exactly what good observability exposes.

Start with outcomes, then work down

A team that only tracks latency and token cost knows its agent is fast and cheap. It doesn't know if it's right.

Here's a layered view, top to bottom:

Layer Track Why Example alert
Outcome Task success, escalation rate, user feedback The only direct measure of "does it do its job" Success drops > 5 pts week over week
Quality Groundedness, correctness, policy compliance Catches confident wrong answers that return 200 Any refund with policy score below threshold
Path Steps per task, loops, plan changes, handoffs Long or looping paths signal confusion before failure Run > 20 steps, or same tool called 5+ times
Tools Tool error rate, arg validity, unexpected tools Most real damage happens through tools A summarize task calls send_email
Retrieval Chunk count, relevance, empty results Bad context drives most hallucinations Top-chunk relevance low on 10% of queries
Cost / speed Tokens and $ per task, p95 latency, retries Agents burn budget quietly Cost per task doubles after a deploy
Safety Blocked inputs, injection signals, PII in output Attacks and leaks never surface as errors Injection detector fires on a tool response

As code, a few of those alerts look like this:

# alerts.yaml
- name: success_rate_drop
  query: success_rate{task_type="refund"} [7d] vs [7d offset 7d]
  condition: drop > 0.05
  severity: page

- name: looping_agent
  query: max(steps_per_run) by (trace_id)
  condition: "> 20"
  severity: ticket

- name: repeated_tool_calls
  query: count(execute_tool) by (trace_id, gen_ai.tool.name)
  condition: ">= 5"
  severity: ticket

- name: unexpected_tool
  query: execute_tool{task_type="summarize", gen_ai.tool.name="send_email"}
  condition: "> 0"
  severity: page

- name: cost_spike
  query: avg(cost_usd_per_task) [1h] vs [24h before last_deploy]
  condition: ratio > 2.0
  severity: page

(Pseudo-query syntax: translate to PromQL, Datadog, or whatever your backend speaks.)

Two habits make these numbers actually useful:

  1. Slice by task type. A refund flow and a returns-label flow can live in the same service and fail in completely different ways. A blended success rate hides both.
  2. Link every alert to a trace. The person who gets paged should land on the exact run that crossed the line, not a graph.

Also: some attacks only show up across many turns. Review sessions, not just single messages.

Agent drift: when nothing changed but everything changed

Agent drift is a change in how an agent behaves without any change to your code. It's caused by a new model version, an edited prompt, a changed tool, updated documents, or a shift in what users ask.

The best-known evidence: researchers from Stanford and UC Berkeley compared the March and June 2023 versions of GPT-4 on the same tasks. Its accuracy at identifying prime numbers fell from 84% to 51%, and many shifts traced back to weaker instruction following (Chen, Zaharia & Zou). Same product name, very different behavior, three months apart.

For agents, drift shows up in four places:

Source What happens
Model updates Provider changes or retires a model; the new one plans differently or calls tools in a different order
Prompt / policy edits A small wording fix solves one case and breaks three others
Tool / data changes An API adds a field; new KB docs change what retrieval returns
Input drift Users ask new kinds of questions, or attackers start probing

How to catch it

1. Pin and tag versions. Pin model versions where you can, and put model, prompt and KB versions on every span (we did this in Part 1).

2. Compare behavior before and after every change. Not just success rate: the shape of behavior.

import pandas as pd

def behavior_profile(df: pd.DataFrame) -> pd.Series:
    """df: one row per run, exported from your trace backend."""
    return pd.Series({
        "success_rate":     df["success"].mean(),
        "avg_steps":        df["steps"].mean(),
        "p95_cost_usd":     df["cost_usd"].quantile(0.95),
        "escalation_rate":  df["escalated"].mean(),
        "pct_issue_refund": (df["tools"].str.contains("issue_refund")).mean(),
        "avg_policy_score": df["policy_score"].mean(),
    })

before = behavior_profile(runs[runs.prompt_version == "refund-v13"])
after  = behavior_profile(runs[runs.prompt_version == "refund-v14"])

diff = (after - before) / before.replace(0, 1e-9)
print(diff[diff.abs() > 0.10].round(2))   # anything that moved > 10%
pct_issue_refund    0.23
avg_steps          -0.31

Fewer steps and 23% more refunds after a prompt edit? That's worth a look before finance notices.

3. Keep a fixed regression set of real production cases and rerun it on every release.

4. When a metric moves and your release log can't explain it, check your provider's changelog and your input distribution.

Hallucination detection: words and actions

In a chatbot, a hallucination is a wrong sentence. In an agent, it can be a wrong action. There are three kinds:

Type Example
Content States a 60-day refund window that doesn't exist in the policy
Action Calls a real tool with an invented order ID, or a tool that doesn't exist
Outcome Tells the user "I've cancelled your subscription" when no tool call did it

Outcome hallucinations are the easiest to miss, and the easiest to catch with traces: compare what the agent says it did with what it actually did.

A cheap claim-vs-trace check

import re

# map phrases in the reply to the tool that must have run
CLAIMS = {
    r"\b(refund(ed)?|processed (a|your) refund)\b": "issue_refund",
    r"\bcancel(l)?ed (your )?subscription\b":       "cancel_subscription",
    r"\b(sent|emailed) (you|a) (copy|receipt|confirmation)\b": "send_email",
    r"\breset (your )?password\b":                  "reset_password",
}

def outcome_hallucinations(final_reply: str, trace: dict) -> list[str]:
    called = {
        s["attributes"].get("gen_ai.tool.name")
        for s in trace["spans"]
        if s["attributes"].get("gen_ai.operation.name") == "execute_tool"
        and s["status"] == "OK"
    }
    reply = final_reply.lower()
    return [
        tool for pattern, tool in CLAIMS.items()
        if re.search(pattern, reply) and tool not in called
    ]

missing = outcome_hallucinations(reply, trace)
if missing:
    flag(trace_id, reason=f"claimed but not executed: {missing}")

It's crude, fast, nearly free, and runs on every trace. Notice it also checks status == "OK": a tool that was called but errored doesn't count as "done."

Argument validation for action hallucinations

from pydantic import BaseModel, Field, field_validator

class IssueRefundArgs(BaseModel):
    order_id: str = Field(pattern=r"^ORD-\d{8}$")
    amount: float = Field(gt=0, le=500)

    @field_validator("order_id")
    @classmethod
    def order_must_exist(cls, v):
        if not orders.exists(v):
            raise ValueError("order_id not found: likely invented")
        return v

Validate before the tool runs (as a guardrail) and log the result on the span, so invented IDs show up in your metrics.

Layer your detectors

Method Catches Trade-off
Groundedness vs. retrieved context Claims not in the docs the agent saw Needs retrieval logging; misses bad source docs
Claim vs. trace "I did X" with no matching call Cheap and precise; actions only
Argument validation Invented IDs, out-of-range values Needs a schema per tool
LLM-as-judge Unsupported or off-policy free text Costs tokens; judge must be calibrated against human labels
Consistency sampling Answers that change across reruns Multiplies cost; use on a sample
User feedback Errors users notice Late and incomplete

A practical setup:

every trace          β†’ rule-based checks (claim-vs-trace, arg validation)
sample + high-risk   β†’ LLM judges (groundedness, policy compliance)
low confidence       β†’ human review queue
reviewed cases       β†’ new eval data

Observability vs. evals vs. red teaming

These get lumped together, but each answers a different question:

Observability Evals Red teaming
Question What did happen? What should happen? What could happen?
Inputs Real traffic Curated test sets Adversarial, often multi-turn attacks
When Continuously, post-deploy Every release + scheduled Pre-launch, after big changes, continuously for high risk
Catches Drift, regressions, cost spikes, incidents Quality regressions Injection, policy bypass, data leaks
Blind spot Only failures real users trigger Only what you thought to write down Doesn't watch live traffic

Reliable teams wire them into one loop:

 red teaming finds a failure ──► becomes an eval (runs every release)
          β–²                              β”‚
          β”‚                              β–Ό
 production traces ◄──────── becomes a monitor (alerts in prod)
 feed new evals & attacks

Observability alone is reactive: you learn after customers do. Red teaming alone is a snapshot. Together they cover before, during and after release.

Picking a tool

Roughly three camps:

  • Open source (Langfuse, Arize Phoenix, OpenLLMetry): control and self-hosting
  • Commercial LLM observability (LangSmith, Datadog LLM Observability): managed storage, dashboards, team features
  • General APM vendors adding agent spans to tools you may already run

Since many now read the OpenTelemetry GenAI format, instrumenting once and sending to more than one backend is increasingly realistic.

Checklist:

  • [ ] Agent-level traces: nested spans for plans, tools, retrievals, handoffs (not a flat list of LLM calls)
  • [ ] OpenTelemetry support: no SDK lock-in
  • [ ] Built-in evals: score traces and turn them into test sets
  • [ ] Data control: self-hosting or regional storage, PII redaction, retention settings
  • [ ] Cost at your volume: per-span pricing grows fast with agents
  • [ ] Security signals: injection attempts, policy violations, unusual tool use

Small team learning what its agents do? Start open source. Multiple teams, compliance requirements, nobody wants to run storage? Commercial starts to make sense.

Getting started: an 8-step plan

  1. Define "good." For each agent: its job, success criteria, and red lines (things it must never do).
  2. Instrument with OpenTelemetry. Spans for model calls, retrievals, tools, guardrails, handoffs, all tagged with versions.
  3. Keep full traces for risky runs. Errors, escalations, payments, data changes, new releases. Redact at capture.
  4. Build one outcome dashboard. Success rate, escalation rate, cost per task, p95 latency, sliced by task type.
  5. Add evals. Score a sample for groundedness and policy compliance; score every high-risk action.
  6. Set a few sharp alerts. Success drops, red-line violations, unexpected tools, cost spikes, each linked to a trace.
  7. Red-team before and after launch. Turn every confirmed finding into an eval and a monitor.
  8. Read traces weekly, as a team. A handful of failed and borderline runs. It's the fastest way to find failure modes your metrics don't measure yet.

Steps 1 and 7 are the ones teams skip most, and they're the ones that decide whether your dashboards measure what matters.

TL;DR

  • Outcomes first, then quality, path, tools, retrieval, cost, safety. Slice by task type; link alerts to traces.
  • Drift happens without code changes. Tag versions, diff behavior profiles, rerun a regression set.
  • Detect content, action and outcome hallucinations; claim-vs-trace is the cheapest win you'll find.
  • Connect observability + evals + red teaming into one loop so every failure becomes a permanent check.

This post is adapted from BotGauge's AI Agent Observability: The Complete Guide.

Which alert has saved you the most pain in production? Drop it below. πŸ“ˆ

πŸ“° Read the original article on Dev.to Security

Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.