One Trace Per Task: Tracing Multi-Agent Systems with OpenTelemetry
In Part 1 we saw that multi-agent systems fail in the gaps between agents, and we built a handoff record to capture what each agent sent, passed and understood. But a handoff record floating on its own isn't much use. Y
In Part 1 we saw that multi-agent systems fail in the gaps between agents, and we built a handoff record to capture what each agent sent, passed and understood.
But a handoff record floating on its own isn't much use. You need it sitting inside one trace that holds the entire task: every agent, every handoff, every tool call, every MCP server, in order. Otherwise you're back to four green traces and one wrong answer.
This post walks through five steps that turn a team of agents into one readable story:
- One trace per task
- A span per agent turn
- A span per handoff
- A clean hierarchy
- Outcomes tagged per agent
Then we'll write a small root-cause walker that reads the trace and points at the agent that broke the task.
Step 1: One trace per task
Create a root span the moment the task starts, and pass its trace context to every agent, sub-agent, tool and MCP server the task touches. OpenTelemetry's context propagation handles this across process and service boundaries using the W3C traceparent header.
# orchestrator.py
from opentelemetry import trace
from opentelemetry.propagate import inject
tracer = trace.get_tracer("multi-agent")
def run_task(user_request: str):
with tracer.start_as_current_span("task") as root:
root.set_attributes({
"app.task.type": "churn_report",
"app.task.goal": user_request[:500],
})
plan = call_agent("planner", {"goal": user_request})
for step in plan["steps"]:
call_agent(step["agent"], step)
Agents over HTTP:
import httpx
def call_agent(name: str, payload: dict) -> dict:
headers = {}
inject(headers) # adds traceparent / tracestate
r = httpx.post(AGENT_URLS[name], json=payload, headers=headers, timeout=120)
return r.json()
# researcher_service.py (receiving side)
from opentelemetry.propagate import extract
@app.post("/run")
def run(request: Request, payload: dict):
ctx = extract(dict(request.headers))
with tracer.start_as_current_span("invoke_agent researcher", context=ctx):
return researcher.run(payload)
Agents over a queue (very common for long-running agent work): put the carrier in the message itself.
# producer
carrier = {}
inject(carrier)
queue.publish({"task": step, "otel": carrier})
# consumer
msg = queue.consume()
ctx = extract(msg["otel"])
with tracer.start_as_current_span("invoke_agent writer", context=ctx):
writer.run(msg["task"])
Do the same for MCP servers and any downstream services. A single missed hop splits the trace again, and the split always seems to happen at exactly the hop you need to see.
Step 2: A span per agent turn
Each agent's work becomes a child span carrying its name, role and version, so you can filter and compare by role later.
AGENT_META = {
"researcher": {"role": "research", "version": "researcher-v7", "model": "your-model-id"},
"writer": {"role": "drafting", "version": "writer-v12", "model": "your-model-id"},
"reviewer": {"role": "verify", "version": "reviewer-v4", "model": "your-model-id"},
}
def agent_span(name: str):
meta = AGENT_META[name]
span = tracer.start_as_current_span(f"invoke_agent {name}")
return span, {
"gen_ai.operation.name": "invoke_agent",
"gen_ai.agent.name": name,
"gen_ai.request.model": meta["model"],
"app.agent.role": meta["role"],
"app.agent.version": meta["version"],
}
The version tag matters more here than in single-agent systems. You might ship a new writer prompt on Monday and a new reviewer on Wednesday. When quality dips Thursday, per-agent versions tell you which change to suspect.
Step 3: A span per handoff
Handoffs get their own spans, not just log lines. That puts them on the timeline, between the agents they connect.
def handoff(sender: str, receiver: str, task: str, context: dict,
permissions: list[str], original_goal: str):
with tracer.start_as_current_span(f"handoff {sender}->{receiver}") as span:
span.set_attributes({
"app.handoff.sender": sender,
"app.handoff.receiver": receiver,
"app.handoff.task_sent": task[:1000],
"app.handoff.context_keys": list(context.keys()),
"app.handoff.context_tokens": count_tokens(context),
"app.handoff.permissions": permissions,
"app.task.original_goal": original_goal[:500],
})
result = call_agent(receiver, {
"task": task, "context": context,
"permissions": permissions, "original_goal": original_goal,
})
# receiver returns its one-sentence restatement (see Part 1)
span.set_attribute("app.handoff.task_understood",
result.get("task_understood", "")[:1000])
return result
Now the comparison from Part 1 (sent vs. understood vs. original goal) can run directly on spans in your trace backend.
Step 4: Model the hierarchy
Multi-agent traces get unreadable fast. A clean, consistent hierarchy keeps them navigable:
task β root: one per user request
βββ invoke_agent planner β agent turn
βββ chat plan β model call
βββ handoff planner->researcher β handoff
β βββ invoke_agent researcher
β βββ chat search_strategy
β βββ execute_tool sql_query β tool call
β βββ execute_tool web_fetch
βββ handoff planner->writer
β βββ invoke_agent writer
β βββ chat draft
βββ handoff writer->reviewer
βββ invoke_agent reviewer
βββ chat review
The pattern: task β agent β (handoff β agent)* β model call β tool call. Pick a shape and stick to it across every agent and team. Consistent span names (invoke_agent X, handoff A->B, execute_tool Y) make queries like "all handoffs into the writer this week" trivial.
Step 5: Tag outcomes per agent
Mark every agent span with its own result, not just the task's. Otherwise every failure gets blamed on whoever acted last, which is usually the writer or the reviewer, and usually wrong.
from opentelemetry.trace import Status, StatusCode
def finish_agent(span, outcome: str, reason: str | None = None):
# outcome: ok | degraded | failed | rejected_input
span.set_attribute("app.agent.outcome", outcome)
if reason:
span.set_attribute("app.agent.outcome_reason", reason)
if outcome in ("failed", "rejected_input"):
span.set_status(Status(StatusCode.ERROR, reason or outcome))
Where do per-agent outcomes come from? Cheap checks you already have: schema validation on each agent's output, the handoff drift score, a tool error, the reviewer's verdict, an eval run on a sample.
Finding the agent that broke the task
When a task fails, read the trace from the root and find the first handoff where the task changed shape. That's usually where the problem started, even if the damage showed up three agents later.
Here's that idea as code, working on spans exported from your backend:
from dataclasses import dataclass
@dataclass
class Suspect:
span_name: str
agent: str
reason: str
def first_bad_hop(spans: list[dict], drift, threshold=0.75) -> Suspect | None:
"""spans: one trace, sorted by start time."""
for s in spans:
a = s["attributes"]
# 1. A handoff where the received task drifted from the goal
if s["name"].startswith("handoff "):
score = drift(a.get("app.handoff.task_understood", ""),
a.get("app.task.original_goal", ""))
if score < threshold:
return Suspect(s["name"], a["app.handoff.receiver"],
f"task changed shape (similarity {score:.2f})")
# 2. An agent that marked its own output as failed or degraded
if a.get("app.agent.outcome") in ("failed", "degraded"):
return Suspect(s["name"], a.get("gen_ai.agent.name", "?"),
a.get("app.agent.outcome_reason", "agent reported failure"))
# 3. A tool error that the agent carried on past
if a.get("gen_ai.operation.name") == "execute_tool" and s["status"] == "ERROR":
return Suspect(s["name"], a.get("gen_ai.tool.name", "?"),
"tool error before task completed")
return None
Run it on the EU-churn example from Part 1:
>>> first_bad_hop(spans, drift)
Suspect(span_name='handoff planner->researcher',
agent='researcher',
reason='task changed shape (similarity 0.71)')
Not the writer. Not the reviewer. The first hop. Run the same function across every failed trace in a week and group by span_name, and you get a ranked list of your weakest handoffs, which is a much better backlog than "the agents are flaky."
Practical notes
- Sample whole tasks, not spans. If you sample, decide at the root and keep or drop the entire task. Half a multi-agent trace is worse than none. Tail-based sampling in the OpenTelemetry Collector can keep 100% of failed or high-risk tasks.
- Truncate, don't drop, large attributes. Store the first N characters of task text and context on the span; put the full payload in logs keyed by span ID.
- Redact at capture. Handoff context often carries customer data. Scrub it before it leaves the process.
- Expect convention churn. OpenTelemetry's GenAI conventions are still in Development. Keep attribute names in one module.
For a broader look at what to collect across single and multi-agent setups, see this complete guide to AI agent observability.
TL;DR
- Propagate context so one task = one trace, across HTTP, queues and MCP servers.
- Give every agent turn and every handoff its own span, tagged with name, role and version.
- Keep a consistent hierarchy: task β agent β handoff β agent β model call β tool call.
- Tag outcomes per agent, then walk the trace for the first hop where the task changed shape.
In Part 3, we'll move from single traces to dashboards: system, role, handoff and verification metrics, the alert that catches a rubber-stamp reviewer, and the security signals for when one compromised agent starts instructing the rest. It all builds on the checklist in this breakdown of multi-agent monitoring metrics and security signals.
How are you propagating context between agents today: HTTP headers, queue messages, or hoping for the best? π§΅
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.