LLM Agent Observability: Traces and Sessions, 10 Problems Reproduced

Observability for an LLM agent starts with two groupings of its model calls: a trace, which is one run of the agent drawn as a timeline, and a session, which is one conversation with its total cost. With three request headers on each call, an LLM gateway (a proxy that sits between your code and the model providers and logs every request) builds both from its request log, with no tracing SDK. We used that setup on a small support agent and on Claude Code, and reproduced ten common agent problems: three that make an agent slow, four that make it expensive and three that break it.

TL;DR

• Three request headers (X-Trace-Id, X-Span-Id, X-Session-Id) turn the Synthorai gateway's request log into traces and sessions with no tracing SDK.
• Sending three independent sub-agent checks together cut their median time from 3.7 s to 1.8 s at the same cost.
• Moving a timestamp off a prompt's first line on GPT-6 Luna raised the cache hit rate from 0% to 73%, from an empty cache.
• One Claude Code run cost $0.1728, and the first call of each of its three agents was 72% of it.

This post is the overview of a seven-part guide series. Each section below summarizes one guide with its numbers and links to it for the code and screenshots. All runs are from October 10 and 11, 2026. The latency figures are medians of five runs; every other figure is from a single run. The code is at github.com/synthorai-io/use-cases.

What are traces, spans and sessions in an LLM agent?

A trace is everything an agent does to handle one user message or one task, a span is one step inside that trace, and a session is the whole conversation, made of many traces. In our support agent, one customer conversation of two turns is one session with two traces, and each trace holds five or six model calls. Some of those calls come from a sub-agent: a second agent that the main agent starts for part of the task, with its own prompt and often a cheaper model.

The ids are strings you choose, up to 128 printable ASCII characters each. Labels are a flat JSON object of strings, at most 10 per request, each value up to 128 bytes. The gateway returns the ids on the response and does not forward any of these headers to the model provider. With the OpenAI Python SDK, per-request headers go in extra_headers:

resp = client.chat.completions.create(
    model="gpt-6-luna",
    messages=messages,
    extra_headers={
        "X-Session-Id": session_id,   # one per conversation
        "X-Trace-Id": trace_id,       # one per user message
        "X-Span-Id": "plan",          # one per call
        "X-Synthorai-Metadata": '{"feature":"support-agent","_user":"u_1002"}',
    },
)

If your agent already sends trace ids through another tool, you may not need to add anything. For each of the three ids a request does not carry, the gateway reads the equivalent header from OpenTelemetry (the open standard for tracing, whose traceparent header carries a trace id), LiteLLM, Langfuse, Open WebUI, Claude Code or the OpenAI Codex CLI. Trace an LLM agent with 3 HTTP headers builds the demo agent step by step: a plan call on GPT-6 Luna, two local tools, three parallel checks on DeepSeek V4.1 Flash, and an answer call. Its demo run made 21 model calls for $0.0017.

Which view answers which question?

Each question in the series is answered by one place in the console or by one way of reading the log from outside it. The console shows each trace as a waterfall, one bar per model call on a shared time axis, so calls that ran in parallel overlap and calls that waited for each other form a staircase.

Every trace page also has a Compare button, which puts two traces on one page with each call paired by name and both waterfalls on one time axis. Each fix in the series was checked that way, with the run before the fix against the run after it.

Why is an LLM agent slow?

Start with the split at the top of the trace. It divides wall time (what the user waited, from the first request to the last response) into time in the gateway, when at least one model request was open, and time in your app, when only your code was running. In the demo trace, 8.5 s of wall time split into 7.0 s in the gateway and 1.6 s in the app, so the model calls were the place to look. The latency guide reproduced three causes and ran each five times:

In the slow-code case the model calls took about as long in both runs and the time in the app is what changed: in the pair of runs shown in the guide, the your app row went from 4.6 s to 1.3 s. A model change cannot shorten that part, so the fix starts in your own profiler or OpenTelemetry spans. The third case ran on GPT-6 Luna. reasoning_effort is the request parameter that sets how much the model reasons before it answers, and with it set to none, output tokens for the five answers fell from 1,150 to 190. Why is my LLM agent slow? has the code and the side-by-side traces for all three.

Why is an LLM agent expensive?

Start in the Traces list sorted by cost, then open the most expensive session and read its cache hit and input columns turn by turn. The cache hit rate is the share of input tokens read from the provider's prompt cache, which bills a prompt prefix it has seen before at a fraction of the input price. The cost guide reproduced four causes. The amounts are fractions of a cent because the demo is small, so the ratios are what carry over:

The first row ran on GPT-6 Luna, starting from an empty cache. The guide adds a second way to lose the cache: an instruction appended to each user message when it is sent but not when it is saved, so every turn rebuilds a prompt the previous turn never sent. Saving the message as sent raised that conversation's hit rate from 0% to 76%. In the history case, three times the input tokens added only 21% to input cost because 71% of them were read from cache. In the last row, three sub-agent checks on GPT-6.1 Sol were 81% of the session's cost, and the same checks on DeepSeek V4.1 Flash cost a tenth as much. Why is my LLM agent expensive? walks through each case.

How do you debug an LLM agent from a user complaint?

You find the user's runs by a label, then read the trace for the one call that went wrong, which often returned status 200. With a _user label on every request, typing metadata._user = u_1002 in the Traces list search returns that user's sessions and traces. The debugging guide then reproduced three failures:

• A 404 with a retry. A call to a model id that does not exist failed in 95 ms at $0. Because the retry kept the trace id and took a new span name (policy-check.retry), the failure and its retry sit next to each other in the waterfall.
• Replies cut off at status 200. With max_tokens=40 and reasoning left on, three calls in a row to DeepSeek V4.1 Flash returned status 200 with exactly 40 output tokens and an empty reply, because the reasoning used up the cap. The trace shows the same span three times, with the output count equal to the cap. Raising the cap to 2,000 and turning reasoning off gave one call that answered in 49 tokens. Raising the cap alone also worked, but that call reasoned for 1,876 tokens and took 22 s.
• A 400 on the second turn. A saved history that had lost a tool call made GPT-6 Luna reject the second turn in 0.5 s at no charge, while the first turn had worked.

The second failure is why every call should log finish_reason, the response field that says why the model stopped. A value of length means the reply hit the max_tokens cap, and it should count as a failure even with status 200. Debug an LLM agent lists the five fields every agent should log, repeated near the end of this post.

Can you trace a coding agent you did not write?

Claude Code can be traced without changing it, because it already sends a session id and an agent id with each request. Pointing it at the gateway takes two environment variables, and a third adds the type of each sub-agent:

export ANTHROPIC_BASE_URL="https://synthorai.io"   # no /v1: Claude Code adds /v1/messages itself
export ANTHROPIC_API_KEY="<your gateway key>"
export CLAUDE_CODE_GATEWAY_HINT_HEADERS=1          # optional: names sub-agent types such as Explore

One Claude Code run on Claude Sonnet 5.5, with two sub-agents, fixing one failing test, made 11 requests for $0.1728. The session page split that by agent: the two sub-agents were 39% of the cost, cache writes were 77% of it, and the first call of each of the three agents made up 72%. The same run repeated half a minute later, with the cache warm, cost $0.0812. Claude Code cost by sub-agent also splits a team's spend by developer and repository.

Can a model read the traces for you?

A model can read them through MCP (the Model Context Protocol, the standard way to give a model tools), because the gateway serves its request log as an MCP server. Which way to read the log depends on who is asking:

In the chat case, Claude Fable 5.1 connected by OAuth, a browser sign-in that replaces pasting an API key, and was granted read access to the request log only. That gave it four read-only tools and a key that expires in 7 days. The tools return cost, tokens and status, and no prompt or reply text. In the program case, GPT-6 Luna found the cache, wrong-model and retry-loop sessions among 26 in one tool call for $0.0015, and Claude Sonnet 5.5 found the same sessions in three tool calls for $0.0631, after looking prices up. Ask Claude about your LLM logs and Give your agent log tools cover the two setups.

What should every agent log?

Five fields cover every problem in the series:

• A trace id per run and a session id per conversation, sent as headers and written to your own logs too.
• A span name per call that names the step and the attempt, such as plan-2 or policy-check.retry.
• Labels for the user, the feature and the release, so a complaint or a bill can be narrowed to the runs behind it.
• The x-request-id response header of every call, which looks up the billed cost of that call later.
• finish_reason and the output token count of every call, with length treated as a failure.

Set a client timeout and a max_tokens value on every call as well, both generous. A cap sized for the visible reply is what caused the empty 200s above, and a reasoning model can take minutes on one long call.

What does gateway-side tracing not cover?

Tracing from the request log covers model calls and nothing inside your own code. The gateway infers the time your code ran from the gaps between requests, as the slow-code case showed. Spans for your tools, retrieval and queues need an OpenTelemetry SDK, and the two views can share a trace id because the gateway reads traceparent when X-Trace-Id is absent. The traces also leave out prompt and response bodies, and they report cost after the fact: a live budget means pricing the usage of each response in your own agent loop as it arrives.

FAQ

What is LLM agent observability?

LLM agent observability is the ability to see what an agent did on each run: which model calls it made, in what order, how long each took, what each cost and which failed. In practice that means grouping calls into traces (one run) and sessions (one conversation), with labels for the user and the feature.

Related reading

• Trace an LLM agent with 3 HTTP headers: the demo agent and the headers, step by step.
• Request headers reference: every header in this post, with the aliases read from other tools.
• Traces and sessions in the console: the list, the waterfall and the session page.
• LLM token usage anatomy: what the token counts in each record mean, per provider.
• Prompt caching, the complete guide: why a prompt prefix hits or misses the cache.