LLM Observability: What to Track in Production (Tokens, Cost, Latency, TTFT and Agent Traces)

LLM observability means every model call in production can be answered after the fact: what was sent, what came back, which provider and model served it, how long it took to the first token and in total, what it cost, who triggered it, and whether the answer was any good. In practice that comes down to one record per call with a fixed set of fields, one trace per agent run with a span for each model call and tool call, low-cardinality metrics for dashboards and alerts, and a sample of production traffic scored by evaluators. The fastest way to get there across many services is to route every call through one capture point, usually a gateway, and add app-side spans only for what that layer cannot see.

This post lists what to capture, how to measure latency the way users feel it, how to structure agent traces with the OpenTelemetry GenAI conventions, how to handle PII and sampling, and which dashboards and alerts to build.

How LLM calls differ from ordinary API calls

Four differences drive everything below:

1. The payload is the debugging signal, and it is often sensitive. To explain a bad answer you need the prompt and output, which is exactly where PII lives.
2. Cost is per request and variable. Two calls to the same endpoint can differ widely in cost depending on input length, output length, and which model served them.
3. Latency has two meanings for streams. Time to first token (TTFT) drives perceived speed; total latency drives agent step time and timeouts.
4. HTTP 200 says nothing about quality. A response cut off at max_tokens, blocked by a content filter, or returning invalid JSON is still a 200.

Put one capture point in front of every call

Hand-instrumenting each SDK call in each service drifts: one service forgets streaming usage, another swallows errors, a third logs raw prompts. Options, by coverage per unit of effort:

1. A gateway in front of all providers. It is language-agnostic and covers every service by changing a base URL.
2. One shared client wrapper per language that every service must import, enforced by lint or code review.
3. Auto-instrumentation libraries per SDK. These work, but versions and attribute names drift across languages.

Start by finding every call site:

grep -rnE "chat\.completions\.create|responses\.create|messages\.create|generate_content|generateContent|streamText|generateText|ChatOpenAI|ChatAnthropic|litellm\.completion" \
  --include=*.{py,ts,tsx,js,mjs,go} --exclude-dir={node_modules,.git,dist,build,.venv} .

What to capture for every LLM call

Emit one record per call, on success and on failure, with the same fields every time:

Compute cost at capture time from the price table in effect, including cached-input discounts. Recomputing later from tokens loses which price applied.

Measure latency the way users feel it

Track three latency numbers separately:

• TTFT (request sent to first content chunk) drives perceived speed in chat UIs.
• Total latency drives agent step time and timeouts.
• Inter-token latency (or output tokens per second) shows provider throughput degrading.

A p95 over mixed streaming and non-streaming traffic is meaningless, so split by stream flag. When comparing models or prompts, normalize total latency by output tokens; a longer answer is not a regression.

For OpenAI-compatible streams, request usage in the final chunk, or tokens and cost will be missing for every streamed call:

stream = client.chat.completions.create(
    model=MODEL, messages=messages, stream=True,
    stream_options={"include_usage": True},
)

const stream = await client.chat.completions.create({
  model: MODEL, messages, stream: true,
  stream_options: { include_usage: true },
});

Make failures and fallbacks loud

• Classify errors by whose fault they are: caller (bad request, unknown model), policy (your own rate limit or budget), provider (429, 5xx, timeout, auth), or internal. A provider 429 means "buy capacity or spread load"; your own 429 means "your limits are too tight." Alerting on raw status codes pages on-call for client typos.
• Record every retry and fallback with the attempt count and which provider and model finally served the request. A fallback that succeeds still changes quality, cost, and latency.
• Count silent failures as errors: finish_reason of length or content_filter, empty outputs, and structured output that fails to parse.
• Never except: pass around an LLM call. Return a typed failure and record it.

Attribute tokens and cost, with safe cardinality

Attach user or tenant, feature, environment, and prompt version to every call; cost questions are always "which feature, which customer, which prompt change." Then put each attribute in the right sink:

If a label can take more than a few hundred values, it does not belong on a metric. One label per user ID will take down a Prometheus server.

Trace agents as one run

Use one trace per agent run or user turn. The root span is the run; child spans are each model call (chat <model>), each tool execution (execute_tool <name>), retrieval, and sub-agent invocation (invoke_agent <name>). Record the step count and the loop's exit reason (final_answer, max_steps, error) on the root span, and alert on runs with abnormal span counts, which is how runaway loops show up.

Use the OpenTelemetry GenAI semantic conventions so any backend can read the spans. They are still marked "development," and names have changed between releases (gen_ai.system became gen_ai.provider.name, and gen_ai.usage.prompt_tokens became gen_ai.usage.input_tokens), so pin the version you emit and check attribute names on a real span before writing queries. Core attributes include gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.response.finish_reasons, gen_ai.conversation.id, and gen_ai.tool.name. Put custom fields such as cost and feature in your own namespace (app.cost_usd, app.feature).

App-side spans cover the agent loop and tools. If model calls go through a gateway that exports OpenTelemetry, propagate traceparent instead of creating a duplicate span per call, so the gateway's span nests under your step span:

from opentelemetry import trace
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor

HTTPXClientInstrumentor().instrument()   # OpenAI and Anthropic Python SDKs use httpx: traceparent is injected
tracer = trace.get_tracer("support-agent")

def run_agent(session_id: str, user_hash: str, messages: list):
    with tracer.start_as_current_span("invoke_agent support-agent") as run:
        run.set_attributes({
            "gen_ai.operation.name": "invoke_agent",
            "gen_ai.agent.name": "support-agent",
            "gen_ai.conversation.id": session_id,
            "app.user_hash": user_hash,
            "app.feature": "support-chat",
        })
        for step in range(1, MAX_STEPS + 1):
            resp = call_model(messages)                      # gateway span joins this trace
            calls = resp.choices[0].message.tool_calls or []
            if not calls:
                run.set_attributes({"app.exit_reason": "final_answer", "app.step_count": step})
                return resp
            for call in calls:
                with tracer.start_as_current_span(f"execute_tool {call.function.name}") as t:
                    t.set_attributes({
                        "gen_ai.operation.name": "execute_tool",
                        "gen_ai.tool.name": call.function.name,
                        "gen_ai.tool.call.id": call.id,
                    })
                    messages.append(tool_message(call.id, TOOLS<a href="call.function.arguments">call.function.name</a>))
        run.set_attributes({"app.exit_reason": "max_steps", "app.step_count": MAX_STEPS})

Treat content as sensitive and sample deliberately

Metadata (model, tokens, cost, latency, status, attribution) is safe to keep at 100%. Content is where PII lives:

• Redact before content leaves the process or gateway, not in the backend after ingestion.
• Allow per-route or per-request opt-out of content capture for regulated flows such as health, payments, and HR.
• Audit every export destination. Turning content off in the log store does not turn it off in your trace exporter.
• Keep content for days to weeks and metadata longer.

Sampling policy:

• Metrics: 100%, always.
• Metadata records: 100% if storage allows; they power cost attribution and feedback lookup.
• Content and full traces: sample, but keep 100% of errors, slow requests above p95, fallbacks, and anything with negative user feedback. Tail-based sampling in an OpenTelemetry Collector can do this; head-based sampling cannot.

Link feedback and score production traffic

Return a request ID to the client on every response, store it with the message in your UI state, and write feedback events (thumbs, edits, regenerate, copy, abandonment) with that ID. Join feedback to records by ID, never by timestamp. Negative feedback should force-retain the full trace and add it to a review queue.

Offline evals catch regressions before release; prompt and model drift happens after. Sample production records daily, stratified by feature and model with errors and negative feedback oversampled, run automated evaluators (schema checks, groundedness against retrieved context, task-specific LLM-as-judge rubrics), and track scores per feature and prompt version over time. Promote failing samples into your offline regression dataset.

Dashboards and alerts

Build one dashboard. The top row shows health (request rate, error rate excluding caller errors, p95 latency, p95 TTFT, and cost per hour), followed by rows for errors, latency, spend, resilience (fallbacks, retries, and key health), and quality from your eval store. These queries use Bifrost's Prometheus metrics; map the names if you emit OpenTelemetry metrics from app code instead:

# Error rate excluding caller mistakes and your own policy refusals (the number to alert on)
sum(rate(bifrost_error_requests_total{error_type!~"(caller|policy)_.*"}[5m]))
  / sum(rate(bifrost_upstream_requests_total[5m]))

# p95 time to first token by model
histogram_quantile(0.95,
  sum by (le, provider, model) (rate(bifrost_stream_first_token_latency_seconds_bucket[5m])))

# Cost per hour by feature (requires a feature label and x-bf-dim-feature headers)
sum by (feature) (increase(bifrost_cost_total[1h]))

# Average input tokens per request (prompt bloat, runaway context)
sum by (model) (rate(bifrost_input_tokens_total[15m]))
  / sum by (model) (rate(bifrost_upstream_requests_total[15m]))

# Share of successful requests served by a fallback
sum(rate(bifrost_success_requests_total{fallback_index!="0"}[5m]))
  / sum(rate(bifrost_success_requests_total[5m]))

# Cost burn versus the same hour last week
sum(increase(bifrost_cost_total[1h])) / sum(increase(bifrost_cost_total[1h] offset 1w))

Alert on the error rate excluding caller and policy errors, p95 TTFT against your baseline, cost burn against last week, fallback share, and agent runs exiting at max_steps. Set thresholds from a week of your own baseline rather than from generic numbers.

Gateway-level capture with Bifrost

Bifrost is an open-source (Apache 2.0) AI gateway. Pointing existing SDKs at it gives every service, in every language, the same per-request record: request logs with content, tokens, cost, and latency; Prometheus metrics at /metrics; OpenTelemetry traces using GenAI conventions; per-request attribution through headers; and correlation headers on every response.

npx -y @maximhq/bifrost                       # or:
docker run -p 8080:8080 -v $(pwd)/data:/app/data maximhq/bifrost
# Logs: http://localhost:8080/logs   Metrics: http://localhost:8080/metrics

import uuid
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/openai",
    api_key="sk-bf-...",
    default_headers={"x-bf-dim-environment": "production", "x-bf-dim-service": "support-api"},
)

request_id = str(uuid.uuid4())
raw = client.chat.completions.with_raw_response.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Summarize this ticket..."}],
    extra_headers={
        "x-request-id": request_id,              # becomes the Bifrost log ID
        "x-bf-dim-feature": "ticket-summary",    # flows to logs, OTel spans, Prometheus
        "x-bf-session-id": "conv-8812",          # groups turns of one conversation
    },
)
print(raw.headers.get("x-bifrost-routing-info-provider"),
      raw.headers.get("x-bifrost-routing-info-is-fallback"))

Verify it end to end:

curl -si http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" -H "x-request-id: obs-check-1" -H "x-bf-dim-feature: smoke-test" \
  -d '{"model":"openai/gpt-4o-mini","messages":[{"role":"user","content":"ping"}]}' | grep -i '^x-bifrost-'
curl -s "http://localhost:8080/api/logs?request_id=obs-check-1" | jq '.logs[0]'
curl -s http://localhost:8080/metrics | grep -E '^bifrost_(upstream_requests_total|cost_total|input_tokens_total)' | head

Install the llm-observability-stack skill

The audit, per-call record, trace structure, and dashboards in this post are packaged as an open-source agent skill, so your coding agent can instrument your app.

• GitHub: github.com/build-with-ai-hq/llm-observability-stack
• skills.sh: skills.sh/build-with-ai-hq/llm-observability-stack

npx skills add build-with-ai-hq/llm-observability-stack

Ask your agent "we have no visibility into our LLM calls" or "add OpenTelemetry to our LLM calls." When the skill loads, the agent:

1. Audits the codebase for LLM call sites, existing instrumentation, prompts in logs, streams missing usage, swallowed errors, and missing user, session, or feature attribution.
2. Applies fixes in order of impact: a single capture point, a fixed per-call record, TTFT versus total latency, error classification and fallback visibility, cost attribution with safe cardinality, agent traces with OpenTelemetry GenAI conventions, PII redaction, sampling, feedback linkage, and production evals.
3. Sets up gateway-level capture with Bifrost where it removes per-SDK instrumentation.
4. Builds dashboards and alert rules with PromQL and verifies the setup end to end.

Related reading

• How to Track Claude Code Usage Across Your Team (Per Developer, Per Project)
• Semantic Caching for LLMs: How to Stop Paying for the Same Answer Twice

FAQ

What is the difference between LLM observability and LLM monitoring?

Monitoring tracks known health signals (error rate, latency, cost) and alerts on them. Observability means you can answer new questions about any individual call or agent run after the fact, which needs per-call records, traces, and content (redacted) as well as metrics.

What metrics should I track for LLM apps?

Request rate, error rate by fault type, p95 total latency, p95 time to first token, input and output tokens, cost per hour by feature and model, fallback share, cache hit rate, and agent step counts. Add eval scores and negative-feedback rate from your quality pipeline.

What is time to first token?

The time from sending a streaming request to receiving the first content chunk. It drives how fast a chat UI feels, and it regresses independently of total latency.

Should I log prompts and completions?

Only with redaction, restricted access, and a short retention period. Keep metadata at 100% and sample content, always keeping errors, slow requests, and negative-feedback cases.

Do I need OpenTelemetry for LLM observability?

Not strictly, but the GenAI semantic conventions give you span and attribute names that most tracing backends understand, which makes it easier to switch tools and join LLM spans with the rest of your traces.