Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 14 min read

Best observability platform for a customer-facing AI support agent (2026)

The best observability platform for a customer-facing AI support agent depends on who built the agent. If you bought it (Intercom Fin, Zendesk AI agents, Sierra, or Decagon), start with that vendor's built-in analytics,

The best observability platform for a customer-facing AI support agent depends on who built the agent. If you bought it (Intercom Fin, Zendesk AI agents, Sierra, or Decagon), start with that vendor's built-in analytics, which already count resolutions and escalations. If you built your own, you need two layers: a tracing tool such as Langfuse, LangSmith, Arize, or Datadog to debug model and tool calls, and a conversation-level tool that judges each whole conversation for resolution, escalation, wrong policy answers, and frustration. Agnost AI is the conversation layer for custom agents and turns recurring failures into pull requests. Dimension Labs fits contact centers running platform bots. Nebuly fits enterprises measuring the ROI of many agents.

Support agents fail differently from coding agents or internal copilots. The model call usually succeeds. The failure is that the customer didn't get what they needed, got a refund promise that breaks policy, or gave up and opened a ticket. Those show up across a conversation, not in one span. This guide, updated for 2026, lists what support teams need to measure, then compares three kinds of tools against that list.

Disclosure: We make Agnost AI. Facts about every other tool come from that vendor's own docs and pricing pages, checked on November 18, 2026. This is a documentation comparison, not a hands-on benchmark. Confirm plan details with each vendor.

What changed since 2025

  • Resolution got stricter. Zendesk introduced resolution tiers on May 18, 2026: Assisted escalation, Contained resolution, and Verified resolution, where an LLM confirms the request was resolved. Only Verified resolutions draw down your allowance. "No follow-up" is no longer treated as proof of success.
  • Support vendors added QA and monitoring. Intercom sells CX Score, Monitors, Custom AI Scorecards, and Incident Detection in a Fin Pro add-on. Decagon's Watchtower reviews interactions against criteria you write in plain language. Sierra's Insights has always-on reviewing and alerting.
  • General tools moved toward sessions, unevenly. LangSmith and Arize AX judge whole threads or sessions. Langfuse deprecated trace-level evaluators, which stopped producing results on Langfuse Cloud after November 16, 2026. Its LLM judges now target observations and experiments.

What a support team needs to see

Need What it means Needs the whole conversation?
Resolution The customer's request was actually completed Yes
Containment No human was needed, and the customer didn't give up and call Yes, plus a repeat-contact check
Escalation When the agent handed off, why, and whether the handoff carried context Yes
Wrong policy answers Refund, cancellation, eligibility, or pricing answers that break your rules Yes, usually
Frustration Repeated questions, "talk to a human", caps, giving up Yes
CSAT correlation Which intents and failure types drag survey scores down Yes, joined to survey data
Tool-call errors Order lookup, refund API, or CRM calls that failed or returned nothing No, traces show these
PII Card numbers, addresses, and health data kept out of logs Handled at export

Containment is the trap. Dimension Labs puts it bluntly: "A conversation that ends without a handoff counts as contained, including the ones where the customer quit and called instead." Measure containment next to repeat contact and frustration, or it rewards an agent that wears customers down. Our post on support conversations the agent resolved but the customer hated shows what that looks like in real transcripts.

Quick comparison

Tool Type Best for Judges whole conversations Resolution and escalation built in Self-host Entry price
Langfuse LLM observability Open-source tracing and evals No (observation-level judges) No Free Free; Core $29/mo
LangSmith LLM observability LangChain and LangGraph agents Thread evaluators, 10 min idle default No Enterprise Free 1 seat; Plus $39/seat/mo
Arize AX LLM observability Managed span, trace, session evals Session-level evals, with sampling No Enterprise Free; Pro $50/mo
Datadog LLM observability in APM Teams already on Datadog Goal Completeness on sessions marked complete No No Free 40K LLM spans; Pro from $160/mo
Agnost AI Conversation-level monitoring Custom-built support agents Every conversation, 15 min idle, no sampling Via rules, intents, and events Enterprise (VPC) Free; Starter $49/mo
Dimension Labs Conversation analytics Contact centers on platform bots Yes Containment and escalation fields Not stated Explore $299/mo
Nebuly Conversation analytics and ROI Enterprise ROI across many agents Yes Resolution rate Yes (cloud or on-prem) Not published
Intercom Fin Built into the agent Teams running Fin Yes Yes No $0.99 per outcome; Pro add-on from $100/mo
Zendesk AI agents Built into the agent Teams running Zendesk AI agents Yes (LLM-verified resolutions) Yes No Not covered here
Sierra Built into the agent Teams running Sierra Yes CSAT and case resolution Not stated Not published
Decagon Built into the agent Teams running Decagon Yes Filters by resolution status and CSAT Not stated Not published

General LLM observability

These tools trace every model call, tool call, and retrieval step. They are the right place to debug why the agent did something. None of them reports resolution, containment, or escalation out of the box. You build those as evaluators or custom scores.

Langfuse

What it is. An open-source platform for tracing, prompt management, evaluations, and datasets, built on OpenTelemetry and free to self-host.

For support agents. Good for seeing every step behind a bad answer, and the SDK masking hooks rewrite span attributes before export, which helps keep PII out. LLM-as-a-judge evaluators target observations and experiments, with sampling. Sessions are not an evaluation target, so resolution and frustration across a conversation are things you assemble yourself.

Pricing. Self-hosting is free. Cloud Hobby is free for 50,000 units and 2 users. Core is $29 a month and Pro $199 a month, with unlimited users.

LangSmith

What it is. LangChain's tracing, evaluation, and monitoring platform. It also accepts OpenTelemetry.

For support agents. Thread evaluators judge a whole conversation after it has been idle (10 minutes by default, 2 minutes minimum). That makes "was the customer's request resolved?" a natural evaluator. Limits: 10 multi-turn evaluators per workspace, at most 500 threads per run, and only runs under a week old.

Pricing. Developer is free for 1 seat and 5,000 base traces a month. Plus is $39 per seat per month with 10,000 base traces. Base traces are kept 14 days. Self-hosting is on Enterprise.

Arize AX

What it is. Arize's managed platform for tracing and online evals. Phoenix is its free self-hosted sibling.

For support agents. Online evals run at span, trace, or session level, so you can write a session judge for resolution or policy compliance. Arize's docs recommend sampling 10-50% for high-volume apps, which means some failed conversations go unjudged unless you pay for 100%.

Pricing. AX Free covers 25,000 spans and 1 GB a month. AX Pro is $50 a month for 50,000 spans and 10 GB. Users and evals are unlimited. Self-hosted AX is Enterprise.

Datadog Agent Observability

What it is. Datadog's LLM Observability, now sold as Agent Observability, with traces linked to Datadog APM.

For support agents. Sensitive Data Scanner is built in at no extra cost, which is the strongest PII story among the general tools. Goal Completeness checks whether the agent resolved the user's intent, but it "runs only on sessions marked as completed." There is no dedicated frustration evaluator. Sentiment runs on input and output, not across a session.

Pricing. Free covers 40,000 LLM spans a month with 15-day retention. Pro starts at $160 a month on an annual commitment for 100,000 LLM spans. More in our Datadog LLM Observability alternatives guide.

Conversation-focused tools

These read whole conversations and report what customers experienced. They don't replace a tracing tool for debugging.

Agnost AI

What it is. Agent monitoring for AI agents in production: chat agents, voice agents (through transcripts), copilots, and MCP servers. It is OpenTelemetry-native and works with the OpenAI Agents SDK, Vercel AI SDK, LangChain and LangGraph, Mastra, Pydantic AI, Google ADK, and Agno, with Python and TypeScript SDKs.

For support agents. Agnost groups traffic by user and session and judges whole conversations, not single traces. It detects failures, user frustration, intents (auto-generated and custom), tool-call errors, loops, and silent wrong answers. Policy checks are plain-language rules, such as "Never promise a refund outside the 30-day window." Once a conversation has been idle 15 minutes, an LLM judges it in a batch and marks each assistant message pass or fail against every rule, with no sampling. Escalation can be tracked as a custom intent ("asks for a human") or an event your app sends. Alerts go to Slack or email. When an alert fires, Auto Improve opens a pull request with a fix tested against past runs, showing the replayed conversations, for a person to review, and re-checks the alert after merge.

Limits. Not open source. No cost tracking. No offline datasets, experiments, or CI gating. Verdicts are pass or fail, not scores. Judging runs in batches after 15 minutes idle, not in real time, so keep checks that must block a reply in your app. It has no CSAT survey of its own: send survey results as events to join them to conversations. Security review is available on request.

Pricing. Free covers 1,000 events a month with 7-day retention. Starter is $49 a month (10,000 events). Pro is $499 a month (1,000,000 events). Enterprise is custom and includes self-hosted VPC deployment and audit logs.

Dimension Labs

What it is. An analytics platform for customer conversations, reviews, surveys, and calls. It writes structured fields to each conversation, such as Bot Containment, Human-Escalation Attempt, Effort Level, and AI Failure Mode (for example repeated answers).

For support agents. Built for CX and contact-center teams. It connects to Zendesk, Intercom, Genesys, Amazon Connect, Twilio, Kore.ai, Cognigy, Rasa, Amazon Lex, and more. Its containment metric explicitly separates customers who were helped from customers who quit.

Limits. Its AI agent analytics page doesn't mention OpenTelemetry or trace ingestion, so it suits platform bots better than custom-built agents. No free plan.

Pricing. Explore is $299 a month for up to 1,000 conversations, Discover $999, Prove $2,999, Scale $6,999, and Enterprise from $150,000 a year. Paid plans have a 14-day trial.

Nebuly

What it is. "The ROI platform for AI agents." It maps each conversation with employee or customer agents to the task behind it and tracks resolution rate, frustration, unmet demand, and churn or revenue risk.

For support agents. Fits enterprises that want to show what their agents are worth across teams. It can run as SaaS, in your AWS, Azure, or GCP account, or on-premises, and says it removes names and emails before analysis. It lists ISO 27001, ISO 42001, and SOC 2.

Limits. Its homepage names no frameworks or OpenTelemetry support, so ask how it connects to a custom agent. It is aimed at ROI and adoption more than debugging.

Pricing. Not published.

Built-in analytics from support-agent vendors

If you run one of these agents, its analytics are your first stop: they already know what a resolution and an escalation are on that platform. If you built your own agent, these don't apply.

Intercom Fin

Intercom reports Fin in Analyze dashboards with a CX Score and KPIs, custom reports, an inbox view of Fin conversations, and an Optimize dashboard of unresolved conversations with suggested improvements. A confirmed resolution is when the customer says the answer worked. An assumed resolution is when the customer "exits the conversation without requesting further assistance" after Fin's last answer. Fin charges $0.99 per outcome, once per conversation. The Pro add-on, from $100 a month, adds CX Score, AI Topics, Trends, AI Recommendations, Monitors, Custom AI Scorecards, and Incident Detection.

Watch for. Assumed resolutions count customers who left. Check a sample for people who gave up.

Zendesk AI agents

The AI agents dashboard shows total conversations, the percentage with an automated resolution, and the percentage escalated to a human. Since May 18, 2026, resolutions are split into three tiers. Assisted escalation: the AI helped before a human finished. Contained resolution: the AI handled it to completion "without the customer requesting further assistance." Verified resolution: an LLM also confirmed the request was satisfactorily resolved. Only Verified resolutions draw down your allowance.

Watch for. The gap between Contained and Verified is a ready-made silent-failure number. Track it.

Sierra

Sierra's Insights reports on CSAT, case resolution, and automatically tagged topics. Its observability features include always-on reviewing and alerting for conversations that need attention, auditing of the reasoning behind each action, and alerts for abuse attempts or performance drops. Explorer answers natural-language questions across conversations. Experimentation covers tests such as hand-off rules.

Watch for. Sierra's pages don't define how case resolution is calculated. Ask.

Decagon

Decagon's Watchtower "reviews every interaction against your custom criteria," written in natural language, for both AI and human agents. It flags issues such as frustration or privacy-policy violations, filters by CSAT, channel, user metadata, and resolution status, and can review past conversations or monitor continuously.

Watch for. The Watchtower page doesn't describe how alerts are delivered or how resolution is defined. Ask.

How to choose

If your situation is… Pick
You run Intercom Fin, Zendesk AI agents, Sierra, or Decagon The vendor's built-in analytics first. Add a separate tool only for gaps you can name
You built your own support agent and need to debug steps Langfuse (open source), LangSmith (LangChain), Arize AX, or Datadog (if you're on Datadog)
You built your own agent and need resolution, escalation, policy, and frustration across every conversation Agnost AI, next to your tracing tool
You want failures turned into reviewed pull requests Agnost AI
You run a contact center on platform bots such as Cognigy, Kore.ai, Lex, or Genesys Dimension Labs
You need to prove ROI across many employee and customer agents, on-prem Nebuly
PII is your first constraint Datadog (Sensitive Data Scanner included), Langfuse masking or self-hosting, or any enterprise self-hosted option

Three checks before you buy:

  1. Ask how the tool defines a resolution. "No follow-up" and "verified by a judge" give very different numbers. Ask for the definition in writing.
  2. Pilot on a week of real conversations. Have a support lead label 50 conversations as resolved, escalated well, escalated badly, or failed, then compare each tool's verdicts. The agent monitoring pilot checklist has the steps.
  3. Score escalations, not just their count. A high escalation rate with good handoffs can be healthier than a low one where customers give up. See when an AI agent should escalate to a human and how to measure handoff quality.

Related reading

Frequently asked questions

What is the best observability platform for a customer-facing AI support agent?

It depends on who built the agent. If you bought one (Intercom Fin, Zendesk AI agents, Sierra, Decagon), start with the vendor's own analytics, which already report resolutions and escalations. If you built your own, pair a tracing tool (Langfuse, LangSmith, Arize, or Datadog) for debugging with a conversation-level tool that judges whole conversations for resolution, escalation, wrong policy answers, and frustration. Agnost AI does the conversation layer for custom agents and opens pull requests for fixes. Dimension Labs fits contact centers on platform bots, and Nebuly fits enterprises measuring ROI across many agents.

What metrics should I monitor for an AI customer support agent?

Resolution rate (verified, not just 'no follow-up'), containment, escalation rate and escalation reasons, wrong or off-policy answers, user frustration, repeat contact within a few days, CSAT on AI-handled conversations, tool-call errors, and PII exposure. Resolution and frustration need the whole conversation to judge, so single-trace evals miss them.

How do I measure AI support agent resolution rate accurately?

Don't count every conversation that ended without a handoff. Intercom counts an assumed resolution when the customer leaves without asking for more help after Fin's last answer, and Zendesk now separates Contained resolutions from Verified resolutions that an LLM has confirmed. For your own agent, judge each conversation with a rule like 'the customer's request was completed and they did not have to repeat themselves', and subtract conversations where the same user comes back about the same issue within 72 hours.

Is Langfuse or LangSmith enough to monitor a support chatbot?

For debugging, yes: both trace every model and tool call, group traces into sessions or threads, and run LLM-as-a-judge evals. For support metrics, you will build most of it yourself. Langfuse's LLM-as-a-judge targets observations and experiments, not sessions. LangSmith's thread evaluators judge whole conversations but are capped at 10 per workspace and 500 threads per run. Neither reports resolution, containment, or escalation out of the box.

Does Intercom Fin or Zendesk have built-in AI agent analytics?

Yes. Intercom reports Fin resolutions, a CX Score, AI Topics, Monitors, and Custom AI Scorecards, with the deeper analysis in a Pro add-on from $100 a month. Zendesk's AI agents dashboard shows total conversations, automated resolution percentage, and escalation percentage, and since May 18, 2026 it splits resolutions into Assisted escalation, Contained, and Verified tiers. These only cover agents running on that vendor's platform.

How do I detect when an AI support agent gives a wrong policy answer?

Write the policy as a plain rule and judge every conversation against it, for example 'Never promise a refund outside the 30-day window' or 'State the cancellation fee before confirming a cancellation.' Trace data won't flag these because the call succeeded. Tools that judge whole conversations (Agnost AI rules, LangSmith thread evaluators, Arize AX session evals, Decagon Watchtower criteria) can catch them. Sample a few flagged and unflagged conversations each week to check the judge.

How do I keep PII out of AI support agent traces?

Redact before data leaves your app. Langfuse's SDKs have masking hooks that rewrite span attributes before export. Datadog includes Sensitive Data Scanner in Agent Observability at no extra cost. If data must stay in your cloud, Langfuse and Arize Phoenix are free to self-host, and LangSmith, Arize AX, and Agnost AI offer self-hosted or VPC deployment on enterprise plans. Nebuly says it strips names and emails before analysis.

Originally published on agnost.ai.

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.