Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 10 min read

Which AI agent observability platform should a startup use? (2026)

A startup should pick AI agent observability by stage. While prototyping, use a free tracing tier: Langfuse Cloud Hobby is the most generous, and LangSmith Developer is the natural pick on LangChain. Once real customers

A startup should pick AI agent observability by stage. While prototyping, use a free tracing tier: Langfuse Cloud Hobby is the most generous, and LangSmith Developer is the natural pick on LangChain. Once real customers use the agent, keep tracing and add a conversation-level tool such as Agnost AI to see where users fail, repeat themselves, or give up. As you scale, add release testing with Braintrust or Langfuse datasets, and only self-host when a customer requires it. Whatever you pick, instrument with OpenTelemetry from day one so switching vendors is an exporter change, not a rewrite.

Most startup advice on this compares feature lists. Feature lists matter less than three questions: what you can afford to run, what you'll actually look at each week, and what breaks when you get your first enterprise customer. This guide, updated for 2026, answers those by stage.

Disclosure: We make Agnost AI. Facts about every other tool come from that vendor's own docs and pricing pages, checked on November 18, 2026. This is a documentation comparison, not a hands-on benchmark. Confirm plan details with each vendor.

What changed since 2025

  • Helicone went into maintenance mode. Mintlify acquired it on March 3, 2026. It still gets security updates, fixes, and new models, but Mintlify says it will help customers migrate elsewhere.
  • Startup programs are worth checking first. Langfuse, LangSmith, and Braintrust all run startup programs. Details are below.
  • Langfuse changed how it judges production traffic. Trace-level evaluators are deprecated and stopped producing results on Langfuse Cloud after November 16, 2026. LLM judges now target observations and experiments.
  • OpenTelemetry GenAI conventions spread. Most tools in this guide accept OTel spans, so your instrumentation is no longer tied to one vendor.

Quick comparison

Tool Best stage Free tier Startup program Self-host First paid plan
Langfuse Prototype onward Hobby: 50,000 units/mo, 2 users, 30 days 50% off Cloud for 12 months Free (Postgres, ClickHouse, Redis, S3) Core $29/mo
LangSmith Prototype onward, on LangChain Developer: 1 seat, 5,000 traces, 14 days Discounts and credits up to $10,000 Enterprise Plus $39/seat/mo
Braintrust First customers to scale, eval-driven teams Starter: 1 GB, 10,000 scores, 14 days, unlimited users 6-12 months of Pro Enterprise (on-prem or hosted) Pro $249/mo
Helicone Existing users only Hobby: 10,000 requests, 1 seat Not stated Apache-2.0, Docker Pro $79/mo
Agnost AI First customers onward Free: 1,000 events/mo, 7 days Not stated Enterprise (VPC) Starter $49/mo

Stage 1: Prototype

Goal. See every step your agent takes while you iterate, for free.

What to instrument first. Do this once and you won't redo it:

  1. A span for every model call and every tool call. OpenTelemetry's GenAI conventions name them with gen_ai.operation.name (for example chat, execute_tool, invoke_agent) and gen_ai.tool.name. Most agent frameworks emit these for you.
  2. A stable conversation ID and user ID on every span. The conventions define gen_ai.conversation.id. Without it, a five-turn conversation shows up as five unrelated traces.
  3. Errors, latency, and tokens. error.type on failed calls, plus gen_ai.usage.input_tokens and gen_ai.usage.output_tokens.

The GenAI conventions are still marked Development, so names can change. Use your framework's built-in instrumentation where it exists. Our OpenTelemetry instrumentation guide shows the setup.

What to use. Langfuse Cloud Hobby or LangSmith Developer. Langfuse gives you more room (50,000 units, 2 users, 30 days of data). LangSmith fits if you build on LangChain or LangGraph, but the free plan is one seat with 14-day retention. Skip conversation-level tools for now: you are the user, and you can read every conversation yourself.

Don't self-host yet. It's free in dollars but not in time. See the self-host section below.

Stage 2: First customers

Goal. Find out where real users fail, before they churn.

This is where tracing alone stops being enough. Your first paying users hit failures you never tested, and most of them aren't errors. The agent answered confidently and wrongly. The user asked the same thing three times. The user gave up. Every span in those conversations is green. See how to catch wrong answers that don't show up as errors.

When to add conversation-level monitoring. Once you can no longer read every conversation yourself each week. For most startups that is somewhere in the first few paying customers.

What to use. Keep your tracing tool for debugging. Add a conversation layer that groups traffic by user and session and judges whole conversations. Agnost AI does this; LangSmith's thread evaluators and Braintrust's trace or group scorers can do part of it if you write the judges. Start with three or four narrow rules tied to things that lose customers, such as "Don't claim an action was completed unless the tool call succeeded."

Upgrade triggers. You'll outgrow free tiers on retention before volume. Seven or fourteen days is too short to compare this week with last month.

Stage 3: Scaling

Goal. Ship changes without breaking what works, and pass enterprise security reviews.

  • Add release testing. Build datasets from production failures and run them before each release. Braintrust is built around this. Langfuse and LangSmith have datasets and experiments too.
  • Add alerts with owners. A weekly dashboard nobody opens is not monitoring. Route alerts on the few metrics that matter to Slack with a named owner. Our guide to AI agent SLOs and KPIs lists which ones.
  • Decide on self-hosting. Usually forced by a customer's data rules, not by cost.
  • Price your real volume. Units, traces, GB, scores, requests, and events don't convert one-to-one. Use the AI agent monitoring pricing guide.

Self-host vs SaaS: the real effort

Self-hosting is free in license cost, but it is infrastructure your team runs.

  • Langfuse runs two app containers (web and worker) plus PostgreSQL, ClickHouse, Redis or Valkey, and S3-compatible storage. Langfuse describes Docker Compose as a single VM "without high availability, scaling, or backups," and points production deployments to Kubernetes with Helm. That means someone owns upgrades, backups, and ClickHouse capacity.
  • Helicone self-hosts with a Docker Compose script under Apache-2.0. Given maintenance mode, don't start a new self-hosted install.
  • LangSmith, Braintrust, and Agnost AI offer self-hosted or VPC deployment on enterprise plans only.

For a five-person startup, SaaS plus a startup discount nearly always costs less than an engineer's week. Self-host when a customer contract requires it.

Langfuse

What it is. An open-source platform for tracing, prompt management, evaluations, and datasets with experiments. MIT-licensed except its ee folders, built on OpenTelemetry.

Best for startups that want the most free room and an open-source escape hatch.

Limits. LLM judges target observations and experiments, with sampling. Sessions are not an evaluation target, so conversation-level judging is something you build. Self-hosting is real ops work.

Pricing. Hobby is free for 50,000 units a month, 2 users, and 30 days of data. Core is $29 a month with 100,000 units, unlimited users, and 90 days. Pro is $199. Startups with no more than $5 million in venture funding, incorporated within 5 years, get 50% off Cloud for 12 months.

LangSmith

What it is. LangChain's tracing, evaluation, and deployment platform. It accepts OpenTelemetry and traces non-LangChain code too.

Best for startups that build on LangChain or LangGraph and want thread-level evaluators that judge whole conversations.

Limits. Per-seat pricing grows with the team. Base traces are kept 14 days. Self-hosting is Enterprise only.

Pricing. Developer is free for 1 seat and 5,000 base traces a month. Plus is $39 per seat per month with 10,000 base traces, then pay as you go. VC-backed startups can get discounted rates and credits up to $10,000.

Braintrust

What it is. An evaluation and observability platform built around experiments, datasets, and scorers.

Best for startups that run releases on evals and want production logs to feed test datasets.

Limits. Scores are metered, so judging all production traffic gets expensive. Pro is a big step from free. Self-hosting is Enterprise only.

Pricing. Starter is free with 1 GB, 10,000 scores, 14-day retention, and unlimited users. Pro is $249 a month with 5 GB and 50,000 scores. New customers at Series A or earlier with at least $100,000 raised get six months of Pro, or twelve months through a partner VC.

Helicone

What it is. An Apache-2.0 AI gateway with logging, cost, and latency analytics. You change your client's base URL to Helicone's gateway, or log asynchronously through OpenLLMetry.

Best for startups that already run it for cost and latency visibility.

Limits. Mintlify acquired Helicone on March 3, 2026, and said it "will continue operating in maintenance mode." Helicone's pricing page now says it "is now part of Experiential Labs. Details to come soon." It is not an eval platform.

Pricing. Hobby is free for 10,000 requests and 1 seat. Pro is $79 a month and Team $799 a month, both usage-based above that.

Agnost AI

What it is. Agent monitoring for AI agents in production: chat agents, voice agents (through transcripts), copilots, and MCP servers. OpenTelemetry-native, with Python and TypeScript SDKs, and it works with the OpenAI Agents SDK, Vercel AI SDK, LangChain and LangGraph, Mastra, Pydantic AI, Google ADK, and Agno. Setup is an OTel exporter config or SDK init.

Best for startups that have real users and need to know where conversations go wrong. Agnost groups traffic by user and session and judges whole conversations. It detects failures, user frustration, intents, tool-call errors, loops, and silent wrong answers. Online evals are plain-language rules judged on every conversation once it has been idle 15 minutes, with no sampling. Alerts go to Slack or email. When an alert fires, Auto Improve opens a pull request with a fix tested against past runs for a person to review.

Limits. Not open source. No cost tracking, no offline datasets or experiments, no CI gating. Verdicts are pass or fail, not scores. Judging runs in batches, not in real time. Use it next to a tracing or eval tool, not instead of one. Security review is available on request.

Pricing. Free covers 1,000 events a month with 7-day retention. Starter is $49 a month (10,000 events). Pro is $499 a month (1,000,000 events). Enterprise adds self-hosted VPC deployment.

How to choose

If you are… Pick
Prototyping, want the most free room Langfuse Cloud Hobby
Prototyping on LangChain or LangGraph LangSmith Developer
Required by a customer to keep data in your cloud Self-hosted Langfuse, or an enterprise plan
Shipping releases on evals and datasets Braintrust (check the startup program)
Live with paying users and can't read every conversation anymore Add Agnost AI next to your tracing tool
Looking for a gateway for cost and latency Not Helicone for new projects; pick a maintained gateway
Unsure Instrument with OpenTelemetry and start on a free tier. Switching later is an exporter change

Related reading

Frequently asked questions

Which AI agent observability platform should a startup use?

Pick by stage. While prototyping, use a free tier with generous limits: Langfuse Cloud Hobby (50,000 units a month), or LangSmith Developer if you build on LangChain. When real customers arrive, keep the tracing tool and add a conversation-level layer such as Agnost AI so you see where users fail and get frustrated, not just which calls errored. When you scale, add Braintrust or Langfuse datasets for release testing, and decide whether to self-host based on customer data rules. Instrument with OpenTelemetry from day one so switching costs only an exporter change.

What is the best free AI agent observability tool for startups?

Langfuse is the most generous: free to self-host, and Cloud Hobby includes 50,000 units a month for 2 users with 30 days of data. LangSmith Developer is free for 1 seat and 5,000 traces with 14-day retention. Braintrust Starter is free with 1 GB, 10,000 scores, and unlimited users. Helicone Hobby is free for 10,000 requests but has been in maintenance mode since March 2026. Agnost AI is free for 1,000 events a month with 7-day retention.

Should a startup self-host Langfuse or use the cloud?

Use the cloud unless a customer contract or data rule requires self-hosting. Self-hosted Langfuse is free, but production runs two app containers plus PostgreSQL, ClickHouse, Redis or Valkey, and S3-compatible storage, and Langfuse describes Docker Compose as a single VM without high availability, scaling, or backups, so production means Kubernetes. That is real ops work for a small team.

Do AI observability vendors have startup programs?

Yes. Langfuse gives 50% off Langfuse Cloud for 12 months to companies with no more than $5 million in venture funding, incorporated within 5 years. LangSmith offers discounted rates and credits up to $10,000 for VC-backed startups. Braintrust gives new customers at Series A or earlier with at least $100,000 in funding six months of Pro, or twelve months if funded by a partner VC.

What should a startup instrument first in an AI agent?

Every model call and every tool call as OpenTelemetry spans, with a stable session or conversation ID and a user ID on each one. Add tool errors, latency, and token counts. Without session and user IDs you can't see a conversation end to end or tell whether one user hit the bug or a hundred did. Prompt versions and the outcome of the user's task come next.

When should a startup add conversation-level monitoring?

When you have real users you don't talk to every day, usually your first paying customers. At that point the failures that cost you are users who got a wrong answer, repeated themselves, or gave up, and those don't show up as errors in traces. A rule of thumb: once you can no longer read every conversation yourself each week, you need a tool to read them for you.

Is Helicone still a good choice for a startup?

Not for new projects. Mintlify acquired Helicone on March 3, 2026, and said it will continue operating in maintenance mode with security updates, bug fixes, and new models, and that it would help customers migrate to another platform. Helicone's pricing page now says it is part of Experiential Labs, with details to come. The code remains Apache-2.0 if you already self-host it.

Originally published on agnost.ai.

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.