Coinbase's Autopilot brings a quality loop to AI customer support

Everyone automates customer support. Almost nobody automates maintaining it.

Coinbase's engineering team just described Autopilot — not a customer-facing bot, but a quality loop around the bots: a pipeline where agents write, test, probe, score, and analyze the versioned procedures that support agents follow. It's a shift from treating support automation as "a collection of prompts that passed a manual spot check" to treating it as software that needs repeatable validation.

Here's what stood out:

• Procedures are code, not prose. Coinbase's bots follow versioned decision trees — data retrieval, branching conditions, end states — expressed in a constrained domain-specific language. Edit a procedure, swap a model, or change a data interface, and outcomes can shift silently. Autopilot treats correctness as a continuing maintenance task, not a one-time test.

• Agents explore, deterministic services grade. A deliberate architectural split: agents do what benefits from language interpretation and exploration (investigating unusual customer behavior, probing malicious attempts), while deterministic services provide the test harness — fixtures, release checks, consistent grading. Discovery without repeatability isn't actionable.

• Shared test contracts make bots comparable. Procedures and test contracts specify five elements: the customer scenario, mock account state, expected behavior, the bot under test, and the aspect being evaluated. One test plan runs against both Coinbase's own bot and partner bots with the same inputs and grading.

• Intent routing decides where failures matter. Coinbase classifies customer intent separately from procedures and product areas, then prioritizes flows by intent volume alongside resolution and CSAT — heavily used intents with weaker outcomes get attention first.

• Humans stay at consequential points. Bots escalate cases needing judgment; people remain in the procedure-writing loop. Autopilot explicitly reduces manual maintenance effort while preserving judgment.

The honest caveat: Coinbase's account gives strong architectural detail but no quantified post-deployment results — no measured reduction in testing time, no resolution or CSAT deltas. The baseline ("one UAT cycle could take days") is a before picture, not a proven before-and-after.

Go deeper

The full analysis on Agentic Affairs walks through the five-element test contracts, the agent/deterministic-service split, intent classification across Help Center, chat and voice, and exactly where the evidence ends and the claims begin.

Originally published on Agentic Affairs (https://agentic-affairs.com)