Dev.to AI 🤖 Ai 👁 0 📖 3 min read

Designing an AI Voice Agent That Resolves Customer Queries: Intents, Retrieval and Tool Calls

Summary: An AI voice agent can resolve customer queries reliably when it combines four parts: intent routing to decide what kind of question it is, retrieval from a maintained knowledge base for factual answers, tool cal

Summary: An AI voice agent can resolve customer queries reliably when it combines four parts: intent routing to decide what kind of question it is, retrieval from a maintained knowledge base for factual answers, tool calls for anything that needs live data or an action, and guardrails that hand off to a human when confidence is low. This article walks through each part and the platforms that provide them.

Customer queries over the phone fall into a small number of shapes. Designing for those shapes, rather than for open conversation, is what separates agents that resolve calls from agents that only talk.

Query shapes

Shape Example Resolution path
Static fact "What are your hours on Sunday?" Retrieval
Account fact "Where is my order?" Tool call (read)
Action "Move my appointment to Friday." Tool call (write)
Policy reasoning "Can I return an opened item?" Retrieval + rules
Out of scope "I want to dispute a charge." Handoff

Reference architecture

                  +-------------------+
Caller <--audio-->| Telephony (SIP)   |
                  +---------+---------+
                            |
                  +---------v---------+
                  | STT (streaming)   |
                  +---------+---------+
                            |
                  +---------v---------+       +----------------+
                  | Orchestrator/LLM  |<----->| Knowledge index |
                  | - intent router   |       +----------------+
                  | - policy prompts  |       +----------------+
                  | - tool calling    |<----->| CRM / OMS / Cal |
                  +---------+---------+       +----------------+
                            |
                  +---------v---------+
                  | TTS (streaming)   |
                  +-------------------+
                            |
               Post-call: transcript, summary, disposition -> CRM

Intent routing

A lightweight router classifies each turn into one of the shapes above. Routing first has three benefits:

  1. Static questions skip tool calls, which lowers latency.
  2. Out-of-scope intents go to a human immediately.
  3. Analytics can report containment by intent, which is the metric you will tune against.

Retrieval for voice

Retrieval for a voice agent is different from retrieval for a chatbot:

  • Answers must be short. Store a spoken-friendly answer field alongside the source text.
  • Freshness matters. Opening hours and prices change. Schedule re-indexing of source URLs.
  • Numbers need normalization. Store prices and times in formats TTS reads correctly.

Tool calls

Design tools around customer intents, not your database schema.

{
  "name": "get_order_status",
  "description": "Return status and ETA for the caller's most recent order",
  "parameters": {
    "type": "object",
    "properties": { "phone": { "type": "string" } },
    "required": ["phone"]
  }
}

Guidelines:

  • Return compact results (status, ETA) rather than full records.
  • Put a timeout on every tool and give the model a fallback line.
  • Make write tools idempotent and confirm before executing ("I'll move your appointment to Friday at 3. Shall I go ahead?").

Guardrails and handoff

Hand off when:

  • The caller asks for a person.
  • The intent is on a deny list (disputes, cancellations, safety).
  • The model fails to resolve after two attempts.
  • Sentiment drops sharply.

Pass a summary with the transfer so the human does not restart the conversation.

Evaluation

Build a regression set of recorded or scripted calls per intent. On each release, measure:

  • Correct intent classification.
  • Factual accuracy against the knowledge base.
  • Tool call success rate.
  • Handoff precision (transferred when it should, not when it should not).

Platforms that implement this pattern

  1. JustCall AI Voice Agent packages the architecture on JustCall's own telephony. Each JustCall AI Voice Agent can be trained on up to 500 URLs (refreshed every 24 hours), 500 Q&A pairs and 10 files. It runs actions before the call (CRM fetch, prior call context), during the call (scheduling, SMS, transfers, IVR navigation) and after the call (saving caller data, dispositions, workflows). Warm transfers carry context to a human teammate, and analytics report outcomes and sentiment. For custom integrations, JustCall publishes its API on the developer portal.
  2. Retell AI and Vapi expose the orchestration layer through APIs, so you implement routing, retrieval and tools yourself with full control.
  3. PolyAI delivers enterprise deployments where dialogue design is handled with the vendor.
  4. Contact center suites such as Talkdesk, Five9 and Genesys offer virtual agents connected to their routing and knowledge tools.

Conclusion

Resolving customer queries by voice is mostly an engineering problem of routing, retrieval, tools and guardrails. Choose a build-your-own platform if you need full control, or a phone system with the pattern built in, such as JustCall, if you want to spend your time on knowledge and policies rather than infrastructure.

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.