Building Production-Grade AI Agents: How I Designed an Autonomous Refund System with Google ADK, HITL Guardrails & Zero Hallucination
Series Navigation: Part 1 (Foundations): Mastering Google ADK: From Zero to Tool-Driven AI Agents Part 2 (This Post): Deep-Dive Architecture, Design Philosophy, End-to-End Flow & System Design GitHub Reference Repo
Series Navigation:
- Part 1 (Foundations): Mastering Google ADK: From Zero to Tool-Driven AI Agents
- Part 2 (This Post): Deep-Dive Architecture, Design Philosophy, End-to-End Flow & System Design
- GitHub Reference Repo: github.com/gouravchawla334/adk-refund-agent
In Part 1, I explored why Google ADK shifts agent development from fragile prompt-chaining to deterministic software engineering. In this post, I want to pull back the curtain on adk-refund-agentβwalking through my exact engineering thought process, features, system design concepts, and how I achieved zero hallucination, strict HITL gates, and robust cost savings.
1. The Core Design Philosophy: Why "Tool-Only" & "Guardrail-First"?
When I set out to build a customer support agent, the most common pitfall I wanted to avoid was giving an LLM a massive system prompt containing return policies and letting it calculate refunds conversationally. In production finance workflows, that approach fails for three distinct reasons:
- LLMs are probabilistic, not calculators: Asking an LLM to subtract a 10% restocking fee from an order subtotal while conditionally waiving it for defective items will eventually result in a hallucinated dollar figure.
-
Prompts are not security boundaries: Telling an LLM "Never refund more than $150" in a system prompt can easily be bypassed via prompt injection. True security must live in deterministic code (
before_tool_callback), never in prompt text. - Blind automation destroys user trust: An autonomous agent shouldn't silently yank money or process a return before confirming what item is being returned and why.
My Architectural Thesis
I designed RefundOrchestratorAgent around a strict separation of concerns:
-
The LLM (
gemini-2.5-flash) is purely a semantic router and conversational translator. It understands user intent, invokes tools, and translates structured JSON results into concise plain English. - Python tools own 100% of state, math, and mutations. Every database lookup, fee deduction, and payment gateway call happens inside isolated, unit-tested Python functions.
- ADK Lifecycle Callbacks act as a deterministic Policy Plane. Code intercepts every model request and tool call before execution to enforce authorization, sequence order, financial caps, and token budgets.
2. End-to-End Execution Flow & Interactive HITL Gates
Rather than executing an entire refund in one opaque step, I designed the agent to guide the customer through five explicit stages punctuated by Human-in-the-Loop (HITL) confirmation gates:
[User: "I want a refund"]
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β STAGE 1: Intake & SQL-Safe Lookup β
β β’ Prompt for Order ID (ORD-XXXXX) β
β β’ Validate regex ^ORD-[0-9]{5}$ -> Query SQLite via parameterized SQL β
β β’ [If missing/invalid] βββΊ Plain-English "Order not available" & STOP β
ββββββββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β STAGE 2: HITL Confirmation Gate #1 (Order Summary & Return Reason) β
β β’ Summarize items & paid amount in plain English β
β β’ Ask WHY the customer is returning the item β
β β’ Call capture_return_feedback(order_id, reason_category, comment) β
ββββββββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β STAGE 3: Parallel Validation & Risk Plane β
β β’ get_return_logistics(order_id) βββΊ Verify DELIVERED_AT_WAREHOUSE β
β β’ fetch_refund_policy(category_id) βββΊ Get return window & fee % β
β β’ check_fraud_score(cust_id, oid) βββΊ Evaluate velocity risk β
β β’ [If not scanned at warehouse] βββΊ Plain-English hold message & STOP β
ββββββββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β STAGE 4: Deterministic Math & HITL Confirmation Gate #2 β
β β’ calculate_refund_amount() computes subtotal, fees & net payout β
β β’ Present exact breakdown & ask explicit confirmation ("Yes / Confirm") β
ββββββββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β STAGE 5: Conditional Execution vs. Finance Escalation β
β β’ before_tool_callback checks: amount <= $150.00 AND fraud_flagged=Falseβ
β ββββΊ PASS: Inject verified amount + idempotency_key βββΊ Payout! β
β ββββΊ FAIL: Block payout βββΊ create_hitl_ticket() for Finance Queue β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
3. Project Features & System Design Implementations
To make this agent robust enough for real-world production, I integrated several key system design patterns and safeguards:
A. Eradicating Hallucinations (Zero-Math LLM Policy)
-
The Design: The LLM is strictly banned from performing arithmetic. All calculationsβincluding subtotals, restocking fee deductions, and shipping reimbursementsβhappen inside
calculate_refund_amount(). -
State Locking: When the calculation tool runs,
after_tool_callbackcaptures the verifiednet_refund_amountand stores it securely intool_context.state["verified_refund_amount"]. Whenexecute_payment_refundis eventually called,before_tool_callbackoverwrites any amount the LLM passed and injects the verified state value directly. The model cannot alter a single cent.
B. Human-in-the-Loop (HITL) Gateways
- Gate #1 (Context & Feedback): Before checking warehouse logs or policies, the agent summarizes the fetched order details in plain English and asks the customer for their return reason. This ensures the customer agrees on which item is being evaluated.
-
Gate #2 (Financial Confirmation): After deterministic math runs, the agent presents the exact itemized breakdown and waits for explicit user consent (
"Yes","Confirm") before authorizing any financial mutation. -
Gate #3 (Automatic Escalation): If a refund exceeds my auto-approval threshold of
$150.00or triggers fraud velocity flags,before_tool_callbackintercepts the payout tool, blocks it entirely, and routes the order to a manual finance review queue viacreate_hitl_ticket().
C. Cost Optimization & Token Efficiency
-
Token Capping: I configured
max_output_tokens=300andtemperature=0.0inGenerateContentConfigto prevent runaway verbose responses and keep API latency low. -
Prompt Caching Optimization: My
prompts.pyfile is kept 100% static (free of dynamic timestamps or user identifiers), maximizing Gemini's prompt caching hit rate across sessions. -
Payload Pruning: In
after_tool_callback, I prune verbose internal database metadata (such as raw database timestamps and audit logs) fromget_order_detailsbefore the response enters the LLM context window. This drastically reduces token bloat over multi-turn conversations.
D. Security & Threat Mitigation (SQL Injection & Domain Locking)
-
SQL Injection Defense: Incoming order IDs are validated against a strict regular expression (
^ORD-[0-9]{5}$) via Pydantic input validators before touching the database. Furthermore, all database operations use strictly parameterized queries (?bindings) rather than raw string formatting. -
Strict Domain Locking: To prevent prompt hijacking or off-topic chat (e.g., asking the agent to write code or tell jokes), my
before_model_callbackintercepts irrelevant queries instantly and responds with a rigid deflection message: "I am a refund agent only and can only help you process returns and refunds." -
Anti-Looping Guardrail: I track consecutive identical assistant replies in
tool_context.state. If the agent repeats the same clarifying question twice, it stops looping and outputs a polite troubleshooting fallback.
4. Local Database & Test Scenarios
To make the repository testable offline in seconds without cloud dependencies, I built refund_agent/db.py to initialize a local SQLite database (local_refund_store.db) seeded with three test cases:
ORD-98765(Happy Path Auto-Approval):-
Data: Apparel order (
$90.00items +$10.00shipping =$100.00), scanned at warehouse (DELIVERED_AT_WAREHOUSE). Behavior: If returned as
NO_LONGER_NEEDED, deducts a 10% restocking fee ($9.00) and auto-refunds$91.00after HITL confirmation. If returned asDEFECTIVE, it waives the restocking fee entirely and refunds$100.00.ORD-55555(Financial Cap HITL Escalation):Data: Electronics order (
$240.00), scanned at warehouse.Behavior: Exceeds my
$150.00auto-approval ceiling.before_tool_callbackblocksexecute_payment_refundand forcescreate_hitl_ticket, returning ticketFIN-HITL-55555.ORD-11111(Logistics Hold):Data: Apparel order (
$80.00), carrier statusIN_TRANSIT.Behavior: Halts gracefully at Stage 3 and informs the customer that the return cannot be refunded until the warehouse scans the package.
5. Clean Modular Folder Structure
I organized the codebase around strict separation of concerns so that as I learn new ADK featuresβlike connecting external databases via MCP, orchestrating Sub-Agents, or running adk eval benchmarksβnew modules plug in cleanly without rewriting core logic:
adk-refund-agent/
βββ .github/workflows/ci.yml # Automated pytest CI on every push
βββ docs/
β βββ PRD.md # Formal functional & non-functional specs
β βββ ARCHITECTURE.md # Callback lifecycle & threat model
βββ refund_agent/ # Core ADK Package (`adk web` discovery root)
β βββ __init__.py # Exports `root_agent`
β βββ agent.py # LlmAgent definition & callback wiring
β βββ config.py # Financial caps ($150), token cap (300) & regex rules
β βββ db.py # Parameterized SQLite store & test seed data
β βββ prompts.py # Static cache-optimized system instructions
β βββ schemas/ # Pydantic input validators & output contracts
β βββ tools/ # Atomic read, compute, and write tool modules
β βββ guardrails/ # ADK before/after callbacks (agent, model, tool)
βββ tests/unit/ # Deterministic unit tests for SQLi, math & caps
βββ .env.example # API key configuration template
βββ pyproject.toml # Dependencies & pytest configuration
βββ README.md # Quickstart & architecture guide
6. Try It Yourself
I invite you to clone the repository, run the unit test suite, and inspect live callback traces in ADK's web UI:
git clone https://github.com/your-username/adk-refund-agent.git
cd adk-refund-agent
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
cp .env.example .env # Add your GOOGLE_API_KEY
pytest # Run unit tests for SQLi defense & refund math
adk web --port 8000 # Test ORD-98765, ORD-55555, and ORD-11111 live!
What should I build in **Part 3? Next up, I plan to upgrade db.py to a live PostgreSQL server using **Model Context Protocol (MCPToolset)* and add an automated adk eval regression suite to CI/CD!*
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.