Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 6 min read

Building Production-Grade AI Agents: How I Designed an Autonomous Refund System with Google ADK, HITL Guardrails & Zero Hallucination

Series Navigation: Part 1 (Foundations): Mastering Google ADK: From Zero to Tool-Driven AI Agents Part 2 (This Post): Deep-Dive Architecture, Design Philosophy, End-to-End Flow & System Design GitHub Reference Repo

Series Navigation:

In Part 1, I explored why Google ADK shifts agent development from fragile prompt-chaining to deterministic software engineering. In this post, I want to pull back the curtain on adk-refund-agentβ€”walking through my exact engineering thought process, features, system design concepts, and how I achieved zero hallucination, strict HITL gates, and robust cost savings.

1. The Core Design Philosophy: Why "Tool-Only" & "Guardrail-First"?

When I set out to build a customer support agent, the most common pitfall I wanted to avoid was giving an LLM a massive system prompt containing return policies and letting it calculate refunds conversationally. In production finance workflows, that approach fails for three distinct reasons:

  1. LLMs are probabilistic, not calculators: Asking an LLM to subtract a 10% restocking fee from an order subtotal while conditionally waiving it for defective items will eventually result in a hallucinated dollar figure.
  2. Prompts are not security boundaries: Telling an LLM "Never refund more than $150" in a system prompt can easily be bypassed via prompt injection. True security must live in deterministic code (before_tool_callback), never in prompt text.
  3. Blind automation destroys user trust: An autonomous agent shouldn't silently yank money or process a return before confirming what item is being returned and why.

My Architectural Thesis

I designed RefundOrchestratorAgent around a strict separation of concerns:

  • The LLM (gemini-2.5-flash) is purely a semantic router and conversational translator. It understands user intent, invokes tools, and translates structured JSON results into concise plain English.
  • Python tools own 100% of state, math, and mutations. Every database lookup, fee deduction, and payment gateway call happens inside isolated, unit-tested Python functions.
  • ADK Lifecycle Callbacks act as a deterministic Policy Plane. Code intercepts every model request and tool call before execution to enforce authorization, sequence order, financial caps, and token budgets.

2. End-to-End Execution Flow & Interactive HITL Gates

Rather than executing an entire refund in one opaque step, I designed the agent to guide the customer through five explicit stages punctuated by Human-in-the-Loop (HITL) confirmation gates:

[User: "I want a refund"]
         β”‚
         β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ STAGE 1: Intake & SQL-Safe Lookup                                       β”‚
β”‚ β€’ Prompt for Order ID (ORD-XXXXX)                                       β”‚
β”‚ β€’ Validate regex ^ORD-[0-9]{5}$ -> Query SQLite via parameterized SQL   β”‚
β”‚ β€’ [If missing/invalid] ──► Plain-English "Order not available" & STOP   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                     β”‚
                                     β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ STAGE 2: HITL Confirmation Gate #1 (Order Summary & Return Reason)      β”‚
β”‚ β€’ Summarize items & paid amount in plain English                        β”‚
β”‚ β€’ Ask WHY the customer is returning the item                            β”‚
β”‚ β€’ Call capture_return_feedback(order_id, reason_category, comment)      β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                     β”‚
                                     β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ STAGE 3: Parallel Validation & Risk Plane                               β”‚
β”‚ β€’ get_return_logistics(order_id)   ──► Verify DELIVERED_AT_WAREHOUSE    β”‚
β”‚ β€’ fetch_refund_policy(category_id) ──► Get return window & fee %        β”‚
β”‚ β€’ check_fraud_score(cust_id, oid)  ──► Evaluate velocity risk           β”‚
β”‚ β€’ [If not scanned at warehouse] ──► Plain-English hold message & STOP   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                     β”‚
                                     β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ STAGE 4: Deterministic Math & HITL Confirmation Gate #2                 β”‚
β”‚ β€’ calculate_refund_amount() computes subtotal, fees & net payout        β”‚
β”‚ β€’ Present exact breakdown & ask explicit confirmation ("Yes / Confirm") β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                     β”‚
                                     β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ STAGE 5: Conditional Execution vs. Finance Escalation                   β”‚
β”‚ β€’ before_tool_callback checks: amount <= $150.00 AND fraud_flagged=Falseβ”‚
β”‚   β”œβ”€β”€β–Ί PASS: Inject verified amount + idempotency_key ──► Payout!       β”‚
β”‚   └──► FAIL: Block payout ──► create_hitl_ticket() for Finance Queue    β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

3. Project Features & System Design Implementations

To make this agent robust enough for real-world production, I integrated several key system design patterns and safeguards:

A. Eradicating Hallucinations (Zero-Math LLM Policy)

  • The Design: The LLM is strictly banned from performing arithmetic. All calculationsβ€”including subtotals, restocking fee deductions, and shipping reimbursementsβ€”happen inside calculate_refund_amount().
  • State Locking: When the calculation tool runs, after_tool_callback captures the verified net_refund_amount and stores it securely in tool_context.state["verified_refund_amount"]. When execute_payment_refund is eventually called, before_tool_callback overwrites any amount the LLM passed and injects the verified state value directly. The model cannot alter a single cent.

B. Human-in-the-Loop (HITL) Gateways

  • Gate #1 (Context & Feedback): Before checking warehouse logs or policies, the agent summarizes the fetched order details in plain English and asks the customer for their return reason. This ensures the customer agrees on which item is being evaluated.
  • Gate #2 (Financial Confirmation): After deterministic math runs, the agent presents the exact itemized breakdown and waits for explicit user consent ("Yes", "Confirm") before authorizing any financial mutation.
  • Gate #3 (Automatic Escalation): If a refund exceeds my auto-approval threshold of $150.00 or triggers fraud velocity flags, before_tool_callback intercepts the payout tool, blocks it entirely, and routes the order to a manual finance review queue via create_hitl_ticket().

C. Cost Optimization & Token Efficiency

  • Token Capping: I configured max_output_tokens=300 and temperature=0.0 in GenerateContentConfig to prevent runaway verbose responses and keep API latency low.
  • Prompt Caching Optimization: My prompts.py file is kept 100% static (free of dynamic timestamps or user identifiers), maximizing Gemini's prompt caching hit rate across sessions.
  • Payload Pruning: In after_tool_callback, I prune verbose internal database metadata (such as raw database timestamps and audit logs) from get_order_details before the response enters the LLM context window. This drastically reduces token bloat over multi-turn conversations.

D. Security & Threat Mitigation (SQL Injection & Domain Locking)

  • SQL Injection Defense: Incoming order IDs are validated against a strict regular expression (^ORD-[0-9]{5}$) via Pydantic input validators before touching the database. Furthermore, all database operations use strictly parameterized queries (? bindings) rather than raw string formatting.
  • Strict Domain Locking: To prevent prompt hijacking or off-topic chat (e.g., asking the agent to write code or tell jokes), my before_model_callback intercepts irrelevant queries instantly and responds with a rigid deflection message: "I am a refund agent only and can only help you process returns and refunds."
  • Anti-Looping Guardrail: I track consecutive identical assistant replies in tool_context.state. If the agent repeats the same clarifying question twice, it stops looping and outputs a polite troubleshooting fallback.

4. Local Database & Test Scenarios

To make the repository testable offline in seconds without cloud dependencies, I built refund_agent/db.py to initialize a local SQLite database (local_refund_store.db) seeded with three test cases:

  1. ORD-98765 (Happy Path Auto-Approval):
  2. Data: Apparel order ($90.00 items + $10.00 shipping = $100.00), scanned at warehouse (DELIVERED_AT_WAREHOUSE).
  3. Behavior: If returned as NO_LONGER_NEEDED, deducts a 10% restocking fee ($9.00) and auto-refunds $91.00 after HITL confirmation. If returned as DEFECTIVE, it waives the restocking fee entirely and refunds $100.00.

  4. ORD-55555 (Financial Cap HITL Escalation):

  5. Data: Electronics order ($240.00), scanned at warehouse.

  6. Behavior: Exceeds my $150.00 auto-approval ceiling. before_tool_callback blocks execute_payment_refund and forces create_hitl_ticket, returning ticket FIN-HITL-55555.

  7. ORD-11111 (Logistics Hold):

  8. Data: Apparel order ($80.00), carrier status IN_TRANSIT.

  9. Behavior: Halts gracefully at Stage 3 and informs the customer that the return cannot be refunded until the warehouse scans the package.

5. Clean Modular Folder Structure

I organized the codebase around strict separation of concerns so that as I learn new ADK featuresβ€”like connecting external databases via MCP, orchestrating Sub-Agents, or running adk eval benchmarksβ€”new modules plug in cleanly without rewriting core logic:

adk-refund-agent/
β”œβ”€β”€ .github/workflows/ci.yml       # Automated pytest CI on every push
β”œβ”€β”€ docs/
β”‚   β”œβ”€β”€ PRD.md                     # Formal functional & non-functional specs
β”‚   └── ARCHITECTURE.md            # Callback lifecycle & threat model
β”œβ”€β”€ refund_agent/                  # Core ADK Package (`adk web` discovery root)
β”‚   β”œβ”€β”€ __init__.py                # Exports `root_agent`
β”‚   β”œβ”€β”€ agent.py                   # LlmAgent definition & callback wiring
β”‚   β”œβ”€β”€ config.py                  # Financial caps ($150), token cap (300) & regex rules
β”‚   β”œβ”€β”€ db.py                      # Parameterized SQLite store & test seed data
β”‚   β”œβ”€β”€ prompts.py                 # Static cache-optimized system instructions
β”‚   β”œβ”€β”€ schemas/                   # Pydantic input validators & output contracts
β”‚   β”œβ”€β”€ tools/                     # Atomic read, compute, and write tool modules
β”‚   └── guardrails/                # ADK before/after callbacks (agent, model, tool)
β”œβ”€β”€ tests/unit/                    # Deterministic unit tests for SQLi, math & caps
β”œβ”€β”€ .env.example                   # API key configuration template
β”œβ”€β”€ pyproject.toml                 # Dependencies & pytest configuration
└── README.md                      # Quickstart & architecture guide

6. Try It Yourself

I invite you to clone the repository, run the unit test suite, and inspect live callback traces in ADK's web UI:

git clone https://github.com/your-username/adk-refund-agent.git
cd adk-refund-agent
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
cp .env.example .env  # Add your GOOGLE_API_KEY
pytest               # Run unit tests for SQLi defense & refund math
adk web --port 8000  # Test ORD-98765, ORD-55555, and ORD-11111 live!

What should I build in **Part 3? Next up, I plan to upgrade db.py to a live PostgreSQL server using **Model Context Protocol (MCPToolset)* and add an automated adk eval regression suite to CI/CD!*

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.