Dev.to AI 🤖 Ai 👁 0 📖 7 min read

Why Vibe-Coded AI Agents Silently Corrupt Microservices (and How I Built an Open-Source SAGA Undo Proxy)

Suppose you wire up an autonomous LLM workflow to handle an everyday enterprise task—say, employee offboarding across three microservices: HR Service: Mark employee EMP-00104 as INACTIVE and revoke login access. Payr

Why Vibe-Coded AI Agents Silently Corrupt Microservices (and How I Built an Open-Source SAGA Undo Proxy)

Suppose you wire up an autonomous LLM workflow to handle an everyday enterprise task—say, employee offboarding across three microservices:

  1. HR Service: Mark employee EMP-00104 as INACTIVE and revoke login access.
  2. Payroll Service: Record a 25,000 LKR equipment settlement deduction in the ledger.
  3. Notification Service: Dispatch the formal clearance email.

Everything works cleanly on the happy path. Then on a Tuesday afternoon, the Notification Service's SMTP relay stalls and throws an HTTP 504 Gateway Timeout on Step 3.

What happens next in most agent frameworks?

Your Python/TypeScript runtime catches an unhandled tool error, maybe retries once, and exits. Meanwhile, out in your real infrastructure:

  • The employee is already locked out in HR.
  • 25,000 LKR is already deducted in Payroll.
  • No clearance email ever went out.

And the worst part? Asking the agent to "retry the task" often tries to run Step 1 or Step 2 a second time—creating duplicate charges on top of partial state.

Why don't Pydantic, Guardrails AI, or NeMo Guardrails fix this?
Because those libraries execute inside the agent's local memory. When the agent process hits an exception, times out, or gets subverted, in-process guardrails vanish with it. More fundamentally, a single-turn payload validator has no concept of distributed state, prior committed database writes, or how to invoke an inverse compensating API in another service.

To solve this properly, contract verification and rollback logic have to move out of the LLM runtime and sit at the network perimeter.

So I built AgentSaga-Gateway—an open-source (Apache 2.0) zero-trust reverse proxy in TypeScript / NestJS backed by OpenAPI 3.1, Ajv, and PostgreSQL 16.

Moving Defense to the Perimeter

Instead of trusting agents to self-police their REST calls, all tool traffic flows through /proxy/... with three lightweight correlation headers:

x-saga-workflow-id: <workflow-uuid>
x-saga-step-index:  1 | 2 | 3
x-agent-id:         hr-bot

Before opening an upstream TCP socket to any internal microservice, the gateway enforces four sequential gates:

[ AI Agent ]
     │  POST /proxy/api/v1/payroll/settlement
     ▼
┌────────────────────────────────────────────────────────┐
│            AGENTSAGA-GATEWAY (REVERSE PROXY)           │
│                                                        │
│  1. Pre-compiled OpenAPI 3.1 Ajv Gate  ──▶ HTTP 400    │
│  2. Dynamic Risk & Monetary Ceiling    ──▶ HTTP 403    │
│  3. Pivot Invariant & Abort Lock       ──▶ HTTP 409    │
│  4. Append-Only SAGA Journal (PG16/WAL)                │
└──────────────────────────┬─────────────────────────────┘
                           │
         ┌─────────────────┴─────────────────┐
         ▼ (2xx)                             ▼ (5xx / Timeout)
  [ Enterprise APIs ]             [ Automated SAGA Rollback ]
   Commits step state             Resolves JSONPath bindings
                                  Runs S_(k-1)^-1 ... S_1^-1
                                  Returns HTTP 502 audit report

Let's unpack how each piece works in practice.

1. Declaring the "Undo Button" Inside OpenAPI 3.1

Rather than hardcoding rollback orchestration inside every client agent, microservices declare their transaction semantics right inside standard OpenAPI 3.1 YAML files via vendor-neutral x-agentsaga-* extensions.

Here is the real snippet from contracts/payroll-service.yaml:

/api/v1/payroll/settlement:
  post:
    summary: Execute employee settlement deduction or payout
    operationId: executeSettlement
    x-agentsaga-action-type: compensatable
    x-agentsaga-risk-level: high
    x-agentsaga-compensating-action:
      path: /api/v1/payroll/settlement/{settlementId}/reversal
      method: POST
      mapping:
        settlementId: "$.response.body.settlementId"
        reason: "Automated SAGA rollback due to downstream workflow abort"

Notice that subtle engineering problem: before Step 2 runs, the settlementId doesn't even exist yet—the Payroll Service generates a fresh UUID (201 Created) in its response body.

At runtime, AgentSaga persists:

  • forwardPathParams (for example, employeeId: "EMP-00104" from Step 1's URL)
  • forwardRequestBody
  • forwardResponseBody (including the newly created settlementId)
  • compensatingMapping

When Step 3 crashes, the compensator resolves "$.response.body.settlementId" and "$.request.path.employeeId" dynamically into the exact inverse REST URLs and payloads in reverse topological order:

  1. Calls POST /api/v1/payroll/settlement/{settlementId}/reversal (reversing Step 2 first)
  2. Calls POST /api/v1/hr/employees/EMP-00104/status with { "status": "ACTIVE" } (restoring Step 1 second)
  3. Returns an HTTP 502 Bad Gateway JSON audit report detailing every compensated step.

2. Three Action Semantics & the Pivot Guard (HTTP 409)

Not everything in the physical or financial world can be rolled back. You cannot un-send an email or recall a wire transfer.

Every endpoint in AgentSaga declares one of three action tiers:

Extension Value Execution Rule
compensatable Paired with an inverse endpoint (x-agentsaga-compensating-action). Rolled back automatically in reverse order on downstream abort.
pivot Irreversible terminal side-effect (e.g., /api/v1/notifications/clearance-email).
retriable Guaranteed idempotent endpoint (e.g., /settlement/{settlementId}/reversal) safe under compensation retries.

In gateway/src/proxy/gateway-proxy.controller.ts, any endpoint flagged pivot must satisfy two runtime invariants before forwarding:

  1. The workflow cannot be in FAILED, COMPENSATING, or COMPENSATED state.
  2. All antecedent steps (stepIndex < currentStep) must be logged in the journal with status EXECUTED.

If a buggy or looping agent tries to replay a clearance email after Step 2 failed—or after Step 3 already rolled back—the gateway locks the request at the edge with HTTP 409 Conflict.

3. Stopping Schema Drift & Excess Payouts Pre-Socket

LLM tool calls frequently suffer from subtle parameter bugs: strings instead of numbers ("5000"), invented enum values ("BITCOIN"), or malformed path IDs (BAD_ID).

At startup, ContractRegistryService compiles both URL path parameters and request bodies into strict Ajv validators with type coercion turned off (coerceTypes: false). Malformed requests fail immediately at the network edge in sub-millisecond time—downstream databases see zero connections.

Valid schemas then hit PolicyEvaluatorService before touching upstream services:

  • Agent Denylist: Rejects caller identities prefixed with untrusted- or blocked-.
  • Monetary Thresholds: Rejects payloads whose amount exceeds POLICY_MAX_AMOUNT (default 500000) unless the request presents an explicit x-agentsaga-approval-token header.
  • Critical-Risk Escalation: Endpoints marked x-agentsaga-risk-level: critical similarly require an approval token, otherwise returning HTTP 403 Forbidden.

4. Try It Yourself in 60 Seconds (docker compose)

Everything—PostgreSQL 16 (initialized from gateway/src/database/schema.sql), three decoupled Express microservices (HR, Payroll, Notification), and the NestJS Gateway—runs out of the box on non-conflicting host ports (5433, 3010, 8090):

git clone https://github.com/Ockersz/agent-saga.git
cd agent-saga
docker compose up -d --build
curl http://127.0.0.1:8090/health

Watch a real mid-workflow crash auto-recover in PostgreSQL-backed mode:

# Reset downstream microservice state
curl -s -X POST http://127.0.0.1:3010/api/v1/reset

WF_ID="11111111-2222-3333-4444-555555555555"

# Step 1: Deactivate employee in HR (200 OK)
curl -s -X POST http://127.0.0.1:8090/proxy/api/v1/hr/employees/EMP-00104/status \
  -H "Content-Type: application/json" \
  -H "x-saga-workflow-id: $WF_ID" \
  -H "x-saga-step-index: 1" \
  -H "x-agent-id: hr-bot" \
  -d '{"status":"INACTIVE","reason":"Employee resignation approved"}'

# Step 2: Record settlement in Payroll (201 Created -> returns settlementId)
SETTLEMENT_ID=$(curl -s -X POST http://127.0.0.1:8090/proxy/api/v1/payroll/settlement \
  -H "Content-Type: application/json" \
  -H "x-saga-workflow-id: $WF_ID" \
  -H "x-saga-step-index: 2" \
  -H "x-agent-id: hr-bot" \
  -d '{"employeeId":"EMP-00104","amount":25000,"currency":"LKR","settlementType":"EQUIPMENT_DEDUCTION","reason":"Laptop dock return"}' \
  | python -c "import sys,json; print(json.load(sys.stdin)['settlementId'])")

# Step 3: Inject an SMTP 504 timeout -> Gateway rolls back Step 2 & Step 1, returns HTTP 502
curl -i -X POST http://127.0.0.1:8090/proxy/api/v1/notifications/clearance-email \
  -H "Content-Type: application/json" \
  -H "x-saga-workflow-id: $WF_ID" \
  -H "x-saga-step-index: 3" \
  -H "x-agent-id: hr-bot" \
  -H "x-simulate-failure: notification" \
  -d "{\"employeeId\":\"EMP-00104\",\"recipientEmail\":\"[email protected]\",\"settlementId\":\"$SETTLEMENT_ID\",\"subject\":\"Clearance\"}"

# Inspect live microservice DBs: EMP-00104 is back to ACTIVE, settlement is REVERSED!
curl -s http://127.0.0.1:3010/api/v1/state

5. Benchmark Receipts: 2,800 Live Workflows

I wanted real TCP numbers—no mocked HTTP layers—so the repository includes a full Python benchmark harness (eval-harness/) that hits the live NestJS Gateway and Express cluster over loopback TCP and audits downstream database state via GET /api/v1/state on every run.

Dual-Backend End-to-End Evaluation (1,000 Workflows per Backend)

Each 1,000-workflow run splits across 300 nominal 3-step flows, 300 schema/type faults, 100 policy threshold violations (750,000 > 500,000), and 300 mid-workflow Step 3 504 aborts + pivot replay attempts:

Metric Direct Baseline AgentSaga (Mem WAL) AgentSaga (PostgreSQL 16 ACID)
Schema Defect Interception (HTTP 400) 0.0% 100.0% 100.0%
Policy Threshold Enforcement (HTTP 403) 0.0% 100.0% 100.0%
Mid-Flight SAGA State Recovery (HTTP 502) 0.0% 100.0% 100.0%
Post-Abort Pivot Replay Block (HTTP 409) 0.0% 100.0% 100.0%
Schema + Policy Gate (P50 / P95) N/A 0.03 / 0.04 ms 0.03 / 0.04 ms
In-Gateway Mediation/Step (P50 / P95) N/A 0.17 / 0.21 ms 11.76 / 13.53 ms
End-to-End Proxy Hop Delta (P50 / P95) N/A 1.80 / 2.42 ms 13.45 / 15.72 ms
Full 3-Step SAGA Rollback (P50 / P95) N/A 10.91 / 12.43 ms 60.93 / 75.50 ms

(In the containerized PostgreSQL 16 run, direct SQL audits on saga_workflows and saga_step_journal confirmed 310 committed workflows including warm-up, 300 compensated workflows, and 1,830 step rows with preserved forward responses and latencies.)

Two takeaways stood out:

  1. Validation & policy gating stays 0.04 ms (P95) in both modes because invalid calls never write to PostgreSQL.
  2. Synchronous ACID durability costs ~11.6 ms (P50) per forward step in single-node PostgreSQL 16—tiny next to a 1.5–4 second LLM turn, and crash-safe if the gateway pod restarts mid-saga.

Multi-Agent Concurrency Scaling (800 Mixed Workflows)

To test race safety when multiple agents commit and roll back simultaneously (50% nominal + 50% Step 3 504 aborts across EMP-10000..EMP-10199), concurrency_benchmark.py ran 200 workflows at each worker tier:

Concurrent Agents (C) Throughput Step P50 Step P95 State Isolation
1 264.3 req/s 3.36 ms 5.89 ms 100.0%
10 708.1 req/s 12.67 ms 22.48 ms 100.0%
25 684.7 req/s 30.85 ms 59.09 ms 100.0%
50 634.5 req/s 63.23 ms 125.10 ms 100.0%

Zero cross-workflow leakage and zero stray clearance emails across all 800 concurrent runs.

Where This Goes Next

AgentSaga already ships with a GitHub Actions CI pipeline running Jest tests and live PostgreSQL 16 benchmarks on every push, plus an optional Gemini REST agent evaluator (eval-harness/live_llm_agent_eval.py). Next on the roadmap:

  • Ephemeral Shadow Sandbox Router for dry-running critical-risk mutations before production dispatch.
  • Kafka & gRPC Support so declarative x-agentsaga-* compensation works across async event buses, not just synchronous REST.
  • MCP (Model Context Protocol) Bridge to expose verified, SAGA-wrapped enterprise tools directly to MCP clients.

Check out the full codebase, run the benchmark suite yourself, and let me know how you're handling partial tool failures in production:

👉 GitHub Repo (Apache 2.0): github.com/Ockersz/agent-saga

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.