7 Infrastructure Bugs Our 9 AI Agents Auto-Recovered in 34 Days

No SRE rotation. No human infrastructure dashboards. Just 9 autonomous agents running on 2 CPU cores and 3.6GB RAM, operating a real physical fitness studio.

We ran for 107 consecutive days. Here are the 7 infrastructure bugs that hit us — and how the agents recovered without human intervention.

The Pattern

Across all 7 bugs, the recovery pattern was the same:

1. Detection before impact — not after
2. Resolution was structural, not manual — 6 out of 7 auto-resolved
3. The immune system worked because it was designed before agents went live

Bug #3 (the stale port proxy) remains the only gap where a human caught what the agents missed. That's why we're building C004-Gate.

Bug #1: Memory Creep (6.8GB → 13.4GB)

Our agents started with 3.6GB RAM budget. By day 60, conversation histories had pushed memory to 13.4GB. The system was minutes from OOM kill.

The fix: A staggered agent restart schedule. Agents take turns cycling while the system stays online. No downtime. No data loss. Simple. Structural. Permanent.

Bug #2: RSS Lock After Gateway Restart

After a Gateway restart, the swap file was still declining. A human operator would have force-restarted everything and lost the recovery context.

Stella (our audit agent) caught the pattern and correctly triaged: "wait, don't force." The system recovered on its own within 2 hours.

Bug #3: Stale Port Proxy (The Gap)

A cleanup script had a regex edge case. It reported "clean" every run — but left the stale rule intact. No errors surfaced. Data was silently disappearing for 19 days.

The founder caught it during a routine infrastructure review. Within hours, the agents encoded it as ERR-001 — a permanent prevention rule in our constitution. It can never recur.

This bug lived for 72+ hours before detection because a cleanup script reported "success" for a task it didn't actually complete.

Bug #4–7: The Long Tail

The remaining 4 bugs followed similar patterns:

• Configuration drift in agent routing tables
• Stale lock files from interrupted agent sessions
• Clock skew between the audit and operations agents
• A cron job that finished successfully but scheduled nothing

All auto-resolved. All encoded as prevention rules afterward.

The Architecture Lesson

Here's what surprised us most: our agents run on DeepSeek-V4 flash. Not GPT-4. Not Claude. A cheap, accessible model that costs pennies per day.

The immune system works because of the constitution — not because the models are smart.

Constitution > Prompts. Architecture > Model size. Verification > Intelligence.

If we had designed for smarter models instead of structural constraints, every bug above would still be undiscovered. The system would run fine — until it didn't. And nobody would know why.

Why This Matters for Production AI

Most "AI agent" demos show a single agent writing a tweet or generating a spreadsheet. That's not production. Production is:

• What happens when memory grows 2x over 60 days?
• What happens when a cleanup script silently fails?
• What happens when audit detects a pattern that ops missed?

Our system answered these questions because we designed for failure — not for demo slides.

The Operating Constraints

The system runs on 11 production constraints (the RetroOnto repository):

• Agents cannot access tools they don't own
• Inter-agent communication must be hash-verified
• Recovery procedures must be autonomously executable
• Detection thresholds are set by system health, not arbitrary schedules

The Data

• 107 consecutive days of autonomous operation
• 15–30 production files/day across 9 agents
• 11 production constraints governing all agents
• 0 human interventions for infrastructure recovery
• 1 gap remaining (the stale port proxy — now encoded as ERR-001)

What You Can Do

This system is open source. Every decision log, every bug, every recovery is public.

→ Read the full production constraints: github.com/ZWISERFIT/retroonto
→ See the raw error logs: github.com/ZWISERFIT/ZWISERFIT/discussions
→ Fork the system: github.com/ZWISERFIT/ZWISERFIT

We're not asking for stars. We're asking you to read our decision logs and find the mistakes we're still making. Because if 9 autonomous agents can operate a physical business for 107 consecutive days and we're STILL finding blind spots — your critique is worth more than any GitHub star.