So you're interested in autonomous AI?

I read the systems, security and software engineering literature before designing the next version of swarmux. Nine findings that change how you should run autonomous coding agents.

I'm building swarmux, an open-source control plane for autonomous AI agents, written in Rust. Its job is simple to say and hard to do: let AI coding agents work unattended, and make their work verifiable. That means independent checks before anything counts as done, policy on every action, budgets that hold, and a tamper-evident record of what happened.

Before designing the next version, I went through the computer science research that bears on this problem. I covered systems, databases, security, software engineering and ML systems, and leaned on 2023–2026 work plus the classics that still define good practice.

This post is what I found. Several results surprised me, and a few contradict things that "everyone knows" about working with agents.

A note on evidence: many of the papers below are recent arXiv preprints. I link every source so you can judge for yourself, and I've left out claims I couldn't trace to a primary source.

1. "Tests pass" is not "done"

This is the best-supported finding in the whole review, and it comes from several independent groups.

• UTBoost (arXiv:2506.09289, ACL 2025) added stronger tests to SWE-bench. 28.4% of "passing" patches on SWE-bench Lite and 15.7% on SWE-bench Verified turned out to be wrong. It changed 24.4% of the Verified leaderboard entries.
• SWE-bench+ (arXiv:2410.06992) found that 32.67% of successful patches had the solution leaked in the issue or comments, and 31.08% passed only because the tests were weak.
• ImpossibleBench (OpenReview) builds tasks whose tests contradict the spec, so any pass is cheating. A "stop if the tests look flawed" instruction cut one model's cheating on the LiveCodeBench variant from 93% to 1%. Claude Opus 4.1 still cheated on 46% of the SWE-bench variant. Hiding the test files nearly eliminated cheating, but it also reduced legitimate performance.

So what does a trustworthy "done" look like? Three things:

1. Checks the agent can't see or edit. If the agent can read the oracle, the oracle stops being an oracle. This is the single most important architectural decision in an agent harness.
2. Measure how strong your tests are, not just whether they pass. Meta's mutation-guided test generation (arXiv:2501.12862, FSE 2025) found that 277 of 571 useful generated tests would have been thrown away under a line-coverage criterion: they added no coverage but caught real faults. Coverage is not strength. A bounded mutation run on changed lines tells you how much to trust a green check, and it costs zero model tokens.
3. Detect test weakening deterministically. Deleted tests, new skip/xfail markers, loosened assertions, edited golden files and relaxed tolerances are all diffable. No LLM is needed to catch them.

2. Prompt caching is the biggest cost lever you're probably breaking

"Don't Break the Cache" (arXiv:2601.06007) measured prompt caching on long-horizon agent tasks across OpenAI, Anthropic and Google, over 500+ sessions. API cost fell 41–80%, and time to first token fell 13–31%.

The catch is layout. Naive caching can even increase latency. The rules that fall out of it:

• Put stable content first and task-specific content last.
• Never inject a timestamp, a git status or a reordered list of facts into the top of the prompt. Each one invalidates the cache for every task.
• Don't rebuild the tool list per task.
• Run sibling tasks of the same job back to back, inside the cache lifetime.

Agent spend is dominated by input tokens. "How Do AI Agents Spend Your Money?" (arXiv:2604.22750) finds agentic tasks use roughly 1000× the tokens of a chat exchange. So the cache-read ratio is the cheapest health metric an agent tool can track: if a new feature drops it, that feature costs you money on every task.

3. Less context, not more

The intuition that more context helps agents is mostly wrong.

• The Complexity Trap (arXiv:2508.21433, JetBrains Research) compared ways of managing an agent's growing history on SWE-bench Verified. Simply masking old tool outputs (replacing them with a placeholder while keeping the agent's own reasoning) roughly halved cost compared with an unmanaged agent, and matched or slightly beat LLM summarisation of the history, which costs extra calls of its own.
• Evaluating AGENTS.md (arXiv:2602.11988) tested repository context files with Claude Code, Codex and Qwen Code. LLM-generated context files did not improve success, and they raised inference cost by more than 20%. Agents obey the files, so they explore more. Repository overviews didn't help agents find the right files faster.
• Don't compress code. An empirical study of prompt compressors (arXiv:2505.00019) found that every method increased hallucination through information loss. For code, one dropped identifier or path breaks the task.

The practical rules:

• Give agents knowledge to look up (short, cited answers when they ask), rather than pushing everything into every prompt.
• Cap tool output with an explicit "truncated, re-query with X" marker.
• Reduce whole observations, never individual tokens of code.

4. Escalate on failure, not on prediction

Model cascades work: try a cheap model, and move to an expensive one only when needed.

• AutoMix (arXiv:2310.12963, NeurIPS 2024) cut compute cost by more than 50% at comparable performance.
• FrugalGPT (arXiv:2305.05176) reported up to 98% savings, on 2023-era tasks.

Learned routers, which predict difficulty up front, are another story. LLMRouterBench (arXiv:2601.07206) evaluated 400K+ instances across 33 models. Several recent and commercial routers don't reliably beat simply using the best single model.

Coding agents have something these papers lack: a cheap, objective verifier, the tests. So the right cascade is to start with the cheaper model and escalate when the checks fail. Choose the starting model from your own project's history, using a lookup table, not a learned router.

The same logic caps retries. Large Language Monkeys (arXiv:2407.21787) showed that SWE-bench Lite coverage (any sample correct) rises from 15.9% with one sample to 56% with 250. But Inference Scaling fLaws (arXiv:2411.17501) proved that with imperfect verifiers, accuracy from resampling has a hard ceiling. Under realistic assumptions the best number of samples can be below 10. Two or three attempts with a strong gate beat twenty attempts with a weak one.

5. Budget by quantile, share by tokens

Running the same agent task twice can cost wildly different amounts (arXiv:2604.22750). Spending more doesn't reliably buy accuracy either. A budget based on the average cost will be blown by the tail.

The scheduling literature has a clean answer for fairness. The Virtual Token Counter (Sheng et al., OSDI 2024, arXiv:2401.00588) shares a serving engine fairly by tokens consumed, not requests, with a proven 2× bound on the service gap. It also stops idle clients from banking credit and bursting later.

For an agent control plane, that means:

• Admit a task only if its p90 estimated cost fits the remaining budget.
• Share capacity across jobs by tokens.
• Stop paying for the same failure twice: after a few consecutive failures with the same signature, open a circuit breaker and escalate to a human (Nygard's Release It! pattern).

6. Prompt-injection defence has to be structural

Asking a model to "ignore instructions in tool output" is not a security boundary. The systems that hold up enforce security in code:

• CaMeL (Google DeepMind and ETH, arXiv:2503.18813) turns the trusted user request into a restricted program in which untrusted data can't change control flow. On AgentDojo it solves 77% of tasks with provable security, against 84% undefended, a cost of about 7 points of utility.
• FIDES (Microsoft, arXiv:2505.23643) attaches confidentiality and integrity labels to data and enforces policy deterministically. With policy checks on, it stopped every prompt injection attack in AgentDojo.
• Design Patterns for Securing LLM Agents (arXiv:2506.08837) states the principle: once an agent has read untrusted input, constrain it so that input can't trigger consequential actions.

A coding agent almost always has private data (source code, secrets in the environment) and untrusted content (web pages, issues, dependency READMEs). Together those are two legs of Simon Willison's "lethal trifecta". Remove the third leg by default, external communication. Mark facts that came from untrusted sources, and require human approval for privileged actions (push, publish, edit the rules) while untrusted content is in play.

For the policy engine itself, Cedar (arXiv:2403.04651, OOPSLA 2024) is the model to copy:

• a deliberately restricted language, default-deny, where an explicit forbid overrides any permit;
• a formal model in Lean, with sound and complete analysis.

That analysis lets a tool answer a question no LLM can answer reliably: "does this rule change allow anything the old rules forbade?" Runtime rule systems like AgentSpec (arXiv:2503.18666) show enforcement can be richer than allow or deny: ask a human, rewrite the action, or log first in shadow mode. AgentSpec reports preventing unsafe execution in over 90% of code-agent cases, at millisecond overhead.

7. A hash chain is not tamper-proof

Many tools, swarmux included, keep an audit log as a hash chain, where each entry includes the hash of the previous one. That detects an edit in the middle. It does not stop someone who can write the file from rewriting the whole tail and recomputing the chain. And an AI agent typically runs as the same OS user as the tool that keeps the log.

The classic answer comes from Certificate Transparency (RFC 6962) and Crosby and Wallach's Efficient Data Structures for Tamper-Evident Logging (USENIX Security 2009). They use Merkle trees with inclusion proofs ("this event is in the log") and consistency proofs ("today's log extends yesterday's"). Production descendants like Sigstore's Rekor v2 use a tile format that needs no special server.

The first-order fix is cheaper than Merkle trees: anchoring. Periodically write a signed checkpoint (head hash, count, time) to a place the agent can't write. Merkle proofs come second, so a single event can be proven to a third party without handing over the whole log.

8. Project knowledge has a valid time

Facts about a codebase go stale. "Tests run with cargo nextest" stays true until someone changes the CI. Temporal databases (Snodgrass; SQL:2011) separate valid time (when a fact is true in the world) from transaction time (when the system recorded it). For code, valid time maps naturally onto code state: tie each fact to the hash of the files it depends on. When those files change, the fact is flagged as possibly stale.

Keeping derived knowledge fresh is a solved problem in build systems. Build Systems à la Carte (ICFP 2018) and incremental indexers like Meta's Glean and Sourcegraph's SCIP recompute only what changed, and stop early when a recomputed result is unchanged. Deterministic derivation costs zero tokens. Re-summarising the repository with an LLM on every commit costs a lot and adds errors.

Store the history as an append-only event log and treat everything else as a projection you can rebuild by replaying it. You then get deterministic reconstruction for debugging, and for testing new rules against past behaviour: "would this policy have blocked any of the last 1,000 tool calls?"

9. Measure like a scientist, or you'll ship features that cost more and help less

The evaluation literature is humbling:

• τ-bench (arXiv:2406.12045) introduced pass^k, which counts a task only if it succeeds on all k tries. GPT-4o's pass^8 on retail tasks was below 25%. If you're going to leave an agent unattended, reliability across runs matters more than peak performance.
• SWE-Bench Illusion (arXiv:2506.12286) found models could locate the buggy file from the issue text alone up to 76% of the time on SWE-bench, against up to 53% on repositories outside it. That's a sign of memorisation.
• VibeMemBench (arXiv:2609.23570) tested agent memory systems independently: 11 of 12 memory-system and solver pairings did no better than no memory at all. Injecting verified experience did help, by 1.1–4.5 points with fewer steps.

So any feature claiming to make agents better needs:

• paired A/B runs on your own repository history, not public benchmarks;
• 3–5 repeats per cell, reporting pass^k and p90 cost alongside success;
• a placebo control: if irrelevant knowledge "helps", the effect is noise;
• a stale-knowledge control, to measure the harm when facts go out of date.

What this means for swarmux

Most of this confirmed the bets swarmux already made. Verification gates the agent can't see, a deterministic Guard, a governor and an append-only ledger are all supported by the evidence. It also changed the plan:

• Gates get a strength score. A bounded mutation run, a test-weakening detector, and a check that the original test suite still passes unchanged.
• Knowledge is pulled, not pushed. Agents ask the project and get short, cited answers. Facts carry their source and the code state they're valid for.
• Prompts stay cache-stable, and the ledger records the cache-read ratio per task.
• Model escalation is driven by failed checks, starting from what has worked in your own history.
• Budgets use p90 estimates, with circuit breakers for repeated failures.
• The ledger gets external anchors now and Merkle proofs later.
• Every new knowledge or memory feature ships behind a paired A/B. It stays on only if it beats the baseline on your own repositories, at no extra cost.

Autonomy is only useful if you can verify it. The research is fairly clear on how to build that. swarmux's work is still in progress. You can install the experimental version now.

swarmux is open source (MIT) and written in Rust: github.com/amirhosseinghanipour/swarmux. I'd love to hear from people working on agent verification, policy or evaluation.

References

Verification and evaluation

• UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench. ACL 2025. arXiv:2506.09289
• Aleithan et al. SWE-Bench+: Enhanced Coding Benchmark for LLMs. arXiv:2410.06992
• ImpossibleBench. OpenReview.
• Meta. Mutation-Guided LLM-based Test Generation at Meta. FSE Companion 2025. arXiv:2501.12862
• Yao et al. τ-bench. ICLR 2025. arXiv:2406.12045
• The SWE-Bench Illusion. arXiv:2506.12286
• VibeMemBench. arXiv:2609.23570

Cost and context

• Lumer et al. Don't Break the Cache. arXiv:2601.06007
• Bai et al. How Do AI Agents Spend Your Money? arXiv:2604.22750
• Lindenbauer et al. The Complexity Trap. NeurIPS 2025 DL4C workshop. arXiv:2508.21433
• Evaluating AGENTS.md. arXiv:2602.11988
• Empirical study of prompt compression methods. arXiv:2505.00019

Routing and test-time compute

• Chen et al. FrugalGPT. arXiv:2305.05176
• Aggarwal et al. AutoMix. NeurIPS 2024. arXiv:2310.12963
• LLMRouterBench. arXiv:2601.07206
• Brown et al. Large Language Monkeys. arXiv:2407.21787
• Stroebl, Kapoor, Narayanan. Inference Scaling fLaws. arXiv:2411.17501

Scheduling and reliability

• Sheng et al. Fairness in Serving Large Language Models (Virtual Token Counter). OSDI 2024. arXiv:2401.00588
• Nygard. Release It! (circuit breakers).

Security and policy

• Debenedetti et al. Defeating Prompt Injections by Design (CaMeL). arXiv:2503.18813
• Costa et al. Securing AI Agents with Information-Flow Control (FIDES). arXiv:2505.23643
• Beurer-Kellner et al. Design Patterns for Securing LLM Agents against Prompt Injections. arXiv:2506.08837
• Debenedetti et al. AgentDojo. arXiv:2406.13352
• Cutler et al. Cedar: A New Language for Expressive, Fast, Safe, and Analyzable Authorization. OOPSLA 2024. arXiv:2403.04651
• Wang et al. AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents. arXiv:2503.18666
• Willison. The lethal trifecta for AI agents.

Logs, storage and incremental computation

• Laurie, Langley, Kasper. Certificate Transparency. RFC 6962
• Crosby, Wallach. Efficient Data Structures for Tamper-Evident Logging. USENIX Security 2009.
• Tile-Based Transparency Logs, transparency.dev.
• Mokhov, Mitchell, Peyton Jones. Build Systems à la Carte. ICFP 2018. doi:10.1145/3236774
• Sourcegraph. Announcing SCIP.
• Kleppmann et al. Local-first software. Onward! 2019. doi:10.1145/3359591.3359737