Why Multi-Agent Systems Get More Expensive — Routing, Handoffs, and Quadratic Cost
Why Multi-Agent Systems Get More Expensive — Routing, Handoffs, and Quadratic Cost A five-step multi-agent workflow. Not one line of business logic changed — only the decision of who goes next was handed to a model. The
Why Multi-Agent Systems Get More Expensive — Routing, Handoffs, and Quadratic Cost
A five-step multi-agent workflow. Not one line of business logic changed — only the decision of who goes next was handed to a model. The overhead went from 750 tokens to 18,000 tokens (750 = five hops × 150-token summaries).
Nobody wrote a bug. The extra 17,250 tokens come from one thing: everything each agent said was re-read by every agent after it.
This article does the math, then gives the four cuts that bring it back to linear.
What matters: two hidden cost centers — routing inputs accumulating at O(N²) and handoff history re-read at O(N²); the three context strategies and their trade-offs; four cuts that bring both back to linear, with before/after numbers; why the stop condition must be chain-scoped; and why guard state belongs to a run, not to the process.
Project context:
lc-agentsin langchainrust — four orchestrators, a handoff protocol, sub-agent isolation, and three guardrails.
1. Building Intuition: How Many Times Is the Same History Read?
One concrete task runs through the whole article:
Split a 20-page quarterly report across five agents: planning, retrieval, financial analysis, risk identification, and drafting.
Each agent produces about 800 tokens; a summary is about 150 tokens; the goal plus the candidate list is about 400 tokens.
A single agent is simple:
Single agent:
context ──▶ model ──▶ output
(one context, read once; history grows linearly)
Split into multiple agents, and it looks like this:
Multi-agent (full-history handoffs):
Agent A ──▶ output 800t ────────────────────────┐
Agent B ──▶ reads all of A ──▶ output 800t ─────┤
Agent C ──▶ reads all of A+B ──▶ output 800t ───┤
Agent D ──▶ reads all of A+B+C ──▶ ...
The problem is visible in the diagram: every hop re-reads everything everyone said before.
And if who goes next is also decided by a model, every round pays for one more full model call — a call whose input also contains all prior outputs.
The same history is read once by the routing model, then read again by the next worker.
Core insight:
The cost explosion in multi-agent systems is not two independent quadratic problems. It is one quadratic term paid twice: routing re-reads the history every round, handoffs re-read the context every hop, and both are reading the same thing.
Swapping in a cheaper model shrinks the unit price; it does not remove the quadratic term.
2. Routing: One Full Call per Round, Inputs Grow as N²
2.1 How the input grows
Round 1: [ goal G | candidates W | output O1 ] → decide next
Round 2: [ goal G | candidates W | output O1 O2 ] → decide next
Round 3: [ goal G | candidates W | output O1 O2 O3 ] → decide next
Round 4: [ goal G | candidates W | output O1 O2 O3 O4 ] → decide next
What gets appended each round is the previous worker's full output.
The input at round k is:
tokens(k) = G + W + (k-1) × O
G = task goal (constant)
W = candidate list (constant)
O = output per worker (~800 tokens)
k = current round
Ignore the summation for a moment. In plain terms: round 1 carries 0 outputs, round 2 carries 1, round 3 carries 2… Over an N-round task, it carries
0 + 1 + 2 + ... + (N-1) = N × (N-1) / 2 outputs
That is where the quadratic term comes from.
This assumes the routing model reads only the goal, the candidate list, and prior outputs — not the source material. That assumption holds in practice: routing does not need to understand the report, only what the previous step accomplished. It is also precisely why fixed routing is worth doing first: this entire block should never have been read by a model.
2.2 Plugging in numbers
A note on accounting: throughout this article, "routing cost" means total input including the constant terms; the pure quadratic component is labeled separately.
| Round | Routing input tokens | Increase over previous |
|---|---|---|
| 1 | 400 | — |
| 2 | 1,200 | 800 |
| 3 | 2,000 | 800 |
| 4 | 2,800 | 800 |
| 5 | 3,600 | 800 |
| Total | 10,000 | — |
With fixed routing, this cost is zero.
When the number of rounds doubles, separate the two components:
N=5 : total 10,000 = constants 2,000 + quadratic 8,000
N=10 : total 40,000 = constants 4,000 + quadratic 36,000
Total input: 4.0× (10,000 → 40,000)
Pure quadratic: 4.5× (8,000 → 36,000)
Latency follows the same shape. Each routing call takes one to three seconds; five rounds add five to fifteen seconds — and the user's total wait may only be twenty seconds.
2.3 Reproducibility is more expensive than tokens
Run the same input twice and the model may route differently. Temperature zero does not guarantee stability: context length, model version, and sampling implementation all matter.
| Consequence | What it looks like |
|---|---|
| No regression testing | A passing case fails; you cannot tell whether the route changed, a worker's output changed, or sampling changed |
| Expensive attribution | The error shows up as "wrong answer", but the divergence happened two or three rounds earlier |
| No capacity planning | Call counts per request are variable; the cost ceiling can only be estimated from the worst path |
Fixed routing is deterministic on all three: call counts are countable, paths are testable, and the cost ceiling equals the maximum number of rounds.
2.4 When model routing is justified
When the next step genuinely depends on what the previous step produced, and that dependency cannot be expressed as a rule.
Open-ended research, cross-system troubleshooting, and scenarios that switch tool families based on intermediate results qualify. If the task can be drawn as a flowchart, or each step's input is a fixed projection of the previous output, letting a model choose the next step is pure waste — and it turns a deterministic problem into a probabilistic one.
2.5 Classifying orchestrators by who decides
| Orchestrator | Who decides | Extra model calls | Reproducible | Fits |
|---|---|---|---|---|
| Sequential pipeline | Code | 0 | Yes | Fixed steps; previous output is next input |
| Fan-out / fan-in | Code | 0 | Yes | N independent views, aggregated at the end |
| Review loop | Verdict | 1 per round | Verdict is constrained | Clear acceptance criteria; rework cheaper than error |
| Supervisor | Model | 1 per round | No | Path truly depends on intermediate results |
langchainrust exposes all four behind the same Orchestrator trait. The interface is identical; the costs are not. The first two add no model calls. The review loop's extra call produces something concrete — a pass/fail verdict plus revision feedback. Only the supervisor's extra cost buys a single line: "who goes next".
What to do: if you can draw the flowchart, put routing in code; only use a model when you cannot, and enumerate the candidates (see §7).
3. Handoffs: Every Hop Re-reads the History
3.1 Why full-history handoffs are quadratic
Full-history handoff:
A ──[ history + all of A's process ]──▶ B ──[ history + A + B's process ]──▶ C
Summary handoff:
A ──[ goal + result + 150t summary ]──▶ B ──[ goal + result + 150t summary ]──▶ C
The bottom variant is linear: every hop carries a fixed 150 tokens.
The top variant: agent 1 reads 0 outputs, agent 2 reads 1, agent 3 reads 2… again N(N-1)/2.
With O=800 and N=10, the handoff total is 36,000 tokens. With summaries (S=150), it is 10 × 150 = 1,500.
One number here looks like a coincidence: the handoff quadratic (36,000) equals the routing quadratic (36,000). They read the same history — the same output is read once by the routing model and again by the next worker. This is one quadratic term paid twice, not two separate problems.
3.2 Saving tokens is a side effect
| Strategy | Handoff tokens | Information fidelity | Prior-failure contamination | Attributability |
|---|---|---|---|---|
| Full history | O(N²) | Highest | Severe | Hard |
| Summary handoff | O(N) | Depends on summary quality | Controllable | Medium |
| Empty context | O(N) | Task text only | None | Easy |
Contamination: the next agent sees that a predecessor failed in some direction and avoids it — even when that direction is right for its own task. This is not model sentiment; it is a directional negative anchor left in the context.
Attribution: when every agent sees the same pile of information, you cannot tell which link's judgment was wrong. A summary handoff reduces each hop's input and output to structured fields, which gives attribution somewhere to land.
3.3 A sub-agent is a third thing
| Orchestrator | Sub-agent | |
|---|---|---|
| Message history | Shared | Empty |
| Input | Goal + accumulated output | Task text |
| Output | String, feeds the next round | {status, summary} |
| What changes | Role | Information boundary |
| Fits | Tasks that share a world state | One-off, outsourceable, verifiable subtasks |
The parent never sees the sub-agent's tool calls, retries, or deliberation. Nesting defaults to two levels — a third level almost never returns more than it costs.
What to do: a handoff carries only "original request + current result + what has already been tried"; summaries are produced upstream against a fixed schema (see §7).
4. Stopping: Why Timeouts and Budgets Are Not Enough
4.1 What each stop condition covers
Timeout: |--2s--|--2s--|--2s--|--2s--| no hop exceeds it → never fires
Budget: ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ stops when money runs out → garbage already produced
Depth: 1 → 2 → 3 → 4 → 5 → 6 → 7 → 8 → 9 → 10 ✕ hard stop
| Mechanism | Governs | Misses | Typical failure |
|---|---|---|---|
| Timeout | Duration of one call | A chain of fast but useless delegations | Every hop takes 0.2s; ten hops stay under the limit |
| Budget | Money | Garbage produced before the budget runs out | Stops only when exhausted, after a dozen contradictory conclusions |
| Depth | Structure: how often the task changed hands | Self-spin inside one hop: an agent loops its own tool calls a hundred times while depth stays at 1 | Needs a separate tool-call cap |
Only depth governs structure — and it is not sufficient on its own. Depth is a guard on the chain; it cannot see inside a link. An agent that spins within its own turn is invisible to it. That case needs a tool-call cap or a run-level budget (see §5).
lc-agents defaults to ten handoffs. The primary agent counts as the first node of the chain, so "hand it back to the primary" is treated as a cycle rather than a legitimate exit.
4.2 Cycle detection granularity
A depth cap guarantees finiteness, not non-repetition:
A → B → A → B → A
Depth never exceeded (5 hops), but A ran 3 times and B ran 2 times
The right granularity is the chain of this run:
A → B → C → A
↑
target A is already in the chain → reject
Wrong #1: a global visited set
A took one job today → no future task may ever route to A again
Wrong #2: only check the previous hop
In A → B → C → A, C's previous hop is B ≠ A → allowed (missed)
4.3 After the guard fires, can the model route around it?
Treated as a tool failure:
model ─▶ handoff fails ─▶ fed back as an observation ─▶ model rephrases and retries ─┐
▲ │
└────────────────────────────────────────────────────────────────────────────────────┘
Treated as a refusal:
model ─▶ handoff rejected ─▶ the entire task ends immediately (no next turn)
The difference is not in the error message; it is in what the executor does next. And the semantics must hold on all three paths — synchronous, parallel batch, and streaming. Miss one, and the guard silently degrades into a warning there.
What to do: compute depth over the whole chain, make the guard a hard abort, and test all three execution paths (see §7).
5. Isolation: Who Owns the Guard State?
5.1 A concurrency incident, end to end
Symptom: a perfectly normal "triage → orders" handoff was rejected during peak concurrency with handoff cycle detected. The same input and the same conversation passed a hundred times in isolation.
The log had exactly one line. The guard only records on rejection; the success path never records which run a chain belongs to. You see "someone was judged cyclic" but not "whose chain was judged".
That is also why replay does not reproduce it: the bug needs the time windows of two runs to overlap, and a single request can never create that interleaving. It is not that load testing was skipped — the agents in load tests are fake, a handoff takes tens of microseconds, and the window is too narrow to collide. In production each hop calls a model and takes seconds, widening the window by five orders of magnitude.
Locating it took three steps:
- Log at both handoff entry and exit with a run id, then interleave both runs' records by timestamp;
- Inject tens of milliseconds of delay around handoffs to widen the window until it reproduces reliably;
- Observe the push order of the two chains — they point at the same memory.
The original implementation was textbook: the visited list was a field on the manager, guarded by a lock. From a memory-safety perspective the code had no data races and could not crash. The problem is that a lock guarantees only one writer at a time; it cannot guarantee the state belongs to this task.
The fix moves the chain into per-run scope:
HANDOFF_CHAIN.scope(RefCell::new(vec![primary]), async move {
executor.invoke(input).await
})
Created when the task starts, destroyed when it ends. Nested handoffs push and pop inside the same scope; other tasks cannot see it.
There is one more measure against recurrence: load tests must put real model calls (or an equivalent injected delay) on the handoff path. No amount of synthetic QPS will collide two chains whose windows are microseconds wide, and this class of bug will live until production.
5.2 Locks and scopes solve different problems
| Mechanism | Guarantees | When it fails |
|---|---|---|
| Mutex | No data race | It does not crash; results are quietly wrong |
| Run scope | A task sees only its own state | — |
The test is simple: if two users hit the system at once, should these two pieces of state see each other? If not, do not make it a process-level field. "Visited set", "steps used", "temporary budget" all belong here.
5.3 The fail-open the fix leaves behind
Outside a run scope, the chain does not exist and both guards are skipped.
This is a deliberate trade-off: an out-of-band call should not fail just because it lacks context. But a silent fail-open and no guard at all read as the same line in an incident report.
So this branch must be counted and exported as a metric, not merely documented. A non-zero value means traffic is bypassing the normal entry point and the guard's safety property is failing there. Documentation expires; metrics do not.
6. Boundaries: When Not to Use Multi-Agent
Splitting has a fixed cost: every handoff is one information loss, one latency penalty, and one new failure point. It pays only when these things genuinely differ:
- The tool sets are completely different, and mixing them makes the model choose wrong;
- The permissions differ — one can read production, one can only query a knowledge base;
- The models differ — the expensive one decides, the cheap one executes;
- Context must be isolated — exploratory subtasks must not contaminate the main line.
This is not an isolated judgment. Two camps in the industry frame the debate.
Anthropic's How we built our multi-agent research system reports that agent-style interactions consume about 4× the tokens of ordinary chat, and multi-agent systems about 15×. Their conclusion: multi-agent fits high-value tasks whose subtasks can run in parallel, and most coding tasks have far less parallelism than research.
Cognition's Don't Build Multi-Agents goes further, using a Flappy Bird reproduction as a counterexample: parallel sub-agents each receive a partial context, and their outputs do not fit together. That is the most direct footnote to the "information fidelity" column in §3.2 — splitting not only costs more, it can make the pieces disagree.
The two are not contradictory: Anthropic says it is worth it for high-value, parallelizable work; Cognition says do not split when context is tightly coupled. Together: parallelism comes from the structure of the task, not from how many agents you spawn.
7. Four Cuts to Bring the Quadratic Term Back to Linear
Who goes next?
│
├─ Can you draw the flowchart up front? ── yes ──▶ sequential / fan-out (0 routing calls)
│
└─ no
│
├─ Can it be expressed as a rule? ── yes ──▶ rule-based routing (keyword / field / return code)
│
└─ no ──▶ Supervisor (1 call per round)
│
├─ candidates must be enumerated, never free text
├─ FINISH is reserved, registration forbidden
├─ parse failure is a hard failure, never a default guess
└─ round cap + per-round accounting
Why FINISH must be reserved: if it can be registered as an ordinary agent, the model can treat "delegate to the end" as a normal call — the termination semantics get hijacked into a business action, and the logs show nothing unusual.
The four cuts map to the previous four chapters; here are just the actions and their costs:
| Cut | Action | What it removes | Cost | Detail |
|---|---|---|---|---|
| 1 | Put routing in code | The entire routing quadratic | Adaptability to unknown paths | 2.4 / 2.5 |
| 2 | Hand off only three things: original request, current result, what has been tried | The entire handoff quadratic | Summary quality determines fidelity | 3.1 / 3.2 |
| 3 | Sub-agents start with empty context and return only {status, summary}
|
Re-reading intermediate work | No shared world state | 3.3 |
| 4 | Depth over the whole chain, abort hard when tripped | Unbounded delegation and loops | Depth cannot see inside one hop | 4.1–4.3 |
If you truly must let a model route, three ways to reduce the bill:
- Return structured results only — name, a summary, an artifact list; never the full text;
- Enumerate candidates, reserve the termination token, fail hard on parse errors — no defaults;
- Cap rounds and account per round — make "routing cost / total task cost" a metric.
8. The Ledger: Before and After, Including the Input Side
8.1 A three-column ledger
Everything above counted only the output side. Plug in the 20-page report from the opening and the ledger changes shape.
This table counts orchestration-layer changes only, and assumes each agent's data access is unchanged (each still reads the full report, about 15,000 tokens). The four cuts change routing and handoffs; they do not change how an agent obtains material. That is exactly why the input side does not move — and why it needs separate treatment:
| Cost item | Before (N=5) | After (N=5) | Does the cut touch it? |
|---|---|---|---|
| Input side (reading the report) | 75,000 | 75,000 | No (needs chunked retrieval / on-demand loading) |
| Routing | 10,000 | 0 | Yes |
| Handoffs | 8,000 | 750 | Yes |
| Total | 93,000 | 75,750 | Saves 18.5% |
N=10:
Before = 150,000 (input) + 40,000 (routing) + 36,000 (handoffs) = 226,000
After = 150,000 (input) + 0 + 1,500 = 151,500
Saves 33%
The conclusion needs to be stated plainly: summary handoffs save the output side; the bulk of the input side needs a different knife.
If every agent reads all 20 pages, that line alone is 75,000 tokens — four times routing and handoffs combined.
The actionable version: have the planning agent produce a retrieval manifest first — which sections each downstream agent needs — and let the retrieval layer return excerpts rather than the full text. That takes the input side from O(N × full text) to O(what is actually needed). It is not the same knife as the four cuts, but the two can be used together. This is a RAG and context-engineering problem, not an orchestration problem.
Mix the two knives and you get numbers like "a 24× saving" that only hold on the output side.
So does that make the "quadratic paid twice" point unimportant?
The opposite. The 75,000 on the input side is the inherent cost of the task — if you need to read the report, nobody can save you from it; you can only turn "read everything" into "read what is needed".
The 18,000 on the orchestration side exists purely because you split the task across agents. It delivers zero business value, and four cuts can take it to zero.
In other words: the input side is paid once, but it is a hard bill; the orchestration side is paid twice, and it can be paid not at all. That is the real weight of the quadratic term.
8.2 What caching saves, and what it cannot
One more layer of reality: input tokens are not always billed at full price. With fixed routing and a stable prompt prefix, cached input is often an order of magnitude cheaper.
This corrects the ledger above; it does not overturn it.
- What caching saves: the unit price of the input side. Fixed routing, a stable system prompt, and an unchanged tool list all raise the hit rate.
- What it cannot save: latency and call-count variance. The quadratic term adds rounds, and every extra round is one more serial model call. Cheaper tokens do not remove that time.
- The multi-agent-specific problem: each agent's context prefix differs, so KV cache cannot be reused across agents. KVCOMM (arXiv:2510.12872) examines exactly this: the core overhead of multi-agent systems is repeated processing of overlapping context, and prefix divergence keeps reuse low. Their approach reaches over 70% reuse and up to 7.8× speedup.
So "fixed routing first" is not only about tokens; it is also serving cache hit rate. In engineering terms, those are the same action. That is why handing routing to a model costs more than the invoice suggests.
8.3 Converting tokens into money and latency
"18,000 tokens" means nothing to a reader. A monthly bill does.
One task adds 17,250 tokens (18,000 − 750). At 100,000 tasks per month, that is 1.725 billion tokens.
At full price (no cache hits), at roughly $0.15–$5 per million input tokens, that is about $260–$8,600 per month.
With the cache hits from §8.2 (one-tenth the input price), the same volume is about $26–$860 per month.
The gap looks like an order of magnitude has been erased by caching. But the cost described in §8.2 has not moved at all:
- The five extra routing rounds are five serial model calls, one to three seconds each. Cheaper tokens do not make that time disappear;
- The number of routing calls per request is variable; P99 is set by the deepest path, and caching does not change that distribution.
Caching saves money. The quadratic term charges time and determinism. They are not on the same account.
9. Metrics: How to Verify the Curve Actually Flattened
Do not just watch the bill. These five metrics can be instrumented directly, with thresholds included:
| Metric | Collection point | Expected value under linear design | Alert condition |
|---|---|---|---|
route.input.tokens.delta_per_round |
Routing entry | ≈ 0 | Increasing for 3 consecutive rounds |
route.calls.per_task |
Per-round routing count | 0 (fixed routing) | Mean > 1 with no business justification |
route.calls.per_task.p99 / p50 |
Same | 1.0 | Ratio > 1.5 |
handoff.guard.skipped_no_scope |
The fail-open branch in §5.3 | 0 | Any non-zero value |
handoff.summary.tokens |
Handoff exit | ≤ threshold (e.g. 200) | Above threshold means summaries are inflating |
The last two deserve a note. A non-zero skipped_no_scope means traffic is bypassing the normal entry and the guard is failing there. A summary.tokens above threshold means the summary is quietly degrading back into "the full text" — which turns "summary quality determines fidelity" from §3.2 into an observable quantity.
One final reminder: cost reduction must not come at the expense of information. A bad summary does not fail on the spot; it surfaces downstream as a wrong answer, which is harder to notice than a larger bill. Before shipping summary handoffs, run the same question set against the full-history version and compare.
10. How These Decisions Look in langchainrust
| Decision in this article | Where it lives in lc-agents
|
|---|---|
| Fixed routing first |
SequentialPipeline / FanOutFanIn make no routing model calls |
| Model routing only when necessary |
Supervisor: one call per round; JSON first, delimiters as fallback, hard failure when parsing fails |
| Summary handoffs |
Handoff carries only the target, the task, and original_request / current_result / conversation_summary
|
| Sub-agent isolation |
SubagentTool: empty context, returns only {status, summary}, nests two levels by default |
| Chain-scoped stopping |
max_handoff_depth defaults to 10; the primary agent is the chain's first node |
| Refusal instead of tool failure | Handoff guards return ToolError::ControlAbort; sync, parallel, and streaming paths each have tests |
| Per-run isolation |
HANDOFF_CHAIN lives in task-local scope, one per run()
|
The trade-off in langchainrust can be stated in one line: give determinism to code, give uncertainty to the model, and keep a narrow interface between them.
11. Summary
The cost, latency, and irreproducibility of multi-agent systems all trace back to one structure: context is re-read across N agents. Routing reads it once; handoffs read it again. The ways to pull it back to linear are the three above — fixed routing, summary handoffs, and a depth cap — plus empty-context sub-agents.
Check these before shipping:
- [ ] Is routing written into code? Use a model only when you cannot, and enumerate candidates
- [ ] Are summaries produced upstream against a fixed schema?
- [ ] Do summaries include "what has been tried, do not retry"?
- [ ] Is the depth cap computed over the whole chain?
- [ ] Does the guard abort hard, consistently on sync / parallel / streaming paths?
- [ ] Is run state isolated per task (task-local)?
- [ ] Does any worker failure fail the whole fan-out?
- [ ] Is
FINISHreserved and unregisterable?
If any box is empty, the tokens you saved will come back in a form that is harder to diagnose.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.