Dev.to AI ๐Ÿค– Ai ๐Ÿ‘ 0 ๐Ÿ“– 3 min read

LLM agent token cost: why a kept chat grows with the square of the steps, and the measured fix (prompt caching counted)

An agent that keeps its whole chat resends everything on every step, so the tokens it sends grow with the square of the number of steps. Holding the state on the host and showing the model the exact state keeps the promp

An agent that keeps its whole chat resends everything on every step, so the tokens it sends grow with the square of the number of steps. Holding the state on the host and showing the model the exact state keeps the prompt the size of the state. Measured on Gemini 3.8 Flash: 85 times fewer prompt tokens a step and 76 times fewer in all over 400 turns, and 4.5 times lower cost per step that ends with the right state.

Naming note: "AXI" here is the AXI host-side runtime for LLM agents, an architecture by Rajnish Choubey. It is not the ARM AMBA AXI chip-bus protocol.

Why does the token bill of a chat-based agent grow so fast?

If step t sends a prompt of a + bยทt tokens (the fixed part plus everything said so far), n steps send nยทa + bยทn(n+1)/2 tokens in all. The second term grows with n squared: twice the turns, about four times the tokens, at the growth rate measured in Test 23. A host that holds the state sends a prompt the size of the state, so n steps send about n times that size: a straight line.

How many tokens does that make, measured?

All runs: Gemini 3.8 Flash, each job run once with the chat kept and once with the host holding the state.

Job Prompt tokens a step, chat kept With a host-held state Tokens sent in all
400 turns of plain, ambiguous sentences (Test 23) 99,809 on average 1,177 (85 times fewer) 39.9 million against 0.53 million (76 times fewer)
Long job with heavy turns (Test 22) 476,835 on average, up to 949,591 2,824 (169 times fewer) 217.5 million (457 turns) against 4.6 million (1,500 steps)
Four ARC-AGI-3 games (Test 24) n/a n/a 25.7 million against 1.95 million (13.2 times fewer)

Doesn't prompt caching fix this?

Caching is the first thing to switch on, and it helps: the service served 97 percent of the kept chat's prompt tokens from its cache. But sent tokens and computed tokens are different bills, so here are both.

  • Computed tokens per exact step, counted after the cache: the kept chat needed 2.3 times as many in Test 23 (whole job) and 1.7 times as many in Test 22 past 500,000 tokens. Caching narrows the gap from 85 times to about 2 times. It does not close it.
  • Open models with prefix caching switched on (vLLM, one NVIDIA L4): the kept chat computed 1.24 to 4.11 times the prompt tokens per step across 11 models, and on a like-for-like 7B pair 1.78 times the tokens and 1.51 times the seconds per step.
  • Caching does nothing for exactness. A cached wrong state is still a wrong state, and that is where most of the cost sits (next section).

What does a correct step cost?

Cost per token hides the real unit. Over the first 400 requests of a 4,000-request Gemini run, at $0.75 in and $3.75 out per million tokens with thinking counted as output:

Way of using the model's replies Steps with the right state (of 400) Spent Cost of one step with the right state
Host layer: check, complete, commit 400 $0.47 $0.0012
Replies applied as written 85 $0.45 $0.0052 (4.5 times as much)
A gate that only refuses bad replies 84 $0.48 $0.0057 (4.9 times as much)

The spend was about the same. What differed was how many steps ended with the right state.

What about energy and capacity?

On a 14B model (Test 44), a step cost 21.8 joules with the exact state shown, 39.4 to 48.0 joules with summaries, a window or retrieval, and 123.1 joules with the whole chat kept. On one NVIDIA L4 with a 7B model (Test 38), 256 sessions ran with every step exact: 105,230 exact steps per GPU hour at 2.458 joules per exact step.

Check the figures yourself

No key, no GPU. This replays the saved replies of the 4,000-request run with the repository's own code and prints the 4,000, 3,954, 85 and 4.5 times figures:

git clone https://github.com/gaurshalendra2412/axi-runtime
cd axi-runtime/python
python3 quickstart_replay.py

Scope

Gemini 3.8 Flash for the sent-token figures; open models on vLLM 0.31.0 and one NVIDIA L4, greedy decoding, no retry; synthetic service graphs; the cost rows are arithmetic added after the results, from the saved runs, not a pre-registered line. Whether the bill follows sent or computed tokens depends on how a company prices cached tokens; the repository does not price them. Joules are the card's board power read during the run. Test pages: github.com/gaurshalendra2412/axi-runtime. White paper: doi.org/10.5281/zenodo.23958737.

Architecture by Rajnish Choubey. Tests built and run by Shalendra Gaur.

๐Ÿ“ฐ Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes โ€” full credit and traffic to the original publisher.