How to Predict Each User's Token Speed on a Shared GPU

Engineer's view. This is capacity planning by request count on a link that is limited by bytes. A model server reads its weights once per step for all users. It also reads each user's stored conversation every step. A few long agent sessions use up that read budget, and every chat on the GPU streams slower.

TL;DR. Every user on a GPU waits for the same shared step, and that step gets longer with the total conversation everyone holds. Plan capacity in tokens in flight, not in users.

Explain Like I'm 5

A librarian answers everyone's letters, one word at a time. Each round, she rereads her big rulebook once. That takes the same time whether five people are waiting or fifty.

Then she rereads each person's letter so far, so she knows what word comes next. Short letters are quick. One person with a hundred-page letter makes every round slower, for everyone.

So the number of people is not what slows her down. The number of pages she has to reread is.

The Problem

You have shipped this bug before, in a system with no model in it.

You sized an API gateway by requests per second. The load test sent small payloads, and it passed with room to spare. Then one customer started uploading large files. The request count barely moved, but the network link filled up. Every request slowed, including the tiny ones. The fix was to meter bytes, not requests.

A model server has the same bug. Say you load-tested Qwen3 8B on one H100 GPU with 40 chat users. Each chat held about 2,000 tokens, and each user saw about 71 tokens per second. That passed.

Then your team shipped an agent feature. Its sessions hold 100,000 tokens of code and tool output. Two of them join the 40 chats. The dashboard shows 42 running requests, only two more than the test. Every user now streams at about 35 tokens per second. Nothing crashed, and GPU memory still has room.

The GPU did not run out of memory. It ran out of reads per step. Your capacity plan counted users, and the cost is in tokens.

The fix: plan capacity in tokens in flight across all users, and admit each request against a speed budget, not a memory budget.

The Fix: Budget the Tokens in Flight, Not the Users

Why is one more user nearly free?

A server writes answers one decode step at a time. Each step makes one token for every running request at once.

To make that token, the GPU reads every weight of the model from its memory. For Qwen3 8B in 16-bit numbers that is 16.4 GB. The read speed, memory bandwidth, caps how fast this goes. An H100 peaks at 3.35 TB per second, and real decoding reaches about 60% of that. So one step takes about 8 milliseconds, and a lone user sees about 120 tokens per second.

The weights are read once per step for the whole batch, so a new user adds no second read. That is why batching looks free.

So what does each user cost?

Each request also keeps a KV cache, the stored keys and values for every token it has seen so far. The model rereads all of it each step to decide the next token. For Qwen3 8B that is 144 KiB per token.

A 2,000-token chat holds 0.29 GB. A 100,000-token agent session holds 14.7 GB, almost the model's size.

Doesn't memory run out first?

At 90% of an 80 GB card, the caches can hold about 377,000 tokens. To keep every user at 50 tokens per second, the step must stay under 20 milliseconds. That allows only about 134,000 tokens in flight. The server accepts the work and streams it slowly. Memory sizing is in How to Work Out How Much GPU Memory One Request Really Needs.

What about a new prompt arriving?

Reading a new prompt, called prefill, is compute work. Servers mix a chunk of it into the same steps. A step then lasts as long as the slower of its two jobs. With a 2,048-token chunk in each step, the 40 chats drop to about 15 tokens per second until the prompt is in.

Does this model hold up?

I fitted it to one published number, then tried to break it. MosaicML measured MPT-7B on one A100 at batch sizes from 1 to 64.

Counting only the weights overstated throughput by 4.7 times. Adding the caches and the prompt work came within 6%.

What This Means for You

When this matters. You run your own model server, such as vLLM, SGLang or TGI, or you rent dedicated capacity. It matters most when long and short requests share a GPU. Agent sessions, code review over a whole repository and long document chats are the long ones. If you only call a hosted API, the provider does this sizing. You see its result as token speed that varies by time of day.

How it affects you. Your load test probably used one prompt length. A test with 40 short chats tells you nothing about 40 chats plus two agent sessions. Your memory dashboard will show headroom while users complain about slow streaming. Adding users who send short prompts is cheap. Adding one long session is not.

What to do about it. First, with no setup: multiply your longest common context by 144 KiB, or your own model's per-token figure. Compare it with your model's size. If one session's cache is near the size of the model, one session costs as much as everyone else.

Next, measure one number on your server: tokens per second for one user alone. From it you get the effective bandwidth, and from that your token budget at your speed target.

Then admit requests against that budget in your gateway. Route long sessions to a separate pool when they would push the short ones under target. The code is in ## Implementing It.

New to KV caches? See How Agent Context Gets Re-Read Every Turn.

Implementing It

The change.

Three roles change: whoever sizes capacity, the gateway in front of the server, and the dashboard that checks it.

Role 1, the capacity planner. Start from the model's config, not a guess. The per-token cache size is two (keys and values), times layers, times key-value heads, times head size, times bytes per number:

KV_TOK = 2 * 36 * 8 * 128 * 2                  # Qwen3 8B: 147,456 bytes per token
WEIGHT_BYTES = 8.2e9 * 2

def measure_effective_bandwidth(inter_token_latency_s, context_tokens):
    # one stream on an idle server; 60% of peak is typical, but measure yours
    return (WEIGHT_BYTES + KV_TOK * context_tokens) / inter_token_latency_s

def max_tokens_in_flight(target_tps, bw, headroom=0.9):
    step = headroom / target_tps               # keep 10% of each step for prompt chunks
    return int((step * bw - WEIGHT_BYTES) / KV_TOK)

Run one request alone, read its inter-token latency, and pass both numbers in. Run it at a short and a long context. If the two bandwidth figures disagree by more than 15%, the model does not fit your server. Check for prefix caching or quantized caches first. vLLM also prints its KV cache size in tokens at startup. Divide the cache memory by it to check your per-token figure.

Role 2, the gateway. Admit by tokens, not by count. vLLM's --max-num-seqs caps requests, which is the wrong unit here. Keep that cap as a safety limit and add this check in front:

class ContextBudget:
    def __init__(self, max_tokens):
        self.max_tokens, self.in_flight = max_tokens, 0

    def try_admit(self, context_tokens):
        if self.in_flight + context_tokens > self.max_tokens:
            return False                       # queue it, or send it to the long-context pool
        self.in_flight += context_tokens
        return True

    def release(self, context_tokens):
        self.in_flight -= context_tokens

Count the prompt plus the output limit at admission, and release on completion. If you skip this, nothing errors. The batch grows past the budget, and the latency median climbs with no line in any log. A rejected long request goes to a second pool of GPUs that serves only long sessions. There, slower streaming is expected.

Role 3, the dashboard. vLLM exports vllm:inter_token_latency_seconds, vllm:num_requests_running and vllm:kv_cache_usage_perc. Plot the predicted step time next to the measured one. Cache usage times cache capacity gives the tokens in flight. Alert when the predicted step time stays over budget for five minutes. Do not alert on request count, which is the number that misled you.

How you know it worked.

The prediction should track the median of vllm:inter_token_latency_seconds within about 15% as load changes. At the budget, median user speed should sit at your target, whatever the request count. If speed falls while kv_cache_usage_perc stays flat, prompt chunks are the cause. Lower the prefill budget instead, as covered in How vLLM Chooses a Prefill Token Budget.

When Budgeting Tokens Is the Wrong Tool

When you only call a hosted API. You cannot see the batch, and nothing you admit changes it. Buy a priority tier or provisioned throughput if streaming speed is in your contract.

When the batch is huge and contexts are short. Past a few hundred requests, the step becomes limited by compute, not reads. The formula then underestimates step time. Watch for measured latency rising faster than predicted.

When the model is not a plain transformer. A mixture-of-experts model reads only the experts the batch touches, so the weight term shrinks at small batches. A model with mostly sliding-window layers reads a capped cache per user. Quantized caches halve the cache term. Refit the per-token figure for each case.

When latency does not matter. An overnight batch job wants throughput. Fill memory and ignore the speed budget.

Three questions before you adopt it:

• What is your longest common context, in tokens, and its cache in GB?
• What does one user alone get, in tokens per second, on your server?
• Do your long and short requests share a GPU today?

Glossary

• decode step — one pass of the model that makes one new token for every running request.
• memory bandwidth — how many bytes per second the GPU can read from its own memory.
• KV cache — the stored keys and values for every token a request has seen, reread every step.
• tokens in flight — the total context held by all running requests on one GPU.
• prefill — reading a new prompt into the KV cache; compute work, not memory reads.
• effective bandwidth — the read speed your server actually reaches, measured from one stream.

Further reading

• LLM Inference Performance Engineering: Best Practices — Databricks / MosaicML engineering (Megha Agarwal, Asfandyar Qureshi, Nikhil Sardana, Linden Li, Julian Quevedo, Daya Khudia)
• How continuous batching enables 23x throughput in LLM inference while reducing p50 latency — Anyscale engineering (Cade Daniel, Chen Shen, Eric Liang, Richard Liaw)
• vLLM metrics design — vLLM project documentation
• Qwen3-8B config.json — Qwen team on Hugging Face

Originally published at theaicommit.com, a daily AI engineering lab for software engineers. The original has an interactive version you can play with, and the full runnable code is on GitHub.