Prefill-Decode Disaggregation: When Splitting LLM Inference Pays
Originally published on AI Tech Connect. What you need to know Prefill — the phase that reads your prompt and produces the first token — is a set of large matrix multiplications over every prompt token at once. It satur
Originally published on AI Tech Connect.
What you need to know Prefill — the phase that reads your prompt and produces the first token — is a set of large matrix multiplications over every prompt token at once. It saturates a GPU's floating-point units. Decode — the phase that emits every token after the first — processes exactly one token per step per sequence, and spends most of its time re-reading the key-value cache out of high-bandwidth memory. It saturates memory bandwidth and leaves the floating-point units largely idle. These are not similar workloads wearing different hats. They are different workloads. When both run on the same GPU pool, a scheduler has to interleave them, and every interleaving choice hurts somebody. Let a long prefill run to completion and every user currently streaming tokens sees their stream…
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.