Tensor vs. Pipeline vs. Data Parallelism: How LLM Inference Scales Across GPUs

Sooner or later, every LLM deployment hits one of two walls. Either the model does not fit on one GPU — a 70B model in FP16 needs roughly 140 GB of weights before you allocate a single byte of KV cache — or it fits, but one GPU cannot serve your traffic at the latency you promised. Both walls are broken the same way: by spreading the work across GPUs. But "spread it across GPUs" hides three very different strategies, and choosing the wrong one buys you expensive hardware doing slow work.

The three strategies answer three different questions:

• Tensor parallelism: how do I make one request faster (or possible at all)?
• Pipeline parallelism: how do I make a model that spans machines workable?
• Data parallelism: how do I serve more requests at once?

This post walks through how each one actually splits the work, what it costs in communication, and how they combine in real deployments — in vLLM terms, the flags behind --tensor-parallel-size, --pipeline-parallel-size, and --data-parallel-size.

Tensor parallelism: split every layer

Tensor parallelism (TP) shards the model within each layer. A transformer's heavy operations are large matrix multiplications, and those matrices can be cut up: the attention block splits naturally by attention heads, and the feed-forward block splits by columns and rows. Each GPU holds a slice of every layer's weights, computes its partial result for a token, and then the GPUs synchronise — an all-reduce operation — so every GPU has the full intermediate result before the next operation starts.

The defining trait is that synchronisation happens inside every layer, for every token. A model with 80 layers needs roughly two all-reduces per layer, and decode generates tokens one at a time, so this chatty pattern runs constantly. That has two consequences:

• It is the only strategy here that reduces per-token latency. More GPUs genuinely cooperate on each token, so time-per-token drops — until communication eats the gains.
• It demands a fast interconnect. All-reduce traffic is large and latency-sensitive. Inside one node, NVLink provides the bandwidth and latency TP needs. Across nodes, over Ethernet or even InfiniBand, the same all-reduces usually cost more than the extra compute buys. The practical rule: TP stays within the node.

TP also shards more than weights. The KV cache is distributed across the TP group (each GPU caches its own attention heads), so TP buys memory headroom for longer contexts and larger batches, not just fit. The constraint to remember: the TP size has to divide the model's attention heads — and with grouped-query attention, the (smaller) number of KV heads — which is why you see TP sizes of 2, 4, and 8, and why some models simply cannot use certain sizes.

Pipeline parallelism: split the layers themselves

Pipeline parallelism (PP) cuts the model the other way: by layer. GPU 1 holds layers 1–20, GPU 2 holds layers 21–40, and so on. Each GPU is a stage. A token's activations are computed by stage 1, handed to stage 2, and the result flows down the pipe. A GPU only talks to its neighbours, and only to pass activations — a far smaller payload than TP's all-reduces.

That one change flips the trade-offs:

• Communication is light and point-to-point, so PP is the strategy that survives crossing node boundaries. It is how genuinely huge models span machines.
• But a single request gets no faster. Your token still passes through every layer sequentially — now with network hops in between. PP alone typically adds latency to an individual request.
• And the pipeline has a bubble. If only one request is in flight, stage 2 sits idle while stage 1 works, stage 3 idles while stage 2 works, and so on. The fix is microbatching: split the work into many small batches so that while stage 1 processes microbatch 3, stage 2 is already working on microbatch 2. With m microbatches and p stages, the wasted fraction is roughly (p − 1) / (m + p − 1) — which is why PP efficiency depends on having enough concurrent work. In LLM serving, continuous batching supplies that stream of work naturally: many requests at different points in the pipe keep the stages fed, and the bubble mostly shows up as ramp-up and low-traffic inefficiency rather than a constant tax.

PP's memory profile is also uneven: each stage holds complete layers (so weights shard cleanly), but activation memory and KV cache concentrate differently per stage, and the first stage additionally carries the embedding. Lopsided stages — say, one stage with more layers or the embedding plus many layers — make the whole pipeline run at the speed of the slowest stage.

Data parallelism: don't split the model at all

Data parallelism (DP) is the simplest strategy and the odd one out: every GPU (or GPU group) holds a complete copy of the model and serves different requests. There is no cross-GPU communication in the forward pass at all — a router sends each incoming request to one replica, and that replica handles it start to finish.

A quick terminology warning: in training, data parallelism means something heavier — replicas process different batches and then synchronise gradients (another all-reduce) every step. In inference, there are no gradients and nothing to synchronise. Inference DP is just replication with a load balancer, which is exactly the "one model, many replicas" deployment shape from any serving stack.

DP's profile follows from that simplicity:

• Throughput scales almost linearly with replicas, because requests are independent. If one replica serves 40 concurrent sequences well, four serve roughly 160.
• Per-request latency does not improve at all. A replica is one copy of the model on its hardware; a token takes as long as it takes.
• Memory cost is the catch. Each replica needs the full weights. DP does nothing to help a model that does not fit — it multiplies the hardware you need by the replica count.
• Failure behaviour is the best of the three. Losing a TP or PP group member takes down the whole serving unit; losing one DP replica just removes capacity until it is replaced.

Side by side

How they combine in practice

Real deployments rarely pick just one, because the three axes multiply. The standard recipe for a large model in production:

• Tensor parallelism inside each node — say TP=8 across one node's GPUs.
• Pipeline parallelism across nodes, when the model still does not fit — say PP=2 across two nodes.
• Data parallelism *on top* — several of those 16-GPU units running as independent replicas behind a router.

Total GPUs = TP × PP × DP, which is why a casual flag change can quietly quadruple a deployment's hardware bill.

In vLLM, that recipe is literally:

vllm serve <model> \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 2 \
  --data-parallel-size 4

One special case worth knowing: mixture-of-experts models add a fourth axis, expert parallelism, which places different experts on different GPUs and routes each token to the experts it selects. If you serve MoE models, that routing — and its load imbalance when a few experts get most of the traffic — becomes a tuning problem of its own.

Gotchas

Parallelism is not a ladder. Teams reach for PP because "TP maxed out," then discover their latency got worse: PP fixes fit, not speed. Diagnose which wall you actually hit — fit, latency, or throughput — and pick the axis that addresses it.

The multiplication surprise. TP × PP × DP means every "small increase" is a multiplier on GPU count, cost, and failure surface. Price the full product before approving the flags.

TP sizes are constrained by the model. If the TP size does not divide the attention heads (or the KV heads under grouped-query attention), the configuration fails or falls back awkwardly. Check the model's head counts before standardising on TP=8 everywhere.

PP starves at low traffic. The bubble math assumes enough microbatches to fill the pipe. A low-traffic internal deployment on a deep pipeline can end up slower and more expensive than a single well-sized replica.

DP hides skew. Replicas are independent, so one slow request never blocks others — but long-context requests pile onto whichever replica took them, and naive routing leaves some replicas saturated while others idle. Route on current load and KV-cache usage, not round-robin.

Every axis shares one failure truth. Within a TP or PP group, one GPU failure kills the unit, and the bigger the group, the more often some member fails. Replication (DP) is not just a throughput tool — it is also your availability story.

The takeaway

Tensor, pipeline, and data parallelism are not competitors; they are answers to three different questions:

• Tensor parallelism makes one request faster and shards a layer too big for one GPU — at the price of constant, bandwidth-hungry synchronisation that belongs on NVLink.
• Pipeline parallelism makes a model too big for one node fit across nodes — at the price of bubbles and added latency.
• Data parallelism makes throughput scale — at the price of a full model copy per replica.

Production systems compose all three, in that order of thinking:

1. Fit the model (TP inside the node, PP across nodes if needed).
2. Hit your latency target (TP, within the node).
3. Replicate for traffic (DP on top).

Choose the axis by the wall you hit, and the flags stop being guesswork.