GDDR7 vs HBM3E for Inference: Bandwidth per Dollar Stops Being the Question

The first instinct when pricing an inference accelerator is to compare bandwidth numbers. HBM3E gives you around 1.2 TB/s per stack and up to eight stacks, so 8 to 9.6 TB/s on a top-end part. GDDR7 on a 384-bit bus gives you roughly 1.5 TB/s. The HBM part looks six times better, costs roughly three times more, and the conclusion writes itself.

Except that for a large share of real inference traffic, 1.5 TB/s is not the bottleneck. The bottleneck is the batch size the SLA allows, and once you fix the batch, the comparison inverts.

Part One: The arithmetic of a single token

A 70B model in FP8 is 70 GB of weights. One token requires reading all of them. On 1.5 TB/s that is 47 milliseconds; on 8 TB/s it is 8.8 milliseconds. If you serve one user at a time, HBM is five times faster and there is no argument.

Now batch. With 32 concurrent sequences, the weight read is amortised across 32 tokens, so per-token time falls to 1.5 milliseconds on GDDR7 and 0.27 milliseconds on HBM. But the key-value cache now has to be read once per sequence per step, and that traffic does not amortise. At 8,000 tokens of context per sequence, a 70B model reads roughly 40 GB of cache per step across the batch. GDDR7 needs 27 milliseconds for that; HBM needs 5. The cache, not the weights, sets the ceiling once you batch, and the cache is where GDDR7's narrower bus starts to hurt.

So the naive story is right for large batches and wrong for small ones. The interesting part is what happens in between, which is where most production traffic lives.

Part Two: Where the crossover actually sits

Work the numbers for a 13B model in FP8 with 4,000-token contexts, a realistic shape for a coding assistant or a document question-answering service.

Weights are 13 GB. Cache is roughly 0.5 MB per token per layer-summed for a 40-layer model, so 4,000 tokens is about 2 GB per sequence. At batch 8, the cache is 16 GB and the weights are 13 GB, so total traffic per step is 29 GB. GDDR7 at 1.5 TB/s takes 19 milliseconds; HBM at 8 TB/s takes 3.6 milliseconds.

Now add the compute floor. On a mid-range accelerator with 400 TFLOPS of FP8, generating 8 tokens of 13B parameters takes 8 × 2 × 13e9 / 400e12, or 0.5 milliseconds. So on GDDR7 the memory system is 38 times slower than compute; on HBM it is 7 times slower. Both are memory-bound, and the ratio between them is the throughput ratio: about 5.3x.

That is the honest comparison, and it is not the comparison that matters commercially, because the GDDR7 part costs a third as much. Per dollar of throughput the two are within 20% of each other for this shape. The decision then turns on rack power and on whether you can fit the model in 24 GB or need 80.

Part Three: The three constraints that actually decide

Power. GDDR7 has a higher energy per bit than HBM because the traces are longer and the signalling is single-ended. At the same bandwidth, GDDR7 draws 1.4 to 1.8 times the memory power. On a 350 W board that can be 30 to 50 watts, which is 10% of the power budget for 60% of the bandwidth. In a power-constrained datacenter, which is now most of them, that trade is often unacceptable regardless of purchase price.

Capacity per package. GDDR7 tops out around 24 to 32 GB per board with a reasonable bus width. HBM3E reaches 141 GB on a single package. If your model plus cache does not fit, you are doing tensor parallelism across boards, which multiplies both the fabric cost and the failure surface. For anything above a 30B model with real context, this constraint alone settles the argument.

Thermals and density. HBM sits under the same lid as the die and is cooled by the same cold plate. GDDR7 chips are scattered around the board and need their own airflow path. In a high-density rack, that difference changes your chassis design, not just your bill of materials.

The pattern is consistent: GDDR7 wins on purchase price, HBM wins on every operational metric. The reason inference clusters keep buying HBM is not that bandwidth is scarce, it is that power, capacity and cooling are, and those three are what you actually run out of.

Part Four: Where GDDR7 does win

Three segments, clearly.

Edge and on-premise inference. A 7B to 13B model on a 24 GB GDDR7 board at 1.5 TB/s serves a small team's traffic comfortably. Power is available, the model fits, and the price difference is a factor of three. This is the fastest-growing segment of the market by unit count.

Batch-tolerant workloads. Document processing, embedding generation, offline classification. These tolerate large batches and long queues, so throughput per dollar dominates and latency is irrelevant. GDDR7's cost advantage compounds here.

Speculative decoding hosts. A draft model running on GDDR7 alongside a large model on HBM is a rational pairing: the draft model is small, its latency is hidden behind the verifier, and its memory cost is a rounding error.

Part Five: What 2030 looks like

GDDR7 will not replace HBM and HBM will not disappear. What changes is the split of the inference market by model size, and that split is moving toward the middle.

My estimate: by 2030, roughly 55% of inference tokens are served by models between 7B and 30B, up from about 35% today, because distillation and task-specific fine-tuning keep proving that a small model with good retrieval beats a large model with bad prompts. That shift favours GDDR7-class memory. HBM volume still grows in absolute terms, driven by frontier models and long-context serving, but its share of units falls.

The memory market as a whole grows from roughly $180 billion today to $400 to $470 billion by 2030, with HBM holding a third of the value on 5% of the bits. That value concentration is the real story: HBM is not a commodity, it is a toll on the ability to serve large models, and tolls are priced by scarcity.

The daily-life angle

This is the difference between a coding assistant that runs on your laptop and one that needs a datacenter. When a 13B model fits in 24 GB and serves at 1.5 TB/s, the same capability shows up in a local tool, an offline device, a hospital system that cannot send data out. Memory choice determines which products can exist at all, and that is a more interesting question than which one benchmarks faster.

For engineers, the practical takeaway is to measure your real batch distribution before choosing hardware. Most teams discover their p50 batch is 1 to 4 and their p99 is 40, which means they are paying for HBM to serve a tail that a queue could absorb.

Buy memory for the batch you actually run, not the batch your benchmark assumes, because the difference is a factor of three in capital cost.

The transferable lesson

Hardware comparisons almost always start with the headline number and end with the wrong answer, because the headline number is measured in isolation and your workload never is. The useful comparison is always per unit of the constraint you are shortest on. In 2015 that was cores. In 2020 it was memory bandwidth. In 2026 it is watts and square metres of floor space.

Figure out which constraint you are actually short of, then price everything against that. The teams that do this consistently end up with configurations that look strange on a spec sheet and work extremely well in production.

SOL: HCqPmPJM3NVtS6YrmVaTZYnuP8XZ2PFgQZKT94VK1LkS
Creating this takes real effort — thank you for the support.