How to Size a GPU Server for Open-Weight AI Models (VRAM, KV Cache, Bandwidth)

Open-weight models such as Llama, Mistral, Qwen, DeepSeek, and gpt-oss can run on infrastructure you control. Before you rent a server, you need to answer one question: what hardware does this model actually need?

This post walks through the numbers.

1. Weight memory

Weight memory ≈ parameter count × bytes per parameter

A 70B model at 16-bit needs at least two 80 GB GPUs. At 4-bit it can fit on a single 48 GB card.

2. KV cache

The KV cache stores attention data for every token of every active request, so it grows with context length and concurrency.

Per-token cache = 2 × layers × KV heads × head dimension × bytes per value

For Llama 3.1 70B (80 layers, 8 KV heads, head dimension 128), that is about 320 KB per token at 16-bit. One 128K-token conversation needs roughly 40 GB of cache on its own.

Tip: Cap max input tokens in your inference engine to prevent out-of-memory crashes in production.

3. Mixture-of-experts models

MoE models lower compute per token, but every expert must stay in memory.

Size your VRAM against total parameters.

4. Memory bandwidth

Tokens/sec ≈ memory bandwidth ÷ bytes read per token

On an H100 SXM (3.35 TB/s), a dense 70 GB model tops out near 48 tokens/sec for a single stream before overheads. Batching raises total throughput across users.

5. Multi-GPU interconnects

• NVLink (H100 SXM): up to 900 GB/s per GPU
• PCIe Gen5 x16: about 64 GB/s each way

Two cards over PCIe often work well. For four to eight GPUs on one model, NVLink servers are worth the extra cost.

6. Fine-tuning

Full fine-tuning with Adam in mixed precision commonly needs about 16 bytes per parameter, so a 7B model needs around 110 GB. LoRA and QLoRA cut this sharply, and the QLoRA authors reported fine-tuning a 65B model on a single 48 GB GPU.

Planning checklist

1. Pick the model and check its license and native precision.
2. Calculate weight memory at your target precision.
3. Add KV cache for your context length and concurrency, plus 10–20% overhead.
4. Compare memory bandwidth with your latency target.
5. Decide on single vs multi-GPU and check interconnects.
6. Size system RAM, NVMe, power, and network.
7. Benchmark on a trial configuration before committing.

Read the full guide

The original article includes a model-to-GPU matching table, host system sizing, and serving engine notes (vLLM, TensorRT-LLM, TGI, SGLang, llama.cpp):

👉 GPU Server Requirements for Open-Weight AI Models