Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 3 min read

The GPU VRAM Calculus: Why your 70B model crashes on 48GB VRAM (and the exact production sizing formula)

A lot of developers and machine learning teams calculate model memory by simply looking at the static weight size on disk. If a 4-bit quantized 70B parameter model is ~38.5GB to 40GB, the natural assumption is: "It wil

A lot of developers and machine learning teams calculate model memory by simply looking at the static weight size on disk.

If a 4-bit quantized 70B parameter model is ~38.5GB to 40GB, the natural assumption is: "It will fit comfortably on dual RTX 3090s or an A6000 (48GB physical VRAM)."

Then they spin up a vLLM or HuggingFace TGI container, fire a modest batch of 8 concurrent user requests with an 8k context window, and immediately hit the dreaded:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.84 GiB...

Here is the exact mathematical reality of where your physical GPU VRAM actually goes, and the production sizing formula to keep your servers rock-solid.

1. The Real Production VRAM Formula

In production serving, your total required memory is never just the weights:

$$\text{VRAM}{\text{Total}} = M{\text{weights}} + M_{\text{kv_cache}} + M_{\text{activations}} + M_{\text{cuda_runtime}}$$

A. Static Weights ($M_{\text{weights}}$)

The memory footprint of the model parameters loaded into memory:

  • FP16 / BF16: $\approx 2 \text{ GB}$ per billion parameters ($70\text{B} \approx 140\text{ GB}$).
  • INT8 / FP8: $\approx 1 \text{ GB}$ per billion parameters ($70\text{B} \approx 70\text{ GB}$).
  • INT4 (AWQ / GPTQ / GGUF): $\approx 0.55 \text{ GB}$ per billion parameters ($70\text{B} \approx 38.5\text{ GB}$).
  • Add a 5% allocation overhead factor ($1.05 \times$).

B. KV Cache ($M_{\text{kv_cache}}$) β€” The Silent Killer

This is the memory required to store the Key and Value attention tensors for past tokens across active requests.
$$\text{Memory}{\text{KV}} = 2 \times L \times H{\text{kv}} \times D_{\text{head}} \times T_{\text{ctx}} \times B \times P_{\text{bytes}}$$

Where:

  • $L$ = Number of layers (e.g., 80 in Llama-3-70B)
  • $H_{\text{kv}}$ = Key-Value heads (8 in Grouped-Query Attention)
  • $D_{\text{head}}$ = Head dimension (128)
  • $T_{\text{ctx}}$ = Maximum context tokens (e.g., 8,192)
  • $B$ = Max concurrent batch requests (e.g., 8)
  • $P_{\text{bytes}}$ = Precision bytes (2 for FP16)

Result: In naive PyTorch, a batch of 8 requests with 8k context requires over 13.4 GB of additional VRAM just for the KV cache!

C. Activations ($M_{\text{activations}}$)

Transient memory required during the forward pass tensor operations. Scales directly with batch size and context length. For 70B models, budget 2GB to 4GB.

D. PyTorch & CUDA Context Overhead ($M_{\text{cuda_runtime}}$)

The CUDA driver, PyTorch runtime context, and cuBLAS workspace grab 0.8GB to 1.5GB the instant the process initializes.

2. Production Checklist to Eliminate OOMs

  1. Adopt PagedAttention (vLLM): Traditional continuous memory allocation suffers 60-80% fragmentation waste. PagedAttention divides the KV cache into virtual non-contiguous memory blocks, slashing waste to under 4%.
  2. Explicit GPU Utilization Ceiling: Set --gpu-memory-utilization 0.90 (leaving 10% strictly for dynamic spikes).
  3. KV Cache Prefix Caching: Enable --enable-prefix-caching so repetitive system prompts (e.g., 2,000 token system instructions) share the exact same physical memory blocks across concurrent users.

3. Open-Source Starter Kit & Configurations

I’ve open-sourced our complete production reference architectures, including:

  • Production docker-compose.yml for vLLM & DeepSeek R1
  • Model Context Protocol (FastMCP) Python tool servers
  • Private PostgreSQL + pgvector knowledge retrieval pipelines

⭐ Check out the GitHub Repository:

github.com/jppeter621/awesome-local-ai-and-mcp-architectures-2026

4. Deep-Dive Field Blueprint

If you are an engineering team or indie developer taking open-weight models and autonomous agents into production, check out our complete 21-page production engineering blueprint:

Happy to answer multi-GPU tensor parallelism and quantization questions in the comments!

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.