How much GPU memory does your model need?

#Start with the weights

The first term is simple:

weights (GB) = parameters (billions) × bytes per parameter

• BF16 or FP16: 2 bytes per parameter. Llama 3.1 70B takes about 140 GB.
• FP8: 1 byte. The same model takes about 70 GB.
• 4-bit formats such as GPTQ, AWQ or MXFP4: about 0.5 byte, slightly more in practice because they also store scaling factors. The same model takes about 35 GB.

Real checkpoints follow the rule. gpt-oss-120b has 116.83 billion parameters, ships its mixture-of-experts weights in MXFP4, and its checkpoint is 60.8 GiB (about 65 GB), roughly 0.56 byte per parameter. DeepSeek-V3 and DeepSeek-R1, with 671 billion parameters, are published with FP8 weights, so the main model alone is about 671 GB.

#Add the KV cache

While generating, the model keeps a key vector and a value vector for every token of every sequence, in every layer. That is the KV cache, and it grows with context length and with the number of sequences served at once:

KV per token (bytes) = 2 × layers × KV heads × head dimension × bytes per value
KV cache (bytes)     = KV per token × tokens of context × concurrent sequences

The 2 counts keys and values. Models with grouped-query attention keep far fewer KV heads than attention heads, which keeps the cache manageable. Llama 3.1 70B has 80 layers, 8 KV heads and a head dimension of 128, so in BF16 each token costs 2 × 80 × 8 × 128 × 2 = 327,680 bytes, about 0.33 MB.

These figures are per sequence; 8K, 32K and 128K mean 8,192, 32,768 and 131,072 tokens. Qwen3-32B handles 32,768 tokens natively and 131,072 with YaRN scaling. Thirty-two concurrent 8K conversations on Llama 3.1 70B need 86 GB of cache, more than the FP8 weights. Serving engines such as vLLM can store the cache in FP8, which halves it. DeepSeek-V3 and R1 use Multi-head Latent Attention, which compresses the cache into a latent vector, so this formula overstates their needs.

#Leave headroom

Weights and cache are not the whole bill. Activations, the CUDA or ROCm context, the serving engine’s buffers and memory fragmentation all take their share. Two rules of thumb keep estimates realistic:

• Count about 92% of a GPU’s memory as usable. An 80 GB H100 gives about 73.6 GB, a 288 GB B300 or MI355X about 265 GB.
• For inference at moderate context, add 20% to the weights, or 30% for 4-bit formats. For long contexts or large batches, compute the KV cache explicitly instead.

That gives 2.4 GB per billion parameters in BF16, 1.2 GB in FP8 and 0.65 GB in 4-bit, the factors our sizing helper uses. Applied to our servers, usable memory looks like this:

#Fine-tuning and full training

Training keeps much more than the weights in memory.

• Full training with Adam in mixed precision needs 16 bytes per parameter before activations: 2 for the 16-bit weights, 2 for the gradients and 12 for the FP32 master weights and Adam’s two moments, as set out in the ZeRO paper. Activations come on top and grow with batch size and sequence length, so we budget 18 bytes per parameter: about 144 GB for an 8B model and 1.26 TB for a 70B model.
• LoRA freezes the base model in 16-bit (2 bytes per parameter) and trains small adapter matrices. With activations, budget about 3 GB per billion parameters: 24 GB for an 8B model, 210 GB for a 70B model.
• QLoRA loads the frozen base in 4-bit and trains adapters on top: about 0.75 GB per billion parameters, or 52.5 GB for a 70B model.

Sharding with FSDP or DeepSpeed ZeRO spreads these states across the GPUs of a server, so what counts is the server’s total memory. See multi-GPU setup.

#Worked examples

Here are these factors applied to real models, with parameter counts from the official model cards and 92% of memory counted as usable:

Three remarks. OpenAI states that gpt-oss-120b fits on a single 80 GB GPU such as the H100; our rule keeps more room for the cache, hence the 96 GB card. When a model is split across several PCIe cards (RTX 4090, RTX 5090, RTX PRO 6000), the GPUs exchange data over PCIe, which is slower than NVLink or Infinity Fabric, so check that your serving engine or trainer handles the split well. And for the 405B model on NVLink, 2× B300 ($8,038) and 4× H200 ($9,116) also fit.

The MI355X appears often because it has the most memory per dollar in our range; its trade-offs are covered in our MI355X article.

#Check it in the sizing helper

The sizing helper on our home page applies these factors: pick a model size and a workload, and it shows the memory needed and the cheapest configurations that fit, with the memory left over. Treat the result as a starting point. Long contexts, large batches and some engines need more than the rule of thumb, so load the model once and watch real usage with nvidia-smi or rocm-smi before you settle on a size.

Once you know the memory you need, compare GPUs on the pricing page, read inference serving for engine setup, and configure a server.