Best GPU for LLM inference in 2026: memory, bandwidth and price

What is the best GPU for LLM inference?

The best GPU for LLM inference is the cheapest one whose memory holds your model and its KV cache, with as much memory bandwidth as your budget allows. Memory capacity decides which model and context fit; memory bandwidth decides how fast tokens come out, because every new token reads the model’s weights from memory.

Among the GPUs we rent, that points to:

• Models up to about 30B in 4-bit: one RTX 4090 or RTX 5090.
• A 70B model in FP8, or gpt-oss-120b: one RTX PRO 6000 (96 GB).
• 70B-class models that need more speed or context on CUDA: the H200 (141 GB at 4.8 TB/s).
• The most memory per dollar, if your model runs on ROCm: the MI355X (288 GB at 8 TB/s).
• The largest models on CUDA: the B300 (288 GB at 8 TB/s, NVLink 5).

We publish no benchmarks: every figure below comes from vendor specifications, our sizing rule and our catalog prices, and the speeds are computed ceilings, not measurements.

VRAM decides which model and context fit

Start with capacity: if the weights and the KV cache do not fit in GPU memory, the model does not run, or must be split across more GPUs. Our sizing rule:

GB needed = billions of parameters × 2.4 in BF16, 1.2 in FP8, 0.65 in 4-bit
usable    = 92% of GPU memory

The factors add 20% to the weights in BF16 and FP8 and 30% in 4-bit, enough for the KV cache at moderate context. A 70B model in FP8 needs about 84 GB: one H100 (80 GB, about 74 GB usable) is too small, while one RTX PRO 6000 (96 GB) or one H200 holds it.

The margin is a starting point. The KV cache grows with the length of each conversation and with the number served at once, and with long contexts or many users it can need as much memory as the weights. Mixture-of-experts models must hold every expert: size them by total parameters. Formulas and worked examples are in how much GPU memory your model needs; the sizing helper lists every server that fits a given model.

Memory bandwidth decides how fast tokens come out

Once the model fits, memory bandwidth sets the pace of each conversation. A model writes one token at a time, and for every token the GPU reads the weights and that conversation’s KV cache from memory. NVIDIA calls this decode phase memory-bound: “The speed at which the data (weights, keys, values, activations) is transferred to the GPU from memory dominates the latency” (NVIDIA). Databricks measures achieved bandwidth as (model size + KV cache size) ÷ time per output token (Databricks), which gives a ceiling for one conversation:

time per token    ≥ (weight bytes + KV cache bytes) ÷ memory bandwidth
tokens per second ≤ memory bandwidth ÷ bytes read per token

A 70B dense model in FP8 reads about 70 GB of weights per token. One conversation is then capped at about 23 tokens per second on an RTX PRO 6000 (1.6 TB/s), 69 on an H200 (4.8 TB/s) and 114 on a B300, B200 or MI355X (8 TB/s), before the KV cache. Real engines stay below the ceiling: Databricks reports 60% of peak bandwidth at batch size 1 on two H100.

• Lower precision raises the ceiling. FP8 halves the bytes read per token compared with BF16, and 4-bit halves them again: quantization lets you “transfer more parameters over the same amount of bandwidth”, as NVIDIA puts it.
• Mixture-of-experts models read only their active parameters. gpt-oss-120b uses 5.1B of its 117B parameters per token (model card), Qwen3.5-397B-A17B 17B of 397B (model card): each token reads a small share of weights that must all sit in memory.
• Many users shift the limit to compute. With batching, “the memory cost of the weights is spread out” over the requests, and for large batch sizes Databricks finds that “the system is compute bound”, as it typically is for the first token of each answer. That is where FP8 and FP4 throughput matters.

VRAM against bandwidth is therefore not a choice: capacity first, then the most bandwidth you can afford at that size.

Number formats, GPU links and software

FP8 and FP4 support

Every GPU we rent runs FP8 in hardware, from the Ada cards (L40S, RTX 4090) to Hopper, Blackwell and the MI355X. FP4 is newer: NVIDIA’s NVFP4 format was “introduced with the NVIDIA Blackwell GPU architecture” (NVIDIA), and the MI355X adds MXFP8, MXFP6 and MXFP4 (AMD). Hopper and Ada have no FP4 tensor cores, but 4-bit weights still save memory and bandwidth on them: vLLM’s Marlin kernels run 4-bit and FP8 checkpoints on Ampere, Ada and Hopper (vLLM), and OpenAI states that gpt-oss-120b, with MXFP4 expert weights, runs on a single 80 GB GPU such as the H100 (model card). Native FP4 math helps most where compute is the limit: large batches and long prompts.

NVLink, Infinity Fabric or PCIe

When a model spans several GPUs, tensor parallelism splits each layer across them, and the GPUs exchange data at every step. NVLink 5 links each B300 or B200 at 1.8 TB/s, NVLink 4 each H200 or H100 at 900 GB/s, Infinity Fabric each MI355X at 1.07 TB/s. The RTX PRO 6000, L40S and GeForce cards use PCIe: NVIDIA quotes 128 GB/s for PCIe Gen5 on the H100 (NVIDIA) and 64 GB/s bidirectional for Gen4 x16 on the L40S (NVIDIA). Without NVLink, vLLM advises pipeline parallelism “for higher throughput and lower communication overhead” (vLLM). Splitting also brings diminishing returns for latency, Databricks notes, so a model that fits on one GPU is best served from one, with a copy per GPU for more throughput.

CUDA or ROCm

The NVIDIA GPUs run the CUDA ecosystem as it is, including TensorRT-LLM, NVIDIA’s own engine for its GPUs (GitHub). The MI355X runs ROCm: vLLM supports the MI350 series, which includes it, from ROCm 7.0 (vLLM). Support still depends on the model and its kernels: our model pages offer a model on the MI355X with vLLM or SGLang only once its ROCm support is confirmed, today gpt-oss-20b, gpt-oss-120b and DeepSeek-R1-0528. See our MI355X article for the trade-offs.

GPUs for LLM inference compared: memory, bandwidth, price

The numbers that matter for inference, for every GPU we rent. Prices are per GPU per month, the same at every server size.

Vendor specifications per GPU, FP8 without sparsity. *NVIDIA does not say whether its RTX PRO 6000 Server Edition figures use sparsity.

• Memory per dollar: a GB costs $4.89 a month on the MI355X, $9.99 on the RTX PRO 6000, $10.28 on the RTX 5090 and $16.16 on the H200.
• Bandwidth per dollar: about 568 GB/s per $100 a month on the MI355X and 545 on the RTX 5090, against 211 on the H200 and 188 on the H100.
• H200 or H100: the same FP8 throughput, but the H200 adds 61 GB and 43% more bandwidth for $500 more a month (H100 vs H200).
• RTX PRO 6000 or H100: more memory for $959 against $1,779, but 1.6 TB/s of bandwidth against 3.35 (RTX PRO 6000 vs H100).
• L40S: the least bandwidth per dollar, about 110 GB/s per $100. An RTX 4090 has more bandwidth for $269 against $789, with half the memory.

How to choose: the cheapest fit for each model size

The cheapest server that holds each class of model by our sizing rule, each class sized at its upper bound. When the cheapest is the MI355X, the cheapest NVIDIA server follows in brackets.

• Up to 14B: one GeForce card. The RTX 5090 costs $60 more than the RTX 4090 for 1.78× the bandwidth and 32 GB instead of 24 (RTX 5090 vs RTX 4090).
• Up to 32B: one RTX 4090 in 4-bit, or two in FP8 over PCIe; one RTX 5090 leaves more room for the KV cache in 4-bit.
• 70B dense: one RTX PRO 6000 is the cheapest single-GPU fit in FP8; two RTX 5090 in 4-bit cost less but split the model. One H200 holds the FP8 weights too, with 45 GB more for context and users.
• 120B-class MoE: gpt-oss-120b in MXFP4 fits one RTX PRO 6000, and each token reads only its 5.1B active parameters. In FP8, a 120B MoE takes one MI355X if it runs on ROCm, or two RTX PRO 6000.
• 400B and above: here 288 GB per GPU pays: one MI355X holds a 397B model in 4-bit, four hold DeepSeek-R1-0528 in FP8. On NVIDIA, they take four to eight RTX PRO 6000 over PCIe, or, for the biggest in FP8, B300 or B200 servers with NVLink. For models not yet confirmed on ROCm, read the NVIDIA figures (see also B300 vs B200 vs H200 vs MI355X).

The 4-bit columns assume that a 4-bit checkpoint exists, official or third-party; check its quality on your own prompts. Native formats vary: MXFP4 expert weights for gpt-oss-120b, INT4 for Kimi K2.6 (model card), FP8 for DeepSeek-R1-0528 (model card) and Mistral Small 4 (model card), BF16 for Qwen3.5-397B-A17B. Our model pages size each model from its official checkpoint plus 20%, so they can differ: Kimi K2.6’s 595 GB checkpoint comes to about 714 GB, and the cheapest NVIDIA fit becomes 4× B300, $16,076.

What the cheapest fit does not tell you

• Mixture-of-experts models need memory for every expert but compute only with the active ones, token by token.
• Long contexts and many users can need more KV cache than our 20–30% margin covers: compute it for your context and concurrency, or take the next size up. vLLM can store the cache in FP8 (--kv-cache-dtype fp8), half the size of BF16 (vLLM).
• The cheapest fit is not the fastest. For a 70B model in FP8, one H200 costs 2.4× an RTX PRO 6000 ($2,279 against $959) for 3.0× the bandwidth. If your users wait on long answers, the speed may be worth it.
• PCIe is the slowest of these links. For a 397B model in 4-bit, one B300 costs $4,019 against $3,836 for four RTX PRO 6000, and needs no split.
• The MI355X price follows an indicative market reference, since no provider publishes an on-demand price for it, and the GPU needs ROCm support for your model and engine. Test one for a month before you scale.

Where CryptGPU fits

CryptGPU rents dedicated GPU servers by the month: 1, 2, 4 or 8 GPUs (GeForce cards: 1, 2 or 4), root SSH access, no other tenant on your cards. Each GPU is priced about 30% below the median public on-demand price of 40 GPU clouds, from $199 a month for an RTX 5080. You pay in Bitcoin, Ethereum, USDT, Monero or Litecoin, with no KYC, and the server is online in under 10 minutes once the payment is confirmed on-chain.

At order time we can preinstall vLLM, SGLang or Ollama with Open WebUI and a model from our catalog, at no extra cost: the open models page lists the server each model needs, and inference serving covers the setup.

A monthly server suits an endpoint that runs all month. It is the wrong choice for a few hours of tests a week, where an hourly cloud costs less, for multi-server clusters joined by InfiniBand, which we do not offer, or for a managed inference API. Since we publish no benchmarks, the reliable way to choose between two GPUs is to rent one for a month and measure tokens per second on your own model.

Questions about GPUs for LLM inference

Is VRAM or memory bandwidth more important for LLM inference?

VRAM comes first: a model that does not fit does not run, or must be split. Among the GPUs that hold it, bandwidth decides how fast each conversation streams.

What is the cheapest GPU server for a 70B model?

By our sizing rule, a 70B model needs about 84 GB in FP8 (1× RTX PRO 6000, $959 a month) and about 46 GB in 4-bit (2× RTX 5090, $658). An H200 costs more but has three times the bandwidth of the RTX PRO 6000.

Is the RTX 5090 good for LLM inference?

For models up to about 45B in 4-bit, yes: 32 GB at 1.8 TB/s for $329 a month. GeForce servers hold up to four cards, linked over PCIe rather than NVLink.

Do I need NVLink to run an LLM?

Only to split one model across GPUs with tensor parallelism. A model that fits on one GPU, or separate copies on separate GPUs, does not use it.

Is the AMD MI355X good for LLM inference?

On paper it offers the most memory per dollar in our range: 288 GB at 8 TB/s for $1,409 a month, $4.89 per GB. In practice it depends on ROCm support for your model and engine: see MI355X vs H200.