Solutions · Inference & serving
GPU servers for LLM inference.
Serve open-weight models with vLLM, SGLang or TensorRT-LLM on dedicated GPUs, at a monthly price you know in advance.
- 1× MI355Xbest fit
- 288 GBGPU memory
- $1,409per month
- From $789cheapest pick
What LLM inference needs from a GPU.
- 1
Memory for weights and KV cache
Weights take about 2 bytes per parameter in BF16, 1 in FP8 and about 0.5 in 4-bit. The KV cache grows with context length and concurrent requests and often needs as much memory again.
- 2
Memory bandwidth
Generating tokens is mostly bandwidth-bound: 8 TB/s on B300, B200 and MI355X, 4.8 TB/s on H200, 1.8 TB/s on RTX 5090 and 1.6 TB/s on RTX PRO 6000.
- 3
Steady cost for steady traffic
Inference endpoints run all month. A flat monthly price removes the per-hour meter from a service that never stops.
Our picks for LLM inference.
The first is the best fit; the other two trade speed, memory or ecosystem for price.
-
Best fit
1× AMD MI355X
288 GB HBM3E · 8 TB/s per GPU
- Interconnect
- Single GPU
- vCPU · RAM
- 32 · 384 GB
- NVMe
- 3.84 TB
$1,409/mo
$2,022 -
Alternative
1× NVIDIA H200
141 GB HBM3e · 4.8 TB/s per GPU
- Interconnect
- Single GPU
- vCPU · RAM
- 24 · 256 GB
- NVMe
- 3.84 TB
$2,279/mo
$3,259 -
Alternative
1× NVIDIA L40S
48 GB GDDR6 ECC · 864 GB/s per GPU
- Interconnect
- Single GPU
- vCPU · RAM
- 12 · 96 GB
- NVMe
- 1 TB
$789/mo
$1,139
How big a server for your model?
FP8 weights at ≈1.2 bytes per parameter and 4-bit weights at ≈0.65, both with a 20–30% margin for the KV cache. Long contexts and many concurrent users need more.
| Model size | FP8 inference | 4-bit inference |
|---|---|---|
| 8Babout 10 GB / 5 GB | 1× RTX 5080$199/mo | 1× RTX 5080$199/mo |
| 32Babout 38 GB / 21 GB | 2× RTX 4090$538/mo | 1× RTX 4090$269/mo |
| 70Babout 84 GB / 46 GB | 1× RTX PRO 6000$959/mo | 2× RTX 5090$658/mo |
| 120Babout 144 GB / 78 GB | 1× MI355X$1,409/mo | 1× RTX PRO 6000$959/mo |
| 405Babout 486 GB / 263 GB | 2× MI355X$2,818/mo | 1× MI355X$1,409/mo |
| 671Babout 805 GB / 436 GB | 4× MI355X$5,636/mo | 2× MI355X$2,818/mo |
Cheapest configuration that holds the model by our rule of thumb, 92% of GPU memory usable. Try other sizes
Tools people use, and what we recommend.
- vLLMPagedAttention, continuous batching, OpenAI-compatible server; CUDA and ROCm builds.
- SGLangFast serving runtime with structured generation and prefix caching.
- TensorRT-LLMNVIDIA-optimized engines with FP8 and FP4 on Hopper and Blackwell.
- llama.cpp / OllamaGGUF quantized models on GeForce cards for development and small deployments.
Tips
- Use FP8 weights on Hopper, Blackwell and MI355X: about half the memory of BF16 with small quality loss for most models.
- Set tensor parallelism to the number of GPUs in the server (2, 4 or 8).
- Bind the API to localhost and reach it through an SSH tunnel, or put an authenticated reverse proxy with TLS in front of it.
Good to know
Mixture-of-experts models must keep every expert in memory even though only a few are active per token: size them by their total parameter count.
Related guides
- Serving LLMs with vLLM on dedicated GPUsvLLM turns a Hugging Face model into an OpenAI-compatible API. Keep it bound to localhost, reach it through SSH or an authenticated TLS proxy, and size it with parallelism and quantization.
- Multi-GPU on one server: topology, NCCL and torchrunHow the GPUs in a server are wired decides how well a job scales. Check the topology, launch with torchrun and measure the interconnect before a long run.
- Docker with GPUs: NVIDIA and ROCm containersContainers keep CUDA, ROCm and framework versions off the host. The host needs a GPU driver, Docker and, for NVIDIA GPUs, the NVIDIA Container Toolkit.
Inference & serving questions.
Other workloads: LLM training, Fine-tuning, Image & video, 3D rendering & VFX, Research & HPC.
Which GPU gives the most memory per dollar?
The AMD MI355X: 288 GB of HBM3E for $1,409 per month, about $4.89 per GB. If your stack runs on ROCm (vLLM and SGLang do), it is the cheapest way to hold very large models.
Can one server serve a 671B mixture-of-experts model?
In FP8 such a model needs about 805 GB by our rule of thumb: 4× MI355X or 4× B300 hold it; 8× H200 do too. In 4-bit, 2× MI355X or 2× B300 are enough.
Do you offer a managed inference API?
No. You get dedicated servers with root access and run the serving stack of your choice.
Start with 1× MI355X.
AMD MI355X, 288 GB of GPU memory, for $1,409 a month.