BTC
ETH
USDT
XMR
LTC
Solutions · Inference & serving
GPU servers for LLM inference.
Serve open-weight models with vLLM, SGLang or TensorRT-LLM on dedicated GPUs, at a monthly price you know in advance.
- 1× MI355Xbest fit
- 288 GBGPU memory
- $1,409per month
- From $789cheapest pick
What LLM inference needs from a GPU.
- 1
Memory for weights and KV cache
Weights take about 2 bytes per parameter in BF16, 1 in FP8 and about 0.5 in 4-bit. The KV cache grows with context length and concurrent requests and often needs as much memory again.
- 2
Memory bandwidth
Generating tokens is mostly bandwidth-bound: 8 TB/s on B300, B200 and MI355X, 4.8 TB/s on H200, 1.8 TB/s on RTX 5090 and 1.6 TB/s on RTX PRO 6000.
- 3
Steady cost for steady traffic
Inference endpoints run all month. A flat monthly price removes the per-hour meter from a service that never stops.
Best GPUs for LLM inference.
The first is the best fit; the other two trade speed, memory or ecosystem for price.
-
Best fit
1× AMD MI355X
288 GB HBM3E · 8 TB/s per GPU
- Interconnect
- Single GPU
- vCPU · RAM
- 32 · 384 GB
- NVMe
- 3.84 TB
$1,409/mo
$2,022 -
Alternative
1× NVIDIA H200
141 GB HBM3e · 4.8 TB/s per GPU
- Interconnect
- Single GPU
- vCPU · RAM
- 24 · 256 GB
- NVMe
- 3.84 TB
$2,279/mo
$3,259 -
Alternative
1× NVIDIA L40S
48 GB GDDR6 ECC · 864 GB/s per GPU
- Interconnect
- Single GPU
- vCPU · RAM
- 12 · 96 GB
- NVMe
- 1 TB
$789/mo
$1,139
How big a server for your model?
FP8 weights at ≈1.2 bytes per parameter and 4-bit weights at ≈0.65, both with a 20–30% margin for the KV cache. Long contexts and many concurrent users need more.
| Model size | FP8 inference | 4-bit inference |
|---|---|---|
| 8Babout 10 GB / 5 GB | 1× RTX 5080$199/mo | 1× RTX 5080$199/mo |
| 32Babout 38 GB / 21 GB | 2× RTX 4090$538/mo | 1× RTX 4090$269/mo |
| 70Babout 84 GB / 46 GB | 1× RTX PRO 6000$959/mo | 2× RTX 5090$658/mo |
| 120Babout 144 GB / 78 GB | 1× MI355X$1,409/mo | 1× RTX PRO 6000$959/mo |
| 405Babout 486 GB / 263 GB | 2× MI355X$2,818/mo | 1× MI355X$1,409/mo |
| 671Babout 805 GB / 436 GB | 4× MI355X$5,636/mo | 2× MI355X$2,818/mo |
Cheapest configuration that holds the model by our rule of thumb, 92% of GPU memory usable. Try other sizes · VRAM of popular open models
Software for LLM inference, and our tips.
- vLLMPagedAttention, continuous batching, OpenAI-compatible server; CUDA and ROCm builds.
- SGLangFast serving runtime with structured generation and prefix caching.
- TensorRT-LLMNVIDIA-optimized engines with FP8 and FP4 on Hopper and Blackwell.
- llama.cpp / OllamaGGUF quantized models on GeForce cards for development and small deployments.
Tips
- Use FP8 weights on Hopper, Blackwell and MI355X: about half the memory of BF16 with small quality loss for most models.
- Set tensor parallelism to the number of GPUs in the server (2, 4 or 8).
- Bind the API to localhost and reach it through an SSH tunnel, or put an authenticated reverse proxy with TLS in front of it.
Good to know
Mixture-of-experts models must keep every expert in memory even though only a few are active per token: size them by their total parameter count.
Related guides
- Serving LLMs with vLLM on dedicated GPUsvLLM turns a Hugging Face model into an OpenAI-compatible API. Keep it bound to localhost, reach it through SSH or an authenticated TLS proxy, and size it with parallelism and quantization.
- Serve LLMs with SGLang on a GPU serverSGLang serves open models through an OpenAI-compatible API, on one GPU or split across eight. This SGLang tutorial installs it on Ubuntu, keeps it on localhost behind an API key and covers NVIDIA and AMD MI355X servers.
- Run Ollama and Open WebUI on a GPU serverOllama runs open models on your GPUs and Open WebUI puts a ChatGPT-style interface in front of them: a self-hosted ChatGPT on a GPU server you control. Keep both on localhost and reach them through SSH or an authenticated TLS proxy.
- Best GPU for LLM inference in 2026: memory, bandwidth and priceBest GPU for LLM inference in 2026: VRAM, bandwidth and price of 10 GPUs, and the cheapest server for each model size, from 8B to 1T.
- Self-host an LLM: a private ChatGPT on your own GPU serverHow to self-host an LLM as a private ChatGPT: pick an open model, size the GPU, run Ollama with Open WebUI or vLLM, lock down access, and what it costs.
Inference & serving questions.
Other workloads: LLM training, Fine-tuning, Image & video, 3D rendering & VFX, Research & HPC.
Which GPU gives the most memory per dollar?
The AMD MI355X: 288 GB of HBM3E for $1,409 per month, about $4.89 per GB. If your stack runs on ROCm (vLLM and SGLang do), it is the cheapest way to hold very large models.
Can one server serve a 671B mixture-of-experts model?
In FP8 such a model needs about 805 GB by our rule of thumb: 4× MI355X or 4× B300 hold it; 8× H200 do too. In 4-bit, 2× MI355X or 2× B300 are enough.
Do you offer a managed inference API?
No. You get dedicated servers with root access and run the serving stack of your choice.
Start with 1× MI355X.
AMD MI355X, 288 GB of GPU memory, for $1,409 a month.