Solutions · LLM training
GPU servers for LLM training.
Pre-train small models, continue pre-training or run full fine-tunes on 8-GPU nodes, at one flat monthly price.
- 8× B300best fit
- 2,304 GBGPU memory
- $32,152per month
- From $11,272cheapest pick
What LLM training needs from a GPU.
- 1
Memory for the whole training state
Mixed-precision training with Adam keeps weights, gradients and optimizer states: about 16 bytes per parameter before activations. That is why training needs 6 to 8 times the memory of inference.
- 2
Bandwidth between GPUs
Data-parallel and sharded training (FSDP, ZeRO) exchange gradients at every step. NVLink 5 moves 1.8 TB/s per GPU on B300 and B200; Infinity Fabric links the eight MI355X.
- 3
Fast local storage
Checkpoints of large models weigh hundreds of GB. Every server includes local NVMe that scales with the number of GPUs.
Our picks for LLM training.
The first is the best fit; the other two trade speed, memory or ecosystem for price.
-
Best fit
8× NVIDIA B300
2,304 GB HBM3e · 8 TB/s per GPU
- Interconnect
- NVLink 5 · 1.8 TB/s
- vCPU · RAM
- 256 · 3,072 GB
- NVMe
- 30.72 TB
$32,152/mo
$45,960 -
Alternative
8× NVIDIA B200
1,440 GB HBM3e · 8 TB/s per GPU
- Interconnect
- NVLink 5 · 1.8 TB/s
- vCPU · RAM
- 224 · 2,304 GB
- NVMe
- 30.72 TB
$27,512/mo
$39,360 -
Alternative
8× AMD MI355X
2,304 GB HBM3E · 8 TB/s per GPU
- Interconnect
- Infinity Fabric · 1.07 TB/s
- vCPU · RAM
- 256 · 3,072 GB
- NVMe
- 30.72 TB
$11,272/mo
$16,176
How big a server for your model?
Full training state at ≈18 bytes per parameter (weights, gradients, Adam states and a margin for activations). Sharding with FSDP or ZeRO-3 spreads it across the GPUs of the server; activation checkpointing lowers the margin.
| Model size | Full training (mixed precision, Adam) |
|---|---|
| 1Babout 18 GB | 1× RTX 4090$269/mo |
| 3Babout 54 GB | 2× RTX 5090$658/mo |
| 7Babout 126 GB | 1× MI355X$1,409/mo |
| 13Babout 234 GB | 1× MI355X$1,409/mo |
| 34Babout 612 GB | 4× MI355X$5,636/mo |
| 70Babout 1,260 GB | 8× MI355X$11,272/mo |
Cheapest configuration that holds the model by our rule of thumb, 92% of GPU memory usable. Try other sizes
Tools people use, and what we recommend.
- PyTorch FSDPSharded data parallelism built into PyTorch.
- DeepSpeedZeRO stages 1–3, offloading and pipeline parallelism.
- Megatron-LM / NeMoTensor and pipeline parallelism for large transformers on NVIDIA GPUs.
- PyTorch for ROCm + RCCLThe same PyTorch code on AMD Instinct MI355X.
Tips
- Measure the interconnect before a long run with nccl-tests (rccl-tests on AMD).
- Write checkpoints to local NVMe, then copy them off the server on a schedule.
- Use BF16 or FP8 mixed precision; keep master weights in FP32.
- Back up everything before the end of your monthly term: disks are wiped if it is not renewed.
Good to know
One 8-GPU server is the right size for pre-training models up to a few billion parameters, continued pre-training and full fine-tuning. Pre-training a 70B model on a trillion tokens takes on the order of 10²³ floating-point operations: months to years on a single server. Each server has up to 8 GPUs; runs that need several servers joined by a high-speed fabric are outside what we offer today.
Related guides
- Multi-GPU on one server: topology, NCCL and torchrunHow the GPUs in a server are wired decides how well a job scales. Check the topology, launch with torchrun and measure the interconnect before a long run.
- Local NVMe, RAID 0 and moving data in and outThe NVMe drives in your server are fast local storage for datasets, checkpoints and model caches. Set them up once, and keep anything you cannot lose in a second place.
- Docker with GPUs: NVIDIA and ROCm containersContainers keep CUDA, ROCm and framework versions off the host. The host needs a GPU driver, Docker and, for NVIDIA GPUs, the NVIDIA Container Toolkit.
LLM training questions.
Other workloads: Fine-tuning, Inference & serving, Image & video, 3D rendering & VFX, Research & HPC.
Can I train a 70B model from scratch on one server?
Not in a reasonable time. Pre-training at that scale needs on the order of 10²³ operations, which takes months to years on one 8-GPU server. A single server is well suited to continued pre-training, full fine-tuning of models up to about 70B on the largest GPUs, and pre-training of small models.
Which GPU trains fastest?
The B300 and B200 have the most tensor throughput and NVLink 5 at 1.8 TB/s per GPU. The MI355X offers the same 288 GB of memory at a much lower price if your code runs on ROCm.
Can I connect several servers together?
Each server holds up to 8 GPUs connected inside the server. High-speed fabrics between several servers are not part of our offer today.
Start with 8× B300.
NVIDIA B300, 2,304 GB of GPU memory, for $32,152 a month.