Solutions · Fine-tuning
GPU servers for fine-tuning.
LoRA, QLoRA or full fine-tunes of open-weight models on dedicated GPUs, billed by the month instead of by the hour.
- 1× H200best fit
- 141 GBGPU memory
- $2,279per month
- From $959cheapest pick
What fine-tuning needs from a GPU.
- 1
Enough memory for the base model
LoRA trains small adapter matrices but still loads the full base model: about 2 bytes per parameter in BF16, or about 0.5 with 4-bit QLoRA.
- 2
Headroom for long sequences
Activations grow with sequence length and batch size. Keep a margin above the weights, or use gradient checkpointing.
- 3
Time to iterate
Fine-tuning is iterative: data cleaning, runs, evaluation, reruns. A monthly server stays yours between runs, with your data and environment in place.
Our picks for fine-tuning.
The first is the best fit; the other two trade speed, memory or ecosystem for price.
-
Best fit
1× NVIDIA H200
141 GB HBM3e · 4.8 TB/s per GPU
- Interconnect
- Single GPU
- vCPU · RAM
- 24 · 256 GB
- NVMe
- 3.84 TB
$2,279/mo
$3,259 -
Alternative
1× NVIDIA RTX PRO 6000
96 GB GDDR7 ECC · 1.6 TB/s per GPU
- Interconnect
- Single GPU
- vCPU · RAM
- 24 · 180 GB
- NVMe
- 1.92 TB
$959/mo
$1,380 -
Alternative
1× NVIDIA H100
80 GB HBM3 · 3.35 TB/s per GPU
- Interconnect
- Single GPU
- vCPU · RAM
- 20 · 200 GB
- NVMe
- 2 TB
$1,779/mo
$2,548
How big a server for your model?
LoRA in BF16 at ≈3 bytes per parameter, QLoRA at ≈0.75 bytes per parameter, both including a margin for activations and adapters.
| Model size | LoRA fine-tune (BF16) | QLoRA fine-tune |
|---|---|---|
| 8Babout 24 GB / 6 GB | 1× RTX 5090$329/mo | 1× RTX 5080$199/mo |
| 14Babout 42 GB / 11 GB | 2× RTX 4090$538/mo | 1× RTX 5080$199/mo |
| 32Babout 96 GB / 24 GB | 4× RTX 5090$1,316/mo | 1× RTX 5090$329/mo |
| 70Babout 210 GB / 53 GB | 1× MI355X$1,409/mo | 2× RTX 5090$658/mo |
| 120Babout 360 GB / 90 GB | 2× MI355X$2,818/mo | 4× RTX 5090$1,316/mo |
| 235Babout 705 GB / 176 GB | 4× MI355X$5,636/mo | 1× MI355X$1,409/mo |
Cheapest configuration that holds the model by our rule of thumb, 92% of GPU memory usable. Try other sizes
Tools people use, and what we recommend.
- Hugging Face TRL + PEFTSFT, DPO and LoRA/QLoRA adapters.
- AxolotlConfiguration-driven fine-tuning on top of PyTorch.
- UnslothMemory-efficient LoRA and QLoRA kernels for NVIDIA GPUs.
- bitsandbytes4-bit and 8-bit quantization used by QLoRA.
Tips
- Start with QLoRA to validate data and hyper-parameters, then move to LoRA in BF16 if quality requires it.
- Turn on gradient checkpointing before adding GPUs: it often removes the need for more memory.
- Keep datasets and adapters on local NVMe; push adapters to your own storage after each run.
Good to know
QLoRA makes 70B-class fine-tuning possible on a single 96–141 GB GPU. Full fine-tuning keeps optimizer states for every parameter and needs 6 to 8 times more memory: see the training page for those sizes.
Related guides
- Docker with GPUs: NVIDIA and ROCm containersContainers keep CUDA, ROCm and framework versions off the host. The host needs a GPU driver, Docker and, for NVIDIA GPUs, the NVIDIA Container Toolkit.
- Local NVMe, RAID 0 and moving data in and outThe NVMe drives in your server are fast local storage for datasets, checkpoints and model caches. Set them up once, and keep anything you cannot lose in a second place.
- NVIDIA drivers, CUDA and Fabric Manager on UbuntuInstall the driver from NVIDIA’s repository, add the CUDA toolkit only if you compile code, and bring up NVSwitch on 8-GPU HGX servers. Commands assume Ubuntu 22.04 or 24.04 LTS.
Fine-tuning questions.
Other workloads: LLM training, Inference & serving, Image & video, 3D rendering & VFX, Research & HPC.
What is the cheapest way to fine-tune a 70B model?
QLoRA on a single GPU with enough memory for 4-bit weights plus activations: about 53 GB by our rule of thumb, so one RTX PRO 6000 (96 GB) fits with room to spare.
LoRA or QLoRA?
QLoRA quantizes the frozen base model to 4 bits and needs about a quarter of the memory. LoRA in BF16 is simpler and can give slightly better quality. Many teams prototype with QLoRA and finish with LoRA.
Can I keep my data on the server between runs?
Yes. The server and its NVMe disks are yours for the paid month; renew to keep them. Back up before the term ends.
Start with 1× H200.
NVIDIA H200, 141 GB of GPU memory, for $2,279 a month.