Fine-tune an LLM with LoRA and QLoRA

LoRA, QLoRA or full fine-tuning: how much GPU memory?

To fine-tune an LLM with LoRA or QLoRA on a GPU server, you train a small adapter on top of a frozen model with Hugging Face TRL and PEFT. LoRA keeps the base model in BF16 and trains low-rank matrices added to its layers; QLoRA does the same on a base model loaded in 4-bit with bitsandbytes. Full fine-tuning instead updates every weight and keeps gradients and optimizer states for all of them. CryptGPU’s sizing rule, which counts 92% of GPU memory as usable and includes a margin for activations, budgets about 3 GB per billion parameters for LoRA, 0.75 GB for QLoRA and 18 GB for full fine-tuning.

Memory needed, then the cheapest server that holds it, per month:

• Start with QLoRA to check your data and settings on the smallest server, then move to LoRA in BF16 if quality requires it.
• One GPU is simpler. With several GPUs the model is split or sharded (see below), and GeForce cards exchange data over PCIe. A 70B QLoRA run (53 GB) also fits on one H100 (80 GB, $1,779 a month) or one RTX PRO 6000 (96 GB, $959), and a 32B LoRA run (96 GB) on one H200 (141 GB, $2,279).
• The MI355X runs ROCm. PyTorch, bitsandbytes, TRL and PEFT work on it (below). On NVIDIA only, the cheapest fit for a 70B LoRA run is 4× RTX PRO 6000, $3,836.

The sizing helper covers other sizes, and How much GPU memory does your model need? explains the rule. A monthly server keeps your data and environment between runs; for a single short run, an hourly cloud can cost less. See GPU servers for fine-tuning.

Set up the environment

If you picked the fine-tuning kit (Axolotl, TRL and PEFT) in the Software step of your order, it is already installed; the steps below set it up by hand. Install the stack in its own Python environment and keep downloads on the NVMe data disk (see Storage and data). With --torch-backend=auto, uv detects the GPU driver and installs the matching PyTorch build, CUDA on NVIDIA and ROCm on AMD (uv PyTorch guide).

curl -LsSf https://astral.sh/uv/install.sh | sh     # then open a new shell so uv is on PATH
uv venv --python 3.12 --seed --managed-python ~/ft-env
source ~/ft-env/bin/activate
uv pip install torch "trl[peft]" bitsandbytes --torch-backend=auto

export HF_HOME=/data/hf               # models, datasets and your token on NVMe
export HF_HUB_DISABLE_TELEMETRY=1     # no usage telemetry from Hugging Face libraries
mkdir -p /data/ft                     # dataset, scripts and outputs
hf auth login                         # gated models only, such as Llama
trl env                               # library versions and the GPUs found

Add the two export lines to ~/.bashrc to keep them. On the MI355X, check that python -c "import torch; print(torch.cuda.get_arch_list())" lists gfx950, or work in a rocm/pytorch container (see Containers). bitsandbytes publishes ROCm wheels that include gfx950, and AMD’s fine-tuning guide uses TRL and PEFT on Instinct GPUs.

Prepare a chat dataset in JSONL

TRL’s SFTTrainer reads each conversation as a messages list of role and content entries and applies the model’s chat template itself (TRL dataset formats). Write one conversation per line in /data/ft/train.jsonl:

{"messages": [{"role": "system", "content": "You answer customer emails briefly."}, {"role": "user", "content": "My order arrived damaged."}, {"role": "assistant", "content": "Sorry about that. Send a photo and we will ship a replacement."}]}
{"messages": [{"role": "user", "content": "Can I change the delivery address?"}, {"role": "assistant", "content": "Yes, until the order ships. Reply with the new address."}]}

Use examples that look like the requests the model will get, remove duplicates, and leave out data you have no right to train on. The script below keeps 5% of the lines for evaluation. Print one example through the chat template to check the roles and special tokens:

python - <<'EOF'
from datasets import load_dataset
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
row = load_dataset("json", data_files="/data/ft/train.jsonl", split="train")[0]
print(tok.apply_chat_template(row["messages"], tokenize=False))
EOF

Option A: QLoRA or LoRA with TRL SFTTrainer

This script fine-tunes Qwen3-8B on one GPU with the current TRL API (TRL 1.16, PEFT 0.21). QLORA = True loads the base model in 4-bit NF4 with bitsandbytes. For Llama 3.1, accept its license on Hugging Face, run hf auth login and change MODEL.

# /data/ft/train_sft.py: LoRA or QLoRA with TRL
import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

MODEL = "Qwen/Qwen3-8B"
QLORA = True                       # False: LoRA on the BF16 model
OUT = "/data/ft/qwen3-8b-lora"

data = load_dataset("json", data_files="/data/ft/train.jsonl", split="train")
data = data.train_test_split(test_size=0.05, seed=42)

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
) if QLORA else None

lora = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                  target_modules="all-linear", task_type="CAUSAL_LM")

args = SFTConfig(
    output_dir=OUT,
    model_init_kwargs={"dtype": torch.bfloat16},
    max_length=2048,
    assistant_only_loss=True,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    per_device_eval_batch_size=2,
    num_train_epochs=2,
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_steps=0.03,             # values below 1 are fractions of the run
    logging_steps=10,
    eval_strategy="steps",
    eval_steps=0.1,                # evaluate and save every 10% of the run
    save_steps=0.1,
    save_total_limit=3,
)

trainer = SFTTrainer(
    model=MODEL,
    args=args,
    train_dataset=data["train"],
    eval_dataset=data["test"],
    quantization_config=bnb,
    peft_config=lora,
)
trainer.train()
trainer.save_model(OUT)            # the LoRA adapter

tmux new -s ft                     # the run survives a dropped SSH session
source ~/ft-env/bin/activate
cd /data/ft
CUDA_VISIBLE_DEVICES=0 python train_sft.py 2>&1 | tee train.log

The values are starting points:

• target_modules="all-linear" puts adapters on all linear layers except the output layer, as TRL’s LoRA Without Regret guide recommends; r=16 and lora_alpha=32 are TRL’s command-line defaults.
• learning_rate=2e-4 is TRL’s suggestion for SFT with LoRA, ten times its full fine-tuning default (PEFT integration). The effective batch, 2 × 8 = 16 sequences, stays under the 32 that TRL advises for LoRA. Unsloth’s guide suggests one to three epochs.
• model_init_kwargs loads the weights in BF16; otherwise TRL loads them in float32. Gradient checkpointing and BF16 training are on by default.
• assistant_only_loss=True computes the loss on the assistant replies only. It needs generation markers in the chat template, which TRL adds for known families such as Qwen3 and Llama 3 (SFT Trainer).
• max_length truncates longer conversations; TRL’s default is 1,024 tokens.

On a multi-GPU server, CUDA_VISIBLE_DEVICES=0 keeps the run on one GPU; a plain python run would spread the model over every GPU.

Option B: Unsloth on a single GPU

Unsloth replaces parts of the model with its own Triton kernels to train with less memory and in less time on one GPU, and it plugs into the same SFTTrainer. Install it in its own environment:

uv venv ~/unsloth_env --python 3.13
source ~/unsloth_env/bin/activate
uv pip install unsloth --torch-backend=auto

# /data/ft/train_unsloth.py
from unsloth import FastLanguageModel
from datasets import load_dataset
from trl import SFTConfig, SFTTrainer

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="Qwen/Qwen3-8B",
    max_seq_length=2048,
    load_in_4bit=True,             # False: LoRA on a 16-bit base
)
model = FastLanguageModel.get_peft_model(
    model, r=16, lora_alpha=16, lora_dropout=0,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    use_gradient_checkpointing="unsloth",
)
data = load_dataset("json", data_files="/data/ft/train.jsonl", split="train")
data = data.map(lambda x: {"text": tokenizer.apply_chat_template(x["messages"], tokenize=False)},
                remove_columns=["messages"])
trainer = SFTTrainer(
    model=model,
    processing_class=tokenizer,
    train_dataset=data,
    args=SFTConfig(output_dir="/data/ft/unsloth", max_length=2048,
                   per_device_train_batch_size=2, gradient_accumulation_steps=8,
                   learning_rate=2e-4, num_train_epochs=1),
)
trainer.train()
model.save_pretrained("/data/ft/unsloth-lora")
tokenizer.save_pretrained("/data/ft/unsloth-lora")

The values follow Unsloth’s hyperparameter guide (rank 16 or 32, alpha equal to the rank or twice it, learning rate 2e-4, effective batch 16). model.save_pretrained_merged("/data/ft/unsloth-merged", tokenizer, save_method="merged_16bit") writes a merged 16-bit model for vLLM. Unsloth’s AMD guide lists the MI350 series (gfx950) on Linux, with separate install steps.

Train on 2, 4 or 8 GPUs

The TRL script also runs on several GPUs (prepare the server as in Multi-GPU). When the model fits on one GPU, data parallelism (DDP) gives each GPU a full copy and its own batches; start one process per GPU:

accelerate launch --num_processes 4 train_sft.py
torchrun --standalone --nproc-per-node=4 train_sft.py     # or with torchrun

The effective batch becomes per-GPU batch × GPUs × accumulation steps (TRL distributed training): on four GPUs, set gradient_accumulation_steps=2 to keep 16.

If the model does not fit on one GPU (LoRA on a 70B model in BF16, full fine-tuning), shard it with FSDP2, which is built into PyTorch and handles LoRA out of the box, or with DeepSpeed ZeRO-3. TRL publishes Accelerate templates for both. Set QLORA = False first: QLoRA with FSDP needs the extra settings in PEFT’s FSDP guide. For full fine-tuning, also remove peft_config and return to TRL’s default learning rate, 2e-5.

curl -LO https://raw.githubusercontent.com/huggingface/trl/main/examples/accelerate_configs/fsdp2.yaml
accelerate launch --config_file fsdp2.yaml --num_processes 8 train_sft.py

uv pip install deepspeed        # builds some of its ops on first use
curl -LO https://raw.githubusercontent.com/huggingface/trl/main/examples/accelerate_configs/deepspeed_zero3.yaml
accelerate launch --config_file deepspeed_zero3.yaml --num_processes 8 train_sft.py

A QLoRA run that needs two GPUs, such as 70B on 2× RTX 5090, can start with plain python train_sft.py instead: Transformers places the layers across both GPUs, which then work one after the other.

Monitor the run and resume

tmux attach -t ft                  # back to the training session
nvidia-smi --query-gpu=index,utilization.gpu,memory.used,memory.total --format=csv -l 5
watch -n 2 amd-smi monitor         # AMD MI355X
tail -f /data/ft/train.log         # loss every 10 steps, eval_loss every 10% of the run

The training loss should fall, and the eval loss with it. If the eval loss climbs while the training loss keeps falling, the model is memorizing its data: keep an earlier checkpoint or train fewer epochs. If the run stops with an out-of-memory error, halve per_device_train_batch_size and double gradient_accumulation_steps, shorten max_length, or switch to QLoRA.

At each save the trainer writes a checkpoint-N folder in the output directory, N being the step, and save_total_limit=3 keeps the last three. After an interruption, change the call to trainer.train(resume_from_checkpoint=True) to continue from the latest one.

Merge the adapter or serve it with vLLM

Serve the adapter next to the base model it was trained on, or merge it into a standalone checkpoint. Run vLLM from its own environment, with an API key and an SSH tunnel, as in Inference serving.

Serve the adapter

vllm serve Qwen/Qwen3-8B --host 127.0.0.1 --port 8000 \
  --enable-lora --lora-modules support=/data/ft/qwen3-8b-lora --max-lora-rank 16

Clients pick the adapter with "model": "support". --max-lora-rank must be at least the adapter’s r, and its default is 16 (vLLM LoRA adapters).

Merge into the base model

python - <<'EOF'
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", dtype=torch.bfloat16)
model = PeftModel.from_pretrained(base, "/data/ft/qwen3-8b-lora").merge_and_unload()
model.save_pretrained("/data/ft/qwen3-8b-merged")
AutoTokenizer.from_pretrained("Qwen/Qwen3-8B").save_pretrained("/data/ft/qwen3-8b-merged")
EOF
vllm serve /data/ft/qwen3-8b-merged --host 127.0.0.1 --port 8000

merge_and_unload() folds the adapter into the weights, so inference carries no adapter overhead (PEFT LoRA guide). The merge loads the base model in BF16, about 2 bytes of memory per parameter, and gives a normal checkpoint that vLLM can also serve in FP8.

Back up checkpoints and check the license

If a term is not renewed, the server stops at the end of the paid month and its disks are wiped. After each run, and well before the term ends, copy the adapters, checkpoints, dataset and scripts off the server and verify the copy, as described in Storage and data:

rclone copy /data/ft s3remote:my-bucket/ft --progress
rclone check /data/ft s3remote:my-bucket/ft

A fine-tune remains subject to its base model’s license. Qwen3-8B is released under Apache 2.0. Llama 3.1 models are gated, and their community license asks, among other terms, that a distributed fine-tune have “Llama” at the beginning of its name and display “Built with Llama”. Read the license of your base model, and the terms of your data, before you publish or sell the result.