Multi-GPU on one server: topology, NCCL and torchrun

#Read the topology first

The links between GPUs decide which kind of parallelism is cheap. Look before you choose:

nvidia-smi topo -m        # connection matrix between GPUs, NICs and CPUs
nvidia-smi topo -p2p r    # which GPU pairs support peer-to-peer reads
amd-smi topology          # AMD MI355X

The CPU Affinity and NUMA Affinity columns show which cores are closest to each GPU. Pin data-loading work to them when the CPU side is the bottleneck.

#Launch 2, 4 or 8 GPU jobs with torchrun

torchrun starts one process per GPU and gives each one RANK, LOCAL_RANK and WORLD_SIZE. Test the setup with a short script before a long job:

# ddp_check.py
import os
import torch
import torch.distributed as dist

local_rank = int(os.environ["LOCAL_RANK"])
torch.cuda.set_device(local_rank)
dist.init_process_group("nccl")

x = torch.ones(1, device="cuda") * dist.get_rank()
dist.all_reduce(x)          # sums the rank numbers over all GPUs
print(f"rank {dist.get_rank()} of {dist.get_world_size()}: sum = {x.item():.0f}")
dist.destroy_process_group()

torchrun --standalone --nproc-per-node=gpu ddp_check.py   # one process per visible GPU
torchrun --standalone --nproc-per-node=4 train.py         # four processes
CUDA_VISIBLE_DEVICES=4,5,6,7 torchrun --standalone --nproc-per-node=4 train.py   # GPUs 4 to 7 only

Every rank should print the same sum, 0 + 1 + … + (n − 1): 28 on eight GPUs. CUDA_VISIBLE_DEVICES lets two independent jobs share a server, each on its own GPUs. torchrun sets OMP_NUM_THREADS=1 for each process unless you set it yourself; raise it if CPU work inside the training processes is slow.

#NCCL essentials

NCCL is the library behind PyTorch’s nccl backend. It detects the topology and picks NVLink, PCIe peer-to-peer, shared memory or the network by itself, and its defaults suit most jobs. These variables show and debug what it chose:

NCCL_DEBUG=INFO torchrun --standalone --nproc-per-node=gpu ddp_check.py 2>&1 | grep -E 'via (P2P|SHM|NET)' | head

#Measure bus bandwidth with nccl-tests

nccl-tests measures what the interconnect delivers. It needs the CUDA toolkit and NCCL’s development files from NVIDIA’s repository (see NVIDIA drivers and CUDA); an NGC PyTorch container already has both.

sudo apt install -y libnccl2 libnccl-dev
git clone https://github.com/NVIDIA/nccl-tests.git
cd nccl-tests
make -j                                          # CUDA_HOME defaults to /usr/local/cuda
./build/all_reduce_perf -b 8 -e 2G -f 2 -g 8     # -g: number of GPUs in the server

-b and -e set the smallest and largest message size, -f the factor between sizes and -g the number of GPUs. Each output line is one message size:

• algbw: data size divided by time.
• busbw: algbw corrected for the number of GPUs (for all-reduce, × 2(n − 1)/n). It reflects the speed of the slowest link in the path (NVLink, PCIe or the link between CPU sockets) and is the figure to compare between servers.
• #wrong must stay at 0. The run ends with an Avg bus bandwidth line.

Read the large-message rows. Record the result while the server is healthy, so you have a reference if performance changes later.

#PCIe cards and NVLink platforms

• 8-GPU H100, H200, B200 and B300 servers (HGX): every GPU reaches every other one through NVSwitch at full NVLink speed, 900 GB/s per GPU on Hopper and 1.8 TB/s on Blackwell. Tensor parallelism across all eight GPUs works well. Fabric Manager must be running (see NVIDIA drivers and CUDA). In 2- and 4-GPU servers of these models, check nvidia-smi topo -m to see how the GPUs are linked.
• MI355X: the GPUs are linked by AMD Infinity Fabric, which RCCL uses (below).
• RTX PRO 6000 and L40S: PCIe only (Gen5 and Gen4). Whether GPUs can exchange data peer-to-peer depends on the card and the platform; check with nvidia-smi topo -p2p r.
• GeForce RTX 5090, 4090 and 5080: PCIe only, no NVLink, and peer-to-peer is usually not available, so NCCL passes data through host memory. It works, but communication costs more.

On PCIe servers, favor data parallelism (DDP or FSDP) with enough work per step, and use gradient accumulation to communicate less often. Tensor parallelism exchanges data at every layer and benefits most from NVLink.

If a job hangs at its first collective on a PCIe server, run it once with NCCL_P2P_DISABLE=1. If that helps, the peer-to-peer path is the problem: report it through the contact form.

#Larger models: FSDP and DeepSpeed

• DDP (DistributedDataParallel): every GPU holds a full copy of the model and optimizer state. Simplest and fastest when everything fits in one GPU’s memory.
• FSDP2 (torch.distributed.fsdp.fully_shard): shards parameters, gradients and optimizer state across the GPUs, so the model only has to fit across all of them. Launched with torchrun like DDP; see PyTorch’s FSDP2 tutorial.
• DeepSpeed ZeRO: stages 1 to 3 shard optimizer state, then gradients, then parameters, with optional offload to CPU memory or NVMe. pip install deepspeed; see Getting started.
• Tensor and pipeline parallelism (for example Megatron-style): split layers, or groups of layers, across GPUs. Used for the largest models; tensor parallelism needs a fast interconnect.

Hugging Face Accelerate and the Transformers Trainer wrap DDP, FSDP and DeepSpeed behind a configuration file. For serving rather than training, see Inference serving.

#AMD MI355X: RCCL

On MI355X the same code runs unchanged. PyTorch’s nccl backend uses RCCL, AMD’s implementation of the NCCL API; torchrun works the same way, and RCCL reads the same NCCL_* variables, such as NCCL_DEBUG=INFO. Select GPUs with HIP_VISIBLE_DEVICES; CUDA_VISIBLE_DEVICES is also accepted.

amd-smi topology                           # links between the GPUs
git clone https://github.com/ROCm/rccl-tests.git
cd rccl-tests
make GPU_TARGETS=gfx950                    # add HIP_HOME=... if ROCm is not in /opt/rocm
./build/all_reduce_perf -b 8 -e 2G -f 2 -g 8

The simplest place to build rccl-tests is a rocm/pytorch container, where ROCm and RCCL are already installed. Installation and GPU selection are covered in ROCm on MI355X.