NVIDIA drivers, CUDA and Fabric Manager on Ubuntu

#Check what is installed

Find out whether a driver is already present and where it came from:

lspci -d 10de: | grep -Ei '3d controller|vga compatible'   # the GPUs on the PCI bus
nvidia-smi                                  # fails if no driver is loaded
cat /proc/driver/nvidia/version             # version of the loaded kernel module
dpkg -l | grep -Ei 'nvidia|cuda' | awk '{print $2, $3}'   # installed packages
mokutil --sb-state                          # is Secure Boot enabled?

The header of nvidia-smi shows the driver version and a CUDA Version. That second number is the newest CUDA release the driver can run, not a toolkit that is installed.

If a working driver is already there and suits your stack, keep it. If you replace it, remove the old packages first and install from a single source: mixing Ubuntu’s packages, NVIDIA’s repository and .run installers is a frequent cause of broken GPU setups.

#Install the driver

NVIDIA’s CUDA repository provides the driver for the data center cards and for the GeForce cards. Install it with the open kernel modules (nvidia-open): NVIDIA supports only the open modules on Blackwell (B300, B200, RTX PRO 6000, RTX 5090, RTX 5080) and recommends them for Hopper and Ada Lovelace.

sudo apt update
sudo apt install -y linux-headers-$(uname -r)

# register NVIDIA's CUDA repository (x86_64; Arm64 servers use sbsa instead)
distro=$(. /etc/os-release; echo "ubuntu${VERSION_ID//./}")   # ubuntu2204 or ubuntu2404
wget https://developer.download.nvidia.com/compute/cuda/repos/$distro/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update

# optional, recommended: lock a driver branch before installing it
sudo apt install -V nvidia-driver-pinning-<branch>

sudo apt install -V nvidia-open
sudo reboot

<branch> is a driver branch number listed on NVIDIA’s version-locking page. The lock keeps apt upgrade from moving you to a new branch in the middle of a project. Since branch 590, Ubuntu package names no longer contain the branch number, so the pinning package is how you stay on one. After the reboot, nvidia-smi should list every GPU.

Alternative: Ubuntu’s own packages

Ubuntu packages the same drivers with precompiled, signed kernel modules. Prefer this route when Secure Boot is enabled: a module that DKMS builds locally is not signed with a key the firmware trusts, and enrolling one requires access to the boot console.

sudo ubuntu-drivers list --gpgpu                 # drivers that match these GPUs
sudo ubuntu-drivers install --gpgpu              # install the recommended one
sudo apt install nvidia-utils-<branch>-server    # nvidia-smi for that branch
sudo reboot

On Blackwell GPUs, make sure the selected driver is an -open variant. Use one source or the other, never both. Details: Ubuntu Server documentation and NVIDIA’s driver installation guide.

#Add the CUDA toolkit if you compile code

PyTorch wheels, vLLM and NGC containers bring their own CUDA libraries; the host only needs a driver that is new enough for them. Install the toolkit when you compile CUDA code yourself: nvcc, custom kernels, nccl-tests, or Python packages that build CUDA extensions.

sudo apt install -V cuda-toolkit                # latest CUDA release in the repository
# or a fixed release: sudo apt install -V cuda-toolkit-<major>-<minor>
echo 'export PATH=/usr/local/cuda/bin:$PATH' >> ~/.bashrc
source ~/.bashrc
nvcc --version

Each CUDA release needs a minimum driver; CUDA 13.x, for example, needs a driver from the 580 series or newer. The CUDA release notes list the requirement for every version.

#Persistence mode

Without persistence, the driver tears down the GPU state when the last process exits and rebuilds it for the next one, which slows the start of every short job. On a server, keep it on:

nvidia-smi --query-gpu=index,persistence_mode --format=csv
sudo systemctl enable --now nvidia-persistenced
sudo nvidia-smi -pm 1

NVIDIA recommends the persistence daemon (nvidia-persistenced) over the older in-driver setting, and nvidia-smi -pm 1 goes through the daemon when it runs. If the query shows Disabled again after a reboot, run systemctl cat nvidia-persistenced: some packages start the daemon with --no-persistence-mode. Override it with sudo systemctl edit nvidia-persistenced (an empty ExecStart= line, then the original command without that flag) and restart the service.

#HGX servers: Fabric Manager and NVLink

On 8-GPU HGX boards (H100, H200, B200, B300) the GPUs talk to each other through NVSwitch chips, which NVIDIA Fabric Manager configures. Without it, nvidia-smi still lists every GPU, but CUDA programs fail at start-up with error 802, “system not yet initialized” (cudaErrorSystemNotReady).

Fabric Manager must be exactly the same version as the driver. Install it from the same repository, keep both on the same branch and upgrade them together.

cat /proc/driver/nvidia/version                               # driver version
apt-cache search --names-only 'fabricmanager|nvlsm|nvlink5'   # what your repository offers

• NVIDIA repository, branch 590 and later (unversioned names, held on your branch by the pinning package): sudo apt install -V nvidia-fabricmanager
• NVIDIA repository, earlier branches: sudo apt install -V cuda-drivers-fabricmanager-<branch>
• Ubuntu’s packages: sudo apt install nvidia-fabricmanager-<branch> libnvidia-nscq-<branch>

HGX B200 and B300 use fourth-generation NVSwitch and also need the NVLink Subnet Manager (nvlsm), libibumad3, infiniband-diags and the ib_umad kernel module; with branch-numbered packages, nvlink5-<branch> installs Fabric Manager and NVLSM together. The Fabric Manager guide lists the exact packages for each generation.

# HGX B200 / B300 only
sudo apt install -V nvlsm libibumad3 infiniband-diags
sudo modprobe ib_umad
echo ib_umad | sudo tee /etc/modules-load.d/ib_umad.conf

# every HGX board
sudo systemctl enable --now nvidia-fabricmanager
systemctl status nvidia-fabricmanager --no-pager
dpkg -l | grep -i fabricmanager                  # same version as the driver?
nvidia-smi -q -i 0 | grep -i -A 2 fabric         # expect State: Completed, Status: Success

Check NVLink

nvidia-smi topo -m     # on an HGX board every GPU pair shows NV# (NV18 on HGX H100)
nvidia-smi nvlink -s   # state and speed of each link
nvidia-smi nvlink -e   # error counters per link

An entry such as PIX, PXB, NODE or SYS between two GPUs of an HGX board means that pair is not using NVLink. To measure bandwidth, run nccl-tests as described in Multi-GPU.

#MIG: split a GPU into instances

Multi-Instance GPU splits one GPU into isolated instances, each with its own memory and compute, for example to serve several small models side by side. In the CryptGPU range:

sudo nvidia-smi -i 0 -mig 1                  # MIG mode on GPU 0 (stop its jobs first)
nvidia-smi mig -i 0 -lgip                    # instance profiles this GPU offers, with IDs
sudo nvidia-smi mig -i 0 -cgi <id>,<id> -C   # create GPU instances and their compute instances
nvidia-smi -L                                # MIG devices and their UUIDs

sudo nvidia-smi mig -i 0 -dci && sudo nvidia-smi mig -i 0 -dgi   # remove the instances
sudo nvidia-smi -i 0 -mig 0                  # back to one whole GPU

Point a process at one instance with CUDA_VISIBLE_DEVICES=MIG-<uuid>. On Hopper and Blackwell, MIG mode and its instances do not survive a reboot; recreate them at boot, for example with NVIDIA’s mig-parted. The MIG user guide lists every profile.

#Common errors and fixes

“NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver”

The kernel module is not loaded. The usual causes are a new kernel without matching headers, so DKMS could not build the module, or Secure Boot rejecting an unsigned module.

lsmod | grep nvidia
dkms status
sudo dmesg | grep -iE 'nvidia|nvrm' | tail -n 20
mokutil --sb-state
sudo apt install -y linux-headers-$(uname -r) && sudo dkms autoinstall && sudo reboot

“Failed to initialize NVML: Driver/library version mismatch”

The driver packages were upgraded while the old kernel module stayed loaded. Reboot. To avoid it during a run, lock the driver branch and upgrade only between jobs.

“CUDA driver version is insufficient for CUDA runtime version”

The program or container was built for a newer CUDA release than your driver supports (compare with the CUDA Version shown by nvidia-smi). Upgrade the driver, or use a build for an older CUDA release.

Error 802 on an HGX server

Fabric Manager is not running, failed at start-up or does not match the driver version. Check systemctl status nvidia-fabricmanager and sudo journalctl -u nvidia-fabricmanager.

Xid errors

The driver reports GPU faults in the kernel log as NVRM: Xid lines:

sudo dmesg -T | grep -i xid

If a hardware-type Xid (48, 64, 74, 79 or 95) returns after a reboot, run sudo nvidia-bug-report.sh and send the Xid lines through the contact form. NVIDIA’s Xid catalog explains every code.