Docker with GPUs: NVIDIA and ROCm containers

#Install Docker Engine

Install Docker Engine from Docker’s own repository, as described in Docker’s documentation:

sudo apt update
sudo apt install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

sudo tee /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF

sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo docker run --rm hello-world

To run docker without sudo, add your user to the docker group (sudo usermod -aG docker $USER, then log in again). Membership in that group is equivalent to root access on the server, so give it only to people who already have sudo.

#NVIDIA Container Toolkit

The toolkit lets containers use the host’s NVIDIA driver. Install the driver on the host first (NVIDIA drivers and CUDA); containers then need no driver of their own.

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update
sudo apt install -y nvidia-container-toolkit

sudo nvidia-ctk runtime configure --runtime=docker   # adds the nvidia runtime to /etc/docker/daemon.json
sudo systemctl restart docker

sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

Choose the GPUs of each container:

docker run --rm --gpus all ubuntu nvidia-smi -L
docker run --rm --gpus '"device=0,1"' ubuntu nvidia-smi -L   # GPUs 0 and 1 only

The toolkit mounts the host’s driver libraries into the container, so the CUDA release inside an image must be supported by the host driver (the CUDA Version shown by nvidia-smi). Installation details: NVIDIA Container Toolkit guide.

#NVIDIA NGC images

NVIDIA’s NGC catalog publishes images for PyTorch, Triton Inference Server, TensorRT-LLM and more, tagged by release (YY.MM). Public images pull without an account.

docker pull nvcr.io/nvidia/pytorch:<yy.mm>-py3
docker run --rm -it --gpus all --ipc=host \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -v /data:/workspace/data \
  nvcr.io/nvidia/pytorch:<yy.mm>-py3

--ipc=host and the two --ulimit flags are what NVIDIA recommends for its framework containers: PyTorch data loaders use shared memory, and Docker’s default is only 64 MB. Each release is built on a specific CUDA version; check the Frameworks Support Matrix against your driver before you pick a tag. For gated content, run docker login nvcr.io with the user name $oauthtoken and an NGC API key as the password.

#ROCm containers on MI355X

ROCm containers need no extra runtime: pass the AMD device nodes. Install the kernel driver on the host first (ROCm on MI355X).

docker run -it --rm \
  --device=/dev/kfd --device=/dev/dri \
  --group-add video --ipc=host \
  --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
  -v /data:/data \
  rocm/pytorch:latest

To limit a container to some GPUs, pass individual render nodes (--device=/dev/dri/renderD128 and so on) instead of /dev/dri. AMD’s Container Toolkit does this mapping for you: install the amd-container-toolkit package from AMD’s repository, run sudo amd-ctk runtime configure, restart Docker, then select GPUs by index:

docker run --rm --runtime=amd -e AMD_VISIBLE_DEVICES=0,1 rocm/dev-ubuntu-24.04 amd-smi list

On MI355X, use images built for ROCm 7.0 or later.

#GPUs in Docker Compose

Compose reserves NVIDIA GPUs through deploy.resources. This file runs vLLM on all GPUs, with the model cache on NVMe and the API published on localhost only:

# compose.yaml
services:
  vllm:
    image: vllm/vllm-openai:latest
    command: ["--model", "Qwen/Qwen3-4B", "--max-model-len", "8192"]
    environment:
      - VLLM_API_KEY=${VLLM_API_KEY}
      - HF_TOKEN=${HF_TOKEN}
    volumes:
      - /data/hf:/root/.cache/huggingface
    ports:
      - "127.0.0.1:8000:8000"
    ipc: host
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all          # or: device_ids: ["0", "1"]
              capabilities: [gpu]

export VLLM_API_KEY=$(openssl rand -hex 32)
docker compose up -d
docker compose logs -f vllm

count and device_ids are mutually exclusive, and capabilities is required (Compose GPU support). On MI355X, map the devices instead:

services:
  rocm:
    image: rocm/pytorch:latest
    command: sleep infinity
    devices:
      - /dev/kfd
      - /dev/dri
    group_add:
      - video
    ipc: host
    cap_add:
      - SYS_PTRACE
    security_opt:
      - seccomp=unconfined
    volumes:
      - /data:/data

#Keep data and images on local NVMe

Containers are disposable; your data should not live inside them. Mount directories from the NVMe data disk (see Storage and data) and keep model caches there too:

docker run --rm -it --gpus all \
  -v /data/datasets:/datasets:ro \
  -v /data/checkpoints:/checkpoints \
  -v /data/hf:/root/.cache/huggingface \
  nvcr.io/nvidia/pytorch:<yy.mm>-py3

Images and named volumes live in Docker’s data directory, /var/lib/docker by default. If the system disk is small, move it to the data disk:

sudo systemctl stop docker docker.socket
sudo mkdir -p /data/docker
sudo rsync -aHAX /var/lib/docker/ /data/docker/
sudo nano /etc/docker/daemon.json     # add "data-root" and keep the "runtimes" entry
sudo systemctl start docker
docker info --format '{{ .DockerRootDir }}'

{
  "data-root": "/data/docker",
  "runtimes": {
    "nvidia": {
      "args": [],
      "path": "nvidia-container-runtime"
    }
  }
}

Check space with docker system df and clean up with docker image prune. Everything on the server, Docker volumes included, is wiped when a term ends without renewal: keep a copy of what matters elsewhere.

#Ports, users and the firewall

• Docker publishes ports through its own firewall rules, before ufw sees the traffic (Docker’s firewall notes). A container started with -p 8000:8000 is reachable from the internet even if ufw blocks port 8000. Publish on the loopback address instead, -p 127.0.0.1:8000:8000, and reach the service through an SSH tunnel or a reverse proxy (see Inference serving).
• The docker group is equivalent to root: anyone who can start containers can mount the host’s filesystem.
• Pin image tags to a version rather than latest, so a rebuild does not change your stack without warning.