Serving LLMs with vLLM on dedicated GPUs

#Choose an engine

• vLLM: broad model support, an OpenAI-compatible server, NVIDIA and AMD (ROCm) builds. Used in the rest of this guide.
• SGLang: another open engine with an OpenAI-compatible API, started with python -m sglang.launch_server --model-path <model> --tp 2; it listens on 127.0.0.1:30000 by default. See the SGLang documentation.
• TensorRT-LLM: NVIDIA’s engine for NVIDIA GPUs; trtllm-serve <model> exposes an OpenAI-compatible endpoint. See the TensorRT-LLM documentation.

Size the GPUs from memory first. Weights take about 2 bytes per parameter in BF16 (a 70B model needs about 140 GB), 1 byte in FP8 and about half a byte in 4-bit formats. The KV cache comes on top and grows with context length and the number of concurrent requests. The sizing helper lists the configurations that fit a given model.

#Install vLLM and serve a first model

Install into its own Python environment, as recommended in the vLLM installation guide. Keep model downloads on the NVMe data disk:

curl -LsSf https://astral.sh/uv/install.sh | sh     # then open a new shell so uv is on PATH
uv venv --python 3.12 --seed --managed-python ~/vllm-env
source ~/vllm-env/bin/activate
uv pip install vllm --torch-backend=auto

export HF_HOME=/data/hf                            # model cache on NVMe
export VLLM_API_KEY=$(openssl rand -hex 32)        # clients must send this key
echo "$VLLM_API_KEY"                               # note it for your clients
vllm serve Qwen/Qwen3-4B --host 127.0.0.1 --port 8000 --max-model-len 8192

Run the server inside tmux (see SSH access). Gated models need a Hugging Face token: hf auth login, or HF_TOKEN in the environment. Test from a second shell on the server, with the same key exported:

curl -s http://127.0.0.1:8000/v1/models -H "Authorization: Bearer $VLLM_API_KEY"

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $VLLM_API_KEY" \
  -d '{"model": "Qwen/Qwen3-4B", "messages": [{"role": "user", "content": "Say hello in five words."}], "max_tokens": 64}'

Any OpenAI client works with base_url="http://127.0.0.1:8000/v1" and api_key set to your key.

#Spread a model over 2, 4 or 8 GPUs

vllm serve Qwen/Qwen3-32B --tensor-parallel-size 2 --host 127.0.0.1 --port 8000
vllm serve <model> --tensor-parallel-size 8 --host 127.0.0.1 --port 8000

• Tensor parallelism splits every layer across the GPUs and exchanges data at each step. The size must divide the model’s attention heads; 2, 4 and 8 are the usual values.
• On NVLink and NVSwitch platforms (8-GPU H100, H200, B200 and B300 servers), tensor parallelism is the default choice.
• Without NVLink (L40S, RTX PRO 6000, GeForce cards), vLLM’s parallelism guide suggests pipeline parallelism (--pipeline-parallel-size) for higher throughput. Measure both on your model.
• If the model fits on fewer GPUs, running several copies usually serves more requests than one large instance: --data-parallel-size, or separate servers pinned with CUDA_VISIBLE_DEVICES on different ports.

Setting up the GPUs for multi-GPU work is covered in Multi-GPU.

#FP8 and 4-bit quantization

# FP8 on the fly from a BF16 checkpoint (no calibration data needed)
vllm serve Qwen/Qwen3-32B --quantization fp8 --host 127.0.0.1

# pre-quantized checkpoints are detected from their config
vllm serve Qwen/Qwen3-32B-AWQ --host 127.0.0.1

# FP8 KV cache: more context or more concurrent requests per GB
vllm serve Qwen/Qwen3-32B --kv-cache-dtype fp8 --host 127.0.0.1

Quality loss depends on the model and the task. Compare the quantized model with the original on your own prompts before switching. vLLM’s quantization pages list which method runs on which hardware.

#Run vLLM in Docker

docker run --rm --runtime nvidia --gpus all \
  -v /data/hf:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" --env "VLLM_API_KEY=$VLLM_API_KEY" \
  -p 127.0.0.1:8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen3-4B --max-model-len 8192

The image runs vllm serve, so arguments after the image name go to vLLM. Publish the port on 127.0.0.1: Docker’s port rules bypass ufw. On MI355X, use vllm/vllm-openai-rocm with --device /dev/kfd --device /dev/dri --group-add=video instead of --gpus (see vLLM’s Docker page and Containers).

#Expose the endpoint safely

vLLM’s security guide is explicit: --api-key protects only the /v1, /v2 and /inference routes, and other routes on the same server, including one that runs inference, are not authenticated. Never publish the vLLM port itself. Choose one of two patterns.

SSH tunnel: for you and a small team

# on your computer; the API is then at http://localhost:8000/v1
ssh -N -L 8000:127.0.0.1:8000 gpu1

Reverse proxy with TLS: for a public endpoint

Point a DNS name at the server, install Caddy from its official repository and forward only the /v1/ routes. Caddy obtains and renews the TLS certificate itself; vLLM checks the API key on every forwarded request.

sudo tee /etc/caddy/Caddyfile >/dev/null <<'EOF'
llm.example.com {
    handle /v1/* {
        reverse_proxy 127.0.0.1:8000
    }
    handle {
        respond 404
    }
}
EOF
sudo systemctl reload caddy
sudo ufw allow 80,443/tcp

#Watch GPU utilization

nvidia-smi dmon -s um                         # utilization and memory, one line per GPU per second
nvidia-smi --query-gpu=index,utilization.gpu,memory.used,memory.total,power.draw --format=csv -l 5
nvtop                                         # interactive view (recent versions also show AMD GPUs)
watch -n 2 amd-smi monitor                    # AMD MI355X
curl -s http://127.0.0.1:8000/metrics | grep -E '^vllm:(num_requests_running|num_requests_waiting|kv_cache_usage_perc)'

vLLM reserves most of the GPU memory for its KV cache at start-up (--gpu-memory-utilization), so high memory use is normal and says nothing about load. GPU-Util in nvidia-smi is the share of time a kernel was running, not how much of the GPU’s compute was used.

Watch the running and waiting requests and the KV cache usage instead (vLLM metrics). Requests that pile up in waiting while the cache is nearly full mean you need more GPUs, a quantized model or shorter contexts.