Serve LLMs with SGLang on a GPU server

SGLang on a GPU server: what you need

To deploy SGLang on a GPU server, install it with uv or run its Docker image, start sglang serve with a Hugging Face model on 127.0.0.1:30000 behind an API key, and send OpenAI-style requests to /v1. Reach it through an SSH tunnel, or publish only /v1/ through a TLS proxy. NVIDIA GPUs need a driver for CUDA 13, from the 580 series or newer (NVIDIA drivers and CUDA); the AMD MI355X runs SGLang’s ROCm images.

If you picked SGLang in the Software step of your order, it is already installed with the model you chose; this guide covers installing and running it yourself. For how SGLang compares with vLLM, Ollama and llama.cpp, see vLLM vs Ollama vs llama.cpp vs SGLang; for vLLM, Inference serving. The examples use two catalog models:

The rule adds 20% to the weights for the KV cache and counts 92% of GPU memory as usable, as on our model pages, which list every server that fits. Long contexts and many concurrent requests need more. Both checkpoints are ungated; gated models need HF_TOKEN in the environment.

Install SGLang with uv or Docker

Run the commands in this guide as root. SGLang needs Python 3.11 or newer (Ubuntu 22.04 ships 3.10), so uv installs Python 3.12 under /opt, where the service account below can read it (installation guide):

nvidia-smi                                         # header: CUDA Version 13.0 or higher
curl -LsSf https://astral.sh/uv/install.sh | sh     # then open a new shell so uv is on PATH
export UV_PYTHON_INSTALL_DIR=/opt/uv-python
uv venv --python 3.12 --managed-python /opt/sglang
source /opt/sglang/bin/activate
uv pip install --prerelease=allow sglang
sglang version

Keep --prerelease=allow: some dependencies publish only pre-releases, and without the flag uv versions older than 0.12.0 silently install SGLang 0.5.9. For Docker, install Docker and the NVIDIA Container Toolkit (Containers) and use a release tag rather than latest: v0.5.22 was the newest when checked on 20 October 2026 (Docker Hub tags).

Launch the server with sglang serve

Create the API key, keep the model cache on the NVMe disk (Storage and data) and start the server inside tmux (SSH access). SGLang downloads the model on first start:

export SGLANG_API_KEY=$(openssl rand -hex 32)
echo "$SGLANG_API_KEY"                    # note it for your clients
export HF_HOME=/data/hf

sglang serve --model-path openai/gpt-oss-120b \
  --host 127.0.0.1 --port 30000 --api-key "$SGLANG_API_KEY" \
  --reasoning-parser gpt-oss --tool-call-parser gpt-oss

Wait for The server is fired up and ready to roll! in the log. 127.0.0.1 and port 30000 are the defaults, written out here and omitted below. The older python3 -m sglang.launch_server takes the same arguments and still works, but sglang serve is now the recommended entry point; sglang serve --help lists every flag (server arguments).

With Docker, the server listens on all interfaces inside the container; -p 127.0.0.1:30000:30000 keeps the published port on the host’s loopback, since Docker’s port rules bypass ufw:

docker run -d --name sglang --restart unless-stopped \
  --gpus all --ipc=host --shm-size 32g \
  -v /data/hf:/root/.cache/huggingface \
  -p 127.0.0.1:30000:30000 \
  lmsysorg/sglang:v0.5.22 \
  sglang serve --model-path openai/gpt-oss-120b --host 0.0.0.0 --port 30000 \
    --api-key "$SGLANG_API_KEY" --reasoning-parser gpt-oss --tool-call-parser gpt-oss
docker logs -f sglang

Test from a second shell on the server, with the same key exported:

curl -s http://127.0.0.1:30000/v1/models -H "Authorization: Bearer $SGLANG_API_KEY"

curl -s http://127.0.0.1:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $SGLANG_API_KEY" \
  -d '{"model": "openai/gpt-oss-120b", "messages": [{"role": "user", "content": "Say hello in five words."}]}'

The model field must match --model-path unless you set a shorter --served-model-name. Any OpenAI client works with base_url="http://127.0.0.1:30000/v1" and api_key set to your key.

SGLang multi-GPU: tensor, expert and data parallelism

# one model split across 2, 4 or 8 GPUs
sglang serve --model-path openai/gpt-oss-120b --tp 2 --api-key "$SGLANG_API_KEY"

# mixture-of-experts model: spread the experts as well
sglang serve --model-path deepseek-ai/DeepSeek-R1-0528 --tp 8 --ep 8 \
  --reasoning-parser deepseek-r1 --api-key "$SGLANG_API_KEY"

# model that fits on one GPU: one copy per GPU
sglang serve --model-path openai/gpt-oss-20b --dp-size 8 --api-key "$SGLANG_API_KEY"

• Tensor parallelism (--tp) splits every layer and exchanges data between the GPUs at each step, so it benefits most from NVLink or Infinity Fabric. If start-up fails with “peer access is not supported between these two devices” (on PCIe cards), add --enable-p2p-check.
• Expert parallelism (--ep) places the experts of a mixture-of-experts model on different GPUs. With the default backend it can be at most --tp; DeepEP and the other all-to-all backends need it equal (expert parallelism). DeepSeek-R1-0528 needs about 827 GB by our rule; SGLang’s cookbook runs its FP8 weights on 8× H200 or 8× B200.
• Data parallelism (--dp-size) runs copies of the model behind one port. SGLang’s tuning guide favors it for throughput when memory allows and recommends its Model Gateway for larger setups.

Topology checks and interconnect tests: Multi-GPU.

Memory settings, FP8 and out-of-memory errors

At start-up SGLang reserves a fixed share of each GPU’s memory, --mem-fraction-static, for the weights and the KV cache pool; the rest holds activations and CUDA graph buffers. The log line before the server is ready shows max_total_num_tokens, the pool size in tokens, and available_gpu_mem: SGLang’s tuning guide aims for 5–8 GB left, raising the fraction if more remains.

SGLang recommends pre-quantized checkpoints over conversion at load time: load Qwen’s own Qwen/Qwen3.8-27B-FP8 without --quantization, and never add the flag to an already quantized checkpoint such as gpt-oss-120b (MXFP4) or DeepSeek-R1-0528 (FP8) (quantization). An FP8 KV cache is only fast with an attention backend that supports it (quantized KV cache). Qwen3.8 27B, a hybrid model, has extra memory flags (--mamba-*): its cookbook page gives complete commands for the H200 and RTX PRO 6000. A “CUDA error: illegal memory access” can be an out-of-memory error in disguise (troubleshooting).

SGLang OpenAI API: structured outputs, tools and reasoning

The server speaks the OpenAI Chat Completions and Completions APIs under /v1, plus an Anthropic-compatible /v1/messages. Start a reasoning model with its reasoning parser and the reasoning arrives in reasoning_content, apart from the answer in content; with a tool-call parser, tool calls arrive as structured tool_calls instead of raw text. Use the parsers each model’s cookbook page names:

Structured outputs constrain the answer to a JSON schema, a regular expression or an EBNF grammar; XGrammar is the default backend:

curl -s http://127.0.0.1:30000/v1/chat/completions \
  -H "Content-Type: application/json" -H "Authorization: Bearer $SGLANG_API_KEY" \
  -d '{"model": "openai/gpt-oss-120b",
       "messages": [{"role": "user", "content": "Capital of France and its population, as JSON."}],
       "response_format": {"type": "json_schema", "json_schema": {"name": "city", "schema": {
         "type": "object",
         "properties": {"name": {"type": "string"}, "population": {"type": "integer"}},
         "required": ["name", "population"]}}}}'

SGLang advises asking for the format in the prompt as well. Details: structured outputs, tool parser, reasoning parser.

SGLang on AMD MI355X

On the MI355X, with 288 GB per GPU, SGLang’s AMD page recommends Docker, with prebuilt ROCm images; the host needs only the AMD kernel driver (ROCm on MI355X). Tags name the release, the ROCm version and the GPU family, mi35x being gfx950. The v0.5.22 release notes list v0.5.22-rocm10-mi35x; Docker Hub also carries ROCm 7.2 builds (-rocm720-mi35x, -rocm724-mi35x): if the container does not see the GPUs, pick the build closest to the ROCm version on the host (amd-smi version).

echo 0 > /proc/sys/kernel/numa_balancing      # AMD page recommendation; lasts until reboot
docker run -d --name sglang --restart unless-stopped \
  --device=/dev/kfd --device=/dev/dri --group-add video \
  --ipc=host --shm-size 16g \
  --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
  -v /data/hf:/root/.cache/huggingface \
  -e SGLANG_USE_AITER=0 \
  -p 127.0.0.1:30000:30000 \
  lmsysorg/sglang:v0.5.22-rocm10-mi35x \
  python3 -m sglang.launch_server --model-path openai/gpt-oss-120b \
    --host 0.0.0.0 --port 30000 --api-key "$SGLANG_API_KEY" \
    --reasoning-parser gpt-oss --tool-call-parser gpt-oss

The image turns on AMD’s AITER kernels by default (SGLANG_USE_AITER=1). The GPT-OSS cookbook turns them off for gpt-oss on the MI355X; keep the default for other models unless their cookbook page says otherwise. The AMD page’s --network=host and --privileged are only for RDMA and left out here. --tp and --ep work as on NVIDIA. Our model pages offer SGLang on the MI355X only for models with confirmed ROCm support: gpt-oss-20b, gpt-oss-120b and DeepSeek-R1-0528.

Run SGLang as a service and reach it safely

To restart SGLang after a crash or a reboot, run it under systemd as an unprivileged user (with Docker, --restart unless-stopped does this). Stop the tmux copy first. The key stays in a root-only file, and systemd substitutes it for ${SGLANG_API_KEY} (systemd.service):

useradd --system --create-home --home-dir /var/lib/sglang --shell /usr/sbin/nologin sglang
mkdir -p /data/hf && chown -R sglang:sglang /data/hf
install -m 600 /dev/null /etc/sglang.env
echo "SGLANG_API_KEY=$SGLANG_API_KEY" > /etc/sglang.env    # the key created above

cat > /etc/systemd/system/sglang.service <<'EOF'
[Unit]
Description=SGLang server
Wants=network-online.target
After=network-online.target

[Service]
User=sglang
EnvironmentFile=/etc/sglang.env
Environment=HF_HOME=/data/hf
ExecStart=/opt/sglang/bin/sglang serve --model-path openai/gpt-oss-120b \
  --host 127.0.0.1 --port 30000 --api-key ${SGLANG_API_KEY} \
  --reasoning-parser gpt-oss --tool-call-parser gpt-oss --enable-metrics
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable --now sglang
journalctl -u sglang -f

SSH tunnel: for you and a small team

# on your computer; the API is then at http://localhost:30000/v1
ssh -N -L 30000:127.0.0.1:30000 gpu1

Caddy with TLS: for a public endpoint

Point a DNS name at the server, install Caddy from its official repository and forward only /v1/. Caddy obtains and renews the certificate itself, as in the vLLM guide:

cat > /etc/caddy/Caddyfile <<'EOF'
llm.example.com {
    handle /v1/* {
        reverse_proxy 127.0.0.1:30000
    }
    handle {
        respond 404
    }
}
EOF
systemctl reload caddy
ufw allow 80,443/tcp

Monitor and upgrade SGLang

curl -s http://127.0.0.1:30000/metrics | grep -E '^sglang:(num_running_reqs|num_queue_reqs|token_usage|cache_hit_rate)'
journalctl -u sglang -f | grep "Decode batch"      # #running-req, token usage, #queue-req
nvidia-smi --query-gpu=index,utilization.gpu,memory.used,memory.total --format=csv -l 5
watch -n 2 amd-smi monitor                         # MI355X

/metrics needs --enable-metrics (production metrics). GPU memory looks full even when idle: the weights and the KV pool are reserved at start-up. Watch token_usage, the share of the KV pool in use, and the queue instead: requests that keep waiting while usage stays above 0.9 mean the pool is the limit (more GPUs or an FP8 KV cache help).

Before upgrading, read the release notes: 0.5.20 retired CUDA 12, and 0.5.22 requires Python 3.11 or newer.

uv pip install --python /opt/sglang/bin/python --upgrade --prerelease=allow sglang
/opt/sglang/bin/sglang version
systemctl restart sglang

To stay on a known version, replace --upgrade --prerelease=allow sglang with --prerelease=allow "sglang==0.5.22". With Docker, remove the container (docker rm -f sglang) and start it again with the new tag. Models in /data/hf survive upgrades, but not the end of an unrenewed term: the disks are then wiped.