Run Ollama and Open WebUI on a GPU server

Ollama and Open WebUI on a GPU server: what you get

To run Ollama and Open WebUI on a GPU server, install Ollama with its official script, which sets it up as a systemd service on 127.0.0.1:11434, pull a model, then start Open WebUI in Docker connected to that local Ollama and open it through an SSH tunnel. Ollama loads models onto the GPUs and serves them through its own API and an OpenAI-compatible one. Open WebUI adds a ChatGPT-style interface with accounts and chat history: together, a self-hosted ChatGPT.

Ollama suits a private chat interface for you or a small team, and quick model switching; vLLM (Inference serving) suits many users or applications calling the API at once. If you picked Ollama with Open WebUI in the Software step when you ordered, both came installed with your model; this guide sets them up by hand. Why self-host: a private ChatGPT on your own GPU server.

Install Ollama on Ubuntu

Install the GPU driver first (NVIDIA drivers and CUDA or ROCm on MI355X): if the installer finds an NVIDIA GPU without a driver, it installs NVIDIA’s cuda-drivers package itself, which mixes driver sources. Then run the script from Ollama’s Linux page:

sudo apt update && sudo apt install -y zstd    # the installer needs zstd to unpack
curl -fsSL https://ollama.com/install.sh | sh
ollama -v                                        # installed version
systemctl status ollama --no-pager               # active (running)
nvidia-smi                                       # NVIDIA: every GPU listed
amd-smi list                                     # AMD MI355X

The script creates an ollama user (in the render and video groups), installs and starts the ollama systemd service on 127.0.0.1:11434 and, on AMD hosts, adds Ollama’s ROCm libraries but not the kernel driver.

• NVIDIA: Ollama needs compute capability 5.0 or later and driver 550 or newer (GPU page). That page has no row for the B200 or B300 (the 0.32.3 release notes mention B200 support through CUDA 12); on either card, check with ollama ps (below) that models load on the GPU.
• AMD: Ollama needs the ROCm v7 driver. The GPU page lists the Instinct MI350X and the gfx950 target, which the MI355X shares, but not the MI355X by name: check that page before you rely on it.

Configure Ollama: OLLAMA_HOST, models on NVMe, GPU settings

Ollama reads environment variables. Set them in a systemd drop-in, not in ollama.service, which the install script rewrites on every run (sudo systemctl edit ollama opens the drop-in in an editor). Models go on the NVMe data disk (Storage and data), owned by the ollama user:

sudo mkdir -p /data/ollama/models
sudo chown -R ollama:ollama /data/ollama

sudo mkdir -p /etc/systemd/system/ollama.service.d
sudo tee /etc/systemd/system/ollama.service.d/override.conf >/dev/null <<'EOF'
[Service]
Environment="OLLAMA_HOST=127.0.0.1:11434"
Environment="OLLAMA_MODELS=/data/ollama/models"
Environment="OLLAMA_NO_CLOUD=1"
Environment="OLLAMA_CONTEXT_LENGTH=32768"
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_KEEP_ALIVE=30m"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"
EOF
sudo systemctl daemon-reload
sudo systemctl restart ollama
journalctl -u ollama -b --no-pager | grep -i "cloud disabled"    # Ollama cloud disabled: true

Multiple GPUs

Ollama loads a model on one GPU when it fits, which its FAQ says usually performs best (less data crosses the PCI bus), and spreads it across all GPUs only when it does not. The other GPUs then hold other models, up to OLLAMA_MAX_LOADED_MODELS.

Pull and run models

ollama pull gpt-oss:20b
ollama run gpt-oss:20b "Explain a KV cache in two sentences."
ollama ls                     # models on disk
ollama ps                     # loaded models: SIZE, PROCESSOR, CONTEXT, UNTIL
ollama stop gpt-oss:20b       # unload it now
ollama rm gpt-oss:20b         # delete it from disk

Without a prompt, ollama run opens a chat. In ollama ps, PROCESSOR should read 100% GPU; a split such as 48%/52% CPU/GPU means part of the model runs on the CPU from system memory (see Troubleshooting). CONTEXT is the context length it was given.

Default tags are usually 4-bit (qwen3.5:9b is Q4_K_M). Tag lists also offer q8_0 and bf16 versions: more memory, closer to the original weights. If they fit, compare them with the default on your own prompts.

Ollama keeps the whole tag in GPU memory, plus a KV cache that grows with the context. CryptGPU’s sizing rule adds 20% to the tag size and counts 92% of GPU memory as usable:

Long contexts, OLLAMA_NUM_PARALLEL above 1 and several loaded models need more. The sizing helper and the model pages show what each catalog model needs.

Run Open WebUI in Docker

Install Docker first (see Containers); Open WebUI itself needs no GPU.

Open WebUI’s quick start reaches Ollama at host.docker.internal, the host’s internal Docker address, not the 127.0.0.1 where Ollama listens. Its connection guide gives the fix: host networking with OLLAMA_BASE_URL=http://127.0.0.1:11434. Docker then publishes no ports and the app listens on the host itself, so bind it to localhost with HOST and PORT, read by the image’s start script:

# once: a fixed secret key (root-only file) keeps users logged in across updates
sudo install -m 600 /dev/null /etc/open-webui.env
echo "WEBUI_SECRET_KEY=$(openssl rand -hex 32)" | sudo tee /etc/open-webui.env >/dev/null

sudo docker run -d --network=host \
  --env-file /etc/open-webui.env \
  -e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
  -e HOST=127.0.0.1 -e PORT=3000 \
  -v open-webui:/app/backend/data \
  --name open-webui --restart always \
  ghcr.io/open-webui/open-webui:main

sudo docker logs -f open-webui       # wait for start-up, then Ctrl-C
ss -tln | grep -E "3000|11434"       # both on 127.0.0.1 only

The open-webui volume holds accounts, chats and settings. Open http://localhost:3000 through the SSH tunnel (next section) and create your account at once: the first account becomes the administrator, and creating it switches off new sign-ups. To add people, an admin turns sign-ups back on under Settings > Admin > Authentication; new accounts wait as Pending until approved.

Remote access to Ollama and Open WebUI

Both services stay on 127.0.0.1. Reach them through SSH, or put an authenticated TLS proxy in front of Open WebUI, never in front of Ollama.

SSH tunnel: for you and a few colleagues

# on your computer; gpu1 is the host entry from the SSH guide
ssh -N -L 3000:127.0.0.1:3000 -L 11434:127.0.0.1:11434 gpu1

Then open http://localhost:3000. The second forward serves the API examples below; if Ollama also runs on your computer, use another local port, such as -L 11435:127.0.0.1:11434. Keys and ~/.ssh/config are covered in SSH access.

Caddy with TLS: for a team

Open WebUI does not terminate TLS, and its hardening guide asks for an extra access-control layer in front of it on the internet, such as an allowlist of known IP ranges. Point a DNS name at the server and install Caddy from its official repository; it obtains the certificate and passes WebSockets and streaming through without extra settings (Open WebUI’s Caddy page).

sudo tee /etc/caddy/Caddyfile >/dev/null <<'EOF'
chat.example.com {
    @team remote_ip 198.51.100.7 203.0.113.0/24
    handle @team {
        reverse_proxy 127.0.0.1:3000
    }
    handle {
        respond 403
    }
}
EOF
sudo systemctl reload caddy
sudo ufw allow 80,443/tcp

# allow the public origin for WebSockets, then recreate the container (see Updates)
echo "CORS_ALLOW_ORIGIN=https://chat.example.com;http://localhost:3000" | sudo tee -a /etc/open-webui.env

Put your team’s addresses after remote_ip; each person still signs in with their own Open WebUI account.

Use Ollama’s OpenAI-compatible API

Ollama serves an OpenAI-compatible API under /v1: chat completions, completions, models, embeddings and responses. Over the tunnel above:

curl -s http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-oss:20b", "messages": [{"role": "user", "content": "Say hello in five words."}]}'

curl -s http://localhost:11434/api/tags     # native API: the models on disk

In an OpenAI client, set base_url to http://localhost:11434/v1/ and api_key to any value, such as ollama: the field is required but ignored.

For people or services without SSH access, use Open WebUI’s own OpenAI-compatible endpoint, /api/chat/completions, behind the Caddy proxy. It checks a per-user API key created under Settings > Account; non-admin users need an admin to allow keys first (API endpoints).

Updates and troubleshooting

Update

curl -fsSL https://ollama.com/install.sh | sh    # update Ollama; the drop-in is kept
ollama -v

# Open WebUI: back up the volume, then replace the container
sudo docker run --rm -v open-webui:/data -v /data/backups:/backup alpine \
  tar czf /backup/openwebui-$(date +%Y%m%d).tar.gz /data
sudo docker rm -f open-webui
sudo docker pull ghcr.io/open-webui/open-webui:main
# then repeat the docker run command from the Open WebUI section

Open WebUI’s update guide warns that database migrations are one-way, hence the backup. To stay on known versions, install Ollama with curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=<version> sh and replace :main with a release tag, :v<x.y.z>. Models and the Open WebUI volume are wiped with the server if the term is not renewed: keep copies elsewhere.

Troubleshooting

More fixes, such as reloading the nvidia_uvm module, are in Ollama’s troubleshooting page.