Self-host an LLM: a private ChatGPT on your own GPU server

Why self-host an LLM?

To self-host an LLM, you run an open-weight model on a GPU server you control and reach it through your own chat interface or API. Your prompts, documents and answers stay on that server instead of passing through a third-party API, you pay a fixed price for the machine rather than per token, and the model changes only when you change it.

• Your data stays on your machine. Conversations, uploaded files and logs sit on the server’s disks. You decide what is logged, who can read it and when it is deleted.
• No per-token bill, no rate limit. The only limits are the GPU’s memory and speed, and heavy use costs the same as light use.
• Your choice of model. Pick among open-weight models from OpenAI, Qwen, Google, Mistral AI, DeepSeek, Z.ai or Moonshot AI, keep the version that works, switch when a better one appears, or fine-tune one on your own data.

There are trade-offs. The closed models behind ChatGPT and similar services are not published as weights, so a self-hosted alternative runs an open model instead. And you run the machine yourself, from updates and security to backups.

How to self-host an LLM in four steps

1. Pick an open-weight model that does your tasks well and whose license fits your use.
2. Size the GPU server. The weights and the KV cache must fit in GPU memory: plan for the size of the weights plus about 20%, more for long contexts and many users.
3. Choose the serving stack: Ollama with Open WebUI for a private chat in the browser, or vLLM or SGLang for an OpenAI-compatible API that serves many requests at once.
4. Lock down access. Keep the model server on 127.0.0.1 and reach it through an SSH tunnel or an authenticated TLS reverse proxy.

With CryptGPU, you can do the first three when you order: choose the server, then pick the engine and a model from our catalog at the Software step, installed at no extra cost.

Step 1: pick an open-weight model

Our open models page lists the 16 models we can preinstall; as root, you can install any other. Four questions narrow the choice.

• What should it do? For chat and writing, Qwen3.8 27B and Gemma 4 31B are dense models that also read images. For reasoning, gpt-oss-120b and gpt-oss-20b let you set the reasoning effort to low, medium or high. Devstral Small 2 and Qwen3-Coder-Next were designed for coding agents.
• Does the license fit? The Qwen models in our catalog, gpt-oss, Gemma 4 31B, Mistral Small 4 and Devstral Small 2 use Apache 2.0; the DeepSeek models and GLM-5.3-Flash use MIT; Kimi K2.6 a modified MIT license and Kimi K3 its own Kimi K3 License. Read the license before commercial use.
• How large? Larger models usually answer better and need more GPU memory. Mixture-of-experts (MoE) models run fast for their size because only part of the model works on each token: gpt-oss-120b has 117 billion parameters, 5.1 billion of them active. Every expert must still sit in GPU memory, so size an MoE model by its total parameters.
• How much context? Long documents need a long context window: 262,144 tokens natively for Qwen3.5 9B, Qwen3.8 27B and Qwen3.6 35B-A3B, 256K for Gemma 4 31B and Mistral Small 4, 1,048,576 for Kimi K3. Every token of context also takes GPU memory for the KV cache.

Then test two or three candidates on your own prompts: the makers’ benchmarks are a starting point, not a verdict on your tasks. Model facts from the cards on Hugging Face, checked on 19 October 2026: gpt-oss-120b, gpt-oss-20b, Qwen3.5 9B, Qwen3.8 27B, Qwen3.6 35B-A3B, Qwen3-Coder-Next, Qwen3.5 397B-A17B, Gemma 4 31B, Mistral Small 4, Devstral Small 2, DeepSeek-R1-0528, DeepSeek-V4-Flash, DeepSeek-V4-Pro, GLM-5.3-Flash, Kimi K2.6, Kimi K3.

Step 2: size the GPU server

GPU memory decides which models your own LLM server can run. Our rule: the size of the weights × 1.2, to leave room for the KV cache and activations, with 92% of GPU memory counted as usable. By parameter count, our sizing helper uses about 1.2 GB per billion parameters in FP8 and 0.65 GB in 4-bit. Here is the rule applied to every model in our catalog, served with vLLM or SGLang from the official weights (in FP8 when they are published in BF16) and with Ollama from its default build:

A dash means no Ollama build in our catalog. For vLLM and SGLang, models whose ROCm support is not confirmed are matched to NVIDIA GPUs only. Ollama’s default builds are quantized: qwen3.8:27b weighs 18 GB, the size of its q4_K_M tag, against 56 GB for bf16 (Ollama library). Expect some loss of quality against the full weights.

Three cautions. The rule assumes moderate context and a few requests at a time: long conversations and many users need more memory for the KV cache, as explained in how much GPU memory an LLM needs. A model split over several GeForce or RTX PRO cards exchanges data over PCIe, slower than NVLink. And a model that only just fits leaves no headroom: one RTX PRO 6000 (96 GB, $959) or one H200 (141 GB, $2,279) gives a mid-size model room to grow. To size any other model, use the sizing helper; to compare GPUs for serving, read the best GPU for LLM inference.

Step 3: Ollama and Open WebUI, or vLLM and SGLang

Sources: Ollama FAQ, Ollama authentication, Open WebUI, vLLM, SGLang, SGLang server arguments.

Ollama and Open WebUI

Open WebUI offers an image with Ollama built in, so one container runs both and Ollama’s API is never exposed. On an NVIDIA server with Docker and the NVIDIA Container Toolkit (see containers), here is the Open WebUI quick start command, adapted to publish the interface on 127.0.0.1 only; replace 203.0.113.10 with your server’s address:

# on the server
docker run -d -p 127.0.0.1:3000:8080 --gpus=all \
  -v ollama:/root/.ollama -v open-webui:/app/backend/data \
  -e WEBUI_SECRET_KEY="$(openssl rand -hex 32)" \
  --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama

# on your computer, then open http://localhost:3000
ssh -N -L 3000:127.0.0.1:3000 root@203.0.113.10

Create the administrator account first, then download a model from the model selector, for example gpt-oss:20b. Chats, users and settings live in the open-webui volume. Our Ollama and Open WebUI guide covers the setup step by step.

vLLM or SGLang

# on the server, with vLLM installed
KEY=$(openssl rand -hex 32); echo "$KEY"
vllm serve openai/gpt-oss-120b --host 127.0.0.1 --port 8000 --api-key "$KEY"

# on your computer: the API is then at http://localhost:8000/v1
ssh -N -L 8000:127.0.0.1:8000 root@203.0.113.10

Any OpenAI client works with that base URL and the key (vLLM docs). SGLang starts with python -m sglang.launch_server --model-path <model>, listens on 127.0.0.1:30000 unless told otherwise and takes the same --api-key option. Open WebUI can also connect to an OpenAI-compatible API, so it can be the chat interface in front of either. Installation, multi-GPU and quantization are in serving LLMs with vLLM.

Step 4: keep it private

A self-hosted model stays private only if strangers cannot reach it. Keep every model server on the loopback address, then add one controlled way in.

• Bind to 127.0.0.1. Ollama does by default, and its local API requires no authentication: never publish port 11434. Start vLLM with --host 127.0.0.1; SGLang binds there by default.
• Publish Docker ports on 127.0.0.1. A plain -p 3000:8080 listens on every interface, and Docker’s rules apply even if ufw blocks the port (Docker docs).
• Use an SSH tunnel for yourself and people who already have SSH access, as in the commands above: nothing new is exposed.
• Use a TLS reverse proxy with access control for a team that should reach the chat from a browser. Caddy obtains and renews HTTPS certificates by itself. Open WebUI does not terminate TLS, and its hardening guide asks for an access-control layer in front of any instance reachable from the internet: a VPN, or a proxy with authentication and an IP allowlist.
• Protect APIs with keys. In vLLM, --api-key protects the /v1 routes and a few others, not every endpoint (vLLM security guide), so let the proxy forward only /v1/, as in our Caddy example. Use a long random key per application and replace it if it leaks.

In Open WebUI, the first account to register becomes the administrator and sign-up then closes; if you reopen it for your team, new accounts wait for the administrator’s approval. The rest is ordinary server care, and it is yours: SSH keys only, and updates for the OS, the engine and Open WebUI (see SSH access).

What a private LLM server costs

The server price is the whole bill: no token is metered. By our sizing rule, gpt-oss-20b fits on one RTX 4090 at $269 a month, about $0.37 an hour, and gpt-oss-120b on one RTX PRO 6000 at $959, about $1.31 an hour. At the top end, the largest open models need eight-GPU servers: eight B300 cost $32,152 a month.

To compare with a per-token API, turn the monthly price into tokens:

break-even (million tokens a month) = server price per month ÷ API price per million tokens

For an API that charges P dollars per million tokens, blended across input and output, the RTX PRO 6000 server breaks even at 959 ÷ P million tokens a month. Two conditions apply: the open model must be good enough for your tasks, and the server must keep up with that volume. Throughput depends on the model, the engine and the number of parallel requests; we publish no benchmarks, so measure it on your own workload.

A shared server divides the same way: 20 people on that RTX PRO 6000 come to $47.95 each a month, however much they send, as long as the server keeps up. For a few hours of use a week, a per-token API or an hourly GPU costs less; a monthly server pays off when it runs most of the day, as shown in monthly vs hourly GPU rental.

Where CryptGPU fits

CryptGPU rents dedicated GPU servers by the month, from one GeForce card to eight data-center GPUs, with root SSH access and no other tenant on your cards. When you order, the Software step can install Ollama with Open WebUI, vLLM or SGLang with a model from our catalog, at no extra cost, and the server is online in under 10 minutes once the payment is confirmed on-chain. You pay in Bitcoin, Ethereum, USDT, Monero or Litecoin, and there is no KYC: an order needs an email address, not an identity.

It is not the right choice if you need a closed model, a managed service or a model for a few hours a week. You administer the server yourself, updates, security and backups included, and we publish no SLA. A server that is not renewed stops at the end of the paid month and its disks are wiped, chat history included, so keep a copy of the Open WebUI volume. Recommended configurations are on the LLM inference page.

Common questions

Is a self-hosted LLM really private?

The model runs on your server, so prompts, documents and answers go to no model provider. What stays private then depends on how you run it: who can log in, what you log, where backups go and which outside services you connect. Open WebUI can search the web through external providers, for example, and they see those queries.

Can a self-hosted model replace ChatGPT?

It depends on your tasks. The closed models behind ChatGPT cannot be self-hosted, so compare open candidates on your own prompts before you switch: small models cost less and respond faster, large ones usually answer better.

How many people can share one server?

It depends on the model, the length of conversations and how often people send requests. Ollama handles one request per model at a time by default; vLLM and SGLang batch concurrent requests. Each active conversation also needs memory for its KV cache, so test with real use before you invite everyone.

Can it answer questions about my documents?

Yes. Open WebUI includes retrieval-augmented generation: you upload files and it adds the relevant passages to the prompt. Long documents also need a long context window and the memory to hold it. Ollama, for instance, picks its default context length from the GPU memory, from 4K tokens below 24 GiB to 256K from 48 GiB, and OLLAMA_CONTEXT_LENGTH changes it (Ollama docs).

Does it run on the AMD MI355X?

Yes, with ROCm builds: SGLang lists the MI355X among its supported GPUs, vLLM supports AMD GPUs, and Ollama supports gfx950, the architecture of the MI355X (Ollama GPU list). Each model page shows whether ROCm support is confirmed for that model. The Docker command above is for NVIDIA GPUs only.