Self-hosted AI coding assistant: a private Copilot on your GPU server

What is a self-hosted AI coding assistant?

A self-hosted AI coding assistant is an open-weight coding model running on a GPU server you control, behind an OpenAI-compatible API that your editor or terminal agent calls instead of a cloud service. Your source code, prompts and completions stay on that server, you pay for the server rather than per seat, and you choose the model.

• Private repositories stay private. Code sent as context reaches your server only, and you decide what is logged.
• No per-seat fee, no rate limit. The monthly price is the same for 2 or 20 developers, as long as the server keeps up; the limits are the GPU’s memory and speed.
• Your choice of model. Keep a version that works on your codebase, switch when a better open model appears, or fine-tune one on your own code.

The trade-offs are real. GitHub Copilot’s plans list closed models from Anthropic, OpenAI and Google (GitHub, checked on 19 October 2026), whose weights are not published, so a self-hosted assistant runs an open model whose quality you have to test on your own code. And you run the server yourself. For a private chat on the same kind of server, see how to self-host an LLM.

Autocomplete, chat and agents need different models

Continue’s documentation notes that “most of the state-of-the-art autocomplete models are no more than 10b parameters, and increasing beyond this does not significantly improve performance”, and that commercial models such as GPT-5 or Claude are not trained with the prompt format autocomplete uses (Continue). Qwen states that fill-in-the-middle “is supported in every version of Qwen3-Coder”, Qwen3-Coder-Next included (Qwen3-Coder), so the agent model can also serve completions if you prefer one model to two. For agents, Ollama advises a context of at least 64,000 tokens for coding tools (Ollama).

Which coding model to self-host

Our catalog has two models built for coding agents, and three general models that can also serve a coding assistant. Memory needed follows our sizing rule, the size of the weights × 1.2 with 92% of GPU memory usable, as on each model page:

• Qwen3-Coder-Next is a mixture-of-experts model with 80 billion parameters, 3 billion active per token, 262,144 tokens of native context and no thinking mode. Qwen cites integration with CLI and IDE platforms such as Claude Code, Qwen Code, Kilo and Cline, and publishes an official FP8 checkpoint, Qwen/Qwen3-Coder-Next-FP8.
• Devstral Small 2 is a dense 24-billion-parameter model published in FP8, with a 256K window. Mistral calls it “light enough to run on a single RTX 4090”: by our rule, Ollama’s 15 GB build fits one, the full FP8 weights need two.
• Qwen3.8 27B thinks by default, which you can turn off per request; gpt-oss lets you set its reasoning effort to low, medium or high. Both also answer non-coding questions.

Larger models in our open models catalog, such as GLM-5.3-Flash and Kimi K2.6, need multi-GPU servers. Test two candidates on your own repositories before you settle. Model facts from the cards on Hugging Face: Qwen3-Coder-Next, Qwen3-Coder-Next-FP8, Devstral Small 2, Qwen3.8 27B, gpt-oss-120b, gpt-oss-20b; Ollama build sizes from its library.

How much GPU memory a team of developers needs

Our rule covers a few requests at moderate context. A team needs more, because every request in flight carries its own KV cache, the keys and values of every token of its context, and agent requests are long. Size for the requests running at the same moment, not for the number of seats:

memory needed = weights + KV cache per request × requests in flight
KV per token  = 2 × attention layers × KV heads × head dimension × 2 bytes (BF16)

Devstral Small 2 has 40 layers with 8 KV heads of dimension 128: about 0.16 MB per token, 10.7 GB for a request of 64K tokens (65,536). Qwen3-Coder-Next keeps a KV cache in only 12 of its 48 layers, with 2 KV heads of dimension 256; the other 36 use Gated DeltaNet, a linear attention whose state this formula does not count. That is about 0.025 MB per token, 1.6 GB per 64K request (Devstral config, Qwen config). With FP8 weights:

• Hybrid attention pays off for teams. Qwen3-Coder-Next has three times the weights of Devstral Small 2, yet needs less memory from 6 requests in flight with a BF16 cache.
• An FP8 KV cache takes half the bytes (--kv-cache-dtype fp8 in vLLM, which recommends calibrated scales for the best quality: vLLM). One RTX PRO 6000 then holds Devstral Small 2 with 11 requests.
• The cheapest fit is not always the simplest. Four RTX 5090 split the model over PCIe; one H200 ($2,279) holds Qwen3-Coder-Next with room for about 31 requests of 64K tokens on a single GPU.

Treat these as estimates: engines need memory of their own, and real requests vary in length. The same sum applies to one developer: with a 64K context, Devstral Small 2 from Ollama takes about 26 GB with Ollama’s default f16 cache (cheapest fit: 1× RTX 5090 at $329 a month), or 20 GB with OLLAMA_KV_CACHE_TYPE=q8_0 (1× RTX 4090 at $269). Once the team is on it, watch the waiting requests and KV cache usage (vLLM metrics). The sizing helper covers any other model, and the best GPU for LLM inference compares bandwidth and price per GB.

Serve the model with tool calling

Agents work through tools, so the engine must parse the model’s tool calls: in vLLM, --enable-auto-tool-choice plus the parser for the model (vLLM). For Qwen3-Coder-Next, with the parser from its model card (vLLM 0.15.0 or later):

# on the server: one H200; on two GPUs, add --tensor-parallel-size 2
KEY=$(openssl rand -hex 32); echo "$KEY"
vllm serve Qwen/Qwen3-Coder-Next-FP8 --served-model-name qwen3-coder-next \
  --host 127.0.0.1 --port 8000 --api-key "$KEY" \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --max-model-len 131072

• --served-model-name gives your tools a short model name, and --max-model-len caps each request at 131,072 tokens (the model accepts 262,144), which bounds the cache per request.
• Devstral Small 2 takes --tool-call-parser mistral --enable-auto-tool-choice, gpt-oss the openai parser.
• SGLang 0.5.8 or later takes the same parser in the model card’s command, plus --api-key, and listens on 127.0.0.1:30000 by default (SGLang).
• With Ollama, ollama pull devstral-small-2 is enough to start, but its API checks no key, it handles one request per model at a time unless you raise OLLAMA_NUM_PARALLEL, and its default context is 4K tokens below 24 GiB of GPU memory. Set OLLAMA_CONTEXT_LENGTH to at least 64000 in the service settings of our Ollama guide.

Installation, Docker and multi-GPU options are in serving LLMs with vLLM.

Connect VS Code, JetBrains or a terminal agent

Every tool below calls the same endpoint, http://localhost:8000/v1 through the SSH tunnel of the next section, with the key and the served model name. VS Code’s built-in chat can use it directly: open Chat: Manage Language Models, choose Add Models, then Custom Endpoint with the Chat Completions API type, and complete the chatLanguageModels.json file VS Code opens:

[
  {
    "name": "GPU server",
    "vendor": "customendpoint",
    "apiKey": "${input:gpuServerKey}",
    "apiType": "chat-completions",
    "models": [
      {
        "id": "qwen3-coder-next",
        "name": "Qwen3-Coder-Next (self-hosted)",
        "url": "http://localhost:8000/v1/chat/completions",
        "toolCalling": true,
        "vision": false,
        "maxInputTokens": 114688,
        "maxOutputTokens": 16384
      }
    ]
  }
]

The two token limits add up to the server’s --max-model-len, and toolCalling makes the model available to agents in chat. VS Code documents that such models work without a GitHub account or a Copilot plan, but not for completions: “Currently, you cannot connect to a local model for inline suggestions” (VS Code docs). On a company Copilot plan, the administrator’s policy must allow your own models.

For tab completions, use Continue’s autocomplete role or Tabby, with a small fill-in-the-middle model next to the agent model. Continue’s example is qwen2.5-coder:1.5b from Ollama, a 986 MB build; it is not in our catalog, but as root you can install any model.

Give the team secure access

Keep the engine on 127.0.0.1 and never publish port 8000 (vLLM), 30000 (SGLang) or 11434 (Ollama). vLLM’s key protects only some routes, and its security guide warns against relying on it alone. Two safe ways in:

An SSH tunnel per developer. Put each developer’s public key on its own line in a tunnel-only account: the key can forward to the API port and nothing else, and deleting the line revokes that person (sshd manual):

# on the server, once
sudo useradd -m -s /usr/sbin/nologin tunnel
sudo install -d -m 700 -o tunnel -g tunnel /home/tunnel/.ssh

# one line per developer, with their public key
echo 'restrict,port-forwarding,permitopen="127.0.0.1:8000" ssh-ed25519 AAAA... alice@laptop' \
  | sudo tee -a /home/tunnel/.ssh/authorized_keys

# on each developer’s computer; the API is then at http://localhost:8000/v1
ssh -N -L 8000:127.0.0.1:8000 tunnel@203.0.113.10

restrict disables forwarding and terminals, port-forwarding turns forwarding back on, permitopen limits it to the API, and the nologin shell gives no command line. Keys and ~/.ssh/config are covered in SSH access.

A TLS reverse proxy for people without SSH: Caddy forwarding only /v1/ (our example), limited to your team’s IP addresses. Everyone then shares the API key, so replace it when someone leaves.

Where CryptGPU fits

CryptGPU rents dedicated GPU servers by the month, from one GeForce card to eight data-center GPUs, with root SSH access and no other tenant on your cards. Qwen3-Coder-Next and Devstral Small 2 are in our catalog: when you order, the Software step can install vLLM, SGLang or Ollama with Open WebUI and either model at no extra cost, and the server is online in under 10 minutes once the payment is confirmed on-chain. The tunnel accounts, editor settings and any autocomplete model are yours to add. The price is per server, not per developer: $269 a month for an RTX 4090, $959 for an RTX PRO 6000, $2,279 for an H200. You pay in Bitcoin, Ethereum, USDT, Monero or Litecoin, with no KYC.

It is not the right choice if you need a closed model, a managed service with single sign-on, or a GPU for a few hours a week. You administer the server, updates and security included, and we publish no SLA. A server that is not renewed stops at the end of the paid month and its disks are wiped, so keep your configuration and any fine-tuned adapters elsewhere.

Common questions

Can a self-hosted model replace GitHub Copilot?

For chat and agents, the tools above give you the same workflow with your own model. Two things differ: Copilot’s closed models cannot be self-hosted, so results depend on the open model you pick, and tab completions need an extension such as Continue or Tabby, since VS Code’s inline suggestions cannot use a local model.

Which coding model runs on a single GPU?

By our rule, gpt-oss-20b from Ollama fits one RTX 4090 ($269 a month), and Devstral Small 2 with a 64K context one RTX 5090 ($329); Qwen3-Coder-Next from Ollama (52 GB, 4-bit) and gpt-oss-120b fit one RTX PRO 6000 ($959). For full FP8 weights and a team, see the table above.

How many developers can share one server?

It depends on how many requests run at the same moment and how long they are, not on the number of seats. Size for your peak with the formula above, then watch vLLM’s waiting requests: requests that queue while the KV cache is full mean you need more memory or shorter contexts.

Does my code leave the server?

The model runs on your server, so prompts and completions reach no model provider. Editors and extensions can still send usage data to their makers, and an agent reaches the internet if you give it tools that do: check their settings. Continue’s final release removed its anonymous telemetry.