Running DeepSeek-V4.1-Flash (763B) on one 96 GB GPU: the three settings that decide your speed

Posted on Oct 20 • Originally published at github.com

DeepSeek-V4.1-Flash has 763 billion parameters. Quantized to Q2_K it is still 264.5 GB, and one RTX PRO 6000 Blackwell has 96 GB of VRAM. We published deepseek-one-gpu, an open source engine that serves it from that one card plus the server's RAM as an OpenAI-compatible API. This piece is for the person who is going to set it up: what decides the speed, and what to check before you blame the engine.

All from the measured results in the repo, RTX PRO 6000 Blackwell Server Edition:

The model routes every token to 6 of 384 experts per layer. The engine keeps the attention, the dense parts and the 5,825 most used experts (69.5 GiB) on the GPU, and the other 9,535 experts (113.9 GiB) in pinned RAM, where CPU threads compute them while the GPU works. In the measured runs about 9 in 10 expert picks landed on the GPU, and the hot set keeps following the traffic: experts swap in and out over PCIe.

That placement is not guessed at start-up. A planning step reads the model headers, your VRAM and your RAM and decides the hot set before the server starts. That is also why the RAM requirement is real: plan for about 280 GB. The planner refuses a host that cannot hold the minimum layout instead of letting it swap to death later.

This one cost us the most time. The CPU half of the work reads expert weights from RAM as fast as the memory bus allows, so the threads must run on the socket that owns the GPU. Look at "NUMA Affinity" in nvidia-smi topo -m and bind the server to it. On the same build that took one user from about 53 to 63 tokens per second.

The trap: inside many containers numactl --preferred is refused by seccomp, and the error is easy to miss. run.sh tries memory-preferred binding, then CPU-only binding, then taskset, and tells you which one it got.

With batching, the engine decodes several requests in the same step. By default the CPU experts use the fork's own numerics ( --cpu-numerics fork ), which is the fast path: batched answers keep the same quality, but they are not bit-identical to a request served alone. If you need identical tokens, ask for "strata_reproducible": "required" per request or start with --cpu-numerics gpu, which costs 25.7% of single-user speed. Most people want the default.

With the prebuilt engine from the v0.1.0 release (CUDA 13, sm_120):

git clone https://github.com/inferencekey/deepseek-one-gpu.git && cd deepseek-one-gpu./run.sh --dry-run # prints every command it would run on this machine./run.sh # downloads the model if missing, plans, binds, serves on:8090 Enter fullscreen mode Exit fullscreen mode Then point any OpenAI client at http://localhost:8090/v1. The quickstart in the README has the build-it-yourself path and every step done by hand.

On fixed-seed subsets the engine scores GSM8K 0.927, IFEval 0.800 and HumanEval 0.890, against 0.960, 0.853 and 0.982 for the full model on DeepSeek's API. The stock llama.cpp fork on the same Q2_K file scores 0.878 on HumanEval, so the gap is the 2-bit quantization, not the offload.

Per token, DeepSeek's API is cheaper than a rented RTX PRO 6000, and the cost comparison in the repo says so. Running it yourself is about data that never leaves your machine and capacity nobody else shares.

If you want it behind API keys on your own worker, the repo also ships an InferenceKey custom backend in integrations/inferencekey.

Credit where it is due: the offload method and the server come from Strata by Niko1221 and contributors, on top of llama.cpp and ggml, with vcruz305's DeepSeek-V4.1 fork and Q2_K GGUF, and DeepSeek's open weights. Everything above, with sources for each number, is in the deepseek-one-gpu repository.