DeepSeek DeepInfer Won't Run on Your RTX 5090. Here's What to Run Instead

Posted on Oct 21 • Originally published at blog.authon.dev

DeepSeek open-sourced DeepInfer, the inference engine behind its own production services, under Apache 2.0. The HN thread passed 2,200 points, and a lot of the excitement I've seen boils down to "finally, fast DeepSeek on my own box."

That isn't going to happen for most of us. Officially, DeepInfer runs on NVIDIA SM100 (B200/GB200) or Huawei Ascend 950, not on an RTX 5090, it serves exactly one 763B model, and it has no HTTP server. You run a Python script and it generates tokens.

I still think the release is worth reading. But the gap between what people think they got and what the README actually describes is big enough to go through point by point.

SM100 means datacenter Blackwell, the B200/GB200 class. Your RTX 5090 is also Blackwell, but it reports compute capability 12.0 (SM120). That's a different target, and the README doesn't list it.

The model is the other wall. The Hugging Face card lists 763B parameters, stored as a mix of I8, FP8, BF16 and F32 tensors. The repo comes to about 510 GB of weights, before you allocate anything for activations. Despite the "Flash" in the name, this is not a small model.

The quickstart shows what kind of tool this is:

# fresh Python 3.12 env export DEEPINFER_ACCELERATOR = nvidia # or ascend./build.sh python examples/offline_inference.py Enter fullscreen mode Exit fullscreen mode There's no deepinfer serve and no OpenAI-compatible endpoint.

One caveat to "you can't run it": someone in the HN thread already linked a community port to older hardware, HalfVulpes/Deepinfer-SM80. It's unofficial, I haven't tested it, and it doesn't make the weights any smaller.

The README is upfront about what this release doesn't include, and it's mostly the stuff that turns an engine core into a service:

You need those pieces before you can put real traffic on a set of fast kernels. Without a KV cache system there's no prefix caching across requests, and prefix caching is a big reason DeepSeek's own API is so cheap on repeated prompts. As HN commenter mobelkh put it, "Deepseek manages to be ridiculously cheap and efficient with prompt caching." That piece didn't ship.

This is my main complaint. The README has prefill and decode Pareto charts, measured on 4 SM100 GPUs for prefill and 64 SM100 GPUs for decode, each set inside one scale-up domain. The page has no table of numbers, and it doesn't compare against vLLM, SGLang or TensorRT-LLM on the same hardware.

I want that baseline because the README says DeepInfer was built primarily on vLLM, with ideas from SGLang and TensorRT-LLM. Without it you can't tell how much of the speed is DeepSeek's engineering and how much is 64 Blackwell GPUs.

Tepix asked the obvious question on HN: "It's a project based on vLLM, explain why you consider it a total game changer?" In the same subthread cmrdporcupine pointed out that vLLM and SGLang can already serve V4.1 Flash, "less efficiently, I'm sure, but on much broader hardware," and that "a lot of what's in here is already in published papers out of DeepSeek already."

If you have the rack, you can reproduce the sweep with examples/sweep/run_sweep.sh. To put a price on that: one HN commenter put B200 cloud rental at "~$6-14/hour". Across the 64-GPU decode setup that comes to about $384–896 per hour.

The eval README gives quality numbers for V4.1 at default settings: AIME 2026 99.9%, GPQA Diamond 93.4%, HLE-Full 42.6% (Pass@1). So the engine gives correct answers, which tells you nothing about how fast it is next to vLLM.

There's real engineering in here, and it's now Apache 2.0 code you can read and port.

Start with the MoE side. DeepInfer does expert parallelism with expert-parallel load balancing (EPLB), alongside attention data parallelism and balanced context parallelism. The kernels come from DeepSeek's earlier open-source releases (FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect) plus custom TileLang kernels. If you work on MoE code paths in vLLM or SGLang, you get to see how DeepSeek fits the pieces together in production.

Then there's batch-invariant, deterministic sampling. DEEPINFER_BATCH_INVARIANT=1 makes outputs independent of whatever else is in the batch. Turn that on, set DEEPINFER_ENABLE_MTP_VARLEN=0, pass a fixed seq_seed in SamplingParams, and the same input gives the same tokens. NVIDIA only. vLLM has its own version ( VLLM_BATCH_INVARIANT=1 ), so the feature itself isn't new. I find it more interesting that DeepSeek runs it in a production engine. If you've ever chased an eval regression that turned out to be batch-composition noise, you'll want this.

Speculative decoding goes through DSpark. The engine drafts tokens, and confidence estimates decide how verification gets scheduled. You enable it with DEEPINFER_ENABLE_MTP_VARLEN=1, and DEEPINFER_USE_NEXTN_DECODE sets the maximum draft length (default 5). Because deterministic mode requires MTP varlen off, you have to choose between the two.

The engine also supports structured output (regex, JSON schema or EBNF via so_args ), turns on async scheduling by default, and uses SWA bounded replay on the decoder to cut prefill work.

If you want V4.1-Flash on your own hardware today, the Hugging Face model page lists the options. You get vLLM and SGLang (both serve OpenAI-compatible APIs), plus quantized builds for llama.cpp, Ollama and LM Studio. DeepInfer isn't on the list.

nvidia-smi --query-gpu = name,compute_cap --format = csv Enter fullscreen mode Exit fullscreen mode 10.0 means you have the hardware the README targets. If you see anything else, use vLLM or SGLang. B300 reports 10.3, and the README doesn't say whether that works. If you get 12.0 you're on consumer Blackwell, and for a 763B model your realistic option is a quantized GGUF in llama.cpp, with a lot of system RAM. Unsloth's guide puts its 1-bit quant at 128GB+ and builds llama.cpp from a PR branch, not a stock release.

I think the best outcome is vLLM and SGLang maintainers reading the EPLB and DSpark code and upstreaming whatever generalizes, whether or not anyone ever runs DeepInfer itself. mobelkh hoped for the same thing, for "the ideas here being incorporated in other Opensource projects." The license allows it, and since DeepInfer started from vLLM, porting back should be easier than usual.

The Ascend support also hints at where DeepSeek is headed: its production engine treats Ascend 950 as a first-class target next to SM100. Or, as mordae put it on HN: "Global south will be buying Ascends soon."

My take: this is a good reference implementation that got hyped as a product. Read the code, but don't plan on running it.

I keep going back and forth on this one. If an inference engine officially runs only on a rack of datacenter Blackwell, is it "open" in a way that matters to you? Or is code you can read and fork enough, even if you never execute it?

Expand arcwrightai arcwrightai arcwrightai Follow Joined Oct 18, 2026 • Oct 21 Dropdown menu Copy link Hide RTX 5090 is still too new for a lot of inference stacks—check if your framework has Blackwell support yet, or fall back to llama.cpp with a recent build.