Serving Claude Code from DeepInfer on 4x B200: the cache, the tail and the memory ceiling

Posted on Nov 5 • Originally published at inferencekey.com

If you serve a coding agent from your own GPUs, the model and the engine are now the easy part. DeepSeek trained DeepSeek V4.1 Flash, wrote DeepInfer and released it under Apache 2.0. Anthropic's Claude Code needs only a few environment variables to talk to another endpoint. What decides whether the machine pays off sits between them: a cache, a queue and a memory budget.

We ran this on one server with four NVIDIA B200 cards in the inferencekey cloud. We gave 100 synthetic agents about 450,000 tokens (the word pieces a model reads) of conversation each. The median wait for the first word of an answer was 2.55 seconds, with zero failed requests. Numbers, method and limits are in the full DeepInfer on 4x B200 research post.

A coding agent resends the whole conversation on every step, and the model must read all of it before replying. A prefix cache keeps the result of that reading for the part already seen, so only the new tail is read. In our runs 96% of the tokens the model read came from the cache.

DeepInfer leaves the cache for you to build, as its own page says. With no cache, our four cards got through about 200K tokens per second, starting each conversation from zero. The inferencekey cache lifted that to about 1.8 million on the same cards: nine times the read rate.

It has two tiers: about 57 million tokens in GPU memory (roughly 120 of our conversations) and 617 million in host RAM for quiet sessions.

Our first adapter, running the engine as four ranks (one process per card), served zero cached tokens. It told the cache to drop each entry when the final token came out, yet the engine only closes a sequence on the step after that. Every request was read in full, silently.

A small quality gate caught it first. Hide one fact in a long document, ask for it with an empty cache, then ask again with the document cached, and compare. Temperature 0 makes the model pick its most likely word every time, so the two answers should match. The fixed backend (0.3.2) served repeats 95 to 99% from the cache, and five documents of up to 200,000 tokens returned the same correct fact both ways. Run a check like this first and log the cache hit rate per request. A silent miss looks like a slow GPU.

Time to first token (TTFT) is how long an agent waits before the answer starts streaming. The 100-agent run had a 2.55 s median TTFT and still failed our bar: 95% of turns starting within 15 s. That 95th percentile, p95, sat at 37 seconds. Warm turns, whose conversation was already cached, queued behind seven cold ones: brand-new 450K conversations read from zero.

Two untested fixes: let cached turns jump the queue, or limit how many fresh sessions start together. Chart p95 TTFT beside the median, split into warm and cold turns.

At 500K tokens per conversation, we had 200 agents ask at once. Their conversations overflowed GPU memory and the median TTFT went past two minutes. No request failed, yet nobody could have worked like that. Host RAM only rescues idle sessions. With agents this busy, the cards hold about 120 such conversations at once.

Size the GPU tier for sessions active at once, not for the ones that exist. The per-step results for 128K, 500K and 1M conversations show what limits each size: compute for new prompts at 128K, GPU memory at 500K.

Claude Code asks for a different model name per role, so map every role to the one model you serve:

export ANTHROPIC_BASE_URL = https://api.inferencekey.com/endpoint/<project>/<workload> export ANTHROPIC_MODEL = deepseek-v4.1-flash export ANTHROPIC_DEFAULT_HAIKU_MODEL = deepseek-v4.1-flash Enter fullscreen mode Exit fullscreen mode The full block adds the key and the Sonnet and Opus roles, and the DeepInfer on 4x B200 guide walks through it. One trap: different values in Authorization: Bearer and x-api-key return a 401. Pointed at this endpoint, Claude Code 2.1.310 implemented a function against a 15-test file in 39 seconds, with 14 tool calls and 15 of 15 passing. A smoke test, not a score.

This part is modelled. A simulator calibrated on 7,879 real turns, load-tested with NVIDIA's aiperf, puts the machine at about 250 agents if 95% of answers must start within 15 s, and about 400 at 30 s. Either way it handles near 450 model calls a minute; more agents only wait longer.

At two agents per power user, 400 agents is about 200 subscriptions. Counted by the peak call rate of our reference user, a very heavy user who also runs autonomous agents, it is about 80. The machine costs 27,900 USD a month, and it only beats Claude Max 20x above about 140 subscriptions.

Before you buy, pull each person's peak calls per second from your team's Claude Code logs.

LiveBench's agentic coding board of 10 October puts DeepSeek V4.1 Flash at 77.3, top of 69 models, with Claude Opus 5.5 at 71.7. Artificial Analysis, a broader index, reverses the order: Opus 5.5 at 58, Flash at 39. We ran neither. Our needle check proves the cache keeps answers intact, not that the model codes well on your repository.

Per-step tables, the cold stress test and the cost derivation are in the research post on one 4x B200 machine taking the Claude Code load of about 200 subscriptions.