Nemotron 3.5 Bolt on one GPU: what the 24 GB covers, the drafter flag and the llama.cpp trap
NVIDIA's Nemotron 3.5 Bolt 74B-A9B 24GB is a 74B agent model that reads text and images. Only about 9B of its parameters (the numbers a model learns in training) work on each token, and a token is roughly a piece of a word. Most are stored at about 2 bits each, so the file is 22.0 GB. We served it
Posted on Oct 29 • Originally published at inferencekey.com
NVIDIA's Nemotron 3.5 Bolt 74B-A9B 24GB is a 74B agent model that reads text and images. Only about 9B of its parameters (the numbers a model learns in training) work on each token, and a token is roughly a piece of a word. Most are stored at about 2 bits each, so the file is 22.0 GB.
We served it on one A40, a 48 GB data-center card on our cloud, with llama.cpp, the open-source program most people use to run models on one machine. The image reader was on, and so was the drafter, a small helper model that speeds up writing. It used about 27 GB. For one user it wrote 136 tokens a second with the drafter and 70 without.
Three choices matter more than the model: what the 24 GB covers, which llama.cpp you compile, and when the drafter pays off. Every figure sits in our Nemotron 3.5 Bolt deployment write-up with the full measurements.
NVIDIA's sum comes to 23.83 GB. That is 22.01 GB for the parameters, 0.87 GB of working memory for roughly 128,000 tokens of conversation, and 0.96 GB for the drafter. The image reader, another 1.68 GB, is deliberately outside the sum.
Two things eat into that margin. First, the unit: a card sold as 24 GB has 24 GiB. Read NVIDIA's figure as decimal and roughly 1.9 GB stays free; read it as GiB and only about 0.2 does. Second, the drafter took close to 2.4 GB on our A40, not 0.96. So a fit on a real 24 GB card stays unproven: we never put Bolt on one.
The recipes NVIDIA publishes for llama.cpp include the image reader, and the smallest of them asks for a 32 GB card. Our 27 GB (25.2 GiB), with the quick-start 98,304 tokens of conversation, counts the image reader too, so it does not check NVIDIA's text-only number. Measure your own server with the drafter on and off, in the unit your card is sold in.
Bolt needs llama.cpp code from release day, 15 October. Kartik Gulia, an NVIDIA engineer, added the model and its image reader to llama.cpp, and ggerganov merged the change in llama.cpp pull request #30520 that same day. Older builds refuse the file.
We hit exactly that. Our worker (the agent that sets up each GPU machine for InferenceKey) pins a llama.cpp release 1,920 commits older than the merge, so we set v0.7.0 by hand.
Then compiling ate the clock. A fresh machine took 764.8 seconds from request to ready, and 705 of those went into building llama.cpp on 9 CPU cores. Our worker calls cmake --build with no parallel jobs unless CMAKE_BUILD_PARALLEL_LEVEL is set. Even with 12 jobs set by hand, the build was 92% of the wait. On a 31-core machine the same build finished in 2 minutes 41 seconds. A second copy on an already built machine was ready in 15.6 seconds. If first starts matter, count CPU cores, not only the GPU.
Bolt comes with the LiLiCorr drafter, about 1.1B parameters in a 1.0 GB file. The drafter proposes several tokens ahead, Bolt verifies the whole batch in a single step, accepts the run up to its first mismatch and writes that token itself. You turn it on when the server starts:
--spec-draft-hf nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr-GGUF \ --spec-type draft-dflash --spec-draft-n-max 6 --spec-draft-ngl 999 Enter fullscreen mode Exit fullscreen mode On short and medium prompts it about doubled the speed: over 25 requests the median was 136 tokens a second against 70, with a floor of 110, and 70% of the drafted tokens were accepted. The per-prompt speed and acceptance table shows where it gains most.
Long documents flip the result: the drafter slows prompt reading by about a quarter. A question over a 25,000-token document took 17.5 seconds without the drafter and 21.5 seconds with it. And v0.7.0 has the per-request drafter control compiled out, so the choice is made once per server. Mixed traffic? Benchmark both shapes first.
The quality numbers are NVIDIA's: Bolt beats Qwen3.6-35B-A3B on most agent benchmarks (Terminal-Bench 2.1: 50.98 against 44.38) and loses on factual recall. NVIDIA's speed table puts it behind Qwen3.6 and Gemma 4. Our side was about 50 requests, one GPU, one request at a time; on a two-tool refund task it called both tools correctly in all 5 runs. We did not test several requests sharing the card, where drafters tend to help less, or vLLM, the data-center server software that is the other route.
NVIDIA built, compressed and released Bolt and its drafter under OpenMDW-1.1, which allows commercial use. Kartik Gulia and ggerganov made llama.cpp able to run it. AI Search's news video pointed us to the family. Our part was wiring it end to end and measuring it. The exact command, per-request timings and worker settings are in the complete Nemotron 3.5 Bolt article on inferencekey.com.