How Much GPU Memory Do You Really Need to Run an Open LLM Locally?

Signup Write New Story How Much GPU Memory Do You Really Need to Run an Open LLM Locally? by Contexte tech by Contexte tech | @ contextetech Building Contexte Tech: free, open-source tools to choose the right AI model, local GPU or API, and the right hardware, from official figures.

Building Contexte Tech: free, open-source tools to choose the right AI model, local GPU or API, and the right hardware, from official figures.

Can we run it on our own machine?" is now one of the first questions teams ask about an open-weight model. Data that cannot leave the building, a predictable monthly bill, or simply the wish to stop depending on a provider: the reasons vary, but the next question is always the same. Which hardware, and how much of it?

Vendor pages and forum threads rarely answer it directly. Yet most of the answer is arithmetic. You need three numbers that are published officially: the model's parameter count, your hardware's memory, and your hardware's memory bandwidth. This article walks through the method step by step, shows where the simple rules break, and applies it to twelve current open models.

A model's weights must sit in GPU (or unified) memory. Their size is simply:

weights (GB) = parameters (billions) × bytes per parameter

The bytes per parameter depend on the precision you load the model in:

So Mistral Small 3.2 (24.0 billion parameters, per its official model card) needs 48 GB of weights in 16-bit, 24 GB in 8-bit and 12 GB in 4-bit. Lower precision costs some quality, and how much depends on the model and the task. Measure it on your own evaluation set before committing.

Weights are not the only thing in memory. For every token in the context, each attention layer stores a key and a value vector: the KV cache. Its size per token is:

KV per token (bytes) = 2 × layers × KV heads × head dimension × bytes per value

Take Mistral Small 3.2. Its official config.json lists 40 layers, 8 key-value heads and a head dimension of 128. In 16-bit, that is 2 × 40 × 8 × 128 × 2 = 163,840 bytes, or 160 KiB per token.

Architecture matters a lot here. Qwen3.8 27B mixes layer types: according to its config, only 16 of its 64 layers use full attention (with 4 KV heads of dimension 256); the other 48 use linear attention, which keeps a fixed-size state instead of a growing cache. Its full-attention cache is 2 × 16 × 4 × 256 × 2 = 64 KiB per token, about 2.1 GB at 32k tokens, less than half of Mistral Small's.

The cache also multiplies with concurrent requests: ten users with 32k-token conversations need ten caches.

A common shortcut is to add 20% to the weights for the cache and runtime overhead. It is a reasonable floor for short contexts and a single user. It is badly wrong for long contexts or many concurrent users. In those cases, compute the cache explicitly.

When a model writes its answer, it generates one token at a time, and for each token it must read the weights it uses from memory. At small batch sizes this step is limited by memory bandwidth, not by compute. That gives a simple ceiling:

max tokens/s (one stream) ≈ memory bandwidth (GB/s) ÷ bytes read per token (GB)

For a dense model, the bytes read per token are the whole weights. For a mixture-of-experts model, only the active parameters are read for each token, which is why these models are fast for their size.

Treat these as ceilings, not forecasts. Real throughput is lower: kernels do not reach peak bandwidth, the KV cache must be read too, and splitting a model across several GPUs adds communication that does not scale linearly. Batching many requests raises total throughput well above the single-stream figure. Use the ceiling to rule hardware out, then benchmark what is left.

Teams usually think in requests per month, not tokens per second. The conversion is direct:

required tokens/s = requests per month × output tokens per request ÷ seconds in a month

For 200,000 requests a month, each producing about 225 words (≈ 300 tokens at 0.75 words per token), that is 60 million tokens over 2,592,000 seconds: about 23 tokens/s on average. Peaks will be higher than the average, and for interactive use you also want each individual answer to stream fast enough to read comfortably. Size for your peak, not your mean.

Applying steps 1 and 2 (weights + 20%, short context) to twelve current open-weight models, with parameter counts from each publisher's official model card:

And the hardware, from the manufacturers' spec sheets:

A few patterns stand out once the numbers are side by side.

Small dense models are now a single-GPU problem. Every model up to about 30 billion parameters fits on one 32 GB card in 8-bit or lower, with single-stream ceilings well above reading speed (Mistral Small 3.2 in 8-bit on an RTX 5090: 1,792 ÷ 24 ≈ 75 tokens/s).

Unified-memory machines trade speed for capacity. A DGX Spark holds the same 8-bit Mistral Small 3.2, but at 273 GB/s its ceiling is about 11 tokens/s: fine for batch jobs, slow for chat. The arithmetic makes that visible before you buy.

Mixture-of-experts changes the trade-off. gpt-oss-120b needs as much memory as a dense 117B model, but reads only 5.1 billion parameters per token. That is why a 128 GB unified-memory box can serve it at interactive speed in 4-bit, something no dense model of that size could do on the same machine.

The largest open models remain server-class. Llama 4 Maverick, Qwen3.5 397B and Nemotron 3 Ultra need hundreds of gigabytes even in 4-bit: a 512 GB unified-memory workstation, or several data-center GPUs.

BYTES = {16: 2, 8: 1, 4: 0.5} def weights_gb(params_b, bits): return params_b * BYTES[bits] def kv_cache_gb(layers, kv_heads, head_dim, context_tokens, users=1, bytes_per_value=2): per_token = 2 * layers * kv_heads * head_dim * bytes_per_value return per_token * context_tokens * users / 1e9 def max_tokens_per_s(bandwidth_gbs, active_params_b, bits): return bandwidth_gbs / (active_params_b * BYTES[bits]) def required_tokens_per_s(requests_per_month, output_tokens): return requests_per_month * output_tokens / (30 * 24 * 3600) # Mistral Small 3.2, 8-bit, 32k context, one user, on an RTX 5090 need = weights_gb(24.0, 8) + kv_cache_gb(40, 8, 128, 32_768) print(f"{need:.1f} GB needed") # 29.4 GB, fits in 32 GB print(f"{max_tokens_per_s(1792, 24.0, 8):.0f} tok/s ceiling") # 75 print(f"{required_tokens_per_s(200_000, 300):.0f} tok/s needed") # 23 Conclusion Sizing hardware for a local LLM comes down to three checks:

All three use figures that publishers and manufacturers print on their official pages. The arithmetic will not replace a benchmark on your own workload, but it tells you which machines are worth benchmarking, and which are ruled out before you spend anything.

If you want to run these checks without the spreadsheet, I built a free tool that applies this method to the models and hardware above: Choose your GPU server for local AI.