How much memory does a local LLM need?

Before downloading a model to run locally, the question is whether it will fit in memory. The estimate is three terms added together and compared against what the machine can actually give the model:

need = weights + KV cache + overhead fits if need <= 0.8 × usable memory Enter fullscreen mode Exit fullscreen mode The numbers below follow the estimate used by the local model browser in the osFoundry desktop app, checked against what that app shows for a 16GB M3 MacBook.

Weights are parameter count times bytes per weight, and the bytes per weight depend on the quantization:

Q4_K_M is 0.56 rather than 0.5 because K-quants store a scale for each small block of weights, which works out to about 4.5 bits per weight.

For a 14B model at Q4_K_M: 14 × 10⁹ × 0.56 = 7.84 × 10⁹ bytes, or about 7.3 GiB.

If you don't recognize the quant name, assume FP16. It's the pessimistic choice, so you won't decide something fits when it doesn't.

During generation the model keeps keys and values for every token in the context. The size is:

2 × layers × kv_heads × head_dim × context_tokens × bytes_per_element Enter fullscreen mode Exit fullscreen mode Qwen2.5 14B has 48 layers, 8 KV heads (it uses grouped-query attention) and a head dimension of 128. With an FP16 cache:

The cache grows linearly with context length, so a model that fits easily at 4K can stop fitting at 32K. Estimates that don't state a context length aren't much use; the one in the screenshot assumes 4K.

llama.cpp also needs compute buffers and scratch space. A rough allowance is 10% of the weights plus about 200 MB. For the 14B model that's about 0.9 GB.

Adding the weights, the cache and the overhead puts a 14B model at Q4_K_M in the 8 to 9 GB range. The app shows 8.4 GB.

This is the part people get wrong most often.

On the 16GB Mac that gives: Q4_K_M 8.4 GB (70%, fits), Q5_K_M 10.4 GB (87%, tight), Q6_K 12.3 GB and Q8_0 14.9 GB (over).

Applied across models at Q4_K_M, the cut-off on this machine falls between 20B (11.9 GB, tight) and 27B (15.9 GB, too big).

Generating each token requires reading every weight from memory once, so an upper bound on generation speed is memory bandwidth divided by model size. A base M3 has about 100 GB/s, so an 8.4 GB model tops out around 12 tokens per second, and a 4.9 GB 8B model around 20. Real numbers come in below that, but the ratio between models holds up well.

So the size estimate answers two questions at once: whether the model loads at all, and roughly how fast it will write once it does.