Your 7B Model Doesn't Need an H100: A Practical GPU Sizing Guide for LLM Inference

Researched October 2026. Sizing rules from public docs and community practice, not my own benchmarks.

The most expensive mistake in LLM inference isn't picking the wrong cloud — it's renting too much GPU. I keep seeing people spin up H100s to serve 7B chatbots. Here's a practical guide to right-sizing your GPU.

Weights in FP16 take roughly 2 bytes per parameter:

Then add ~20–30% headroom for KV cache (grows with context length and batch size). Long-context workloads can double your memory needs — that's the part most guides skip.

FP8 / AWQ / GPTQ roughly halve the memory requirement:

If your accuracy budget allows it, quantization is the single biggest cost lever you have — bigger than switching cloud providers.

Combining sizing with cloud prices (researched Oct 2026): a 7B model on a Vast.ai spot 4090 (~$0.35/hr) vs. an H100 on-demand (~$2.50+/hr) is a 7x cost difference for a workload that might not even notice.

I'm building a free tool that asks about your use case and ranks GPU options: https://qisuanai.com/advisor

What are you serving, and on what? Curious where these rules break down in practice.