Local LLM vs Cloud API: The VRAM Math, Unit Economics, and Routing Architecture

Posted on Oct 27 • Originally published at xsypher.com

Almost nobody should deploy exclusively on local silicon or entirely behind a provider's API key. The functional engineering question is determining which segments of your workload belong on internal hardware and which justify managed endpoints.

Cloud models remain superior at deep multi-step reasoning, massive context processing, and complex tool orchestration. However, modern open-weight releases mean keeping sensitive data offline is no longer a compromise on utility. The architectural decision hinges on three constraints: physical memory limits, token volume break-even points, and regulatory data isolation.

Compute ratings dictate generation speed, but available memory dictates whether a model initializes at all. Weights must reside in memory, and the key-value (KV) cache for your context window sits directly on top.

At 16-bit precision, weights consume approximately two bytes per parameter. A 70B model requires roughly 140 GB of memory before processing a single token of context. Four-bit quantization (such as Q4_K_M) reduces that requirement by approximately 4x:

Model Scale | FP16 VRAM | Q8_0 VRAM | Q4_K_M VRAM ------------|------------|------------|------------ 7B | ~14 GB | ~7.5 GB | ~5 GB 13B | ~26 GB | ~13.5 GB | ~9 GB 32B | ~64 GB | ~34 GB | ~22 GB 70B | ~140 GB | ~72.5 GB | ~44 GB Enter fullscreen mode Exit fullscreen mode Mixture-of-Experts (MoE) architectures change the operational dynamic. OpenAI's gpt-oss-120b features 117B total parameters but activates only 5.1B per token. Post-trained using MXFP4 quantization, these architectures read like large models on disk while generating tokens with the compute overhead of a smaller model.

Local inference trades operational expense for capital expenditure and maintenance.

Consider an enterprise workstation equipped with 24–32 GB VRAM priced at $2,500. Amortized across 36 months, base hardware depreciation sits near $69 per month. Factoring in three hours of active daily generation at 400W and an idle draw of 60W at $0.18/kWh, power adds approximately $13 per month. Total baseline cost: ~$83/month prior to human maintenance.

Against hosted cloud pricing (assuming a blended ratio of 75% input tokens and 25% output tokens):

Cloud Tier | Blended Rate ($/1M) | Monthly Break-Even Volume ------------------|---------------------|-------------------------- Utility / Fast | ~$0.85 | ~98 Million tokens Mid-Tier | ~$4.50 | ~18 Million tokens Frontier | ~$10.00 | ~8 Million tokens Enter fullscreen mode Exit fullscreen mode Replacing lightweight classification APIs with local deployments rarely yields positive ROI. Replacing consistent frontier workloads pays back hardware capital within months.

Beyond cost, data isolation follows two discrete topologies:

[User Payload] │ ▼ [Routing Policy Engine] ├── (Sensitive / Structured) ──> [Local On-Premises Runtime] │ │ │ (Zero Egress) │ │ └── (Complex / High-Context) ───> [Provider API via TLS] │ (Contractual Isolation) Enter fullscreen mode Exit fullscreen mode Local execution establishes a physical boundary: zero network egress, zero third-party data retention policies to audit, and zero dependency on external uptime SLAs.

Hosted APIs offer contractual boundaries. While enterprise agreements frequently specify zero-retention policies and exclude inputs from training pipelines, these safeguards are legal abstractions rather than architectural certainties.

When deploying on-premises runtimes, teams must account for three persistent constraints:

The resilient approach routes routine extraction, summarization, and private internal records through quantized local infrastructure, while offloading high-order reasoning and extreme context pipelines to commercial providers.