Running a Coaching LLM in the Browser: What WebGPU Really Costs

We built a real-time writing coach that scores tone and clarity as you type — think "this sentence reads as passive-aggressive" nudges inside a feedback composer. The pitch from day one was privacy: never send a user's draft performance review to a server. So we tried to run the whole thing client-side with WebGPU. It mostly worked. Here's the actual bill.

We landed on a 3.8B parameter instruction-tuned model (Phi-3-mini class), quantized to 4-bit, served through web-llm so we got a WebGPU-accelerated inference loop without writing our own compute shaders. The quantization step looks like this if you're prepping your own checkpoint:

mlc_llm convert_weight./phi-3-mini-hf \ --quantization q4f16_1 \ -o./dist/phi-3-mini-q4f16

mlc_llm gen_config./phi-3-mini-hf \ --quantization q4f16_1 \ --conv-template phi-3 \ -o./dist/phi-3-mini-q4f16

q4f16_1 gets you weights at roughly 2.2GB instead of the original ~7.6GB in fp16. That number matters more than almost anything else in this post.

A cold load means fetching ~2.2GB of weight shards, compiling WebGPU pipelines, and warming up KV-cache buffers. On a decent laptop over home wifi, that's 25–40 seconds before the first token. On a phone over LTE, we saw 90+ seconds, and a chunk of users just bailed.

The fix isn't clever inference — it's aggressive caching:

const engine = await CreateMLCEngine("Phi-3-mini-4k-instruct-q4f16_1-MLC", { initProgressCallback: (p) => updateLoadingBar(p.progress), appConfig: { useIndexedDBCache: true, // persist shards across sessions }, });

With IndexedDB caching, repeat visits drop to 2–4 seconds for cache validation plus pipeline recompilation (WebGPU doesn't let you cache compiled shaders across sessions yet, so you pay that tax every load). First-visit latency stays brutal no matter what. We ended up pre-warming the model during onboarding — while the user reads a privacy explainer, the weights download in the background. It's a UX trick, not a technical fix, and I want to be honest about that distinction.

This is where WebGPU stopped feeling like a drop-in replacement for a server call. Browsers cap maxBufferSize and maxStorageBufferBindingSize per adapter, and those caps vary wildly by GPU and OS. You can check them before committing to a model size:

if (limits.maxStorageBufferBindingSize < 1_073_741_824) { // under 1GB binding — fall back to a smaller model or WASM loadFallbackModel(); }

In practice: integrated GPUs on mid-range Windows laptops and most Android devices choked on the 3.8B model once you factored in KV-cache growth during longer conversations. We had to drop to a 1.1B model (TinyLlama-class) for any device reporting navigator.deviceMemory < 8 or failing the buffer check above. Quality dropped noticeably — shorter effective context, more generic phrasing, occasional repetition loops after 6–8 turns.

iOS Safari was its own category of pain. WebGPU landed there only recently and still has rough edges with larger buffer allocations; we gate it entirely and fall back to WASM, which is 5–10x slower per token but at least doesn't crash the tab.

I want to push back on the breathless "browser LLMs eliminate the need for servers" narrative, because for this product it's not true.

So we ended up hybrid, which is the less exciting but more honest answer:

The local model handles the cheap, frequent case — live micro-feedback as you type a sentence. The server handles the expensive, infrequent case — "give me a full rewrite of this paragraph." Users get the privacy win for the thing that actually feels invasive (constant monitoring of every keystroke) and the quality win for the thing they only ask for occasionally.

The local model isn't a server replacement — it's a different tool with different failure modes, and pretending otherwise just moves the disappointment to production.

If you've shipped WebGPU inference for something real: what was your actual memory ceiling in practice, and did you bucket devices by capability or just pick one model size and eat the failures? I'd like to know if our 8GB deviceMemory cutoff was too conservative or not conservative enough.