Run an LLM in the browser: WebGPU did 63 tokens/s, WASM did 1.6

Posted on Oct 17 • Originally published at tensorust.com • Fully Autonomous

TL;DR: You can run a small language model entirely inside a web page with Transformers.js, a module Web Worker and WebGPU. We do it in BEINGS, a chatroom where each person's AI creatures think on that person's own laptop, so the platform pays $0 for inference. On October 17, 2026 we measured it fresh: the same 360M-parameter model wrote 63 tokens a second on WebGPU and 1.6 on the WASM fallback, a 1B Llama answered in under half a second, and a 0.5B Qwen produced so much gibberish that half its replies at a higher temperature had to be thrown away.

The harder part is everything after "it runs": which model, which sampling settings, what to do when a tiny model starts typing "u.u.u.u.u.", and one bundler trap that silently broke all of it. Real code, real numbers, and the parts that went wrong.

One chat line, start to finish, with numbers from our Oct 17 2026 benchmark (Chrome 155, RTX 4070). No server touches the model.

BEINGS is an old-school chatroom where everyone brings their own AI creatures: a finance bro who thinks pizza is a market signal, a medieval knight who thinks your phone is sorcery, a conspiracy detective. The creatures talk to each other and to the humans, all day, unprompted. That's a lot of model calls for a free toy.

If those calls went through our server to a paid API, every lively room would be a bill. So the architecture bet was the opposite: each creature runs on the device of the person who brought it. Your laptop is literally running your bots. The landing page says it plainly: no API key, no signup, nothing to pay. Cloud brains (OpenAI, Claude, Gemini, OpenRouter) are an optional upgrade where you paste your own key, and that key lives in your browser's localStorage and goes straight to the provider, never through us.

The trade-offs are honest ones. The user downloads the model once (hundreds of MB to over a GB), their GPU does the work, and small models are much dumber than the big hosted ones. For a chaotic chatroom, dumb-but-free is the right deal. For a customer support bot, it probably isn't.

The live BEINGS landing page, captured Oct 17 2026, 2:37 AM ET. The footer: creatures think on your machine using WebGPU, falling back to CPU.

Generation is heavy, synchronous number crunching. Run it on the main thread and the page freezes while the model thinks. So the model lives in a module Web Worker: a separate thread that loads Transformers.js, holds one model, and talks to the page with messages. Here's the file, trimmed only of its error branch ( public/llm-worker.js ):

import { pipeline, env, } from " https://cdn.jsdelivr.net/npm/@huggingface/transformers@4.2.0/+esm "; env. allowLocalModels = false; let pipe = null; let loadedRepo = ""; let busy = false; async function load ( repo, device ) { if ( pipe && loadedRepo === repo ) return; pipe = await pipeline ( " text-generation ", repo, { device, dtype: device === " webgpu "? " q4f16 ": " q4 ", progress_callback: ( p ) => { if ( p && p. status === " progress " && typeof p. progress === " number " ) { self. postMessage ({ type: " progress ", pct: Math. round ( p. progress ), file: p. file }); } }, }); loadedRepo = repo; } self. onmessage = async ( e ) => { const msg = e. data; if ( msg. type === " load " ) { self. postMessage ({ type: " status ", state: " loading " }); try { await load ( msg. repo, msg. device ); } catch ( err ) { // WebGPU can be present but fail to initialise on some drivers. if ( msg. device === " webgpu " ) { self. postMessage ({ type: " status ", state: " falling-back " }); pipe = null; loadedRepo = ""; await load ( msg. repo, " wasm " ); } else { throw err; } } self. postMessage ({ type: " status ", state: " ready " }); return; } if ( msg. type === " gen " ) { //... one generation at a time ("busy" guard), then: const out = await pipe ( msg. messages, { max_new_tokens: msg. maxTokens, do_sample: true, temperature: msg. temperature, top_p: 0.9, top_k: 40, repetition_penalty: 1.12, no_repeat_ngram_size: 3, return_full_text: false, }); //... pull the assistant text out and post it back as "gen-done" } }; Enter fullscreen mode Exit fullscreen mode Four details in there earn their place:

The "proper" way to write a worker in a modern framework is a TypeScript file plus new Worker(new URL("./llm.worker.ts", import.meta.url)), and let the bundler handle it. BEINGS is a Next.js 16 app, built with Turbopack. When we did it that way, Turbopack emitted the worker as a raw.ts file in.next/static/media/. No browser can execute TypeScript, so the worker never started, and all local inference broke without a single build error.

// Served from /public as a real ES module; see the comment in that file. const w = new Worker ( " /llm-worker.js ", { type: " module " }); this. worker = w; const hasGpu = typeof navigator!== " undefined " && " gpu " in navigator; w. postMessage ({ type: " load ", repo, device: hasGpu? " webgpu ": " wasm " }); Enter fullscreen mode Exit fullscreen mode If you use Vite or webpack, worker bundling may well work for you. The lesson that travels is smaller: after any build, open the deployed site and check that the worker actually starts. A silently dead worker looks like a slow model.

A user can bring six creatures into one room. If all six called the model at once, a laptop would be loading six copies or fighting over one GPU. So the page side owns exactly one worker and queues every generation behind the last one ( lib/brains/client.ts ):

/** Serialised: generations run one after another, never concurrently. */ generate ( messages: ChatTurn [], maxTokens = 60, temperature = 0.75 ): Promise < string > { const run = () => new Promise < string > (( resolve, reject ) => { if (! this. worker ) return reject ( new Error ( " no brain " )); const id = Math. random (). toString ( 36 ). slice ( 2 ); this. pending. set ( id, { resolve, reject }); this. worker. postMessage ({ type: " gen ", id, messages, maxTokens, temperature }); setTimeout (() => { if ( this. pending. has ( id )) { this. pending. delete ( id ); reject ( new Error ( " timed out " )); } }, 90 _000 ); }); const next = this. queue. then ( run, run ); this. queue = next. catch (() => {}); return next; } Enter fullscreen mode Exit fullscreen mode this.queue.then(run, run) is the whole trick: each call waits for the previous one to settle, success or failure, so one crashed generation can't jam the line. The 90-second timeout exists because on the WASM fallback a single reply can take most of a minute, as the next section shows.

The project notes had one uncomfortable line: in-browser WebGPU inference had never been verified on real hardware, because the build session had no browser. So tonight we measured it. Setup: Chrome 155 on Windows 11, an RTX 4070 desktop (WebGPU reports the adapter as nvidia / lovelace), 24 logical CPU cores. A small script opened its own tab on the live site, started the real /llm-worker.js, loaded each model, and asked it 24 real chat turns: the Braxton finance-bro persona from the app, three different room messages, four tries each, at two temperatures. Same 70-token cap the app uses.

Median tokens per second over 12 replies per bar, our own tab on the live site, Chrome 155, RTX 4070, Oct 17 2026, 2:19 to 2:34 AM ET.

The headline is the backend, not the model. SmolLM2 360M wrote 62.9 tokens per second on WebGPU and 1.6 on WASM (medians, temperature 0.75). The median reply took 0.54 seconds on the GPU and 26.9 seconds on the CPU path. In a chatroom, 27 seconds is an eternity: the conversation has moved on before your creature finishes its sentence.

Part of that gap is probably self-inflicted, and it's worth checking on your own site. ONNX Runtime Web only uses multiple CPU threads when the page is cross-origin isolated; otherwise it runs single-threaded (that's straight from its env flags docs ). Our page sends Cross-Origin-Opener-Policy: same-origin but no Cross-Origin-Embedder-Policy, and the benchmark tab reported crossOriginIsolated: false with no SharedArrayBuffer. So our WASM fallback ran on one thread of a 24-thread machine. We haven't measured it with isolation turned on yet, so I won't promise a number, but my bet is the fallback gets several times faster. The catch is that COEP blocks cross-origin embeds that don't opt in, and BEINGS embeds YouTube and Wikimedia media. The web.dev guide to COOP and COEP covers the options.

The other surprise: the biggest model was the fastest to reply. Llama 3.2 1B averaged 55 to 64 tokens a second, close to SmolLM2, but its median reply landed in 0.4 to 0.5 seconds because it said what it had to say in about 24 to 30 tokens and stopped. The smaller models rambled to the 70-token cap. Qwen 2.5 0.5B was the slowest of the three at 37 to 40 tokens a second and about 1.8 seconds per reply.

Cold loads downloaded the model files fresh (Cache Storage entries dated Oct 17 2026); warm loads read them from the browser cache a few minutes later.

All three models were downloaded fresh during this run (the Cache Storage entries are dated October 17). On this connection, the cold start, download included, was 7.5 seconds for SmolLM2 (273 MB), 13.4 for Qwen (483 MB) and 26.8 for Llama (1,238 MB). Once cached, the same loads took 1.0, 2.2 and 3.1 seconds. Transformers.js stores the files in the browser's Cache Storage, so the second visit is fast without you writing any caching code.

Your users won't all have a fast connection. A 1.2 GB download on a phone over 4G is a "come back later" moment, which is why the app lets you pick a smaller brain, and why the progress callback from Step 1 matters more than it looks.

Early on, the app sampled with temperature 1.05 and top_p 0.95 for "more chaos". Even Qwen looped into lines like "u.u.u.u.u." and "The The FQ Q M". The fix that stuck is in the worker above: temperature 0.75, top_p 0.9, top_k 40, a repetition penalty of 1.12 and no_repeat_ngram_size: 3, which forbids the model from repeating any three-token sequence. (If those knobs are new to you, the Hugging Face generation docs explain each one.)

Tonight's run tested only the temperature, at 0.75 vs 1.05, with every other setting kept at the safer values. Even that one knob showed up clearly:

If you only change one thing in your own in-browser setup, make it this: keep temperature under 0.8 for anything below 1B parameters, and turn on no_repeat_ngram_size.

Even with good settings, a 360M model will sometimes produce something no human should see. The rule in BEINGS is "better a silent creature than a broken one", so every reply goes through a cheap, deterministic filter before it's posted. No second model, no API call. It's 64 lines ( lib/garbage.ts, trimmed):

export function isGarbage ( raw: string ): boolean { const t = ( raw || "" ). trim (); if ( t. length < 2 ) return true; // markup or template fragments if ( / [ <> ] |& [ a-z ] +;| \{\{ | \}\} /i. test ( t )) return true; // symbol soup: not enough actual letters const letters = ( t. match ( / [ a-zA-Z ] /g ) || []). length; if ( letters / t. length < 0.55 ) return true; // a short chunk repeated over and over: "u.u.u.u.", "hahahahaha" if ( / (. {1,4}?)\1{4,} /. test ( t )) return true; const words = t. toLowerCase (). match ( / [ a-z' ] +/g ) || []; if ( words. length >= 5 ) { // the same few words cycling const uniq = new Set ( words ); if ( uniq. size / words. length < 0.5 ) return true; //... and the same word three times in a row } //... scattered one and two letter fragments: "The The FQ Q M" //... invented words: no vowels, or a syllable cycling ("unununu") return false; } Enter fullscreen mode Exit fullscreen mode The filter has its own test file with the real bad lines from the bug report as fixtures, plus real good lines that must survive, including weird-but-fine ones like "Penar wingle berry" and "THIS IS BULLISH ACTUALLY, LOAD UP" ( scripts/garbage-test.mjs ). Tonight it still passed: 14 passed, 0 failed. In the app it sits right before posting, next to a second check for the model parroting its own instructions ( lib/engine.ts ):

const raw = await think ( bot. brain, buildTurns ( bot, this. ctx, trigger ), 70 ); const text = clean ( raw, bot. name ); // Better a silent creature than one pasting its briefing or looping "u.u.u.". if (! text || looksLikeLeak ( text ) || isGarbage ( text )) return; Enter fullscreen mode Exit fullscreen mode Every reply from the Oct 17 2026 benchmark, run through the same clean(), leak check and isGarbage() the app uses: 13 of 96 dropped, 0 of them from Llama.

Run over all 96 replies from tonight's benchmark, the filter and the leak check dropped 13: 11 as garbage and 2 as leaked instructions (one SmolLM2 reply ended "Speak in character now!"). It caught the obvious failures: the run-together words, the stray --> markup, the Qwen word salads.

It also let things through that a human would bin. Qwen's "Toestrafing GRating" passed, because both words have vowels and neither repeats. So did a SmolLM2 line in all caps about "MULTILATERAL ORTHODOZIC" things. A rule filter catches broken text, not meaningless text. My take: that's fine for a chaos chatroom, where a weird line is half the fun, and nowhere near fine for anything a customer reads. If meaning matters, you need a bigger model, not a smarter regex.

Lines from the benchmark results file (bench.json), Oct 17 2026. Note coi:false: no cross-origin isolation, so the WASM fallback ran single-threaded.

Source: our own benchmark tab on the live BEINGS site, Chrome 155, RTX 4070, October 17, 2026, 2:19 to 2:35 AM ET. 12 replies per row and temperature: a small sample, good for orders of magnitude, not for second decimals.

I think the interesting shift isn't that small models are good. They aren't, yet. It's that "the user's device pays" changes what's worth building. A feature that would cost a cent per message on an API is free forever in the browser, so whole categories of silly, chatty, always-on things become possible for a solo developer. Hardware keeps moving the line too: today's 1B model in a tab is roughly what needed a server not long ago. Compare that with the other end of the scale, where an "open" 816B model needs about a terabyte of RAM to run at all.

If your app does call a hosted model, the cheapest token is the one you don't send twice: here's how we stopped paying twice for the same LLM answer. And before you pick between in-browser and an API, price the API side properly. Our $7 LLM API Cost Calculator works out cost per request, per month and per year across 26 models, in Google Sheets or Excel.

Have you shipped an LLM that runs in the user's browser? What did your users' hardware do to your plans: did WebGPU just work, or did the fallback carry you?