LLMs in the browser with WebGPU

Browsers can now run small LLMs entirely client-side: no server, no API key, no data leaving the device. That unlocks zero-install demos, offline tools and privacy-preserving features, with the constraints of a sandboxed, memory-limited environment. This lesson explains how it works and how to ship something solid.

#What WebGPU is

WebGPU is the modern browser API for the GPU. Unlike WebGL, which was designed for drawing, WebGPU supports general-purpose compute shaders, so matrix multiplications for LLMs run on the GPU. It works through your normal browser permissions: no install, no drivers to manage, and the page cannot reach beyond the sandbox.

Availability changes quickly. It is in Chromium-based browsers on supported platforms, and arriving in recent Safari and Firefox builds, with gaps on some devices and operating systems. Never assume it exists: feature-detect.

#The stack

The same ideas as native runtimes apply: weights in a compact quantised format (typically 4-bit), a tokenizer, a KV cache and a sampling loop. The difference is where it runs and how it is delivered.

#WebLLM in a few lines

import { CreateMLCEngine } from "@mlc-ai/web-llm";

const engine = await CreateMLCEngine("Qwen2.5-0.5B-Instruct-q4f16_1-MLC", {
  initProgressCallback: (p) => console.log(p.text),
});
const reply = await engine.chat.completions.create({
  messages: [{ role: "user", content: "Hello!" }],
});
console.log(reply.choices[0].message.content);

Try it live in the playground. Model identifiers and the API evolve, so check the project's current documentation for the exact model list and options.

#Limits to design around

• Download size: 300 MB to 2 GB on first visit. Weights are cached afterwards (Cache Storage or IndexedDB), but a first-time visitor on mobile data will notice.
• GPU memory per tab is limited, and browsers cap single buffers and total allocations. Stay at 3B or smaller for broad compatibility; 7B+ models work only on powerful desktops.
• Speed: expect a fraction of native speed. A 0.5 to 1B model may produce tens of tokens per second on a decent laptop GPU; larger ones are much slower. Prompt processing is relatively slow on integrated GPUs.
• Browser support: WebGPU is not everywhere. Always feature-detect navigator.gpu.
• Run it in a Web Worker so the UI stays responsive.
• Mobile browsers have tighter memory limits and may reload the tab under pressure, losing the loaded model.
• Background tabs are throttled. Long generations in a hidden tab can stall.
• Storage quotas: browsers limit how much a site can cache; large models may be evicted. Request persistent storage where appropriate.
• Security headers for WASM threads: multi-threaded WebAssembly needs cross-origin isolation (COOP and COEP headers), which affects embedding third-party content.

#Good use cases

Privacy-sensitive summarisation, offline tools, product demos, educational sandboxes, classification and rewriting features where you do not want server cost, and "try before you install" experiences. Poor fits: anything needing a large model, heavy reasoning, long contexts or shared state between users.

#Feature detection and fallback

async function pickBackend() {
  if ("gpu" in navigator) {
    try {
      const adapter = await navigator.gpu.requestAdapter();
      if (adapter) return "webgpu";
    } catch {}
  }
  return "wasm";            // slower CPU runtime, or fall back to a server API
}

Fallback options, in order of preference: a smaller model on WebGPU, a WASM runtime (slower but works), a server API (with consent and a privacy notice), or a clear message explaining what is unsupported. Test in at least one browser without WebGPU.

#A worker-based implementation

Keep the engine off the main thread:

// llm.worker.ts
import { WebWorkerMLCEngineHandler } from "@mlc-ai/web-llm";
const handler = new WebWorkerMLCEngineHandler();
self.onmessage = (msg: MessageEvent) => handler.onmessage(msg);

// main.ts
import { CreateWebWorkerMLCEngine } from "@mlc-ai/web-llm";

const engine = await CreateWebWorkerMLCEngine(
  new Worker(new URL("./llm.worker.ts", import.meta.url), { type: "module" }),
  "Qwen2.5-0.5B-Instruct-q4f16_1-MLC",
  { initProgressCallback: (p) => updateProgressBar(p.progress, p.text) },
);

const stream = await engine.chat.completions.create({
  messages: [{ role: "user", content: "Summarise: ..." }],
  stream: true,
});
for await (const chunk of stream) {
  appendToUI(chunk.choices[0]?.delta?.content ?? "");
}

Streaming the reply token by token makes a slow model feel responsive.

#Performance expectations

Rough numbers, which vary enormously by device and browser:

The same bandwidth rule applies: speed is about memory bandwidth divided by model size (memory bandwidth), minus browser overhead. Measure on the lowest-end device you intend to support (measuring speed).

#UX patterns that work

1. Ask before downloading. Show the size ("Download a 600 MB model for offline use?"), and let the user cancel.
2. Show real progress, and explain that it only happens once.
3. Cache and verify. After the first load, start instantly; if the cache is evicted, handle re-download gracefully.
4. Stream the output, and provide a Stop button.
5. Disable the input until the model is ready, and show a clear loading state.
6. Persist the conversation locally if appropriate, but tell users where data lives.
7. Offer a lighter model for weak devices, chosen automatically from detected memory when possible.
8. Handle tab reloads and crashes: the model must reload; do not lose user input.

#Privacy and security

• Data stays on the device, but only if your own page does not send it elsewhere: analytics, error trackers and third-party scripts can capture prompts. Audit them.
• Model provenance: models fetched from a CDN are code-adjacent data. Pin versions and use subresource integrity or checksums where possible, and host the files yourself for production (supply chain).
• Prompt injection applies as usual if the page feeds untrusted content to the model (prompt injection); a browser-based model with no tools has a small blast radius, which is a real advantage.
• Licences: check that the model licence permits redistribution through your site (licences).

#Deployment checklist

• Feature detection and a tested fallback
• Engine runs in a Web Worker
• Download size shown, with progress and cancel
• Weights cached, with a plan for eviction
• Self-hosted model files with versions pinned and integrity checked
• Tested on a low-end laptop and a mid-range phone
• No third-party scripts that capture prompts
• A clear statement of what stays on the device
• Memory use checked in the browser's task manager

#Troubleshooting

#Common mistakes

1. Picking a model too big for typical user devices.
2. Forgetting feature detection.
3. Running on the main thread.
4. Auto-downloading hundreds of MB without consent.
5. Testing only on a development machine with a discrete GPU.
6. Sending prompts to analytics.

#Exercise

You are building an offline "summarise this page" browser extension for general users.

1. Which model size would you target and why?
2. List three things you would add to protect the user experience.

Answers:

1. A 0.5 to 1.5B model at 4-bit (about 350 MB to 1 GB): it fits in most devices' GPU memory limits, downloads in reasonable time and is fast enough for short summaries.
2. Ask before downloading and show progress, run the model in a worker with streaming output and a Stop button, and feature-detect WebGPU with a WASM or "not supported" fallback. Also cache the model and pin its version.