WebLLM is an LLM inference engine that runs open models entirely inside a web browser, with no server, using WebGPU for hardware acceleration. Its interface follows the OpenAI chat completions API, so the same calls (streaming, JSON mode, seeding, and preliminary function calling) work against local open-source models, and inference stays on the user's device.
It is a companion project to MLC LLM and reuses its model artifacts and compilation flow. Built-in model families include Llama, Phi, Gemma, Mistral, and Qwen, and custom models compiled to the MLC format can be added. JSON-mode structured generation runs in the WebAssembly part of the model library, and optional Subresource Integrity hashes verify downloaded config, WASM, and tokenizer files before loading.
WebLLM ships as an npm package or through a CDN, and its engine can run in a dedicated Web Worker or a Service Worker to keep heavy computation off the UI thread and manage model lifecycle. It also supports building Chrome extensions, and WebLLM Chat is a full chat application built on it. Downloaded weights are cached in the browser through the Cache API, IndexedDB, the Origin Private File System, or an experimental cross-origin storage extension.
Features
- In-browser inference: models run client-side with WebGPU acceleration and no server
- OpenAI-compatible API: chat completions with streaming, JSON mode, seeding, logit-level control, and preliminary function calling
- Structured JSON generation: JSON mode with custom schemas implemented in WebAssembly
- Built-in models: Llama, Phi, Gemma, Mistral, and Qwen families
- Custom models: compile and deploy your own weights in the MLC format
- Workers: run the engine in a Web Worker or Service Worker
- Chrome extensions: examples for basic and advanced browser extensions
- Cache backends: Cache API, IndexedDB, OPFS, or a cross-origin storage extension
- Integrity verification: SRI hashes checked on model artifacts before loading