BigMoeOnEdge runs Mixture-of-Experts (MoE) language models that are larger than a device's RAM, on CPU only, with phones as the main target. Each generated token uses only a few experts, so the engine keeps the always-needed weights in memory and reads the experts a token routes to straight from flash storage at the moment they are needed. Output is byte-identical to running the same model fully resident.
It is built on llama.cpp's public API by hooking its evaluation callback, so every quantization format, tokenizer, and chat template llama.cpp supports works unchanged. Adding a new MoE architecture is one row in a registry, with expert counts and layouts read from the model file at runtime. The largest model run so far is DeepSeek V4 Flash (284B parameters, about 91 GB on disk) on a phone with 12 GB of RAM, at under 1 token per second.
| Model on a 12 GB phone (Q4_K_M) | Size | Best measured tok/s | mmap baseline tok/s |
|---|---|---|---|
| gpt-oss-120b | ~60 GB | 2.2 | 0.09 |
| Qwen3.6-35B-A3B | 22.3 GB | 5.8 | 0.1 (unstable) |
| Qwen3-30B-A3B | 18.5 GB | 5.2 | 2.0 (unstable) |
| Gemma-4-26B-A4B | 17.0 GB | 5.0 | 0.4 |
Streaming settings change how weights are fetched but never the math. A separate group of lossy settings (fewer active experts, dropping cold experts, preferring cached experts, route-ahead) trades quality for speed on devices far past their memory. A decision mode scores a fixed set of choices by log-probability in a single prefill, for agents picking a UI action or a router picking a tool. Prefill can optionally run on a Snapdragon Hexagon NPU, and the same engine builds on Linux, macOS, and Windows desktops, streaming from an SSD.
The project describes itself as an engineering package of ideas from AirLLM, LLM in a flash, FlexGen, PowerInfer, and EdgeMoE rather than a new technique. Correctness is checked by byte-identity tests on a synthetic MoE model, and speed figures come from one phone and one laptop.
Features
- Expert streaming: reads only the routed experts from flash per token, leaving the rest of the model on disk
- Expert cache: keeps frequently used experts in RAM with automatic or fixed sizing and a ceiling
- Parallel and direct I/O: multiple read lanes, page-cache bypass, and optional overlap of reads with compute
- Dense-weight policies: controls how always-needed weights are held so the OS cannot reclaim them mid-generation
- Lossy speed settings: reduced active experts, cold-expert dropping, cache-preferring routing, and route-ahead
- Speculative decoding: MTP and n-gram drafting that verifies several tokens per pass without approximation
- Decision mode: scores multiple-choice options in one prefill, with probabilities usable as an abstention threshold
- NPU prefill: optional Hexagon NPU prefill, 2.8 to 9.5 times faster than CPU on measured prompts
- Telemetry: per-token split of flash I/O, cache management, and compute, plus cache hit rate and bytes read
- Supported architectures: Qwen MoE families, Gemma 4, gpt-oss, NVIDIA Nemotron, Liquid LFM2, DeepSeek V4, Ling 3.0, and the Qwen4 preview
- Android demo app: model downloader with multi-shard support, live telemetry, and every streaming setting exposed
