Magnitude is an open-source inference engine that runs open-weight models locally and serves them to the agents you already use. It compiles and tunes its kernels on your device before a model runs, so they fit the exact chip, rather than shipping kernels precompiled for broad hardware classes.
It ships as a desktop app for macOS, Windows, and Linux that bundles a CLI. You pick a recommended model in the app, download it, and connect an agent. Prompts, files, and models stay on the machine, and no internet connection is needed once a model is downloaded.
Supported hardware is Apple Silicon, NVIDIA, and AMD GPUs, or a CPU alone, with no fixed minimum. One-click connections cover Pi, OpenCode, Hermes, OpenClaw, Codex, Claude Code, Oh My Pi, and Cline, and any other agent connects through an OpenAI-compatible API.
Published benchmarks against llama.cpp (Qwen 3.6 35B A3B, 4-bit, 64k context, no speculative decoding):
| Backend | Prefill (tok/s) | Decode (tok/s) |
|---|---|---|
| Metal, Mac M4 Pro 48 GB | 466 → 507 | 30 → 57 |
| CUDA, DGX Spark | 2,033 → 2,507 | 49 → 58 |
Features
- On-device kernel tuning: compiles and tunes kernels on your hardware before a model runs
- Hand-optimized kernels: kernels written for popular open-weight model families
- Lower memory use: uses 27% less memory per agent and frees it when agents stop
- Concurrent sessions: sessions share prefix caches so parallel agents avoid slowdown
- Agent connections: one-click setup for Pi, OpenCode, Hermes, OpenClaw, Codex, Claude Code, Oh My Pi, and Cline
- OpenAI-compatible API: any other agent connects through a standard endpoint
- Local and private: no token costs, and prompts, files, and models stay on the machine
- Hardware coverage: Apple Silicon, NVIDIA, and AMD GPUs, or CPU only
