Rapid-MLX runs open-weight models locally on Apple Silicon Macs. It serves them through MLX kernels rather than a llama.cpp or Metal shim, and exposes OpenAI- and Anthropic-compatible HTTP endpoints, so a client that accepts a custom local endpoint can point at it without an adapter. The project ships a CLI, a Python package, and a macOS desktop app that bundles the same engine.
rapid-mlx chat starts a terminal REPL with slash commands and downloads the weights on first run; rapid-mlx serve starts a local HTTP server on port 8000. The server implements /v1/chat/completions, /v1/responses, /v1/messages, /v1/embeddings, /v1/audio/*, and /v1/videos. rapid-mlx launch claude-code patches a client's local config to route to the server, and rapid-mlx agents --setup writes a config template for the other supported clients.
The base install is text-only at about 460 MB. Vision, audio, video, embeddings, and speculative decoding are opt-in extras; the video runtime needs Python 3.11 or later and ffmpeg, while core text and audio run on 3.10. Audio covers 44 aliases spanning text-to-speech, transcription, voice cloning, voice design, forced alignment, and text-to-music. Video generation runs Wan 2.1/2.2, CogVideoX-Fun, and LTX-2.3 across eight checkpoints, one clip at a time.
Twelve agent command-line clients and three Python frameworks are wire-verified against real weights each release. Five are treated as tier 1 — Claude Code, Codex CLI, Hermes, Aider, and DeepSeek Harness — and an integration script drives each of them through the same multi-step bug-fix task against a local 35B model; a failure blocks the release from tagging. Cursor is not configured for a local endpoint, as it routes bring-your-own-key requests through its own servers.
Single-request serving measurements on a 256 GB M3 Ultra, each row the median of three requests at an 8K prompt and 256-token decode with the prefix cache cleared:
| Model | Shape | Median TTFT | Prefill | Decode |
|---|---|---|---|---|
| qwen3.8-27b-4bit | 27B dense | 24.66s | 330.8 tok/s | 43.4 tok/s |
| qwen3.8-flash-next-4bit | 180B total / 6B active | 9.40s | 867.9 tok/s | 23.0 tok/s |
| glm5.3-flash-4bit | 320B total / 18B active | 22.78s | 359.6 tok/s | 27.8 tok/s |
Features
- Apple Silicon engine: pure MLX kernels with continuous batching, a radix prompt cache, and a quantized live KV cache, on M1 through M4
- Drop-in APIs: OpenAI and Anthropic wire formats on one local server, including chat completions, responses, messages, embeddings, audio, and video routes
- One-command client setup: rapid-mlx launch writes local config for Claude Code, Cline, and Continue.dev; agents --setup covers the rest
- Model catalog: 226 aliases across 170 text, 2 text-diffusion, 2 image, 8 video, and 44 audio entries, browsable with rapid-mlx models and info
- RAM-tier recommendations: the installer, desktop picker, and rapid-mlx recipe share one catalog, from lfm2.5-2.6b-4bit at 8-15 GB to qwen3.8-27b-4bit at 32 GB and above
- Speculative decoding: a verified multi-token-prediction path enables automatically on eligible builds, with --no-spec-decode as the opt-out
- Audio stack: text-to-speech, transcription with word-level timestamps, voice cloning, voice design, forced alignment, and text-to-music
- Video generation: text-to-video and image-to-video through an OpenAI-compatible videos API with job polling
- Tool-call recovery: automatic recovery for models that emit tool calls as plain text, with --tool-call-parser as an explicit override
- Benchmarking: rapid-mlx benchmark run produces reproducible local measurements that stay private unless you share a run
- Diagnostics: rapid-mlx doctor self-check, plus documented paths for slow decode, out-of-memory, and port conflicts
- Telemetry off by default: anonymous telemetry requires an explicit opt-in and excludes prompts, completions, paths, and keys
