mlx-serve is a native Zig inference server for Apple Silicon that runs MLX-format models and, through an embedded llama.cpp, any GGUF model, with no Python at runtime. It exposes OpenAI-compatible, Anthropic-compatible, and Ollama-compatible HTTP APIs on one port, so clients written for any of those wires connect to http://localhost:11234 unchanged. It needs macOS 26.2 or later on Apple Silicon.
Beyond text, the same server generates images, video, music, speech with voice cloning, and 3D models natively on MLX, reachable as the /v1/images, /v1/audio, /v1/video, and /v1/3d endpoints or requested inline in a chat. Defaults and memory requirements per modality:
| Modality | Default model | Approx. RAM |
|---|---|---|
| Image | FLUX.2-klein 4B 4-bit | 8 GB |
| Video | LTX-Video 2.5 4-bit | 24 GB |
| Speech | Qwen3-TTS 1.7b, with voice cloning | 8 GB |
| Music | ACE-Step 1.5 XL Turbo 8-bit | 8 GB |
| 3D | Hunyuan3D-2.1 8-bit, shape plus PBR texture | 16 GB |
MLX Core, a signed macOS menu-bar app, bundles the same server binary with a full interface: a model browser with resumable downloads, multi-session chat with PDFs and images, an agent mode with ten built-in tools and per-tool approvals, an MCP client and marketplace, folder retrieval-augmented generation, persistent memory, named agents with their own model, voice, and workspace, a hands-free voice mode, and a settings window covering every server flag. Agent shell commands can be confined to an isolated Linux virtual machine.
On an Apple M4 Max loading identical MLX weights, the project reports geometric-mean decode throughput 26% above LM Studio and 25% above oMLX, with prefill 36% and 10% higher. Speculative decoding comes in four forms — prompt-lookup decoding, model-shipped draft companions, a Gemma 4 drafter, and native Qwen multi-token prediction — alongside continuous batching, KV-cache quantisation, and prefix and tokenize caches.
Features
- Model coverage: native MLX dispatch for Gemma 3 and 4, Qwen 3 through 3.8, Llama 3.x, Mistral, DeepSeek V4 Flash, Hunyuan 3, and embedding models, with everything else auto-routed to GGUF
- Four API surfaces: OpenAI chat completions and Responses with a WebSocket transport, Anthropic Messages, and the Ollama API on one port
- Serving features: streaming, tool calling with schema-driven auto-repair, JSON-schema constrained decoding, logprobs, vision, and reasoning content
- Command line: a run command downloads and chats, pull fetches, list shows local models, and serve loads them on demand by name
- Coding-agent launchers: one-click or command-line setup for Claude Code, pi, oh-my-pi, OpenCode, Codex, aider, and Zed
- Media generation: image, video, music, speech with voice cloning, and 3D from the same server
- LAN model sharing: another Mac's models are reachable over Bonjour with no configuration
- Web console: chat playground, live monitor, media tools, and API reference in the browser
- Agent sandbox: agent shell commands run in an isolated Linux virtual machine that boots in under a second
- Single binary: a roughly 7 MB Zig binary with no Python at runtime
- Speculative decoding: four greedy-equivalent modes with adaptive gates
- macOS app: MLX Core wraps the server with model management, chat, agents, and settings
