oMLX is a local LLM inference server for Apple Silicon Macs, built on MLX and mlx-lm and managed from a native macOS menu bar app. It serves text LLMs, vision-language models, OCR models, embedding models, and rerankers from the same process, and exposes OpenAI-compatible and Anthropic Messages endpoints so coding agents such as Claude Code, OpenClaw, and OpenCode can run against local models.
Its main mechanism is a tiered KV cache aimed at agent workloads, where the prompt prefix changes many times per session. Cache blocks are paged with prefix sharing and copy-on-write, hot blocks stay in RAM, and cold blocks are written to SSD in safetensors format. A request with a previously seen prefix restores those blocks from disk instead of recomputing them, including after a server restart. Concurrent requests run through mlx-lm's BatchGenerator with continuous batching.
Models load side by side with LRU eviction, pinning, per-model idle timeouts, and a total memory limit. A web admin dashboard handles model downloads from Hugging Face, per-model settings and profiles, a built-in chat, benchmarks, and one-click configuration of agent tools. Source builds can also split one model across several Macs as an experimental feature.
| Endpoint | Purpose |
|---|---|
| /v1/chat/completions | Chat completions with streaming |
| /v1/completions | Text completions with streaming |
| /v1/messages | Anthropic Messages API |
| /v1/embeddings | Text embeddings |
| /v1/rerank | Document reranking |
| /v1/models | Model listing |
Features
- Tiered KV cache: hot blocks in RAM and cold blocks on SSD, restored on matching prefixes across requests and restarts
- Continuous batching: concurrent requests through mlx-lm's BatchGenerator, with a configurable concurrency limit
- Multi-model serving: LLMs, VLMs, OCR, embedding, and reranker models in one server, with LRU eviction, pinning, and per-model TTL
- API compatibility: OpenAI chat and completions endpoints plus the Anthropic Messages API, with streaming usage stats and vision inputs
- Claude Code support: reports the model's real context window for auto-compaction and keeps long prefills alive with SSE keep-alive
- Tool calling: auto-detected tool-call formats for Llama, Qwen, Gemma, GLM, MiniMax, Mistral, and Kimi K2 families, plus JSON schema output and MCP tools
- Vision and OCR: multi-image chat, video input for some Qwen checkpoints, and auto-detected OCR models such as DeepSeek-OCR and GLM-OCR
- Admin dashboard: model management, per-model settings and profiles, built-in chat, and one-click benchmarks, usable offline
- Model downloader: searches and downloads MLX models from Hugging Face inside the dashboard
- Agent integrations: one-click setup for OpenClaw, OpenCode, Codex, Hermes Agent, Copilot, Pi, and DeepSeek Harness
- Menu bar app: native SwiftUI app that starts, stops, and monitors the server, with usage history and auto-restart
- Multi-Mac inference: experimental splitting of one model across Macs over Ring or Thunderbolt links
