oMLX

oMLX is an LLM inference server for Apple Silicon Macs built on MLX and mlx-lm, packaged as a native SwiftUI menu-bar app (also available through Homebrew or from source) that starts, stops, and monitors the server. One process serves text LLMs, vision-language models, OCR models, embeddings, and rerankers, and exposes OpenAI- and Anthropic-compatible endpoints on localhost:8000 alongside a built-in chat UI. It requires macOS 15 or later, Python 3.11 to 3.13, and an M-series chip.

Concurrent requests are handled by continuous batching through the mlx-lm BatchGenerator. The KV cache is block-based with prefix sharing and copy-on-write, in the manner of vLLM, and operates in two tiers: a hot tier in RAM and a cold tier on SSD in safetensors format. When the hot cache fills, blocks are offloaded to disk and restored on a later request with a matching prefix, including after a server restart, so past context stays reusable when a conversation changes mid-way.

Multiple models load in the same server with LRU eviction when memory runs low, manual load and unload, pinning, per-model idle timeouts, and a total memory limit (default: system RAM minus 8 GB). Per-model settings cover sampling parameters, chat template kwargs, aliases, type overrides, and named profiles that can be exposed as : ids on the same engine. A Claude Code optimization scales reported token counts so auto-compaction triggers at the right time on smaller-context models and keeps SSE connections alive during long prefill.

The admin dashboard at /admin provides monitoring, model management, chat, a prefill and generation benchmark, a Hugging Face model downloader, and one-click integration setup for OpenClaw, OpenCode, Codex, Hermes, Copilot, and Pi, in eight interface languages with vendored assets for offline use. Source builds can also split one model across several Macs over Ring or Thunderbolt RDMA as an experimental feature.

Type	Models
LLM	Any model supported by mlx-lm
VLM	Qwen3.5 series, GLM-4V, Pixtral, and other mlx-vlm models
OCR	DeepSeek-OCR, DOTS-OCR, GLM-OCR
Embedding	BERT, BGE-M3, ModernBERT
Reranker	ModernBERT, XLM-RoBERTa

Features

• API endpoints: /v1/chat/completions, /v1/completions, /v1/messages (Anthropic Messages API), /v1/embeddings, /v1/rerank, and /v1/models, with streaming usage stats and adaptive thinking
• Tiered KV cache: hot RAM tier plus SSD cold tier with prefix restoration across restarts
• Continuous batching: configurable maximum concurrent requests (default 8)
• Multi-model serving: LRU eviction, pinning, per-model TTL, manual load and unload, process memory enforcement
• Tool calling: all mlx-lm function-calling formats, JSON schema validation, and MCP tool integration, with auto-detected parsers for Llama, Qwen, DeepSeek, Gemma, GLM, MiniMax, Mistral, Kimi K2, and Longcat
• Vision and OCR: multi-image chat, base64, URL, and file inputs, tool calling with vision context, auto-detected OCR models
• Admin dashboard: monitoring, chat, benchmark, model downloader, per-model settings, integrations
• Menu-bar app: native Swift/SwiftUI with persistent serving stats, auto-restart on crash, and auto-update
• Native custom kernels: optional Metal kernels for GLM-5.2, MiniMax M3, and Qwen3.5 families
• Multi-Mac inference: experimental pipeline sharding across unequal-memory Macs