exo connects several devices into a single AI cluster so that models too large for one machine can run locally, and so that adding devices increases speed rather than just capacity. Devices running exo discover each other on the network without manual cluster configuration, and each node exposes an API and dashboard at http://localhost:52415.
Placement is topology-aware: exo tracks each device's resources and the latency and bandwidth of the links between them, then decides how to split a model across the cluster. Both pipeline and tensor sharding are supported, with documented speedups of up to 1.8x on two devices and 3.2x on four. On macOS 26.2 and Thunderbolt 5 hardware, exo uses RDMA over Thunderbolt for lower inter-device latency, reported as 8 microseconds on that path.
Inference runs on MLX, with MLX distributed handling communication between nodes. On macOS exo uses the GPU; on Linux it currently runs on CPU, with accelerator support described as under development. Models are loaded on demand into instances that you create, wait on, and delete through the API, and custom models can be added from the Hugging Face hub.
Four API surfaces are served so existing clients work unchanged: OpenAI Chat Completions, the Claude Messages format, the OpenAI Responses format, and Ollama endpoints. A built-in dashboard manages the cluster and chats with loaded models, and exo-bench measures prefill and generation speed across placement configurations. Environment variables control model directories, offline operation, image-model support, cluster namespace isolation, and tracing.
Features
- Automatic device discovery: nodes running exo find each other with no manual cluster configuration
- Topology-aware auto parallel: placement accounts for device resources and real-time link latency and bandwidth
- Tensor and pipeline sharding: up to 1.8x speedup on two devices and 3.2x on four
- RDMA over Thunderbolt 5: low-latency transport between Macs on macOS 26.2 or later
- MLX backend: MLX for inference and MLX distributed for cross-device communication
- Four compatible APIs: OpenAI Chat Completions, Claude Messages, OpenAI Responses, and Ollama endpoints
- Custom models: load models from the Hugging Face hub as custom model cards, with remote code execution off by default
- Built-in dashboard: cluster view and chat interface served from every node
- Benchmarking: exo-bench reports prompt and generation tokens per second and peak memory per placement
- macOS app: background app for macOS 26.2 or later, with namespace-based cluster isolation
