mistral.rs is an LLM inference engine written in Rust that runs text, vision, video, audio, speech generation, image generation, and embedding models across 45+ architectures in one engine. It detects a model's architecture, weight format, and chat template automatically for supported Hugging Face checkpoints and GGUF files, and a single binary covers interactive chat, an API server with a built-in web UI, and benchmarks. It is built on Hugging Face's Candle and is not affiliated with Mistral AI.
The server exposes OpenAI-compatible endpoints and Anthropic-compatible Messages endpoints from the same process, along with Prometheus metrics. Beyond plain completion, it runs a server-side agentic loop: tool calling with grammar enforcement, web search, persistent Python and shell sessions, OpenAI-compatible skills and file inputs, and an MCP client for external tools. The engine can also be embedded in-process through a Python package or a Rust crate.
Quantization is a central feature. In-situ quantization (ISQ) quantizes Hugging Face models at load time, a prebuilt UQFF format is used when available, and GGUF, GPTQ, AWQ, HQQ, FP8, and bitsandbytes formats are supported, with per-layer topology and a tuning command that recommends quantization and device mapping for the detected hardware. Performance features include continuous batching, PagedAttention with prefix caching, FlashAttention on CUDA, Metal on Apple Silicon, and multi-GPU and multi-node inference.
Published v0.8.2 benchmarks compare mean decode tokens per second against llama.cpp (Q8) and vLLM (BF16) on Gemma 4 models:
| Model | Hardware | Precision | mistral.rs | Comparator |
|---|---|---|---|---|
| Gemma 4 E4B | H100 SXM | Q8 | 223.1 | llama.cpp 183.0 |
| Gemma 4 26B-A4B | H100 SXM | Q8 | 199.8 | llama.cpp 183.9 |
| Gemma 4 E4B | H100 SXM | BF16 | 174.4 | vLLM 153.0 |
| Gemma 4 26B-A4B | H100 SXM | BF16 | 138.7 | vLLM 148.0 |
Features
- Automatic model loading: detects architecture, weight format, and chat template for Hugging Face checkpoints, GGUF files, and UQFF quantizations
- Multimodality: text, vision, video, audio, speech generation, image generation, and embeddings in one engine
- Dual API compatibility: OpenAI-compatible endpoints and Anthropic-compatible Messages endpoints, including streaming, tool use, and token counting
- Quantization: in-situ quantization, UQFF, GGUF (2 to 8 bit), GPTQ, AWQ, HQQ, FP8, and bitsandbytes, with per-layer topology
- Hardware tuning: a tune command recommends quantization and device mapping, and a doctor command diagnoses CUDA, Metal, and connectivity issues
- Agentic runtime: server-side tool loop with web search, persistent Python and shell sessions, skills, file inputs, and custom tool hooks
- MCP client: connects to external MCP tools over process, HTTP, or WebSocket transports
- Throughput: continuous batching, PagedAttention, prefix caching, FlashAttention V2 and V3, and multi-GPU or distributed inference
- Adapters: LoRA and X-LoRA with per-request selection, plus AnyMoE for building mixture-of-experts on any base model
- Web UI: built-in chat interface showing reasoning, code execution, plots, and files, with message editing that branches the session
- SDKs: in-process inference from a Python package or a Rust crate
- Observability: Prometheus metrics with per-request counts and latency
