mistral.rs is an LLM inference engine written in Rust. One binary covers interactive chat, an API server, benchmarks, and a web UI through the run, serve, and bench commands, and it loads Hugging Face checkpoints, GGUF files, and UQFF quantizations. Architecture, weight format, and chat template are detected automatically for supported models, with flags for explicit selection.
The engine is multimodal: text, vision, video, and audio input, speech generation, image generation, and embeddings run in the same process. mistralrs serve exposes OpenAI-compatible /v1 endpoints and Anthropic-compatible /v1/messages endpoints side by side, a Prometheus /metrics endpoint, and a browser UI at /ui that shows reasoning, code execution, plots, and files inline. mistralrs tune recommends quantization and device mapping for the detected hardware, and Python and Rust SDKs wrap the same engine.
A server-side agentic loop executes tools and feeds results back automatically, with web search, persistent Python sessions with matplotlib capture, sandboxed shell sessions with approval controls, OpenAI-compatible Skills bundles, file inputs, an MCP client, a tool dispatch URL, and Python or Rust tool callbacks.
Q8 prefill tokens per second, mistral.rs UQFF q8 versus llama.cpp GGUF Q8_0 (v0.8.2, mean across prompt lengths from 128 to 16384 tokens):
| Model | Hardware | mistral.rs | llama.cpp |
|---|---|---|---|
| Gemma 4 E4B | GB10 | 7395.7 | 3973.7 |
| Gemma 4 E4B | B200 | 27705.6 | 11992.4 |
| Gemma 4 E4B | H100 SXM | 26220.6 | 11702.1 |
| Gemma 4 26B-A4B | GB10 | 2947.0 | 2178.5 |
| Gemma 4 26B-A4B | B200 | 12725.3 | 8503.4 |
| Gemma 4 26B-A4B | H100 SXM | 12362.3 | 8055.1 |
Features
- Model formats: Hugging Face checkpoints, GGUF (2 to 8 bit), and UQFF, with --quant selecting a published artifact
- Quantization: in-situ quantization (ISQ) for Hugging Face models, plus GPTQ, AWQ, HQQ, FP8, and BNB, per-layer topology, and automatic method selection per hardware
- Multimodality: text, image, video, and audio input, speech and image generation, and embeddings in one engine
- Dual API compatibility: OpenAI /v1 and Anthropic Messages endpoints from one server
- Batching and attention: continuous batching on all devices, FlashAttention V2/V3 on CUDA, PagedAttention on CUDA and Apple Silicon, prefix caching including multimodal, and multi-GPU distributed inference
- Adapters and MoE: LoRA and X-LoRA with per-request adapter selection, and AnyMoE for building mixture-of-experts on any base model
- Multiple models: load and unload models at runtime
- Agentic runtime: tool calling with grammar enforcement and strict schema mode, web search, Python and shell execution, Skills, file inputs, session management, and custom tool hooks
- MCP client: connects to external tool servers over process, HTTP, or WebSocket transports
- Observability and tuning: Prometheus metrics, mistralrs tune hardware recommendations, and mistralrs doctor diagnostics
- Block diffusion: DiffusionGemma text generation integrated with paged attention, prefix caching, ISQ, and tool calling
- SDKs: Python package and Rust crate
