LocalAI is a self-hosted inference engine that runs LLMs, vision, voice, image, and video models on local hardware behind one API. It exposes drop-in OpenAI-, Anthropic-, and ElevenLabs-compatible APIs (plus an Ollama-compatible API), so clients built for those services can point at it instead. A GPU is optional: it runs on NVIDIA, AMD, Intel, Apple Silicon, Vulkan, NVIDIA Jetson, or CPU only.
The core is small and each backend ships as its own OCI image, pulled only when a model needs it. More than 60 backends wrap engines such as llama.cpp, vLLM, SGLang, transformers, whisper.cpp, diffusers, and MLX, and the project maintains its own native C/C++ backends for text generation (vllm.cpp), speech recognition, text-to-speech, speaker and face recognition, object detection, depth estimation, 3D reconstruction, and PII filtering. Models load from a model gallery, Hugging Face, the Ollama registry, YAML configs, or any OCI registry.
Beyond serving, LocalAI includes built-in agents with tool use, RAG, MCP, skills, and an Agent Hub, a terminal agent that reads files and runs commands with approval, and a WebUI. It also handles multi-user deployments with API keys, OIDC, quotas, and per-user usage metrics, and it scales out through a distributed mode with VRAM-aware routing and autoscaling.
Features
- API compatibility: OpenAI, Anthropic, ElevenLabs, and Ollama APIs across every backend, including an OpenAI-compatible tools API
- Modalities: text generation, embeddings, reranking, vision, object detection, image and video generation, text-to-speech, and speech-to-text
- Realtime API: speech-to-speech conversations with tool calling over WebRTC
- Backend gallery: install or remove backends on the fly as OCI images, with automatic GPU detection
- Built-in agents: autonomous agents with tool use, RAG, MCP, skills, SSE streaming, and the Agent Hub community hub
- Terminal agent: a CLI chat agent that answers questions, reads files, and runs commands after approval
- Multi-user platform: API key and OIDC authentication, role-based access, per-user quotas, and usage metrics
- Distributed mode: horizontal scaling with PostgreSQL and NATS, VRAM-aware routing, autoscaling, and P2P inference
- Fine-tuning and quantization: in-UI fine-tuning with TRL, export to GGUF, and on-the-fly quantization
- Constrained grammars: grammar-constrained output for structured generation
- Privacy features: NER-based PII filtering and redaction
