LocalAI is a self-hosted inference server that runs language, vision, voice, image, video, and 3D models on your own hardware behind OpenAI-, Anthropic-, Ollama-, and ElevenLabs-compatible APIs. Existing clients keep working after a URL change, with the model now on your machine, and no GPU is required.

The core is a single binary; each inference engine (llama.cpp, vLLM, SGLang, transformers, whisper.cpp, diffusers, MLX, and others, more than 60 in total) is packaged as a separate OCI backend image that LocalAI pulls only when a model asks for it. Backend selection is a line in the model's config, and GPU capabilities are detected automatically. Models load from the built-in gallery, Hugging Face, the Ollama registry, a YAML config, or an OCI registry.

Beyond serving models, LocalAI includes a multi-user platform with OIDC login, API keys, per-user quotas, and role-based access; a distributed mode that scales horizontally over PostgreSQL and NATS with VRAM-aware routing and autoscaling; in-UI fine-tuning with TRL that exports to GGUF; on-the-fly quantization; and built-in agents with tool use, retrieval-augmented generation, MCP, and skills, plus a terminal agent (local-ai chat) that reads files and runs commands with approval prompts.

The project also develops a set of native C/C++/GGML engines with no Python at inference, including vllm.cpp (a C++ port of vLLM), parakeet.cpp (NeMo Parakeet ASR), several TTS ports (MOSS, Magpie, Voxtral, VibeVoice), speaker and face recognition backends, object detection (RF-DETR, LocateAnything), depth estimation, and image-to-3D reconstruction (FreeSplatter, TRELLIS.2).

Features

API compatibility: OpenAI (chat, tools, Responses, Realtime), Anthropic, Ollama, and ElevenLabs endpoints
Modalities: text generation, embeddings, reranking, vision, object detection, text-to-speech, speech-to-text, image and video generation, and speech-to-speech over WebRTC
Hardware: NVIDIA (CUDA 12/13, Jetson L4T), AMD (ROCm), Intel (oneAPI/SYCL), Apple Silicon (Metal), Vulkan, and CPU-only
Composable backends: engines install and uninstall on demand from the backend gallery as OCI images, signed with cosign
Constrained output: grammar-constrained generation and structured output
Multi-user: OIDC, API keys, per-user quotas, usage attribution, and role-based access
Distributed mode: PostgreSQL and NATS cluster with prefix-cache-aware routing, layer-split inference, and mTLS
Agents: built-in agents with tools, RAG with source citations, MCP apps and client-side tools, SSE streaming, and a community Agent Hub
Fine-tuning and quantization: in-UI TRL fine-tuning with GGUF export and an on-the-fly quantization backend
Privacy: data stays on your infrastructure, with a PII redaction tier backed by privacy-filter.cpp
Web UI: integrated React interface with canvas mode, chat history, a visual pipeline editor, and configurable branding