llama.cpp runs large language model and vision-language model inference in plain C/C++ with minimal setup, locally or in the cloud. It is built on the ggml library and has no external dependencies, and it loads models in the GGUF format, including ones pulled directly from Hugging Face. Integer quantization from 1.5-bit up to 8-bit lowers memory use and speeds up inference.
It targets a wide range of hardware. Apple Silicon is a first-class target through ARM NEON, Accelerate, and Metal, x86 builds use AVX, AVX2, AVX512, and AMX, and there are backends for NVIDIA, AMD, Intel, and other GPUs through CUDA, HIP, SYCL, Vulkan, OpenCL, and WebGPU, among others. CPU and GPU hybrid inference can partially accelerate a model larger than total VRAM.
Alongside the library, it ships a command-line tool, a completion tool, and a server that exposes an OpenAI-compatible REST API, so coding agents, editors, and scripts that speak the OpenAI API connect to a local model. The server also has a built-in web UI for chatting with a model in the browser. GBNF grammars constrain generated output to a defined structure.
Features
- C/C++ inference: LLM and VLM inference with no dependencies, built on the ggml library
- GGUF models: loads GGUF-format models, including ones pulled directly from Hugging Face
- Quantization: 1.5-bit through 8-bit integer quantization to cut memory use and speed inference
- Wide hardware support: backends for Apple Silicon (Metal), NVIDIA (CUDA), AMD (HIP), and Intel (SYCL) GPUs, plus Vulkan, OpenCL, and WebGPU
- Hybrid inference: splits a model across CPU and GPU to run models larger than total VRAM
- OpenAI-compatible server: a REST server that coding agents, editors, and scripts reach as if it were the OpenAI API
- Built-in web UI: chat with a model in the browser against the local server
- GBNF grammars: constrain generated output to a defined structure
