llama.cpp runs LLM and VLM inference in plain C/C++ with no dependencies, targeting minimal setup across a range of hardware, locally and in the cloud. It is built on the ggml tensor library and uses the GGUF model format, with integer quantization from 1.5-bit to 8-bit to reduce memory use.

The llama cli tool runs a model interactively (including vision sessions), and llama serve starts an OpenAI-compatible API server with a built-in web UI; both can pull models directly from Hugging Face with -hf. Apple silicon is optimized through ARM NEON, Accelerate, and Metal, x86 through AVX/AVX2/AVX512/AMX, and RISC-V through RVV and related extensions. Custom CUDA kernels serve NVIDIA GPUs, with AMD via HIP and Moore Threads via MUSA, and CPU+GPU hybrid inference offloads part of a model when it exceeds available VRAM.

Backend	Target devices
BLAS / BLIS	All
CANN	Ascend NPU
CUDA	NVIDIA GPU
HIP	AMD GPU
Hexagon (in progress)	Snapdragon
IBM zDNN	IBM Z and LinuxONE
MUSA	Moore Threads GPU
Metal	Apple silicon
OpenCL	Adreno GPU
OpenVINO (in progress)	Intel CPUs, GPUs, and NPUs
RPC	All
SYCL	Intel GPU
VirtGPU	VirtGPU APIR
Vulkan	GPU
WebGPU	All
ZenDNN	AMD CPU

Features

CLI inference: llama cli runs text and vision models from the terminal, fetching GGUF weights from Hugging Face on demand
OpenAI-compatible server: llama serve exposes a REST API plus a built-in web UI
Quantization: 1.5-, 2-, 3-, 4-, 5-, 6-, and 8-bit integer formats for lower memory use
Hybrid inference: splits a model between CPU and GPU when it exceeds VRAM
Backends: CUDA, HIP, MUSA, Metal, Vulkan, SYCL, OpenCL, WebGPU, CANN, zDNN, ZenDNN, and RPC for distributed execution
Grammar-constrained output: GBNF grammars restrict generation to a defined format
Multi-GPU: documented multi-device usage and performance tuning
Platform builds: Docker images, Android builds, and an Apple XCFramework
Coding-agent pairing: the pi-llama plugin lets the Pi coding agent discover a local llama serve instance without configuration