vLLM is a library for LLM inference and serving, originally developed in the Sky Computing Lab at UC Berkeley. It manages attention key and value memory with PagedAttention, batches incoming requests continuously, and adds chunked prefill and prefix caching to raise serving throughput. Model execution uses CUDA or HIP graphs, torch.compile-driven kernel generation, and attention kernels such as FlashAttention, FlashInfer, FlashMLA, and Triton.
Models load from Hugging Face; more than 200 architectures are supported, including decoder-only LLMs (Llama, Qwen, Gemma), mixture-of-experts models (Mixtral, DeepSeek-V3, GPT-OSS), hybrid attention and state-space models (Mamba), multimodal models (LLaVA, Qwen-VL, Pixtral), embedding and retrieval models, and reward and classification models. An OpenAI-compatible API server is included, alongside Anthropic Messages API and gRPC support.
| Area | Support |
|---|---|
| Quantization | FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, compressed-tensors, ModelOpt, TorchAO |
| Parallelism | Tensor, pipeline, data, expert, and context parallelism |
| Speculative decoding | n-gram, suffix, EAGLE, DFlash |
| Hardware | NVIDIA, AMD, and Intel GPUs; x86, ARM, and PowerPC CPUs; plugins for Google TPU, Intel Gaudi, IBM Spyre, Huawei Ascend, Rebellions NPU, Apple Silicon, MetaX GPU |
Features
- PagedAttention: block-based management of attention key and value memory
- Continuous batching: incoming requests join running batches, with chunked prefill and prefix caching
- Quantization formats: FP8, MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, and more
- Speculative decoding: n-gram, suffix, EAGLE, and DFlash methods
- Distributed inference: tensor, pipeline, data, expert, and context parallelism
- Disaggregated serving: separate prefill, decode, and encode stages
- API servers: OpenAI-compatible endpoints plus Anthropic Messages API and gRPC
- Structured outputs: constrained generation with xgrammar or guidance, and tool-calling and reasoning parsers
- Decoding algorithms: parallel sampling, beam search, and streaming outputs
- Multi-LoRA: serves multiple LoRA adapters for dense and MoE layers
- Hardware coverage: NVIDIA, AMD, and Intel GPUs, CPUs, and plugin backends for TPUs, Gaudi, Ascend, and others
