vLLM is a library for LLM inference and serving, originally developed in the Sky Computing Lab at UC Berkeley. It manages attention key and value memory with PagedAttention, batches incoming requests continuously, and adds chunked prefill and prefix caching to raise serving throughput. Model execution uses CUDA or HIP graphs, torch.compile-driven kernel generation, and attention kernels such as FlashAttention, FlashInfer, FlashMLA, and Triton.
Models load from Hugging Face; more than 200 architectures are supported, including decoder-only LLMs (Llama, Qwen, Gemma), mixture-of-experts models (Mixtral, DeepSeek-V3, GPT-OSS), hybrid attention and state-space models (Mamba), multimodal models (LLaVA, Qwen-VL, Pixtral), embedding and retrieval models, and reward and classification models. An OpenAI-compatible API server is included, alongside Anthropic Messages API and gRPC support.
| Area | Support |
|---|---|
| Quantization | FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, compressed-tensors, ModelOpt, TorchAO |
| Parallelism | Tensor, pipeline, data, expert, and context parallelism |
| Speculative decoding | n-gram, suffix, EAGLE, DFlash |
| Hardware | NVIDIA, AMD, and Intel GPUs; x86, ARM, and PowerPC CPUs; plugins for Google TPU, Intel Gaudi, IBM Spyre, Huawei Ascend, Rebellions NPU, Apple Silicon, MetaX GPU |
Features
- PagedAttention: block-based management of attention key and value memory
- Continuous batching: incoming requests join running batches, with chunked prefill and prefix caching
- Quantization formats: FP8, MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, and more
- Speculative decoding: n-gram, suffix, EAGLE, and DFlash methods
- Distributed inference: tensor, pipeline, data, expert, and context parallelism
- Disaggregated serving: separate prefill, decode, and encode stages
- API servers: OpenAI-compatible endpoints plus Anthropic Messages API and gRPC
- Structured outputs: constrained generation with xgrammar or guidance, and tool-calling and reasoning parsers
- Decoding algorithms: parallel sampling, beam search, and streaming outputs
- Multi-LoRA: serves multiple LoRA adapters for dense and MoE layers
- Hardware coverage: NVIDIA, AMD, and Intel GPUs, CPUs, and plugin backends for TPUs, Gaudi, Ascend, and others
Built with this
Integrated by
BetterWright
A persistent, policy-guarded Playwright browser for AI agents — network policy, encrypted credential vault, proof screenshots, and CAPTCHA s
BrainDB
Postgres and pgvector long-term memory for LLM agents with typed entities, graph traversal, temporal decay, and a REST API
CascadeFlow
Cascading runtime for AI agents. Optimize cost, latency, quality, and policy decisions inside the agent loop.
CyberStrike
Terminal AI agent for authorized offensive security that drives recon, exploitation, and reporting across 15+ LLM providers
