LMCache is a KV cache management layer for LLM serving. It stores the key-value cache that an inference engine computes for a prompt in tiers outside GPU memory (CPU RAM, local disk, and remote stores) so that reusable text is not recomputed, which lowers time-to-first-token and GPU load for long-context and multi-round workloads.
It runs as a companion to engines such as vLLM: the engine points at an LMCache configuration file, and cached KV entries are then shared across requests, sessions, and engine instances. In a multi-instance deployment, an LMCache backend server holds the cache so that a second vLLM engine serving the same long context answers with lower response delay.
Beyond exact prefix reuse, CacheBlend allows KV reuse for text at any position in a prompt, not only the prefix. The library also supports disaggregated prefill (separate prefill and decode workers), peer-to-peer KV sharing across nodes, and a serialisation interface for cache compression and quantisation.
Features
- Tiered KV offloading: CPU RAM, local SSD, and remote backends behind GPU memory
- Storage backends: Redis and Valkey, S3-compatible storage, NIXL, Mooncake, Aerospike, Bigtable, and custom plugins
- Prefix caching: reuse of computed prefixes across requests and engine instances
- CacheBlend: non-prefix KV reuse at any prompt position
- Disaggregated prefill: separate prefill and decode workers exchanging KV state
- P2P sharing: KV cache exchange between nodes in a cluster
- Compression and quantisation: a SERDE interface for compact cache storage
- Hardware: NVIDIA CUDA and AMD ROCm GPUs
- Operations: metrics, logging, tracing, and Kubernetes deployment with an operator
