llm-d is a Kubernetes-native stack for distributed large language model inference. Model servers such as vLLM and SGLang run a model on accelerators; llm-d adds the orchestration above them (request routing, KV-cache management, disaggregated serving, flow control, and autoscaling) to turn single-node engines into cluster-wide serving. It is a Cloud Native Computing Foundation (CNCF) sandbox project founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA.
The project is organised around well-lit paths: tested deployment recipes with Helm charts and kustomize overlays for common production patterns. These include an optimized baseline, routing by predicted latency, precise prefix-cache-aware routing, a tiered prefix cache, prefill/decode disaggregation, wide expert parallelism for mixture-of-experts models, flow control and fairness, and autoscaling of an inference pool. The same patterns apply across GPUs, TPUs, XPUs, CPUs, and NPUs.
Results reported from production deployments and partner benchmarks:
| Reported gain | Technique | Setup |
|---|---|---|
| 3x output throughput, 2x faster TTFT | Prefix-cache-aware routing vs round-robin | Llama 3.1 70B on 4x AMD MI300X |
| 40% lower TTFT and ITL | Predicted-latency scheduling vs heuristics | NVIDIA GPUs (Google) |
| Up to 70% more tokens/sec | Prefill/decode disaggregation vs standard vLLM | GPT-OSS on NVIDIA B200 (AWS) |
| 10–30% throughput | Disaggregated serving on identical hardware | GPT-OSS-120B and Llama 3.3 70B on AMD MI300X (Oracle) |
| 50k tokens/sec cluster throughput | Wide expert parallelism | 16x16 NVIDIA B200 |
| 13.9x throughput at 250 concurrent users | Hierarchical KV offloading vs GPU-only | 4x NVIDIA H100 |
Features
- Request routing: prefix-cache-aware and load-aware balancing, with predicted-latency scheduling
- KV-cache management: tiered offloading to CPU or disk and global indexing of cache state for multi-turn workloads
- Large-model serving: prefill/decode disaggregation and wide expert parallelism over accelerator interconnects for models such as DeepSeek-R1 and GPT-OSS
- Flow control: multi-tenant fairness and SLO-aware autoscaling driven by inference signals, including scale-to-zero
- Batch processing: OpenAI-compatible Batch APIs with asynchronous processing for offline inference
- Multimodal serving: graduated to production in the 0.8 release, alongside initial reinforcement-learning support
- Accelerator portability: consistent patterns across GPUs, TPUs, XPUs, CPUs, and NPUs
- High availability: active-active configurations and UCCL-based transport resilience
- Reproducible benchmarks: benchmark workflows and published results on Prism
