llm-d is a Kubernetes-native stack for distributed LLM inference. Model servers such as vLLM and SGLang run a model efficiently on an accelerator, and llm-d adds the orchestration layer above them: request routing, KV-cache management, disaggregated serving, flow control, and autoscaling across a cluster. It targets open models on GPUs, TPUs, XPUs, CPUs, and NPUs across most infrastructure providers.
The project is organized around well-lit paths: tested, benchmarked deployment recipes with Helm charts for common production patterns. They include an optimized baseline, routing by predicted latency, prefix-cache-aware routing, a tiered prefix cache, prefill/decode disaggregation, wide expert parallelism for mixture-of-experts models such as DeepSeek-R1 and GPT-OSS, flow control and fairness for multi-tenant serving, and autoscaling of an inference pool.
| Reported result | Setup |
|---|---|
| 3x output throughput, 2x faster time to first token | Prefix-cache-aware routing versus round-robin, Llama 3.1 70B on 4× AMD MI300X |
| 40% lower time to first token and inter-token latency | Predicted-latency scheduling versus heuristics on NVIDIA GPUs |
| Up to 70% more tokens per second | Prefill/decode disaggregation versus standard vLLM, GPT-OSS on NVIDIA B200 |
| 10–30% throughput gain | Disaggregated serving, GPT-OSS-120B and Llama 3.3 70B on AMD MI300X |
| 50k tokens per second cluster throughput | Wide expert parallelism on 16×16 NVIDIA B200 |
| 13.9x throughput at 250 concurrent users | Hierarchical KV offloading versus GPU-only, 4× NVIDIA H100 |
llm-d is a Cloud Native Computing Foundation sandbox project. Reproducible benchmarks are published on the project's Prism site.
Features
- Intelligent routing: prefix-cache-aware and load-aware balancing, with predicted-latency scheduling to cut latency and raise throughput
- KV-cache management: tiered offloading of the KV cache to CPU memory or disk and precise global indexing of cache state
- Prefill/decode disaggregation: prefill and decode run on separate workers to improve time to first token and throughput
- Wide expert parallelism: serves large mixture-of-experts models across many accelerators over fast interconnects
- Flow control: fairness and admission control for multi-tenant serving
- SLO-aware autoscaling: scales the inference pool from real-time inference signals, including scale to zero
- Batch processing: OpenAI-compatible Batch APIs and asynchronous processing for large offline workloads
- Cache-aware LoRA routing: routes requests to replicas that already hold the needed LoRA adapter
- Hardware portability: tested on NVIDIA and AMD GPUs, Google TPUs, Intel XPUs, CPUs, and emerging NPUs
- Well-lit paths: benchmarked deployment recipes with Helm charts for common production patterns
