Dynamo is an inference orchestration layer that turns a cluster of GPUs into one coordinated serving system. It sits above existing inference engines (SGLang, TensorRT-LLM, and vLLM) rather than replacing them, and coordinates their workers across multiple GPUs and nodes for LLM, reasoning, multimodal, and video generation workloads. The core is written in Rust with Python for extensibility.
It targets deployments that span more than one GPU or node: separating prefill and decode into independently scaled GPU pools, routing requests by KV cache overlap to avoid recomputing prefill, autoscaling to meet latency targets at the lowest cost, and starting new replicas quickly. For a single model on a single GPU, an inference engine on its own is usually enough.
Dynamo exposes an OpenAI-compatible API, either through its own Frontend and Router or through a Kubernetes Gateway API Inference Extension gateway. It deploys as containers, a Python package, or on Kubernetes, where a zero-config request (in beta) takes a model, hardware, and latency targets and works out the topology. Pre-built recipes cover models such as Qwen3, DeepSeek-R1, and Kimi, and agentic workloads can pass per-request hints for priority, expected output length, and speculative prefill, with LangChain and NeMo Agent Toolkit integrations.
| Result | Context |
|---|---|
| 7× higher throughput per GPU | DeepSeek R1 on GB200 NVL72 with Dynamo vs B200 without |
| 7× faster model startup | ModelExpress weight streaming, DeepSeek-V3 on H200 |
| 2× faster time to first token | KV-aware routing, Qwen3-Coder 480B |
| 80% fewer SLA breaches | Planner autoscaling at 5% lower total cost of ownership |
Features
- Disaggregated serving: prefill and decode run in separate GPU pools that scale independently
- KV-aware routing: routes requests by worker load and KV cache overlap to skip redundant prefill
- KV Block Manager: offloads KV cache from GPU to CPU, SSD, and remote storage such as S3 or Azure blob
- ModelExpress: streams model weights GPU-to-GPU over NIXL and NVLink for faster replica cold starts
- Planner: an SLA-driven autoscaler that profiles workloads and right-sizes GPU pools
- Grove: a Kubernetes operator for topology-aware gang scheduling across racks, hosts, and NUMA nodes
- AISimulate: predicts serving behavior and searches deployment configurations offline
- Fault tolerance: canary health checks and in-flight request migration when a worker fails
- Multimodal and video: disaggregated encode, prefill, and decode for images, plus FastVideo and SGLang Diffusion video generation
- Agentic inference: per-request priority, output-length, and speculative-prefill hints with session metadata for tracing
- Agent skills: bundled skills let coding agents deploy, troubleshoot, benchmark, and tune a Dynamo deployment
