NVIDIA NeMo Switchyard is a library that decides which model handles each request an AI agent makes. It pairs efficient models with more capable ones so you can trade off task success, cost, and latency on your own workload. You choose the model pool, Switchyard supplies the routing decision, and your gateway or application owns the surrounding service.
A routing algorithm examines the request or the agent's recent tool activity, and some algorithms call a model to judge the task. The gateway or application then sends the request to the selected model, and routing can change as the agent continues its work. Evaluate the whole agent, model pool, and routing configuration against a single-model baseline, since a cheaper model call does not guarantee a cheaper completed task.
You use Switchyard through a gateway (hosted on OpenRouter, or as plugins for LiteLLM and NeMo Relay), through a standalone OpenAI- and Anthropic-compatible proxy for demos and evaluation, or embedded as a library in a Rust or Python harness. Guides cover the Pi and Oh My Pi agents. The software is pre-1.0, and APIs, configuration, and routing behavior can change between releases.
| Routing choice | How it chooses |
|---|---|
| Auto | The default preset: Execution routing, efficient-first, with a confidence threshold and no classifier call |
| Task | A model judges whether the efficient model can handle the task |
| Execution | Uses recent tool activity and outcome signals to choose a model as the agent runs |
| Composite | A classifier sets the default model tier when execution signals are uncertain |
Features
- Per-request routing: picks a model from your pool for each agent request, balancing task success, cost, and latency
- Routing algorithms: Auto, Task, Execution, and Composite, plus plan/execute, escalation, advisor, sub-agent, and random strategies
- Gateway integrations: hosted on OpenRouter, with plugins for LiteLLM (experimental) and NeMo Relay
- Local proxy: a demo server compatible with the OpenAI and Anthropic APIs, including a local Codex service
- Embeddable library: routing inside your own Rust application or Python harness, where you own model calls, credentials, and retries
- Agent guides: integration guides for Pi and Oh My Pi
- Benchmarking: tooling to run your own baseline and routing comparisons, plus soak tests for latency and routing overhead
