Switchyard

Switchyard is a Rust proxy and library for LLM traffic. It sits between a client and one or more model backends, translating between the OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages formats so a coding agent keeps speaking its native API while the request is served by vLLM, NVIDIA NIM, Ollama, or any OpenAI-compatible endpoint.

The same proxy chooses which backend handles a request. Routing algorithms are typed and composable, and a route can also spread traffic across several models for A/B benchmarking or run an algorithm written by the operator. Prometheus metrics cover requests, errors, latency, tokens, and routing overhead.

There are two ways to use it. The standalone server reads a routes.toml configuration and exposes an HTTP proxy. The switchyard-libsy crate embeds the routing algorithms in a Rust application instead: it never calls a model itself, but decides which target to use and hands each model call back to the caller, so it drops into an existing proxy, gateway, or agent runtime without owning an HTTP stack. A companion plugin runs Switchyard-configured routes inside NeMo Relay.

The project describes itself as pre-alpha, with the API and algorithms expected to change before v1.0. Component maturity is stated per crate: libsy at beta, switchyard-llm-client and switchyard-runner at alpha, and switchyard-server as a demo server rather than a production one.

Routing strategy	Use it when	Route type
LLM Classifier	Request content should decide whether a turn needs the weak or strong tier	llm_classifier
Stage Router	Signals already in the conversation, such as tool results and errors, should route most turns without an extra model call	stage_router
Escalation Router	Every turn runs on the weak tier first and a judge decides whether to resend it to the strong tier	llm_classifier with mode = "escalation"
Composite	One algorithm sets the configuration of another before handing off	composite
Random	A fixed traffic split is needed for A/B tests, baselines, or cost experiments	random
Passthrough	One target is registered under one model ID with no routing decision	passthrough

Features

• Protocol translation: converts between OpenAI Chat, Anthropic Messages, and OpenAI Responses formats
• Multi-backend routing: random, LLM-as-classifier, signal-driven stage routing, or a custom algorithm
• Operational metrics: Prometheus counters for requests, errors, latency, tokens, and routing overhead
• Embeddable library: switchyard-libsy decides the target and returns each model call to the host application
• NeMo Relay plugin: runs Switchyard routes inside NeMo Relay while Switchyard owns provider HTTP dispatch
• Configuration validation: --dry-run checks a routes.toml file before the server starts