Shepherd Model Gateway (SMG) is a Rust inference gateway that sits between applications and a fleet of LLM workers. It exposes one endpoint, manages worker lifecycles, and routes requests across self-hosted engines and cloud providers, with multi-tenancy controls, chat-history storage, MCP tool execution, and observability built into the gateway layer. It installs as a Docker image, a Helm chart, a PyPI package, or a Cargo crate, and smg launch --worker-urls ... starts it against a set of workers.
Cache-aware routing tracks each worker's KV-cache state in radix trees so requests with shared prefixes land on workers that already hold them, with load modeling that accounts for queued token work and KV pressure. A native streaming gRPC pipeline to the engines supports prefill/decode disaggregation, a separate encode stage for vision inputs, and data-parallel-aware routing. Multi-node deployments form a mesh with SWIM gossip and CRDT-replicated state.
Features
- Self-hosted engines: vLLM, SGLang, TokenSpeed, TensorRT-LLM, MLX on Apple Silicon, and any OpenAI-compatible server such as Ollama
- Cloud providers: OpenAI, Anthropic, Google Gemini, xAI, OCI Generative AI, AWS Bedrock, Azure OpenAI, and OpenAI-compatible providers such as Groq and Together
- API coverage: OpenAI Chat Completions, Completions, Embeddings, Rerank, Classify, Responses, and Conversations; Anthropic Messages; Gemini Interactions; Realtime over WebSocket and WebRTC; audio transcription; and tokenize/detokenize
- Routing policies: cacheaware, leastload, poweroftwo, consistenthashing, prefixhash, bucket, round_robin, random, manual, and passthrough
- Model parsers: 21 tool-call parsers and 16 reasoning parsers with automatic detection for DeepSeek, Qwen, Kimi, GLM, Llama, Mistral, Command, Nemotron, and others
- MCP integration: tool discovery and execution over stdio, SSE, and streamable HTTP, with approval policies and audit logging in the Responses and Messages APIs
- Kubernetes discovery: pod watchers with label selectors, per-role prefill/decode/encode selectors, and router peer discovery
- Chat history: pluggable storage with schema migrations for PostgreSQL, Oracle, Redis, or in-memory
- Access control: priority admission scheduling with preemption, per-tenant controls, API-key auth, and OIDC on the control plane
- WASM plugins: custom WebAssembly middleware for request and response handling
- Resilience: circuit breakers, retries with backoff and jitter, and rate limiting
- Observability: 90+ Prometheus metrics, OpenTelemetry tracing with W3C trace context propagated into engines, and structured JSON logs with request correlation
