Helicone AI Gateway is a Rust service that fronts many LLM providers behind a single OpenAI-compatible API. Your application keeps using the OpenAI SDK in any language, changes its base URL to the gateway, and names models as provider/model (for example openai/gpt-4o-mini or anthropic/claude-3-5-sonnet); the gateway holds the provider keys and forwards the request. It is available as a hosted service or self-hosted with npx @helicone/ai-gateway or Docker.
Routers are defined in a YAML config. Each router sets a load-balancing strategy over a list of models, rate limits, and cache directives, and a request targets a router by path (/router/ ). Routing strategies include model-latency, provider-latency (power-of-two-choices with PeakEWMA), weighted, and cost-based selection, with fallbacks when a provider is down or rate-limited. The hosted version provides a configuration wizard instead of hand-written YAML.
Reported figures for the gateway itself are a P95 latency under 5 ms, about 64 MB of memory, roughly 3,000 requests per second, a binary of about 30 MB, and a cold start around 100 ms. Tracing goes to Helicone's observability platform and to OpenTelemetry for logs, metrics, and traces.
Features
- Unified interface: OpenAI-style requests to OpenAI, Anthropic, Google, AWS Bedrock, Google Vertex, and 20+ providers, 100+ models
- Load balancing: model-latency, provider-latency, weighted, and cost strategies, aware of provider uptime and your rate limits
- Fallbacks: automatic failover across providers within a router
- Rate limiting: per user, team, or global, by request count, token usage, or dollar amount
- Response caching: in-memory, Redis, or S3 backends with cache directives such as max-age and max-stale
- Observability: built-in Helicone integration plus OpenTelemetry export
- Authentication: optional control-plane API key for the self-hosted gateway
- Deployment: hosted at ai-gateway.helicone.ai, or self-hosted via npx, Docker, Kubernetes, and cloud platform guides
