Bifrost is an AI gateway written in Go that fronts more than 20 model providers, among them OpenAI, Anthropic, AWS Bedrock, Google Vertex, Azure, Cerebras, Cohere, Mistral, Ollama, and Groq, behind one OpenAI-compatible HTTP API. Applications point at the gateway instead of a provider, and the gateway handles provider selection, key rotation, and retries on their behalf.
You run it in one of three ways: as a standalone HTTP gateway started from npx or Docker with a built-in web UI for configuration and monitoring, as a Go SDK embedded directly in a Go service, or as a drop-in replacement where an existing OpenAI, Anthropic, or Google GenAI SDK is repointed at a gateway path such as /openai or /anthropic. Configuration can come from the web UI, an API, or a config file, and secrets can be referenced from environment variables.
Beyond routing, the gateway acts as an MCP client so models can call external tools such as filesystems, web search, and databases, and a plugin layer hosts middleware for governance, logging, semantic caching, telemetry, and mock responses. Published benchmark figures at a sustained 5,000 requests per second report 11 µs of added latency per request on a t3.xlarge instance (59 µs on a t3.medium) with no failed requests.
Features
- Unified interface: one OpenAI-compatible API in front of every configured provider, including text, image, audio, and streaming requests
- Fallbacks and load balancing: automatic failover between providers and models, with requests distributed across multiple API keys
- Semantic caching: responses are reused for requests that are semantically similar, to cut cost and latency
- MCP tool calling: models can invoke external tools through the Model Context Protocol
- Governance: virtual keys, usage tracking, rate limits, and hierarchical budgets for teams and customers
- Observability: Prometheus metrics, distributed tracing, and request logging
- Custom plugins: a middleware architecture for analytics, monitoring, and custom logic
- User provisioning: OAuth 2.0 / OIDC login with directory sync for teams and roles
- SDK integrations: OpenAI, Anthropic, Bedrock, Google GenAI, LiteLLM, and LangChain SDKs connect by changing a base URL
- Enterprise deployment: clustering, adaptive load balancing, guardrails, and an MCP gateway for private deployments
