LiteLLM is an AI gateway that exposes more than 100 large language model (LLM) providers — OpenAI, Anthropic, Gemini, Bedrock, Azure, and others — through a single interface that follows the OpenAI request and response format. You can use it as a Python SDK inside your own code or deploy the proxy server as a central gateway for a team or organization.
The SDK gives you the completion() call with a consistent output shape regardless of provider, OpenAI-compatible exceptions, a Router that retries and falls back across multiple deployments, and observability callbacks for services such as Lunary, MLflow, and Langfuse. The proxy adds virtual keys, per-project and per-user spend tracking, guardrails, caching, load balancing, and an admin dashboard.
Beyond chat completions, the gateway routes Agent2Agent (A2A) calls to agents built on LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore, and Pydantic AI, and bridges Model Context Protocol (MCP) servers so their tools can be called from any model via /chat/completions or from IDEs such as Cursor.
Supported endpoints include /chat/completions, /responses, /messages, /embeddings, /images, /audio, /batches, /rerank, and /a2a, with per-provider coverage varying by endpoint. Published Terraform modules deploy the componentized stack (gateway, backend, UI, managed Postgres, Redis, object store) on AWS (ECS Fargate, Aurora, ElastiCache) or GCP (Cloud Run, Cloud SQL, Memorystore), and Docker images are signed with cosign.
Features
- Unified API: one OpenAI-format interface across 100+ providers, so switching providers does not require rewriting call sites
- Python SDK: direct library use with a Router for retry and fallback across deployments, application-level load balancing, and cost tracking
- AI gateway (proxy): a self-hosted service with authentication, virtual keys, multi-tenant spend management, per-project logging, guardrails, and caching
- Admin dashboard: a UI for monitoring usage and managing keys, models, and settings
- A2A agent gateway: register agents and call them through the proxy with the A2A SDK, using virtual keys for access
- MCP gateway: load MCP server tools in OpenAI tool format in the SDK, or register MCP servers on the proxy and call their tools from any model
- Endpoint coverage: chat, responses, messages, embeddings, images, audio transcription and speech, moderation, batches, and rerank
- Observability callbacks: logging integrations for Lunary, MLflow, Langfuse, and others
- Terraform deployment: registry modules for AWS and GCP that provision the gateway, backend, UI, database, cache, and secret storage
- Stable releases: Docker images tagged -stable pass a 12-hour load test before publication
- Signed images: every GHCR image is signed with cosign and verifiable against a pinned public key
Built with this
AnimaWorks
One of the hardest problems in multi-agent systems is forgetting. Most frameworks hoard context until the window explode
ContextGem
Ask an LLM which clause a finding came from and it writes you a citation that reads right and points nowhere. ContextGem
Crawl4AI
Reliable, large-scale web extraction, now built to be drastically more cost-effective. than any of the existing solutions.
OpenKB
VectifyAI’s OpenKB compiles raw documents into an interlinked Markdown knowledge base before agents query it. PDFs, docs
Integrates with
Langfuse
Open-source platform for tracing, evaluating, and debugging LLM applications, self-hosted or cloud
LangGraph
Low-level orchestration framework for long-running stateful agents, with durable execution, interrupts, and persistent memory
MLflow
Open-source platform to trace, evaluate, monitor, and deploy LLM applications, agents, and ML models
Pydantic AI
Python agent framework from the Pydantic team for building type-safe, model-agnostic generative-AI applications and workflows
Integrated by
Agent Memory
neo4j-labs/agent-memory gives agents three memory layers in one graph: conversation histo...
BrainDB
Postgres and pgvector long-term memory for LLM agents with typed entities, graph traversal, temporal decay, and a REST API
CascadeFlow
Cascading runtime for AI agents. Optimize cost, latency, quality, and policy decisions inside the agent loop.
Dspy CLI
Been a DSPy fan for a while because it matched how I already think about LLM systems: not as prompts, but as programs yo
Alternatives
FreeLLMAPI
OpenAI-compatible proxy that stacks the free tiers of 16 LLM providers (~1.7B tokens/month) behind one /v1 endpoint — plus any custom OpenAI
GoModel
AI gateway written in Go. Lightweight unified OpenAI-compatible API for OpenAI, Anthropic, Gemini, Groq, xAI & Ollama. LiteLLM alternative w