LiteLLM

LiteLLM gives applications one interface to call more than 100 LLM providers in the OpenAI request and response format, so switching providers does not mean rewriting code or learning a new SDK. It ships in two forms: a Python SDK that you import directly, and a self-hosted AI Gateway (proxy server) that a platform team runs as a central service for an organization.

The Python SDK exposes the same completion() interface for every provider, returns a consistent output format, and maps provider errors to OpenAI-compatible exceptions. Its Router adds retries and fallbacks across multiple deployments, application-level load balancing, and cost tracking, and observability callbacks send traces to tools such as Lunary, MLflow, and Langfuse.

The AI Gateway adds authentication and authorization, virtual keys, per-project and per-user spend tracking, per-project logging, guardrails and caching, and an admin dashboard. The gateway reports 8 ms P95 latency at 1,000 requests per second. It runs as a container, with Helm charts and Terraform modules for AWS (ECS Fargate, Aurora, ElastiCache) and Google Cloud (Cloud Run, Cloud SQL, Memorystore).

Beyond chat completions, supported endpoints include responses, messages, embeddings, images, audio, batches, rerank, and A2A. The gateway also acts as an MCP gateway that exposes registered MCP servers to clients such as Cursor and lets models call MCP tools through the chat completions endpoint, and as an A2A gateway for registered agents. An agents API runs Claude Code, Codex, OpenCode, or Deep Agents against any model, with every model call routed through the gateway.

Supported providers include OpenAI, Anthropic, Google Gemini and Vertex AI, AWS Bedrock and SageMaker, Azure, Mistral, Cohere, DeepSeek, xAI, Groq, Together AI, Fireworks AI, Hugging Face, IBM watsonx, Nvidia NIM, Moonshot, Meta's Llama API, and local runtimes such as Ollama, vLLM, LM Studio, and Llamafile.