LiveKit Agents is a framework for building realtime, programmable participants that run on servers: voice and multimodal agents that join LiveKit rooms alongside human users and can see, hear, and respond. Typical uses include multimodal assistants, telehealth consultations, inbound and outbound call centers, realtime translation, game characters, and robotics.
An agent is an LLM-based application with instructions and function tools. An AgentSession holds agents and manages the interaction with end users, mixing any combination of speech-to-text (STT), LLM, text-to-speech (TTS), or a realtime speech API through provider plugins. An AgentServer coordinates job scheduling and dispatches agents to user sessions. Agents can hand off to one another with shared state, for example an intake agent passing a conversation to a specialist.
Models run either through provider keys directly (plugins for OpenAI, Google, Deepgram, Cartesia, and others) or through LiveKit Inference, a unified API on LiveKit Cloud that hosts models such as Gemma without separate API keys. Agents connect to LiveKit Cloud or a self-hosted LiveKit server, and clients reach them through LiveKit's WebRTC SDKs or its telephony stack. The Python library has a JS/TS counterpart, AgentsJS, and LiveKit Agent Builder prototypes voice agents in the browser without code.
A terminal console mode runs an agent with local audio input and output, a development mode adds hot reloading, and a production mode runs the agent server for deployment to LiveKit Cloud or any custom environment.
Features
- Provider plugins: mix and match STT, LLM, TTS, and realtime APIs per agent
- Job scheduling: built-in task distribution with dispatch APIs that connect end users to agents
- Multi-agent handoff: agents transfer a conversation to one another with shared user data
- Semantic turn detection: a transformer model detects when a user has finished speaking, which reduces interruptions
- Telephony: agents make and receive phone calls through LiveKit's telephony stack
- Client data exchange: RPCs and data APIs pass structured data between agents and client apps
- MCP support: tools from MCP servers attach to an agent in one line of code
- Test framework: native pytest integration with event expectations and LLM judges for agent behavior
- Video avatars: add an AI avatar through providers such as Tavus, Bithuman, and LemonSlice
- Self-hostable stack: the whole stack, including LiveKit server, runs on your own servers