LiveKit Agents is a Python framework for building programmable participants that run on a server and join realtime sessions: voice agents that hear, see, and reply, and text-only agents built with the same code. A Node.js equivalent, AgentsJS, lives in a separate repository.
An Agent holds instructions and function tools; an AgentSession manages the interaction with end users and combines any mix of speech-to-text, LLM, text-to-speech, and realtime API components; an AgentServer schedules jobs and launches agents for user sessions. Model components come either from provider plugins (for example Deepgram, OpenAI, and Cartesia) or from LiveKit Inference, a unified API for models hosted through LiveKit Cloud. Agents connect to LiveKit Cloud or a self-hosted LiveKit server, and a console mode runs an agent locally with microphone and speaker for testing.
Multi-agent workflows hand off between Agent subclasses, with per-agent model overrides and shared typed user data. A test framework runs a session against scripted user input and asserts on the resulting events, including LLM-judged checks on assistant replies.
Features
- Pluggable models: mix STT, LLM, TTS, and realtime API providers per session, or use LiveKit Inference for hosted models
- Job dispatch: built-in scheduling and dispatch APIs that connect end users to agents
- WebRTC clients: LiveKit client SDKs for browsers, mobile, and desktop platforms
- Telephony: make and receive phone calls through LiveKit's SIP stack
- Client data exchange: RPCs and data APIs between agents and client apps
- Semantic turn detection: a transformer model decides when the user has finished speaking
- MCP support: attach tools from MCP servers to an agent
- Testing: native pytest integration with event assertions and LLM judges
- Multi-agent handoff: transfer a session between agents with different instructions and models
- Video avatars: integrations with Tavus, Bithuman, LemonSlice, and others
- Coding-agent aids: a LiveKit Docs MCP server and an agent skill for AI coding assistants