Pipecat is a Python framework for building real-time voice and multimodal conversational agents. An agent is a pipeline of processors that connects a transport (WebRTC or WebSockets), speech-to-text, a language model, text-to-speech, and conversation logic, and the framework handles the streaming, interruption, and coordination between them so that the agent responds with low latency.
Each pipeline is one agent, and several agents can be composed into a multi-agent system: specialists hand off to each other, fan out in parallel, or run as sidecar workers over a shared bus, either in one process or distributed across processes and machines. Pipecat Flows, built into the framework, adds predefined or dynamic conversation paths with state management for structured dialogs such as customer intake and guided support flows.
The service layer is pluggable. Transports include Daily, LiveKit, SmallWebRTCTransport, Vonage, a FastAPI WebSocket server, WhatsApp, and local audio, with serializers for telephony providers (Twilio, Telnyx, Plivo, Exotel, Genesys, Vonage). Language model services cover Anthropic, OpenAI (including the Responses API), Gemini, Grok, DeepSeek, Mistral, Qwen, Ollama, OpenRouter, Groq, Cerebras, Fireworks AI, Together AI, NVIDIA NIM, AWS, Azure, and others, and speech-to-speech services include OpenAI Realtime, Gemini Multimodal Live, AWS Nova Sonic, Azure Voice Live, Grok Voice Agent, and Ultravox. Further categories cover speech-to-text, text-to-speech, video avatars (HeyGen, Tavus, Simli, LemonSlice), memory (Mem0), vision and image generation (fal, Google Imagen, Moondream), audio processing (Silero VAD, Krisp, Koala, RNNoise), and OpenTelemetry and Sentry for metrics.
Around the core library sit client SDKs for JavaScript, React, React Native, Swift, Kotlin, C++, and ESP32, a shadcn component registry (Pipecat UI), a CLI that scaffolds a project for an AI coding assistant (Claude Code or Codex) to build and then monitors and deploys the agent, Whisker (a real-time pipeline debugger), Tail (a terminal dashboard), and Claude Code skills for scaffolding and deploying to Pipecat Cloud. Agents run on a local machine first and move to the cloud when ready.
Features
- Composable pipelines: an agent is built from modular processors for transport, speech recognition, language model, speech synthesis, and conversation handling
- Multi-agent systems: pipelines hand off to each other, fan out in parallel, or run as sidecar workers over a shared bus, locally or distributed across machines
- Real-time transports: WebRTC through Daily, LiveKit, Vonage, or SmallWebRTCTransport, plus WebSocket servers, WhatsApp, and local audio
- Telephony serializers: Twilio, Telnyx, Plivo, Exotel, Genesys, and Vonage
- Language model services: Anthropic, OpenAI, Gemini, Grok, DeepSeek, Mistral, Qwen, Ollama, OpenRouter, Groq, Cerebras, Fireworks AI, Together AI, NVIDIA NIM, AWS, Azure, Baseten, Perplexity, SambaNova, Sarvam, Novita, Nebius, Crusoe, and Inception
- Speech-to-speech models: OpenAI Realtime, Gemini Multimodal Live, AWS Nova Sonic, Azure Voice Live, Grok Voice Agent, and Ultravox
- Speech services: more than twenty speech-to-text providers (Deepgram, AssemblyAI, Whisper, Soniox, Speechmatics, and others) and thirty text-to-speech providers (Cartesia, ElevenLabs, Rime, Kokoro, Piper, and others)
- Structured conversations: Pipecat Flows defines predefined or dynamic conversation paths with state management
- Video, vision, and memory: HeyGen, Tavus, Simli, and LemonSlice avatars, Moondream and Google Imagen for vision and images, and Mem0 for memory
- Audio processing and metrics: Silero VAD, Krisp Viva, Koala, ai-coustics, and RNNoise, with OpenTelemetry and Sentry for analytics
- Client SDKs: JavaScript, React, React Native, Swift, Kotlin, C++, and ESP32, plus the Pipecat UI shadcn component registry
- Tooling: a CLI that scaffolds, monitors, and deploys agents, Whisker for real-time pipeline debugging, Tail as a terminal dashboard, and Claude Code skills for Pipecat Cloud deployment
