AgentOps is an observability and developer-tool platform for AI agents that covers building, evaluating, and monitoring from prototype to production. A Python SDK records sessions, LLM calls, tool use, and multi-agent interactions, and a dashboard shows step-by-step execution graphs, session replays, chat viewers, and summary analytics. The application (dashboard and API backend) is open source and can be self-hosted.
Instrumentation starts with an init call at program start and an end-session call at the end, which captures analytics on all LLM calls. For finer structure, decorators from agentops.sdk.decorators mark a @session root span, @agent classes, @operation or @task functions, and @workflow functions, and they nest to build a span hierarchy. All decorators record inputs and outputs, handle exceptions, and support async, generator, and custom-attributed functions.
Framework integrations are switched on by setting an API key in the environment and initializing the client. CrewAI installs with an optional AgentOps extra, AG2 (formerly AutoGen) and Camel AI pick up the key automatically, LangChain uses a LangchainCallbackHandler, and the OpenAI Agents SDK is supported in both Python and TypeScript.
Features
- Replay analytics and debugging: step-by-step agent execution graphs and session replays with metadata drill-down
- LLM cost management: spend tracked per foundation-model provider
- Decorator-based spans: @session, @agent, @operation/@task, and @workflow with nesting for hierarchy
- Framework integrations: OpenAI Agents SDK (Python and TypeScript), CrewAI, AG2, Agno, LangGraph, LangChain, Camel AI, LlamaIndex, and Microsoft
- Provider integrations: OpenAI, Anthropic (Python SDK 0.32.0+), Cohere (5.4.0+), Mistral, Groq, and Ollama, including streaming and async clients
- Self-hosting: run the dashboard and API backend on your own infrastructure
- MCP server: an AgentOps MCP server published on Smithery
