Opik is an observability and evaluation platform for LLM applications and AI agents, built by Comet. It covers the application lifecycle from the first trace in development through production monitoring: trace trees for multi-step agents and tool calls, datasets and experiments, LLM-as-a-judge metrics, prompt management, online evaluation rules, guardrails, and an Agent Optimizer SDK that tests prompt variations against your metrics and datasets.
The full platform (server backend, web application, tracing, datasets, experiments, evaluations, prompt management, online evaluation, and agent optimization) can be self-hosted with Docker Compose or Kubernetes and Helm, or used as the hosted Comet.com cloud. Clients are a Python SDK, a TypeScript SDK, a REST API, and first-party OpenTelemetry support, so any language with an OpenTelemetry SDK (Java, Ruby, .NET) can send traces. Wrapping a function with the @track decorator logs it and its nested calls, which covers whole agent and pipeline traces, and the SDK runs locally or against the cloud.
Opik is framework-agnostic. Direct integrations exist for agent frameworks (Google ADK, AutoGen, AG2, Agno, CrewAI, LangChain, LangGraph, LlamaIndex, Mastra, Pydantic AI, Semantic Kernel, Smolagents, Strands Agents, OpenAI Agents SDK, Microsoft Agent Framework, Spring AI), visual builders (Dify, Flowise, Langflow, n8n), gateways (LiteLLM, OpenRouter, TrueFoundry), model providers (OpenAI, Anthropic, Gemini, Bedrock, Cohere, DeepSeek, Mistral, Ollama, Groq, Together AI, watsonx, xAI Grok), and coding agents (Claude Code sessions through a plugin, Cursor conversations, OpenClaw agent runs). An MCP server lets Claude Code, Cursor, VS Code Copilot, Codex, or OpenCode read traces, score outputs, save prompts, and run experiments from chat.
Evaluation runs through datasets and experiments, with built-in LLM-as-a-judge metrics (hallucination, moderation, answer relevance, context precision) and heuristic metrics, plus custom metrics, and a PyTest integration runs evaluations in CI on every commit. In production, dashboards track feedback scores, trace counts, token usage, latency, cost, and error rates, online evaluation rules score incoming traces automatically, and the platform is sized for 40M+ traces per day.
Positioning against the field, as stated by the project:
| Capability | Opik | LangSmith | Phoenix | Arize AX | Weights & Biases (Weave) | Langfuse | Braintrust |
|---|---|---|---|---|---|---|---|
| Self-hosted deployment | Yes | Enterprise only | Yes | Enterprise only | Enterprise only | Yes, core | Enterprise only |
| Free tier (cloud or self-hosted) | Yes, both | Yes, cloud | Yes, self-hosted | Yes, cloud | Yes, cloud | Yes, both | Yes, cloud |
| Prompt management | Yes | Yes | Partly | Partly | Partly | Yes | Yes |
| Framework-agnostic | Yes | Partly, built around LangChain | Yes | Yes | Yes | Yes | Yes |
Features
- Agent tracing: full trace trees of LLM calls, tool executions, retrieval steps, and conversation logs for multi-step agents, with feedback-score annotation from the SDK or the UI
- Evaluation: datasets and experiments, LLM-as-a-judge metrics (hallucination, moderation, answer relevance, context precision), heuristic metrics, and custom metrics
- CI/CD evaluation: a PyTest integration runs LLM evaluations on every commit
- Production monitoring: dashboards for feedback scores, trace counts, token usage, latency, cost, and error rates, with online evaluation rules scoring incoming traces, sized for 40M+ traces per day
- Agent Optimizer: six optimization algorithms test prompt variations for each step of an agent against your metrics and datasets
- Guardrails: validation features for safe-use policies, logged as traces
- Prompt playground and management: versioned prompts and a playground for trying prompts and models
- Framework integrations: Google ADK, AutoGen, AG2, Agno, CrewAI, LangChain, LangGraph, LlamaIndex, Mastra, Pydantic AI, Semantic Kernel, Smolagents, Strands Agents, OpenAI Agents SDK, Dify, Flowise, Langflow, n8n, and more
- Provider and gateway integrations: OpenAI, Anthropic, Gemini, Bedrock, Cohere, DeepSeek, Mistral, Ollama, Groq, Together AI, watsonx, xAI Grok, LiteLLM, OpenRouter, and TrueFoundry
- Coding-agent hooks: a Claude Code plugin logs sessions, a Cursor integration logs conversations, OpenClaw agent runs are traced, and an MCP server drives Opik from Claude Code, Cursor, VS Code Copilot, Codex, or OpenCode
- SDKs and OpenTelemetry: Python and TypeScript SDKs, a REST API, and OpenTelemetry ingestion for Java, Ruby, .NET, and other languages
- Deployment: hosted on Comet.com, or self-hosted with Docker Compose or Kubernetes and Helm, including the backend
