OpenLIT is a self-hosted platform for tracing, evaluating, and improving AI agents and coding agents on OpenTelemetry. It treats an agent as a model plus a harness (tools, context, prompts, memory, hooks, guardrails, and feedback loops) and supports a loop of running the agent, observing failures in traces, evaluating them, fixing the harness, and verifying the fix.
OpenLIT does not run the agent loop itself. Python, TypeScript, and Go SDKs instrument an application in-process and export traces and metrics asynchronously over OTLP, following the OpenTelemetry GenAI semantic conventions, so no proxy sits in the request path. An OTLP receiver writes the telemetry to ClickHouse, and the OpenLIT dashboard reads it from there. Because the output is standard OTLP, the same data can also go to Grafana, Datadog, New Relic, SigNoz, or an existing OpenTelemetry Collector.
A command-line tool adds coding-agent observability: it ingests the hook events of Claude Code, Codex, and Cursor and maps them to OpenTelemetry spans without code changes, capturing prompts, LLM calls, file reads and edits, shell commands, sub-agent activity, token usage, and cost.
| Harness step | OpenLIT feature |
|---|---|
| Observe LLM calls, tool calls, MCP requests, retrieval, and agent steps | Tracing |
| Evaluate quality, safety, and cost | LLM-as-a-judge, programmatic, and human-feedback evals, plus CI gates |
| Guard the agent at runtime | Guardrails for prompt injection, sensitive topics, and topic restriction |
| Fix the harness without redeploying | Prompt Hub, Context, Rule Engine, Vault |
| Compare models and prompts | OpenGround |
| Account for cost and hardware | Cost tracking, custom pricing, GPU monitoring |
Features
- OpenTelemetry tracing: traces LLM calls, prompts and responses, tool calls, retrieval, embeddings, vector database operations, agent steps, latency, token usage, and errors
- Coding-agent sessions: a CLI maps Claude Code, Codex, and Cursor hook events to OpenTelemetry spans and shows them in a Coding Agents dashboard
- Cost tracking: breaks down spend by model, provider, user, session, agent, and environment, with custom pricing for fine-tuned models
- Evals: LLM-as-a-judge checks for hallucination, bias, toxicity, safety, instruction following, faithfulness, and more, run on production traces or offline as CI gates
- Guardrails: detects and blocks prompt injection, jailbreak attempts, and sensitive or off-topic prompts, with each check recorded in the trace
- Prompt Hub and Context: versioned prompts and reusable RAG content kept outside application code and fetched at runtime
- Rule Engine: conditional rules on trace attributes choose which prompts, evaluations, and context apply at runtime
- Vault: stores LLM API keys and other secrets for retrieval through the SDK or API
- OpenGround: runs one prompt across several providers to compare cost, latency, and output
- GPU monitoring: collects utilization, memory, power, and temperature for NVIDIA, AMD, and Intel GPUs and correlates them with traces
- Integrations: auto-instruments 70+ LLM providers, agent frameworks, and vector stores such as ChromaDB, Pinecone, Qdrant, and Milvus
- Connectors: reads from ClickHouse, Grafana Tempo, Loki, Prometheus, and Jaeger
