Tracely

Tracely is a self-hostable observability and evaluation stack for AI agents that turns recorded production runs into regression tests. Traces arrive over OTLP, evaluators grade each one as it lands, failures are clustered into issues, and a failing trace can be promoted to a hermetic case that replays in continuous integration and blocks the pull request that would reintroduce it. Everything downstream — scores, clusters, cases, gate verdicts and trends — is computed from the trace rather than from a separately authored dataset.

On ingest, agent semantics such as agent.id, conversation.id, turn and step are promoted to indexed columns, so spans group into conversation threads rather than a flat list. Evaluators appear as columns on the trace table: structural checks that need no model, and LLM-as-judge evaluators at conversation, run or span level, with @VARIABLE prompt templates, per-agent and per-environment targeting, deterministic sampling and advisory modes. A calibration screen compares judge verdicts against human labels and reports per-evaluator agreement.

Promoting a failure bundles the recorded input with the tool and model outputs as fixtures and attaches a fail-to-pass contract: the case must fail on the old code and pass on the fix. Multi-turn behaviour is covered by scenarios instead — a scripted conversation, or an adversarial goal a red-team model pursues, where achieving the goal counts as a failure. The tracely command-line tool provides simulate, replay and gate, and a GitHub Action posts a commit status and a pull-request comment.

Instrumentation is a Python software development kit (SDK) that wraps OpenTelemetry: tracely.init(instrument="auto") detects OpenAI, Anthropic, Google, Mistral and LangChain calls without span code, with @observe and explicit agent, tool, llm, retriever and guardrail context managers for custom spans. Any OTLP/HTTP exporter can post to /v1/traces instead, which is how non-Python services report. Alerts are built as flows: a trigger followed by conditions, Slack, email, webhook, LLM-prompt and Python-expression steps. Each backend also serves an MCP endpoint at /mcp.

The stack is FastAPI with Pydantic v2, Celery and Redis for workers, ClickHouse for traces and scores, Postgres with pgvector as the registry, S3-compatible blob storage as the durable source of truth, and a Next.js front end.

Features

• OTLP ingest: blob-first durability, agent and conversation semantics as indexed columns, three message conventions normalised
• Evaluators as columns: structural checks plus LLM judges at conversation, run or span level, with live streaming verdicts
• Failure clustering: structural signatures and semantic embeddings group repeated failures into one issue with a count and a suggested evaluator
• Regression cases: one click freezes a failing trace into a fixture bundle with a fail-to-pass contract, assertions and a reference trajectory
• Scenarios: multi-turn conversations Tracely drives against your agent endpoint, or adversarial red-team goals
• CI gates: tracely simulate, replay and gate plus a GitHub Action that sets a commit status and upserts a pull-request comment
• Hermetic replay: cases re-run against recorded tool and model outputs, so gate runs need no API keys
• Alert flows: triggers wired to conditions, Slack, email, templated webhooks, an LLM step and Python expressions, with test runs that show what each step sent
• Judge calibration: label verdicts against human review to measure agreement, missed failures and over-flagging
• Conversation views: waterfall timeline, step-by-step replay, and a Fleet view that renders a multi-agent conversation as a room
• Self-hosting: Docker Compose or a one-click Railway template for API, worker, UI, Postgres, ClickHouse, Redis and object storage