Kitaru turns recorded agent runs into replayable evaluations. Production runs are captured through an adapter or imported from an existing trace store as sessions, then replayed against a new model, prompt, or code change so you can see what improved and what broke before shipping. It comes from the team behind ZenML and ships Python and TypeScript SDKs talking to one self-hosted server.
Replay re-executes your own code rather than scoring a transcript: the agent runs again with tool calls answered from the recording, so external side effects are not repeated. An unchanged replay gives a baseline, a forked replay isolates the effect of a single change, cohorts freeze the population of sessions so a run keeps meaning the same thing over time, and experiments replay a cohort against a candidate.
The intended driver is your coding assistant. Agent skills carry the method and an MCP server exposes bounded operations, so the assistant sweeps sessions with built-in deterministic evaluators, interviews you over the ones that matter, pins your judgments to exact trace locations, drafts an evaluator, runs the experiment, and reports what regressed. Results are presented as two runs side by side rather than a single blended score.
The server is FastAPI and Postgres, provisioned locally with Docker through kitaru login --local, and is distributed with a Docker image and Helm chart. No user code executes on it: replays, imports, and evaluations run on workers in your own environment, using your virtualenv, credentials, and network.
| Framework | Language | Package |
|---|---|---|
| PydanticAI | Python | kitaru-pydantic-ai |
| LangGraph, LangChain agents, Deep Agents | Python | kitaru-langgraph |
| OpenAI Agents SDK | Python | kitaru-openai-agents |
| Mastra | TypeScript | @zenml-io/kitaru-mastra |
| Vercel AI SDK | TypeScript | @zenml-io/kitaru-vercel-ai |
Features
- Sessions from real runs: wrap the agent once to record, or import traces you already collect
- Code-level replay: the real agent re-runs with recorded tool results, so no external action repeats
- Baselines and forks: an unchanged replay is the baseline; a forked replay isolates one change
- Cohorts and experiments: a frozen session population is replayed against a candidate change
- Assistant-driven loop: agent skills plus an MCP server give a coding assistant bounded operations
- Human judgment first: your answers are pinned to trace locations and used to calibrate evaluators
- Framework adapters: PydanticAI, LangGraph, OpenAI Agents SDK, Mastra, and the Vercel AI SDK
- Trace importers: Langfuse, LangSmith, Braintrust, Logfire, and Arize Phoenix, plus custom importers
- Self-hosted: one server with Docker image and Helm chart; workers run in your environment and traces stay put
- Regression gating: production traffic replayed against a change can gate it in CI
Integrates with
AI SDK
TypeScript toolkit for LLM calls, tool use and multi-step agent loops.
Braintrust
Eval, logging and prompt-playground platform for LLM and agent applications.
Claude Code
Terminal-based agentic coding tool that reads your codebase and runs tasks through natural-language commands
Codex
Lightweight coding agent that runs in your terminal