Kitaru

Kitaru turns recorded agent runs into tests. Production runs are recorded or imported as sessions, then replayed against a new model, prompt, or code change to show what improved, what regressed, and what it costs before the change ships. It comes from the team behind ZenML and has Python and TypeScript SDKs that talk to the same server.

Replay re-executes the real agent code, with tool calls answered from the recording so side effects such as payments do not repeat. An unchanged replay gives a faithful baseline, and a forked replay isolates the effect of one change. Sessions come from a one-line wrapper around the agent or from traces already collected in Langfuse, LangSmith, Braintrust, Logfire, Arize Phoenix (Phoenix), or MLflow, which stay the system of record.

Evaluation starts with human judgment. A coding assistant, driven by Kitaru's skills and MCP server, sweeps sessions with built-in deterministic evaluators, interviews you about the sessions that matter, and pins your answers to exact trace locations. Those judgments calibrate evaluators, cohorts freeze the population under test, and experiments replay the cohort against the change, which can also gate a change in CI as a regression suite.

The server is self-hosted (a Docker image and Helm chart are included) or available as a managed cloud, and no user code runs on it: replays, imports, and evaluations run on workers in your own environment with your own credentials, so traces never have to leave your systems.