Braintrust is a platform for tracing, evaluating, and monitoring LLM and agent applications. Production traces carry prompts, responses, and tool calls; experiments score outputs with LLM judges, code, or human review against versioned datasets; and results are compared across prompts and models to catch regressions before a release ships.
The JavaScript and TypeScript SDK is published as the braintrust package with an autoevals scorer library. An evaluation is a file declaring its data, task, and scorers, run through the braintrust eval command; the bt CLI is installed alongside the SDK. Integration packages cover browser environments, OpenTelemetry span export, and Temporal client, worker, and workflow tracing, while the LangChain and OpenAI Agents integrations now live in the main package. The Python SDK is maintained in a separate repository.
Features
- Tracing: prompts, responses, and tool calls captured from production runs, with search across logs and latency, cost, and quality tracking
- Evals: experiments over versioned datasets, scored by LLM judges, code, or humans, with side-by-side prompt and model comparison
- Regression gating: compare a candidate release against expectations before it reaches users
- Trace-to-dataset: turn production traces into eval datasets so regression tests come from real failures
- Custom views and annotation: review interfaces shaped per task type, plus facets and topic clustering over traces
- JavaScript SDK: the braintrust package for logging, tracing, and evals, with the autoevals scorer library and a bt CLI
- Integrations: browser SDK with AsyncLocalStorage polyfill support, an OpenTelemetry span processor, and a Temporal plugin with workflow interceptors
- IDE access: an MCP server for querying logs, running evals, and updating prompts from an editor
