Autoevals is a Python and TypeScript library for scoring AI model outputs. It collects LLM-as-a-judge, heuristic, statistical, and embedding-based evaluation methods behind one interface: you pass an input, an output, and an expected value, and each scorer returns a score between 0 and 1 with optional scorer-specific metadata such as a rationale.
Many of the model-graded evaluators are adapted from OpenAI's evals project, implemented so each can run on a single example, have its prompt tweaked, and have its output debugged. You can also define your own model-graded evaluator from a prompt template and a mapping of choices to scores (with optional chain-of-thought), or write a plain scoring function that uses no LLM at all.
Model-graded scorers call OpenAI by default and work with any OpenAI-compatible API, configured globally or per evaluator with a custom client. Requests can also route through the Braintrust Gateway, which gives access to other providers such as Anthropic, with request caching and optional logging. Logging and comparing results in Braintrust is optional and not required to use the library.
The project exists to handle the parts of evaluation that are hard in practice: normalizing metrics to a 0–1 range, parsing model-graded outputs, and swapping methods behind a uniform interface for comparison.
Features
- LLM-as-a-judge scorers: Battle, Closed QA, Humor, Factuality, Moderation, Security, Summarization, SQL, Translation, and fine-tuned binary classifiers
- RAG evaluations: context precision, relevancy, recall, and entity recall, plus faithfulness, answer relevancy, similarity, and correctness
- Heuristic scorers: Levenshtein distance, exact match, numeric difference, and JSON diff
- Composite and embedding scorers: semantic list contains, JSON validity, and embedding similarity
- Custom model-graded evals: build a classifier from a prompt template and a choice-to-score map, with optional chain-of-thought
- Custom scorers: plain functions that return a score object without calling an LLM
- Uniform score object: every scorer returns a name, a 0–1 score, and scorer-specific metadata
- Provider choice: OpenAI by default, any OpenAI-compatible endpoint, or other providers through the Braintrust Gateway
- Two languages: matching Python and TypeScript APIs with synchronous and asynchronous calls