Open RAG Eval is a Python toolkit (Python 3.9 or higher) for measuring the quality of retrieval-augmented generation (RAG) pipelines. Its core metrics, UMBRELA and AutoNuggetizer, score retrieval and answer quality without golden chunks or golden answers, so you can evaluate a pipeline before building a labeled reference set. Optional golden-answer metrics apply when reference answers do exist.
An evaluation run is described in a YAML config: a connector calls your RAG system with a set of input queries and collects answers and retrieved passages, one or more evaluators apply metrics, and per-query scores and intermediate outputs are written to a results file. Connectors ship for Vectara, LlamaIndex, and LangChain, and you can evaluate any other pipeline by supplying its output in the toolkit's CSV format.
Three evaluators are built in. TRECEvaluator scores each query independently with TREC-RAG retrieval and generation metrics, including UMBRELA and an HHEM factual-consistency score. GoldenAnswerEvaluator compares generated answers to reference answers using embedding cosine similarity and claim-level factual correctness computed with natural language inference. ConsistencyEvaluator measures agreement across repeated generations for the same query using BERTScore and ROUGE-L. Evaluators can be combined in one run, with ConsistencyEvaluator listed last because it depends on the other evaluators' scores.
When you have no queries to start from, the toolkit generates them from corpus documents, with weights that control the mix of directly answerable, reasoning-required, unanswerable, and partially answerable questions, and it can generate expected answers at the same time. Results are explored through CLI plotting, a Streamlit viewer, or a web viewer that accepts the results file, and a Flask server exposes /api/v1/evaluate and /api/v1/evaluate_batch for programmatic use.
Features
- Reference-free core metrics: UMBRELA and AutoNuggetizer score retrieval and answers without golden chunks or golden answers
- YAML-configured runs: connector, evaluators, metrics, judge models, and output paths are declared in a config file
- Connectors: Vectara, LlamaIndex, and LangChain out of the box, plus a CSV format for any other pipeline
- Golden-answer evaluation: semantic similarity plus factual correctness as precision, recall, and F1 over decomposed claims
- Consistency evaluation: BERTScore on multilingual xlm-roberta-large with baseline rescaling, and ROUGE-L for English
- Factual-consistency backends: the open-source HHEM model by default, or the Vectara factual-consistency API, with a configurable detection threshold
- Synthetic query generation: generates queries from Vectara corpora, local files, or CSV, with per-type weights and optional expected answers
- Judge-model choice: OpenAI, Anthropic, Gemini, and Together models can serve as the LLM judge
- Custom prompt templates: the Vectara connector accepts a prompt template inline or from a file
- Reporting: per-query scores with intermediate outputs, a consolidated report joined on query ID, and plots comparing several runs
- Web API: Flask endpoints for single and batch evaluation
- Extensibility: metrics, connectors, and evaluators are pluggable classes with defined interfaces
