Evidently is a Python library for evaluating, testing, and monitoring machine learning and LLM systems across the lifecycle, from one-off experiments to a hosted monitoring service. It works with tabular and text data and covers predictive tasks such as classification and regression as well as generative tasks such as RAG.
The core objects are Reports and Test Suites. A Report computes and summarizes evaluations from presets or individual metrics and can be viewed interactively in Python, exported as JSON, a dictionary, or HTML, or sent to the monitoring UI. Adding pass/fail conditions to a Report turns it into a Test Suite for regression testing, CI/CD checks, or data validation; conditions use operators such as gt and lt, or can be auto-generated from a reference dataset.
For LLM evaluation, a Dataset takes row-level descriptors such as sentiment, text length, or keyword matches, and LLM-as-a-judge evaluators can be configured alongside them. A monitoring dashboard visualizes metrics and test results over time; it can be self-hosted from the open-source package or used through Evidently Cloud, which adds dataset and user management, alerting, and no-code evaluations.
| Area | Example checks |
|---|---|
| Text descriptors | Length, sentiment, toxicity, language, special symbols, regex matches |
| LLM outputs | Semantic similarity, retrieval relevance, summarization quality, model- and LLM-based evals |
| Data quality | Missing values, duplicates, min-max ranges, new categorical values, correlations |
| Data drift | 20+ statistical tests and distance metrics |
| Classification | Accuracy, precision, recall, ROC AUC, confusion matrix, bias |
| Regression | MAE, ME, RMSE, error distribution, error normality, error bias |
| Ranking and RAG | NDCG, MAP, MRR, hit rate |
| Recommendations | Serendipity, novelty, diversity, popularity bias |
Features
- Reports: preset or custom evaluations rendered in notebooks or exported as JSON, dictionary, or HTML
- Test Suites: pass/fail conditions on any Report, with auto-generated conditions from a reference dataset
- 100+ built-in metrics: from data drift detection to LLM judges, plus a Python interface for custom metrics
- LLM descriptors: row-level evaluators such as sentiment, length, and content checks, with LLM-as-a-judge options
- Monitoring UI: a self-hosted dashboard service that tracks metrics and test results over time
- Evidently Cloud: a managed option with dataset and user management, alerting, and no-code evals
- Open architecture: data export and integration with existing tooling
