DeepEval is a Python framework for testing and evaluating LLM applications in the style of Pytest. A test case holds an input, the application's actual output, and optionally an expected output and retrieval context, and metrics score it from 0 to 1 against a pass threshold. Tests run from the command line or in CI, or through an evaluate function in a notebook.
Evaluation works at three levels: an application end to end as a black box, a full agent trajectory across every decision and action, or individual steps such as LLM calls, tool use, retrieval, and sub-agent handoffs. Tracing captures the ordered sequence of model decisions, tool calls, and intermediate steps, either through manual instrumentation or through integrations for LangChain, LangGraph, Pydantic AI, CrewAI, Google ADK, AI SDK, Mastra, OpenAI Agents SDK, LlamaIndex, AWS AgentCore, and the OpenAI and Anthropic clients.
Metrics are scored by an LLM of the user's choice (OpenAI by default, or a custom model), by statistical methods, or by NLP models that run locally. They cover custom criteria (G-Eval, a graph-based DAG builder, and JevEval), agents, RAG, multi-turn conversation, MCP use, multimodal output, and safety. DeepEval also generates synthetic datasets, optimizes prompts from evaluation results, and runs standard benchmarks such as MMLU, HellaSwag, DROP, BIG-Bench Hard, TruthfulQA, HumanEval, and GSM8K. Results can sync to Confident AI, a hosted platform from the same team, for shared reports and production monitoring.
Features
- Pytest-style tests: test cases with thresholds that pass or fail in a test run or CI pipeline
- Custom metrics: G-Eval for any criteria, DAG for deterministic graph-based judging, and JevEval scored with calibrated probabilities
- Agent metrics: task completion, tool correctness, goal accuracy, step efficiency, plan adherence, plan quality, and argument correctness
- RAG metrics: answer relevancy, faithfulness, contextual recall, precision, and relevancy, plus a RAGAS average
- Conversation metrics: knowledge retention, conversation completeness, turn relevancy, turn faithfulness, and role adherence
- MCP metrics: task completion and server usage for MCP-based agents, single and multi-turn
- Multimodal metrics: text-to-image, image editing, coherence, helpfulness, and reference
- Safety and format checks: hallucination, summarization, bias, toxicity, JSON correctness, and prompt alignment
- Tracing: component-level evaluation of traced spans through manual instrumentation or framework integrations
- Synthetic data: generates single and multi-turn evaluation datasets
- Benchmarks: runs any LLM on MMLU, HellaSwag, DROP, BIG-Bench Hard, TruthfulQA, HumanEval, and GSM8K
- Coding-agent skill: a DeepEval skill lets a coding agent generate a dataset, write the eval suite, run it, and iterate on failures
