DeepEval

DeepEval is an open-source framework for evaluating LLM applications. It follows the shape of Pytest but targets LLM systems: you build a test case from an input, the application's actual output, an expected output, and any retrieval context, attach one or more metrics, and assert on the resulting scores against a threshold. Metrics run on your own machine, using an LLM judge, statistical methods, or local NLP models, and each score comes with a reason.

Evaluation works at three levels. You can treat the application end-to-end as a black box, score a complete agent trajectory across every decision and action, or score individual components such as LLM calls, tool use, retrieval, and sub-agent handoffs. The @observe decorator and evals_iterator() capture the ordered sequence of steps in a run, so trajectory metrics see the whole path an agent took rather than only its final answer.

Metrics cover several families: custom criteria (G-Eval and the graph-based DAG builder), agentic behaviour (task completion, tool correctness, goal accuracy, step efficiency, plan adherence and quality, argument correctness), retrieval-augmented generation (answer relevancy, faithfulness, contextual recall, precision, and relevancy, plus RAGAS), multi-turn chat (knowledge retention, conversation completeness, turn relevancy and faithfulness, role adherence), Model Context Protocol usage, multimodal output, and safety checks such as hallucination, bias, toxicity, summarization, JSON correctness, and prompt alignment.

The same suites run in any CI/CD environment, and models can be benchmarked directly on MMLU, HellaSwag, DROP, BIG-Bench Hard, TruthfulQA, HumanEval, and GSM8K. Python 3.9 or later is required.

Features

• Pytest-style assertions: assert_test and deepeval test run turn metric thresholds into passing or failing tests
• Custom criteria: G-Eval scores any written criterion with an LLM judge; DAG builds a deterministic, graph-based judge
• Component-level tracing: @observe and updatecurrentspan attach test cases to individual spans for step-by-step scoring
• Trajectory evaluation: metrics such as task completion score the full ordered sequence of model decisions and tool calls
• Standalone metrics: any metric can be called directly on a test case outside a test run, returning a score and a reason
• Synthetic data: generates single-turn and multi-turn datasets for evaluation
• Prompt optimisation: prompts are tuned automatically from evaluation results
• Custom metrics: user-defined metrics plug into the same ecosystem as the built-in ones
• Framework integrations: LangChain, LangGraph, Pydantic AI, CrewAI, Mastra, AI SDK, LlamaIndex, Strands, Google ADK, AWS AgentCore, OpenAI, OpenAI Agents, and Anthropic clients
• Benchmarks: MMLU, HellaSwag, DROP, BIG-Bench Hard, TruthfulQA, HumanEval, and GSM8K in a few lines of code
• Environment loading: .env.local then .env load at import time, with DEEPEVALDISABLEDOTENV=1 to opt out
• Hosted platform: Confident AI adds shareable reports, dataset management, production tracing, and an MCP server for driving evals from an editor