Verdict is a declarative Python framework for specifying and executing compound LLM-as-a-judge systems. Instead of one model call, a judge is composed from primitives — Unit, Layer, and Block — into an arbitrary dependency graph, so reasoning, verification, debate, and aggregation steps stack and swap without rewriting the pipeline.
It addresses documented failure modes of single LLM judges: inconsistent output formats, mode collapse, miscalibrated confidence, and biases toward answer position, data frequency, model family, length, style, tone, and safety. The approach is to spend more inference tokens on judging — scaling judge-time compute — using architectural patterns drawn from the scalable-oversight, automated-evaluation, and generative-reward-modeling literature, such as hierarchical reasoning verification and debate-aggregation.
A pipeline chains units with >>. A judge unit takes a scale (DiscreteScale((1, 5)), BooleanScale(), or a categorical set), a prompt, and a model selected with .via() including retries and temperature; a Layer repeats a sub-pipeline a fixed number of times, and a pooling unit such as MaxPoolUnit aggregates by vote. One worked example replaces a reasoning-model call with two gpt-4o-mini calls: the first scores politeness with an explanation, the second checks that the explanation actually references the source conversation.
Named applications are automated evaluation of AI applications, run-time guardrails over production systems, verifiers that rank, prune, and select candidates during test-time compute scaling, and reward signals for reinforcement learning where rewards are not otherwise verifiable. The accompanying paper is VERDICT: A Library for Scaling Judge-Time Compute (arXiv:2502.18018).
Features
- Composable primitives: Unit, Layer, and Block combine into an arbitrary dependency graph
- Judge protocols: hierarchical reasoning verification, debate-aggregation, and voting pools taken from the research literature
- Interchangeable parts: models, prompts, extraction methods, and judge protocols swap independently
- Concurrency model: coordinates thousands of simultaneous LLM calls in one run
- Client-side rate limiting: experiment results are not lost partway through a run
- DSPy integration: a Verdict judge can serve as the metric when optimizing a DSPy system
- Scales and explanations: discrete, boolean, and categorical scales with optional generated explanations
