LLM-as-a-Verifier is a Python framework that scores agent outputs and trajectories with fine-grained rewards between 0 and 1, without training a reward model. Where LLM-as-a-Judge reduces a model's judgment to a single discrete score, this method takes the expectation over the full probability distribution of the verifier model's score tokens, and scales verification along three axes: score granularity, repeated evaluation, and decomposition into separate criteria.
The library exposes three calls. select picks the best of N candidates, compare returns raw rewards for a pair, and track scores an agent's progress at each step of a finished run. A ProgressTracker scores steps live while an agent runs, so a hopeless rollout can be stopped early. Best-of-N selection uses a Probabilistic Pivot Tournament: a ring pass cancels positional bias, the top candidates become pivots, and the remaining comparisons are made only against pivots, cutting cost from O(N²) to O(Nk) pairwise verifications.
Every entry point accepts images, so a multimodal verifier can score visual outputs, vision-language agents, and robot rollouts with per-step camera frames. Verifier backends include DeepSeek, Gemini through Vertex, and any OpenAI-compatible server that returns logprobs, such as Qwen served with vLLM. Prompts keep the criterion at the end so trajectories form a shared prefix, which raises the cache hit rate from 5.2% to 78.4% on Terminal-Bench 2.1, and per-call token accounting reports cached and uncached usage.
Published test-time scaling results, with Gemini 2.5 Flash as the verifier:
| Benchmark | Harness | Pass@1 | With verifier | Oracle |
|---|---|---|---|---|
| Terminal-Bench V2 | Capy | 83.1% | 86.5% | 92.1% |
| SWE-Bench Verified | mini-SWE-agent | 76.1% | 78.2% | 84.4% |
| MedAgentBench | AgentBench | 70.2% | 73.3% | 75.0% |
A self-verification experiment on Terminal-Bench 2.1, where the same model generates and verifies trajectories, reaches 88.0% best-of-5 against a 78.7% Pass@1. The fine-grained rewards can also serve as dense rewards for reinforcement learning. A companion project, TurboAgent, brings the verifier to Claude Code as an API proxy that generates several candidate responses in parallel and selects the best.
Features
- Fine-grained rewards: continuous 0–1 scores from the full logprob distribution of score tokens
- Best-of-N selection: picks the best candidate or trajectory with a cost-efficient pivot tournament
- Pairwise scoring: raw rewards for a direct comparison of two candidates
- Progress tracking: scores a trajectory at every step, offline or live while the agent runs
- Criteria decomposition: scores each candidate against several named criteria with repeated evaluations
- Multimodal inputs: images, video frames, and robot camera feeds alongside text trajectories
- Prefix-cache optimization: prompt layout that reuses cached trajectory prefixes across criteria and repeats
- Token accounting: reports input, cached, output, and reasoning tokens per run
- Benchmark reproduction: bundled trajectories and runners for Terminal-Bench, SWE-Bench Verified, and MedAgentBench
- RL rewards: dense rewards usable for off-policy and on-policy reinforcement learning
