Terminal-Bench measures how well AI agents complete end-to-end tasks in real terminal environments — compiling code, training models, setting up servers, and similar work — without human intervention. It has two parts: a dataset of tasks and an execution harness that connects a language model to a sandboxed terminal.
Each task carries an instruction in English, a test script that checks whether the agent finished the job, and a reference solution that solves it. The suite is in beta at roughly 100 tasks and is distributed as a Python package driven by a tb command; running the harness also needs uv and Docker.
Results can be submitted to a public leaderboard, which tracks resolution rate, cost, and token use per model and agent. The leaderboard corresponds to the Terminal-Bench-Core v0.1.1 task set, selected by passing a dataset name and version to the harness. New users are pointed at Harbor, a newer framework that runs Terminal-Bench 2.0.
Features
- Task dataset: around 100 tasks, each with an English instruction, a verification test script, and a reference solution
- Execution harness: connects a language model to a sandboxed terminal and runs it through tb run
- Versioned datasets: named, versioned task sets chosen with --dataset-name and --dataset-version, listed in a registry
- Leaderboard: runs against Terminal-Bench-Core v0.1.1 can be submitted for public ranking
- Concurrency: --n-concurrent controls how many task instances run at once
- Extensibility: contributors add tasks in a documented task format, and adapters connect other benchmarks to the harness
