Harbor

Harbor is a Python framework for evaluating and optimising agents and language models in sandboxed environments. It runs arbitrary agents against a dataset of tasks, lets you build and share your own benchmarks and environments, and generates rollouts for reinforcement-learning optimisation. It comes from the team behind Terminal-Bench.

A run names a dataset, an agent, and a model: harbor run --dataset terminal-bench@2.0 --agent claude-code --model anthropic/claude-opus-4-1. Runs execute locally in Docker by default; passing --env moves execution to a sandbox provider such as Daytona, and --n-concurrent scales a run into many parallel environments. harbor datasets list shows the supported third-party benchmarks, which include SWE-Bench and Aider Polyglot.

Harbor is the official harness for Terminal-Bench 2.0. It installs with uv tool install harbor or pip install harbor, and a separate cookbook repository carries end-to-end examples.

Features

• Arbitrary agents: evaluate agents such as Claude Code, OpenHands, and the Codex CLI against a dataset
• Own benchmarks: build and share your own tasks and environments
• Parallel sandboxes: thousands of concurrent environments through providers including Daytona, Modal, Blaxel, Novita Sandbox, and Tensorlake
• Local or cloud execution: Docker by default, with --env selecting a cloud sandbox provider
• Third-party datasets: listed benchmarks include Terminal-Bench, SWE-Bench, and Aider Polyglot
• RL rollouts: runs can generate rollouts for reinforcement-learning optimisation
• Official Terminal-Bench harness: the supported way to run Terminal-Bench 2.0