Harbor is a framework from the creators of Terminal-Bench for evaluating and optimizing agents and language models in sandboxed container environments. It runs arbitrary agents such as Claude Code, OpenHands, and Codex CLI against task datasets, and the same runs double as rollouts for reinforcement-learning optimization.
A run names a dataset, an agent, and a model, and executes locally in Docker or on a cloud sandbox provider. Supported providers include Daytona, Modal, LangSmith, Blaxel, Novita Sandbox, Tensorlake, Runta, and Sail, which lets a single experiment fan out across thousands of environments in parallel. Harbor is the official harness for Terminal-Bench 2.0, and the dataset list also covers third-party benchmarks such as SWE-Bench and Aider Polyglot. Builders can define and share their own benchmarks and environments, and a separate cookbook repository holds end-to-end examples.
Features
- Agent evaluation: runs arbitrary agents, including Claude Code, OpenHands, and Codex CLI, against benchmark datasets with a chosen model
- Sandboxed environments: each task runs in a container, locally in Docker or on a cloud sandbox provider
- Parallel providers: Daytona, Modal, LangSmith, Blaxel, Novita Sandbox, Tensorlake, Runta, and Sail run thousands of environments concurrently
- Benchmarks: the official harness for Terminal-Bench 2.0, with third-party datasets such as SWE-Bench and Aider Polyglot, versioned by name
- Custom environments: build and share your own benchmarks and task environments
- RL rollouts: evaluation runs generate rollouts for reinforcement-learning optimization
