Tau Bench

Tau Bench (τ-bench) is a benchmark for tool-agent-user interaction. It emulates dynamic conversations between a user, simulated by a language model, and a language agent that has domain-specific API tools and policy guidelines, and it scores whether the agent completes the task correctly. It reports pass^k, the rate at which an agent succeeds on all of k repeated attempts at the same task, to measure consistency.

This repository holds the original airline and retail domains, and its task versions are no longer updated. The maintained successor, τ³-bench (in the tau2-bench repository), carries fixed airline and retail tasks plus a banking domain and a voice evaluation modality.

Agents can use tool-calling, Act, or ReAct strategies, and the user simulator can run as a plain LLM, ReAct, a verify strategy that checks each simulated reply with a second LLM call, or a reflection strategy. Models from OpenAI, Anthropic, Google, Mistral, and AnyScale are supported for both the agent and the user simulator. An auto error identification tool assigns each failure to the user, agent, or environment and classifies the fault type, and historical trajectories for both domains are included so results can be studied without rerunning the benchmark.