Source-linked AI summary
$τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Shunyu Yao, Noah Shinn, Pedram Razavi, Karthik Narasimhan
TL;DR
Existing benchmarks under-test human interaction and domain-policy adherence, both needed for real-world agent deployment. τ-bench evaluates dynamic user–agent–tool conversations with policy-guided APIs and database-state scoring, and finds current agents unreliable and inconsistent.
Problem
Existing language-agent benchmarks often omit human-in-the-loop interaction and domain-specific guidelines required for realistic deployment evaluation.
Method
τ-bench combines realistic databases and APIs, domain policies, LM-simulated users, annotated task outcomes, database-state evaluation, and the pass^k reliability metric.
Results
Current agents perform poorly and inconsistently: gpt-4o achieves ∼61% pass^1 on τ-retail and ∼35% pass^1 on τ-airline, while τ-retail pass^8 reaches ∼25%.
Takeaways & Limitations
τ-bench shows that even state-of-the-art language models are far from reliable for real-world agent use and need stronger consistency and rule-following abilities.
Takeaways & Limitations
The simulated user may contain ambiguities, lack domain knowledge, or have limited reasoning, calculation, memory, and instruction-alignment capacity.
Abstract
from arXiv · showhide
Existing benchmarks do not test language agents on their interaction with human users or ability to follow domain-specific rules, both of which are vital for deploying them in real world applications. We propose $τ$-bench, a benchmark emulating dynamic conversations between a user (simulated by language models) and a language agent provided with domain-specific API tools and policy guidelines. We employ an efficient and faithful evaluation process that compares the database state at the end of a conversation with the annotated goal state. We also propose a new metric (pass^k) to evaluate the reliability of agent behavior over multiple trials. Our experiments show that even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail). Our findings point to the need for methods that can improve the ability of agents to act consistently and follow rules reliably.
1 Introduction
τ-bench addresses the gap between simplified agent benchmarks and real-world deployment needs by testing multi-turn human and API interaction, domain-policy adherence, and consistency. It combines modular benchmark construction with database-state evaluation and finds that current agents remain unreliable.
- Existing benchmarks often omit human interaction and domain-specific guidelines, despite their importance for realistic agent evaluation.
- τ-bench measures whether agents can interact with simulated users and APIs while consistently following domain-specific policies.
- The benchmark focuses on customer service through τ-retail and τ-airline, using LM-generated data and user simulation with manual annotation and verification.
- Database-state comparison with annotated ground truth enables objective evaluation while allowing stochastic variation in conversations.
- ∼61% pass^1 on τ-retail and ∼35% pass^1 on τ-airline were achieved by gpt-4o with function calling, while τ-retail pass^8 fell to ∼25%.The experiments also identify difficulties with database reasoning, ad-hoc policies, and compound requests.
2 Related Work
Prior benchmarks typically evaluate conversation or tool use in simplified or single-step settings. τ-bench combines these capabilities with dynamic user interaction and domain-policy adherence.
- τ-bench unifies conversational and tool-use evaluation while testing consistent adherence to domain-specific policies.
- Tool-use benchmarks such as BFCL, ToolBench, MetaTool, and ToolEmu primarily assess function-call generation, often in single-step interactions.
- Many agent benchmarks evaluate autonomous interaction with environments while providing all information upfront and omitting human-in-the-loop interaction.
- Task-oriented dialogue benchmarks commonly use static trajectories, rule-based or symbolic simulators, or crowdsourced human interactions.
- Prior LM user-simulation work has not used such simulators to benchmark agent reliability.
3 τ-bench: A benchmark for Tool-Agent-User Interaction
τ-bench models each task as interaction among an agent, databases, APIs, policies, and a simulated user. It evaluates success through final database state, required user-facing information, and consistency across repeated trials.
- Task formulation: Each task is formulated as a POMDP whose state combines database and user states, while actions combine database and user interactions.
- Environment: Databases are hidden and accessed through deterministic Python API actions that produce database transitions and observations.
- Environment: Domain policies describe databases, procedures, and restrictions, with some rules enforced by APIs and others requiring agent reasoning.
- User simulation: The simulated user maintains the task prompt and conversation history, generates stochastic responses, and ends an episode with a stop signal.
- Task instances: Task instances pair hidden user-simulation instructions with ground-truth database writes and optional user-answer outputs, then evaluate the resulting episode.
- Evaluation: Episode reward is binary and requires both an identical final database outcome and responses containing all necessary information.
- Evaluation: pass^k measures the chance that all k independent trials succeed, capturing reliability across stochastic conversations with the same underlying task.
4 Benchmark Construction
τ-bench is built from modular domain components and manually validated task scenarios across retail and airline customer service. Its evaluation prioritizes realistic, diverse interactions while preserving objective database-based scoring.
- Construction: τ-bench separates shared environment and user-simulation classes from domain-specific databases, APIs, policies, and task instances.
- Construction: Construction proceeds through manual schema/API/policy design, LM-assisted data generation, and manual task annotation with agent-run validation.
- Domains: The benchmark instantiates τ-retail and τ-airline, chosen for diverse tasks, realistic customer-service relevance, and manageable synthetic data and policies.
- Domains: τ-retail tests order changes, returns, exchanges, addresses, and information requests under one-time and product-compatibility constraints.
- Domains: τ-airline includes 300 flights between 20 US cities and more complex constraints involving payments, baggage, changes, cancellations, membership tiers, and cabin classes.
- Key Characteristics: LM-generated user utterances are open-ended and natural-sounding even when the underlying user instructions are synthetic.
- Key Characteristics: Although schemas and rules are simplified, they support diverse tasks, while repeated trials on a small set of high-quality tasks reveal model and method differences.
- Key Characteristics: Unique ground-truth database outcomes enable fast, faithful rule-based evaluation instead of relying entirely on human judgments.
5 Experiments
Experiments compare models and agent methods on τ-bench, then analyze consistency, costs, and failure modes. Results show substantial gaps in task success, robustness, database reasoning, rule following, and compound-request handling.
- Model comparison: gpt-4o is the best function-calling model, but all models remain far from solving τ-bench, especially τ-airline.The reported τ-airline result for gpt-4o is 35.2% task success.
- Method comparison: Natively supported function calling consistently outperforms text-formatted methods, while reasoning traces improve ReAct over Act-only methods.Adding a “think” function did not improve function-calling performance.
- Agent consistency via pass^k: pass^k declines sharply as the number of trials increases: gpt-4o function calling falls below 25% at pass^8 despite over 60% average task success.The benchmark therefore distinguishes average success from reliable repeated success.
- Failure breakdown: Among 40 failed tasks, 36 were attributed to agent issues after four instruction problems were fixed.The analysis samples 115 gpt-4o function-calling trajectories in τ-retail, yielding pass^1=65.2%.
- Failure 1: Complex database reasoning: Wrong arguments or information account for about 55% of failures, exposing challenges in database reasoning, numerical reasoning, and interpreting user intents.Agents may select the correct tool but fill arguments incorrectly, omit required information, or provide incorrect information.
- Failure 2: Domain understanding and rule following: Wrong decision-making causes 25% of failures when agents misunderstand domain-specific knowledge or rules and choose the wrong tool action.One example violates the policy requiring all exchange items to be collected before a single exchange call.
- Policy ablation: Removing domain policies reduces pass^1 by 4.4% for gpt-4o and 5.5% for gpt-3.5-turbo in τ-retail, where rules are simpler and closer to commonsense.The ablation suggests successful τ-retail behavior often relies on intuitive tool use rather than fully leveraging policy documents.
6 Discussion
τ-bench evaluates agents in dynamic human-and-tool interactions and finds that current systems remain unreliable, inconsistent, and weak at following domain rules. The discussion identifies consistency, rule following, memory, and information focus as central challenges.
- τ-bench evaluates agent reliability in dynamic interactions with humans and tools while testing domain-specific rule following.
- The benchmark’s simulated users can have typos, ambiguities, incomplete domain knowledge, or limited reasoning capacity.
- Future improvements include systematic simulator checks, more complex policies, additional success metrics, and alternative data-curation methods.
- Current LM function-calling agents lack sufficient consistency and rule-following ability for reliable real-world applications.
- Agents need stronger long-horizon information tracking, memory, and focus on relevant context when facts conflict.
Checklist
The checklist reports that the paper addresses its contributions, limitations, societal impacts, reproducibility materials, training details, error bars, compute, asset attribution, licensing, and participant-related documentation as applicable.
- The authors report that they accurately reflect the paper’s contributions and scope and discuss its limitations and potential negative societal impacts.
- For the benchmark experiments, the authors report providing reproducibility materials, training details, error bars, and compute information.
- For existing and newly curated assets, the authors report citing creators, describing licenses, including assets, and discussing consent or personally identifiable information as applicable.
- The human-subjects checklist items concerning participant instructions, risks, institutional review, and compensation are marked not applicable.
A Additional Results
The additional results characterize τ-bench tasks across a broad range of retail difficulties and use per-task success rates to identify especially difficult cases. Figure 7 orders tasks by gpt-4-turbo success rate, with at least 40 trials per task.
- τ-bench tasks span a well-balanced and diverse spectrum of difficulties.
- The task-success plot is used to find zero-success tasks and examine their annotations in a targeted way.
- Figure 7 sorts each τ-retail task by gpt-4-turbo success rate using at least 40 trials per task.
B.1 Stage I: design of database schemas, APIs, and policies
τ-retail defines domain-specific databases, APIs, and policies that constrain authentication, information access, and order actions. Its exchange API validates order status, item compatibility, availability, payment coverage, and updates the order state.
- The exchange API verifies that an order exists and is delivered before processing an exchange.
- Exchange validation checks item counts, same-product available variants, price differences, and an existing payment method with sufficient gift-card balance.
- After validation, the API records the exchange request, exchanged items, replacement items, payment method, and price difference in the order.
- τ-retail supports user authentication, profile and order information, pending-order cancellation or modification, and delivered-order returns or exchanges.
- Consequential actions require explicit user confirmation, while tool calls must be performed one at a time without simultaneous user responses.
C.1 Data examples
The τ-retail examples show linked user, order, product, and variant records with identifiers, options, availability, prices, fulfillment data, and payment history. The data is generated programmatically, mostly using gpt-4-produced code.
- The database examples are generated by code, with most generation code produced by gpt-4 and prompts produced by the authors.
- User records include names, payment methods, and lists of associated order identifiers.
- Product records contain variants with item identifiers, configurable options, availability, and prices.
- Order records connect users, addresses, item records, fulfillment tracking, status, and payment history.
C.2.1 Task 0: wrong decision
In Task 0, the agent initially performs one exchange without explaining the policy that a delivered order can be exchanged only once. A later exchange attempt fails because the order is no longer accepted as delivered by the API.
- The agent omits the one-exchange-per-order policy and submits the keyboard exchange before informing the user about that restriction.
- The agent identifies an available full-size clicky keyboard but reports that no model satisfies all requested features, leading the user to prioritize clicky switches over RGB backlighting.
- After confirmation, the keyboard exchange succeeds and refunds a $3.17 price difference to the credit card.
- The subsequent thermostat exchange returns the error “non-delivered order cannot be exchanged,” after which the agent transfers the case to a human agent.
C.2.2 Task 7: wrong argument
In Task 7, the agent struggles to reason over inventory preferences and ultimately submits an exchange argument that does not match the user’s selected lamp. The tool processes the request with a different replacement item than the user specified.
- The agent fails to reason over the lamp inventory to identify the unique option implied by the user’s brightness and power-source preferences.
- The user explicitly selects desk-lamp item 5320792178 while postponing the water-bottle decision.
- The agent warns that exchanging the lamp now would prevent a later exchange of the water bottle because delivered-order exchanges can occur only once.
C.2.3 Task 42: partially solve compound requests
The agent authenticated Mei Patel, found the relevant jigsaw order, corrected both addresses, and checked shipment status, but only partially completed the compound request by failing to update all order addresses.
- Task outcome: The task required checking all orders to correct wrong addresses, but the agent only fixed the jigsaw order address as the user suggested.Thus, the conversation addressed the selected order but did not complete the broader address-correction request.
- Authentication and order lookup: The agent authenticated Mei Patel using her name and ZIP code after she could not provide the registration email.It then located her recent orders using her user ID.
- Authentication and order lookup: The agent identified the jigsaw order as #W4082615 and reported its shipping address as 443 Maple Drive, Suite 394, Fort Worth, TX 76165.The user corrected the first address line to 445 Maple Drive.
- Address correction: After confirmation, the agent updated Mei Patel’s default address and order #W4082615 to 445 Maple Drive, Suite 394, Fort Worth, Texas, 76165.The dialogue includes separate modify_user_address and modify_pending_order_address calls.
- Order status: The agent correctly reported that order #W4082615 was pending and had not been processed or shipped.This supported the user’s request to determine whether the jigsaw had shipped.
- Product exchange: The agent found two 500-piece jigsaw options, including Item ID 1096508426 with an Art theme, beginner difficulty, and a price of $46.13.It asked the user to confirm an exchange and provide a payment method for any price difference.
D.2 Task 0: successful trajectory
The trajectory shows an agent adapting to a user’s flight constraints, selecting a valid one-stop itinerary, and completing a policy-compliant booking with specified payment methods.
- Requirements gathering: The agent gathered the user ID, trip type, date, cabin, passenger, and insurance preferences before searching for flights.The user requested a one-way economy trip from New York to Seattle on May 20 without travel insurance.
- Flight search: After the user rejected flights departing before 11:00 AM EST, the agent presented one-stop options departing after that time.The displayed options used JFK-to-ATL and ATL-to-SEA flight legs.
- Itinerary selection: The user selected HAT136 from JFK to ATL and HAT039 from ATL to SEA, and the agent summarized a $255 economy itinerary with three free checked bags and no insurance.The booking summary identified Mia Li as the passenger and listed the selected flight segments.
- Payment policy: Because only one travel certificate could be used per reservation, the agent selected the $250 certificate and charged the remaining $5 to the credit card ending in 7447.The user had requested both certificates first, with the larger certificate preferred if only one was allowed.
- Booking outcome: The booking succeeded with reservation ID HATHAT, recording the two selected flights, the $250 certificate, the $5 card payment, three bags, and no insurance.The database result and confirmation message indicate successful completion.