$τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains

Motivation

• Real-world agents must gather information through long-horizon human and API interaction, obey complex domain rules, and behave reliably across equivalent requests.Existing benchmarks commonly provide all information upfront and omit human-in-the-loop interaction and domain-specific guidelines.Source: §1 · p. 1

Benchmark

• τ-bench evaluates whether agents interact with simulated users and programmatic APIs while consistently following domain-specific policies.The benchmark focuses on τ-retail and τ-airline customer-service scenarios built from realistic databases, APIs, policy documents, user scenarios, and ground-truth annotations.Source: §1 · pp. 1–2

Evaluation

• Its evaluation compares each episode’s final database state with the annotated goal state, enabling objective scoring despite varied conversational phrasing.The benchmark is constructed through manual schema and API design, LM-assisted data generation, and manual scenario generation and verification.Source: §1 · p. 2
• The pass^k metric measures whether an agent solves a task consistently across k independent trials, extending evaluation beyond single-trajectory success.This captures robustness to stochastic variation in conversations that can nevertheless lead to the same expected database state.Source: §1 · p. 2

Findings

• ∼61% pass^1 on τ-retail and ∼35% on τ-airline were achieved by gpt-4o with function calling, while τ-retail pass^8 fell to ∼25%.These results expose fragile performance under stochasticity and partial information, alongside failures in database reasoning, policy adherence, and compound requests.Source: §1 · p. 2

• τ-bench unifies conversational interaction, tool use, and consistent adherence to domain-specific policies, unlike benchmarks that evaluate these capabilities separately.Source: §2 · p. 2

Benchmarks for language agents and tool use

• Prior tool-use benchmarks assess function-call accuracy but generally provide only single-step instructions containing all required information, whereas τ-bench requires agents to gather information and authorization from users.This comparison includes BFCL, ToolBench, MetaTool [11], and ToolEmu [16].Source: §2 · pp. 2–3

Task-oriented dialogue

• τ-bench uses language models to simulate realistic user utterances and long-context conversations, producing diverse yet faithful scenario variations for testing agent consistency.Stochastic sampling enables repeated runs with the same scenario while varying the dialogue.Source: §2 · p. 3

User simulation with LMs

• Unlike prior LM-based user simulators focused on realism, τ-bench uses realistic simulation to assess the reliability and robustness of agents in large-scale human interactions.Related applications include NPCs, human-like societies [15], collaborative tasks [21], online shopping [6], and web search [28].Source: §2 · p. 3

Task formulation

• Each task is a POMDP in which the agent jointly interacts with hidden databases through APIs and a simulated user under a domain-specific policy.The state combines database and user states, while actions and observations include database/API and user interactions.Source: §3 · p. 3, Figure 2 · p. 4

Databases and policies

• Domain databases expose deterministic Python API transitions, while policies specify procedures and restrictions that may be enforced by APIs or require agent judgment.For example, invalid payment IDs trigger API errors, whereas baggage allowances must be inferred and entered by the agent.Source: §3 · p. 3

User simulation

• A gpt-4-0613 simulator generates stochastic user messages from a hidden task prompt and conversation history, ending the episode when it emits ###STOP###.The simulated user cannot see the agent’s API-tool interaction history.Source: §3 · p. 3

Task instances

• Each task uniquely specifies the intended database writes and may specify required answers, enabling evaluation from the final database state and agent-to-user messages.Task instructions hide user identity, intent, and preferences from the agent, while annotations support simulation and evaluation.Source: §3 · p. 4, Figure 2 · p. 4

Reward

• The binary reward requires both an identical final database and all necessary user-facing information, although it may not detect policy violations such as acting without confirmation.The benchmark’s rule-based reward is designed to be fast and faithful while challenging current agents.Source: §3 · p. 4

Pass^k metric

• pass^k is the probability that all k independent trials succeed, capturing agent reliability across conversational variations with fixed task semantics and database transitions.The benchmark reports average reward across tasks as its main comparison metric, with pass^1 = pass@1 = E[r].Source: §3 · pp. 4–5

4 Benchmark Construction

• τ-bench domains combine shared environment and user simulation classes with domain-specific databases, APIs, documentation, policies, and task instances, built through three mostly linear stages.The process includes manual design, LM-assisted data generation, and iterative task annotation and validation with agent runs.Source: §4 · p. 5

4.1 Domains

• τ-retail covers order cancellation, modification, returns, exchanges, address changes, and information requests under constraints designed to test rule following and information collection.Pending and delivered orders have one-time action limits, and items cannot be exchanged for another product type.Source: §4.1 · pp. 5–6
• τ-airline supports booking, modification, cancellation, and refunds across 300 flights linking 20 US cities, with complex policy constraints requiring multi-hop reasoning.Constraints involve payment combinations, baggage allowances, changes, cancellations, membership tiers, and cabin classes.Source: §4.1 · p. 6

4.2 Key Characteristics

• LM-generated user utterances are open-ended and natural-sounding, while τ-bench’s simplified schemas, APIs, and rules support diverse, creative tasks despite trading quantity for quality.The benchmark uses a small set of high-quality tasks across multiple trials to reveal model and method differences.Source: §4.2 · p. 6
• By enforcing a unique database outcome from domain policies and user desires, τ-bench replaces subjective human judgments with fast, objective database-state comparisons.This design addresses diverse trajectories and multifaceted success criteria that otherwise make evaluation difficult.Source: §4.2 · p. 6
• τ-bench’s modular codebase supports adding domains, entries, functionalities, rules, APIs, tasks, and evaluation metrics when they remain consistent with existing domain data.The released codebase is intended to encourage community creation of new tasks and domains.Source: §4.2 · p. 6

5.1 Main results

• gpt-4o is the best function-calling model, but all models remain far from solving τ-bench; on τ-airline, even gpt-4o solves only 35.2% of tasks.Open-weight llama-3-70b and mistral-8x22b remain substantially behind proprietary gpt-4o and claude-3-opus.Source: §5.1 · pp. 6–7, Table 2 · p. 7
• pass^8 falls below 25% for gpt-4o function calling despite its average task success exceeding 60%, revealing sharply declining reliability across repeated trials.The benchmark therefore evaluates both average success pass^1 and consistency pass^k.Source: Figure 4 · p. 7, §5.1 · p. 7
• Natively supported function calling consistently outperforms text-formatted methods, while ReAct reasoning traces improve over Act-only; adding a “think” function does not help.The experiments compare function calling with ReAct and Act-only, while noting that self-reflection and planning are unsuitable for real-time user-in-the-loop interaction.Source: §5 · p. 6, §5.1 · p. 7
• On τ-retail, pairing gpt-4o function calling with gpt-4 simulation costs $0.38 and $0.23 per task, respectively, with 95.9% of agent cost from input prompts.One trial per task costs around 200 dollars, primarily because domain policies and function definitions create long system prompts.Source: §5.1 · p. 7

5.2 Research challenge analysis

• Among 115 τ-retail gpt-4o trajectories, 40 failed, yielding pass^1=65.2%; 36 failures were agent issues after four user typos or ambiguities were fixed.The 36 agent failures are further categorized in Figure 5.Source: Figure 5 · p. 7, §5.2 · p. 7
• Wrong information or arguments comprise ~55% of failures, reflecting complex database and numerical reasoning challenges; hallucinated IDs occur at 0.46 calls per τ-retail task for gpt-4o versus 2.08 for gpt-3.5-turbo FC.Agents may select the right tool but fill arguments incorrectly, omit required information, calculate wrong totals, or provide misleading results.Source: §5.2 · pp. 7–8
• Incorrect domain decisions account for 25% of failures, while removing policy reduces pass^1 by 4.4% for gpt-4o in τ-retail and 22.4% in τ-airline.The contrast suggests simpler retail rules can be handled commonsensically, whereas complex airline rules require stronger policy processing.Source: Table 3 · p. 8, §5.2 · p. 8
• Function-calling agents partially resolve 19% of compound-request cases, omitting explicit or implicit actions and exposing weaknesses in long-context memory, consistency, and systematicity.For example, agents may stop after checking one order when the user requests corrections across all orders.Source: §5.2 · p. 8

Discussion

• τ-bench evaluates reliability in dynamic human-and-tool interactions through simulated users, automated testing, and domain-specific rule assessment, yet state-of-the-art models remain unreliable.The benchmark is intended to assess consistent behavior in realistic settings involving both humans and tools.Source: §6 · p. 8

Directions for improvement

• Simulated users may introduce typos, ambiguities, incomplete domain knowledge, or limited reasoning, though these variations can reflect diverse real-world users whose needs agents must handle.Future work can improve annotation and simulation while preserving realistic variation in user skill and knowledge.Source: §6 · p. 9
• Future work should strengthen outcome checks, domain policies, success metrics, annotation methods, and user simulation while considering possible societal consequences.The authors note that manual annotation is difficult, task curation may contain implicit bias, and broader metrics could check whether rules are followed.Source: §6 · p. 9

Challenges for agents

• Current language-model function-calling agents lack sufficient consistency and rule-following ability for reliable real-world applications.Improvement priorities include long-horizon information tracking, memory, and focusing on relevant context amid conflicting facts.Source: §6 · p. 9

• The authors affirmed that the paper accurately reflects its contributions and scope, describes limitations, discusses potential negative societal impacts, and conforms to ethics guidelines.Source: Checklist · p. 12
• Theoretical-results checklist items on stating assumptions and providing complete proofs were marked N/A.Source: Checklist · p. 12
• For experiments, the authors affirmed providing reproducibility materials, training details, error bars, and compute-resource information.Source: Checklist · p. 12
• For existing and newly released assets, the checklist confirms creator citations, license information, inclusion of new assets, and coverage of consent and data-content issues where applicable.Source: Checklist · p. 12
• All crowdsourcing and human-subjects checklist items were marked N/A, including participant instructions, risks and IRB approvals, wages, and compensation.Source: Checklist · p. 12

A Additional Results

• τ-bench tasks have a well-balanced and diverse spectrum of difficulties, and zero-success tasks are examined through their annotations.The difficulty distribution is used to identify tasks requiring targeted annotation analysis.Source: A Additional Results · p. 13
• Figure 7 sorts τ-retail tasks by gpt-4-turbo success rate, with at least 40 trials per task for reliable estimates.The figure presents per-task success rates rather than aggregate benchmark performance.Source: Figure 7 · p. 13

B.1 Stage I: design of database schemas, APIs, and policies

• The benchmark models retail and airline domains with structured databases and APIs supporting domain-specific user assistance and database updates.Retail agents handle orders, returns, exchanges, addresses, profiles, and products; airline agents handle flight reservations and related booking changes.Source: Table 4 · p. 13, Retail policies · p. 15, Airline policies · p. 17

Retail policies

• Retail agents must authenticate users before providing information, serve only one user per conversation, and require explicit confirmation before consequential database updates.They must also use at most one tool call at a time and deny or transfer requests outside their permitted scope.Source: Retail policies · pp. 15–16
• Retail order operations are status- and inventory-constrained: cancellations and modifications apply to pending orders, while returns and exchanges apply to delivered orders.Modifications and exchanges are limited to available variants of the same product, require complete item lists and payment handling for price differences, and can be called only once.Source: Retail policies · pp. 16–17

Airline policies

• Airline agents must obtain user and itinerary details, collect passenger and payment information within reservation limits, and obtain explicit confirmation before booking updates.Flights marked delayed, on time, or flying cannot be booked; reservations allow at most five passengers and restrict payment methods to those already in the user profile.Source: Airline policies · pp. 17–18
• Airline policies additionally govern baggage allowances by membership and cabin, charge $50 for each extra bag, and offer $30-per-passenger travel insurance with specified cancellation coverage.The agent must avoid unsupported information, make only one tool call at a time, deny policy-violating requests, and transfer only out-of-scope requests.Source: Airline policies · pp. 17–18

B.2 Stage II: data generation

• Stage II combines GPT-4-generated code, execution-based refinement, manual polishing, random sampling, and language-model-generated lists to create scalable synthetic user databases.The approach can sample 10,000 users with minimal human effort, while the shown generator creates 500 user profiles containing realistic addresses, contact fields, and payment methods.Source: §B.2 · pp. 20, 22

C.1 Data examples

• The displayed data examples are generated by code, with most code and the prompt used to generate it produced by GPT-4.Source: §C.1 · p. 23
• τ-retail user records include names, addresses, gift-card balances, identifiers, and associated order IDs.Source: §C.1 · p. 23
• τ-retail product records represent items such as office chairs with product IDs, variants, configurable options, availability, and prices.Source: §C.1 · p. 23
• τ-retail order records connect orders to users and addresses while listing purchased items, prices, and item-level options.Source: §C.1 · p. 24
• Order records also include fulfillment tracking, processing status, and payment history linked to a gift-card payment method.Source: §C.1 · p. 24

C.2 Task and trajectory examples

• Tasks are not cherry-picked, and the example trajectories use a GPT-4o function-calling agent while keeping JSON task annotations hidden from that agent.Source: §C.2 · p. 25

C.2.1 Task 0: wrong decision

• The agent omits the policy that only one exchange per order is possible and processes the keyboard exchange without informing the user.The exchange request changes the order status to “exchange requested” for the mechanical keyboard.Source: §C.2.1 · pp. 25, 28
• The agent correctly identifies that no available keyboard satisfies all requested clicky, RGB, and full-size attributes, then offers available alternatives.The user selects the full-size clicky model without backlight, item ID 7706410293.Source: §C.2.1 · pp. 27–28
• After the user requests a thermostat exchange, the agent finds one available Google Assistant-compatible model and receives confirmation to proceed.The available option is item ID 7747408585, priced at $249.01.Source: §C.2.1 · p. 29
• The second exchange fails with a “non-delivered order cannot be exchanged” error despite the earlier order record showing delivered status, so the agent transfers the case to human agents.The transfer succeeds after the agent reports the status inconsistency.Source: §C.2.1 · p. 30

Task 7: wrong argument

• The agent fails to reason reliably over the complex product inventories needed to identify the user’s desired exchange options.The task specifically requires a larger water bottle and a less-bright lamp, prioritizing AC adapter over battery and USB.Source: §C.2.2 · pp. 31–32
• The agent presents available larger water bottles and less-bright desk lamps, including AC-adapter options that match the user’s preferences.It lists 1000ml water bottles and medium- or low-brightness AC-adapter lamps, including item 5320792178 and item 1569765161.Source: §C.2.2 · p. 34
• After explaining that a delivered order permits only one agent exchange, the agent processes the desk-lamp exchange alone as requested.The final state records exchange_items ["8384507844"] and exchange_new_items ["5320792178"], with a $2.70 refund to PayPal.Source: §C.2.2 · p. 35

C.2.3 Task 42: partially solve compound requests

• The agent partially solves the compound request by correcting only the jigsaw order’s address rather than all order addresses.The task requires checking all orders, but the agent fixes only the jigsaw order address suggested by the user.Source: §C.2.3 · p. 36
• After authenticating the user with their name and zip code, the agent found recent orders and received confirmation to update the user and order addresses to 445 Maple Drive, Suite 394, Fort Worth, Texas, 76165.Source: §C.2.3 · pp. 37–39
• The agent reports the pending order as unshipped, identifies available 500-piece jigsaws, and exchanges the original puzzle for the beginner Art-themed item 1096508426.The exchange leaves the order pending with the new 500-piece beginner puzzle and processes a $7.98 PayPal refund.Source: §C.2.3 · pp. 40–42

D.1 Data examples

• The airline examples comprise synthetic users.json and flights.json records generated by code, with most code generated by GPT-4 from author-generated prompts.The examples include user profiles, payment instruments, memberships, reservations, flight schedules, statuses, seat availability, and prices.Source: §D.1 · pp. 43–44
• A flight-database entry records availability and prices for basic economy, economy, and business cabins.The example lists 2, 17, and 20 available seats, priced at 91, 186, and 321, respectively.Source: §D.1 · p. 44
• The reservation example also stores three passengers, payment history, creation time, baggage totals, and insurance status.It lists one gift-card payment, five total bags, zero nonfree bags, and no insurance.Source: §D.1 · p. 45
• A reservation entry includes a reservation and user identifier, round-trip route, business cabin, and outbound and return flight details.The example specifies DFW–LAX and LAX–DFW segments with flight numbers, dates, and prices.Source: §D.1 · p. 45

Outcome

• The completed reservation records flights HAT136 and HAT039, economy cabin, three bags, no insurance, and the confirmed payment history.The tool returns reservation ID HATHAT with the selected one-stop itinerary from JFK to SEA.Source: §D.2 · pp. 46, 50

Trajectory

• The agent gathers trip type, date, cabin, passenger, and insurance details before searching and booking the requested economy itinerary.The user specifies a one-way May 20 trip, economy class, one passenger from the profile, and no travel insurance.Source: §D.2 · pp. 46–47
• The agent adapts to the user's request for a post-11:00 AM departure by presenting one-stop JFK–SEA alternatives.The user rejects two direct flights as too early, and the agent returns options departing at 6:00 PM and 7:00 PM EST.Source: §D.2 · pp. 48–49
• The agent follows the one-certificate safety rule by applying the $250 certificate and charging the remaining $5 to card 7447.The user requests two certificates but accepts using the larger certificate when only one is permitted.Source: §D.2 · p. 49