Source-linked AI summary
$τ^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
Victor Barres, Honghua Dong, Soham Ray, Xujie Si, Karthik Narasimhan
TL;DR
Existing conversational-agent benchmarks largely keep users passive, whereas real-world support requires users to act in a shared environment. τ 2-bench addresses this with dual-control Dec-POMDP evaluation, compositional tasks, and constrained user simulation, finding substantial performance declines when agents must guide users.
Problem
Existing benchmarks model users as passive information providers, unlike support scenarios where users actively modify shared-world state.
Method
τ 2-bench introduces a telecom dual-control Dec-POMDP, compositional verifiable-task generation, and user simulation constrained by tools and observable environment state.
Results
Agents show an approximately 20% pass^1 performance decrease when shifting from autonomous operation to guiding a user.
Takeaways & Limitations
Communication and coordination are critical bottlenecks for agents solving user requests in dual-control settings.
Takeaways & Limitations
τ 2-bench does not explicitly model the expert-novice gap, including adapting explanations to a naive user’s mental model.
Abstract
from arXiv · showhide
Existing benchmarks for conversational AI agents simulate single-control environments, where only the AI agent can use tools to interact with the world, while the user remains a passive information provider. This differs from real-world scenarios like technical support, where users need to actively participate in modifying the state of the (shared) world. In order to address this gap, we introduce $τ^2$-bench, with four key contributions: 1) A novel Telecom dual-control domain modeled as a Dec-POMDP, where both agent and user make use of tools to act in a shared, dynamic environment that tests both agent coordination and communication, 2) A compositional task generator that programmatically creates diverse, verifiable tasks from atomic components, ensuring domain coverage and controlled complexity, 3) A reliable user simulator tightly coupled with the environment, whose behavior is constrained by tools and observable states, improving simulation fidelity, 4) Fine-grained analysis of agent performance through multiple ablations including separating errors arising from reasoning vs communication/coordination. In particular, our experiments show significant performance drops when agents shift from no-user to dual-control, highlighting the challenges of guiding users. Overall, $τ^2$-bench provides a controlled testbed for agents that must both reason effectively and guide user actions.
1 Introduction
τ 2-bench addresses the gap between single-control conversational benchmarks and real-world support scenarios by giving users tools to act in a shared environment. It combines dual-control evaluation, compositional tasks, reliable simulation, and diagnosis of reasoning versus coordination failures.
- Existing conversational benchmarks leave simulated users passive, limiting them to providing preferences and goals while agents alone interact with the world.Real-world technical support instead requires users to perform actions such as restarting phones or disabling airplane mode.
- τ 2-bench introduces a dual-control environment where users can communicate, take actions, and call tools alongside agents.The design supports selective information hiding, non-verbal environment manipulation, and structured task specification.
- The telecom domain formalizes shared, dynamic agent-user interaction as a Dec-POMDP and exposes coordination and communication challenges absent from limited-agency evaluations.Both players have distinct tools to observe, act upon, and verify shared state.
- A programmatic task generator composes diverse, verifiable tasks from atomic scenarios while providing domain coverage and explicit control over complexity.Task complexity can be controlled through factors such as solution-step count and issue type.
- Tool- and state-constrained user simulation improves reliability, with telecom showing a 16% error rate and 6% critical-error rate.The corresponding retail figures from τ-bench are 40% and 12%, respectively.
- Agents show an approximately 20% pass^1 performance decrease when shifting from autonomous operation to guiding a user.The ablation separates pure reasoning failures from communication and decentralized-control failures, identifying the latter as a critical bottleneck.
2 Related Work
τ 2-bench extends prior conversational-agent benchmarks by giving both the agent and user state-changing abilities over a shared world. This enables more complex scenarios and finer diagnosis of failures while retaining an asymmetric focus on the agent’s ability to guide the user.
- τ-bench evaluates reliable multi-turn, task-oriented conversations under domain rules across retail and airline customer-service workflows.Its pass^k metric measures the fraction of k independent runs that succeed.
- Follow-up benchmarks vary planning supervision and programmatic test construction through workflow knowledge, policy graphs, and structured synthetic suites.FlowBench injects workflow knowledge, while IntellAgent builds suites from structured policy graphs.
- τ 2-bench generalizes the paradigm by allowing both users and agents to change shared-world state through tool calls.This supports more complex domains and fine-grained analysis of agent failure points.
- Unlike supervision-focused user simulation work, τ 2-bench uses the environment itself to constrain and shape simulator behavior for increased reliability.The approach makes user behavior more predictable through available tools and observable state.
- Its evaluation remains centered on whether the agent elicits information and performs correct actions, rather than on solving a symmetric multi-agent problem.The framework can represent collaborative, competitive, or hybrid interactions involving user agency and mistakes.
3 τ 2-bench: Evaluating Agents in a Dual-Control Environment
τ 2-bench models agent-user interaction as a Dec-POMDP in which both players communicate, use tools, and receive observations. Its telecom implementation combines state-based dynamics, automatically composed tasks, and environment-constrained simulation.
- 3.1 The Dec-POMDP Formalism: The Dec-POMDP models a conversational agent and user who communicate, use tools, and receive observations while coordinating in interactive scenarios.This formalism targets settings such as technical troubleshooting.
- 3.1 The Dec-POMDP Formalism: The global state combines agent and user database states with an interaction history recording actions, observations, and messages.Telecom examples include CRM data for the agent and phone status for the user.
- 3.1 The Dec-POMDP Formalism: Each player can either call tools affecting its database or send a message, with only one player acting per turn.Agent and user tools include customer lookup and airplane-mode toggling, respectively.
- 3.1 The Dec-POMDP Formalism: Observations consist of tool outputs or messages from the other player, and transitions update both world state and interaction history.Agent tool calls can change user-state-relevant conditions such as roaming availability.
- 3.1 The Dec-POMDP Formalism: The environment assigns global reward from the resulting state, with telecom success determined by whether the user’s issue is fixed.Success is verified through the user’s database state.
- 3.1 The Dec-POMDP Formalism: User tools are designed to produce human-readable outputs, support reactive use, and preserve complexity asymmetry between agent and user.Constraining tools and environment effects makes simulated behavior more predictable and reduces reliance on extensive prompting.
- 3.2 Domain and task creation: Composite tasks select at most one atomic subtask from each mutually exclusive group and verify final-state assertions after all solution functions are applied.The construction also checks that tasks remain unresolved until every required solution function is executed.
- 3.2 Domain and task creation: The telecom generator combines 15 atomic subtask groups across three intents into 2285 tasks, then subsamples 114 balanced tasks.The number of subtasks serves as a difficulty proxy because more diagnostic and resolution steps are required.
4 Experiments
Experiments evaluate four language models across telecom, airline, and retail, then ablate reasoning load, dual control, policy detail, task complexity, issue type, persona, and simulator reliability. Telecom is substantially harder, while active-user coordination and longer or more complex tasks reduce success.
- Experimental setup: Four language models were evaluated with deterministic function-calling runs, using gpt-4.1-2025-04-14 as the user simulator.Each task was run four times at temperature 0.
- Cross-domain performance: 34% pass^1 for gpt-4.1 in telecom, versus 74% in retail and 56% in airline.Other models achieved around 50% pass^1 in telecom, while telecom pass^k declined more rapidly as k increased.
- Ablation analysis: 18% and 25% pass^1 drops occurred for gpt-4.1 and o4-mini, respectively, when shifting from No-User to Default dual control.The comparison isolates communication and coordination overhead from autonomous reasoning and tool use.
- Ablation analysis: Workflow policies slightly improved Default and No-User performance but hurt Oracle Plan performance for both gpt-4.1 and o4-mini.The authors hypothesize that workflow details can confuse agents already given the ground-truth action sequence.
- Task complexity: Agent performance drops as required actions increase, reaching close to zero for more than 7 actions in Default mode.No-User scores were higher overall, but the gap narrowed from about 0.3–0.4 to below 0.2 as action counts increased.
- Task complexity and issue types: Performance trends downward as tasks combine more sub-tasks, and harder mobile_data_issue and mms_issue types produce higher failure rates across models.Service_issue is easiest, while the other issue types require more complex, multi-stage reasoning and conditional logic.
- User persona: Agents perform better on Easy-persona tasks than Hard-persona tasks, while tasks without persona information are often comparable to or worse than Hard.The result supports evaluating systems with explicit user personas.
- Simulator reliability: The telecom user simulator has a 16% error rate with 6% critical errors, lower than retail’s 40% and airline’s 47% error rates.The structured telecom environment and available affordances constrain behavior toward more consistent interactions.
5 Conclusion
τ^2-bench highlights coordination and communication as critical bottlenecks beyond pure reasoning, while identifying important directions for improving user simulation, domain curation, and expert-novice modeling.
- τ^2-bench identifies coordination and communication as critical bottlenecks over pure reasoning for solving user requests.
- Applying curated tools to users can avoid critical errors, but this approach has not been investigated in the airline and retail domains.
- Extending benchmark domain coverage still relies heavily on human experts, motivating further automation of domain curation.
- τ^2-bench does not explicitly model the expert-novice gap in customer support, where experts adapt explanations to users’ mental models.
Broader Impact
Standardized benchmarks for LLMs and AI agents support transparent evaluation, research coordination, fairness, and societal control amid rapid technological advances.
- Standardized benchmarks provide transparent evaluation frameworks and common tasks that help research groups coordinate their efforts.
- These benchmarks are presented as important for fostering fairness and ensuring societal control as AI technologies advance rapidly.
A Telecom Domain
The telecom domain samples tasks across three user intents, varying action and subtask complexity, and assigns personas representing different technical expertise and comfort with technology.
- Task coverage: Table 3 reports the number of sampled tasks for each intent and number of subtasks.
- Task coverage: The telecom domain contains three user intents: service_issue, mobile_data_issue, and mms_issue.
- Task coverage: Tasks vary in the number of actions required to resolve issues, providing an axis for comparing task complexity.
- User personas: Two user personas represent different levels of technical expertise and comfort with technology.
- User personas: The Easy persona describes a user familiar with the domain, while the Hard persona represents a user with low technical knowledge.
- User personas: Easy users prefer clear guidance and provide detailed feedback, whereas Hard users may become overwhelmed, share information selectively, and need reassurance.
A.2 Example Task
The example telecom task tests resolving a No Service issue through user actions and an environment assertion that the phone becomes connected.
- Task specification: The example task tests a service_issue involving airplane mode being on and an unseated SIM card.
- Task specification: The user’s objective is to resolve a phone showing No Service and consider the issue fixed when the status bar shows signal.
- User scenario: The user follows agent guidance, expresses mild frustration after the first unsuccessful attempt, and may need another status-checking tool call.
- Initial state: The task initializes John Smith’s user information and turns on airplane mode while unseating the SIM card.
- Evaluation criteria: The user can toggle airplane mode and reseat the SIM card as specified actions.
- Evaluation criteria: Success requires the user-side assert_service_status check to return expected_status connected with assert value true.
A.3 Example Trajectory 1: Default mode
The default-mode trajectory shows the agent authenticating the customer, diagnosing a no-service problem, and guiding the user through corrective phone actions until connectivity is restored.
- Customer authentication: The agent first requests identifying information and retrieves the customer record using the provided phone number.The user provides a phone number and name; the system then calls get_customer_by_phone.
- Diagnosis: Network checks reveal Airplane Mode ON, an invalid SIM card, and no cellular service or signal.The returned status also shows Wi-Fi OFF and no connected network.
- Troubleshooting: The agent instructs the user to turn Airplane Mode OFF and then checks the resulting no-signal state.The user performs the toggle, but the phone still shows no signal.
- Troubleshooting: After the user checks the SIM and reports it is invalid or unrecognized, the agent instructs them to reseat it.The user performs the requested action through the reseat_sim_card tool.
- Resolution: Reseating the SIM restores full signal strength, 5G connectivity, and enabled mobile data, resolving the issue.The user confirms that the status bar now shows full signal and that the problem is resolved.
B.1 Verifying Implementation
Implementation verification standardizes tool interfaces and domain data models while adding a mock domain for isolated unit testing.
- Implementation verification: A unified tool formalism represents agent capabilities consistently across domains.Each domain environment also specifies its own data model.
- Implementation verification: A mock domain enables isolated verification of core benchmark functionalities.
B.2 Verifying Tasks
Task verification combines structured specifications, categorized assertions, iterative simulation review, and programmatic generation with automatic verification.
- Task specification: Task descriptions include capability metadata, structured user instructions, and initialization options for controlling starting states.The instruction dimensions include intent, specific instructions, and known or unknown information.
- Task evaluation: Evaluation criteria separately check database state, communication content, natural-language conditions, and required actions.
- Task review: Each task undergoes simulation-based review to identify underspecification, overspecification, and non-unique solutions before instructions are refined.Reviewers may intervene when transient agent or simulator errors would otherwise halt a simulation prematurely.
- Task generation: Programmatic task generation with automatic verification is used for the newly introduced domain.
- User simulation: User simulations follow scenario instructions, alternate between messages and tool calls, and ground responses in requested tool results.The simulator can terminate with STOP, TRANSFER, or OUT-OF-SCOPE tokens according to task conditions.
- Task examples: The task examples specify domain goals, known and unknown information, conditional actions, and completion criteria.Examples cover airline booking, retail order handling, and telecom troubleshooting scenarios.
E.1 Common Error Types and Failure Modes (retail)
Retail simulator errors include turn-taking violations, premature termination, ungrounded references, and missing constraints, with critical failures concentrated in the latter two task-blocking categories.
- Error types: 11/20 annotated retail errors are conversation-structure rule violations, such as mixing tool calls with natural language in one turn.
- Error types: 3/20 errors involve premature termination after user confirmation, preventing the agent from completing the transaction.
- Error types: 2/20 errors are ungrounded references, while 4/20 are missing constraints that omit required instructions.
- Error impact: Premature termination and missing constraints account for most task-critical errors, whereas structural violations and ungrounded references are usually recoverable.
E.2 Common Error Types and Failure Modes (airline)
Annotated airline dialogues show recurring simulator errors, with missing constraints identified as the main task-critical failure. The section also includes troubleshooting workflows for service and mobile-data issues.
- Missing constraint (11/47), conversation-structure rule violation (19/47), ungrounded reference (15/47), and premature termination (2/47) are reported among annotated airline errors.
- Most task-critical airline errors stem from missing constraints.
- The section presents troubleshooting workflows for service_issue and mobile_data_issue.
E.3 Common Error Types and Failure Modes (telecom)
Manual analysis of eight annotated telecom errors finds premature termination in every case. The simulator returns the transfer token before the assistant performs the transfer tool call.
- Premature termination accounts for all annotated telecom errors (8/8, 100%).
- The simulator returns ###TRANSFER### before the assistant calls the transfer tool.
- The error is critical only when premature termination prevents the expected transfer and causes task failure.
E.4 Illustrative Cases
Illustrative cases show task-critical failures across retail, airline, and telecom settings. In each case, simulator behavior prevents the assistant from completing an expected action or satisfying a task constraint.
- Retail: In the retail case, the simulator terminates after confirmation, preventing order modification and receipt issuance.
- Airline: In the airline case, the simulator books the return trip instead of requesting human transfer when the outbound trip is unavailable.
- Telecom: In the telecom case, the simulator emits ###TRANSFER### before the agent performs the actual transfer.