PaperlayerView exact revision on arXiv

Source-linked AI summary

ToolRACER: A Robust Agentic Conversation Emulation Resource for Agent Training and Evaluation

Arkajyoti Chakraborty, Aryan Tayal, Ishika Agarwal, Tanner Sorensen, Justin Chiu, Alessandro Di Bari, Neha Gupta, Andreas Stolcke

arXiv:2610.09163v1cs.CLcs.AI

TL;DR

Task-oriented agents face interactions that depart from predictable scripts, while existing benchmarks underrepresent adversarial trajectories. ToolRACER generates validated multi-turn conversations with user, assistant, and tool emulation, and ToolRACERBench-trained models perform better than agents trained with current state-of-the-art benchmarks across function-calling evaluations.

  • Problem

    Task-oriented dialogue systems can struggle with deviations from expected user behavior, and robust agent training needs diverse multi-turn trajectories.

    Source: §1 · p. 1

  • Method

    ToolRACER generates synthetic multi-turn agent traces through user, assistant, and tool simulation, evaluator validation, and refinement, including adversarial trajectory types.

    Source: §1 · p. 2

  • Results

    ToolRACERBench-trained agents perform better than agents trained with current state-of-the-art benchmarks, with evaluation across function-calling benchmarks.

    Source: §5 · p. 9

  • Takeaways & Limitations

    ToolRACERBench provides a training and testing resource spanning diverse personas and Happy, Unhappy, and Impossible interactions.

    Source: §5 · p. 9

Abstract

from arXiv · show

Task-oriented conversational agents remain fragile under real world conversation scenarios as they rarely follow a predictable script, especially when users exhibit non-cooperative behavior. Existing function-calling benchmarks often emphasize successful, cooperative interactions and underrepresent adversarial conversation trajectories, thereby limiting the training resources available for developing robust agents. We present ToolRACER, a synthetic data generation pipeline that coordinates user, assistant and tool emulation models to generate and validated multi-turn interactions between a user and an agent. Using \sysn, we construct ToolRACERBench a robust multi-turn conversation benchmark spanning six domains, ranging over 55 varied personas, generating a validated corpus of 5.6K conversation trajectories, with approximately 66\% of conversations containing failure-prone conversation scenarios. We inject adversarial behaviors, producing validated conversational interaction trajectories that capture realistic, robust scenarios. We evaluate models trained on ToolRACERBench against internal benchmarks, as well as on function calling benchmarks such as $\tau^2$-bench, BFCLv3 and ACEBench to evaluate agentic accuracy and robustness. Models trained on ToolRACERBench improve end to end agentic accuracy across $\tau^2$-bench and ACEBench, demonstrating significant gains when mixed with in-domain dataset in small language models for agent capability tasks.

1 INTRODUCTION

Task-oriented dialogue systems can struggle with deviations from expected user behavior, while robust agent training needs diverse, multi-turn trajectories. ToolRACER introduces a validated synthetic-data pipeline and a taxonomy of Happy, Unhappy, and Impossible interactions.

1 INTRODUCTION

  • ToolRACER generates multi-turn agent traces through user, assistant, and tool simulation, multi-stage evaluation, and refinement of failed conversations.The resulting ToolRACERBench is intended for training and testing tool-calling conversational agents.Source: §1 · p. 2
  • The pipeline leverages tool-execution feedback and evaluator rubrics to generate validated, diverse synthetic conversations.Source: §1 · p. 2
  • Happy, Unhappy, and Impossible trajectories distinguish non-adversarial achievable tasks, adversarial achievable tasks, and non-achievable tasks.Source: §1 · p. 2
  • The introduction frames evaluation against APIGen-MT and cross-domain function-calling benchmarks, with training-data mixture ablations on ToolRACERBench-test.Source: §1 · pp. 2–3

2 RELATED WORK

Related work covers function-calling benchmarks, robustness benchmarks, and persona-driven approaches. The authors identify user-centric robustness as a remaining gap in prior benchmark coverage.

Function-calling benchmarks

  • Earlier function-calling benchmarks added long-context conversations and multi-agent or shared-environment settings, but did not fully capture unpredictable real-world interactions.Source: §2 · p. 3

Robust Function-Calling Agents

  • Robustness benchmarks add irrelevant functions, ambiguous or incomplete instructions, or goal changes, while the authors identify user-centric robustness as still lacking.Source: §2 · p. 3

Trajectory-Driven Robustness

  • Prior persona-based work diversifies evaluation or training, and ToolRACER likewise uses diverse personas and situations for tool-use scenarios.Source: §2 · p. 3

3 TOOLRACER: METHODOLOGY FOR ROBUST SYNTHETIC MULTI-TURN DATA GENERATION

ToolRACER formulates conversations as multi-turn exchanges of user utterances, assistant actions, and tool observations. Its methodology simulates adversarial interactions, validates generated traces, and retries failed conversations using evaluator feedback.

3.1 PROBLEM FORMULATION

  • Each turn comprises a user utterance, assistant action, and possible tool-call observations, continuing until completion, termination, or the interaction budget is exhausted.Source: §3.1 · p. 3

3.2 OVERVIEW

  • ToolRACER injects user and tool adversity into trajectories, validates generated interactions, and uses failure reasons to guide refinement; the loop recovers 65% of failed conversations.Source: §3.2 · p. 4, Figure 2 · p. 4

3.2.1 SCENARIO GENERATION AND TOOL DEFINITION

  • Scenarios specify task objectives, success criteria, and user/tool seed data; each draws from 55 personas and injects adversarial user and tool behaviors.Source: §3.2.1 · p. 4

3.2.2 ROBUST MULTI-AGENT DIALOG SIMULATION

  • Three agents simulate dialogue: the assistant selects tool calls or utterances, the user responds from a persona, and the tool returns observations that may model system failures.Source: §3.2.2 · pp. 4–5

3.2.3 MULTI-STAGE EVALUATION

  • Generated conversations enter the dataset only after passing checks for syntax, faithfulness, role confusion, and task success.Source: §3.2.3 · p. 5

3.2.4 REFINEMENT LOOP

  • Failed conversations are retried with refinement hints derived from judge failure reasons, at most twice, and are discarded if they still fail validation.Source: §3.2.4 · p. 6

3.3 TAXONOMY OF ROBUST TRAJECTORIES

ToolRACERBench classifies trajectories by path type and includes adversarial or failure-prone interactions; its quality assessment reports higher scores on five of six automatic evaluations than APIGen-MT.

3.3 TAXONOMY OF ROBUST TRAJECTORIES

  • Approximately 66% of the final dataset contains failure-prone conversations classified as Happy, Unhappy, or Impossible paths.Source: §3.3 · p. 6
  • The taxonomy distinguishes solvable Happy paths, adversarial but achievable Unhappy paths, and Impossible paths with non-achievable tasks.Happy paths use function-call chains toward a single objective; Unhappy paths stress planning and clarification, while Impossible paths include tool failures or conditions requiring recovery or refusal.Source: §3.3 · p. 6
  • Unhappy-path behaviors include incomplete information, constraints revealed over multiple turns, goal revisions, and adversarial attitudes that challenge assistant suggestions.Source: §3.3 · p. 6
  • Impossible-path scenarios include backend errors, empty results, timeouts, and policy constraints that require recovery or principled refusal.Source: §3.3 · pp. 6–7

3.4 DATA QUALITY ASSESSMENT

  • ToolRACERBench scores higher on five of six automatic dataset-quality evaluations than APIGen-MT, while using fewer turns and tool calls per dialogue.The authors report a lower entailment rate, which they suggest could reflect users changing goals or tools failing and requiring plan revisions.Source: Table 2 · p. 7, §3.4 · p. 7

4 EXPERIMENT

The experiments evaluate ToolRACERBench-trained models on cross-domain function-calling benchmarks and ToolRACERBench-Test. Results report gains in τ²-bench and ACEBench evaluations, alongside a process-accuracy drop on longer multi-turn ACEBench tasks.

4.1 EXPERIMENTAL SETUP

  • Training uses 3.7K ToolRACERBench trajectories, splits them at assistant responses, and trains models to predict assistant responses using conversation history.The setup also reports LoRA rank 32, learning rate 1e-4, and batch size 32; models include Qwen3-4b-Inst-2507 and other open-source function-calling models.Source: §4.1 · p. 7

4.2 RESULTS AND ANALYSIS

  • 39.20% τ²-bench macro average is reported for ToolRACERBench fine-tuning alone, with gains in both domains and performance above the base model and APIGen-MT checkpoint.The authors also report a 52.5% gain when combining ToolRACERBench with the in-domain dataset.Source: Table 3 · p. 8, §4.2 · p. 9

4.1 EXPERIMENTAL SETUP

  • Evaluation covers τ²-bench retail and airline, ACEBench's 50-task agent category, BFCLv3, and turn-level action accuracy across Happy, Unhappy, and Impossible paths.Turn-level evaluation uses a 1.8K-trajectory test set; function calls are evaluated with F1 and responses with semantic similarity.Source: §4.1 · p. 8

4.2 RESULTS AND ANALYSIS

  • Over 10% higher action accuracy follows training on Unhappy and Impossible paths, while Impossible-path training excels on the hallucination category.The authors also report that the APIGen-MT checkpoint's F1 score drops in this evaluation.Source: §4.2 · pp. 8–9, Table 4 · p. 9
  • ACEBench evaluation shows improved end-to-end accuracy over the base model and contemporary open-source models, but lower process accuracy, especially on multi-turn tasks over six turns.Table 5 reports ACEBench agent performance averaged across multi-step and multi-turn tasks.Source: Table 5 · p. 9, §4.2 · p. 9

5 CONCLUSION

The conclusion presents ToolRACER as a synthetic data pipeline and ToolRACERBench as a training and testing benchmark for robust task-oriented conversational agents. It reports performance above current training benchmarks and generalization across function-calling benchmarks.

5 CONCLUSION

  • ToolRACER is a synthetic data generation pipeline for training robust task-oriented conversational agents.The authors also release ToolRACERBench as a training and testing benchmark.Source: §5 · p. 9
  • ToolRACERBench covers diverse user personas and Happy, Unhappy, and Impossible interactions, distinguishing ideal achievable tasks from non-ideal or unachievable ones.The conclusion describes Happy users as ideal, Unhappy users as non-ideal with achievable tasks, and Impossible tasks as unachievable.Source: §5 · p. 9
  • The authors report that ToolRACERBench-trained agents outperform agents trained with current state-of-the-art training benchmarks and generalize across function-calling benchmarks.Source: §5 · p. 9

A TOOLRACER EFFICIENCY

The efficiency results describe low first-attempt validation success and recovery through refinement. Recovery occurs across all three conversation types, though Unhappy paths are hardest to repair.

A TOOLRACER EFFICIENCY

  • 26.3% of simulations pass validation checks without feedback, showing the difficulty of producing valid conversations in one attempt.Source: A TOOLRACER EFFICIENCY · p. 11
  • 65.4% of initially failed conversations are recovered by the refinement loop using evaluator feedback to repair rejected trajectories.Source: A TOOLRACER EFFICIENCY · pp. 11–12
  • Recovery is observed across Happy, Unhappy, and Impossible conversation types, although Unhappy paths are the most difficult to repair.The pipeline efficiency table measures success rate.Source: A TOOLRACER EFFICIENCY · p. 12, Table 6 · p. 12

B PROMPTS

The appendix identifies prompts used to initialize the tool and user agents and evaluation judges. The user-agent prompt is split across two figures.

B PROMPTS

C SCENARIO CONFIGURATIONS

The simulator uses scenario configurations to generate varied failure-prone conversations, with metadata for Happy, Unhappy, and Impossible trajectories.

D TOOL DEFINITION

The section presents tools from the home services domain.

E PERSONA CATALOG

The section gives an example of a persona used in ToolRACERBench.

Loading 2610.09163v1…