PaperlayerView exact revision on arXiv

Source-linked AI summary

Trajectory2Task: Training Robust Tool-Calling Agents with Synthesized Yet Verifiable Data for Complex User Intents

Ziyi Wang, Yuxuan Lu, Yimeng Zhang, Pei Chen, Ziwei Dong, Jing Huang, Jiri Gesi, Xianfeng Tang, Chen Luo, Qun Liu, Yisi Sang, Hanqing Lu, Manling Li, Jin Lai, Dakuo Wang

arXiv:2601.20144v3cs.CL

TL;DR

Tool-calling research lacks realistic, verifiable data for ambiguous, changing, and infeasible user intents. Trajectory2Task generates such data from executed trajectories, benchmarks seven LLMs, and finds that trajectory-based SFT improves robustness across scenarios and transfers to unseen domains.

  • Problem

    Existing benchmarks and training data underrepresent ambiguous, changing, and infeasible multi-turn user intents, limiting systematic study of robust tool-calling behavior.

  • Method

    Trajectory2Task explores executable tool-use trajectories, converts filtered trajectories into user-facing tasks with controlled intent adaptations, and validates task–trajectory pairs.

  • Results

    Trajectory-based SFT with fewer than 3k successful trajectories improves robustness and stability in complex scenarios and generalizes to unseen domains.

  • Takeaways & Limitations

    The results suggest that trajectory-based training improves general tool-use decision behavior beyond narrow task memorization.

  • Takeaways & Limitations

    The framework relies on an LLM-driven user simulator and is instantiated only on TAU2-BENCH, so it does not exhaustively represent real user behavior or tool ecosystems.

Abstract

from arXiv · show

Tool-calling agents are increasingly deployed in real-world customer-facing workflows. Yet most studies on tool-calling agents focus on idealized settings with general, fixed, and well-specified tasks. In real-world applications, user requests are often (1) ambiguous, (2) changing over time, or (3) infeasible due to policy constraints, and training and evaluation data that cover these diverse, complex interaction patterns remain under-represented. To bridge the gap, we present Trajectory2Task, a verifiable data generation pipeline for studying tool use at scale under three realistic user scenarios: ambiguous intent, changing intent, and infeasible intents. The pipeline first conducts multi-turn exploration to produce valid tool-call trajectories. It then converts these trajectories into user-facing tasks with controlled intent adaptations. This process yields verifiable task that support closed-loop evaluation and training. We benchmark seven state-of-the-art LLMs on the generated complex user scenario tasks and observe frequent failures. Finally, using successful trajectories obtained from task rollouts, we fine-tune lightweight LLMs and find consistent improvements across all three conditions, along with better generalization to unseen tool-use domains, indicating stronger tool-calling ability.

1 Introduction

Real-world tool-calling must handle ambiguous, changing, and infeasible user goals, yet existing benchmarks and training data provide limited coverage of these interactions. Trajectory2Task addresses this gap with verifiable trajectories and tasks, benchmarks model failures, and uses trajectory-based SFT to improve performance and transfer.

  • Motivation: Real-world user intents can omit critical details, evolve during interaction, or violate system constraints and policy.These conditions require agents to reason under partial information, ask clarifying questions, revise plans, and handle unsupported requests.
  • Motivation: Existing benchmarks mainly assume stable goals and assess static tool-calling success, leaving complex real-world scenarios systematically under-investigated.The gap concerns ambiguity, intent changes, and infeasible requests in multi-turn interactions.
  • Approach: Trajectory2Task generates verifiable complex-scenario tasks by first executing exploratory tool-use trajectories, then adapting them into user-facing tasks with reachable states and ground-truth solutions.The pipeline supports controlled ambiguous, changing, and infeasible intents and automatic validation.
  • Evaluation and training: Benchmarking seven strong LLMs reveals substantial limitations in dynamic scenarios, while successful trajectories expose how agents should ask, adapt, call tools, and handle infeasible requests.The generated trajectories include action sequences and accompanying reasoning tokens for downstream training.
  • Evaluation and training: Trajectory-based SFT improves complex-intent tool calling and transfers gains to unseen domains, suggesting broader decision behavior rather than narrow task memorization.The approach trains agents on successful trajectories that demonstrate when to ask, adapt plans, and act.

2 Related Works

Prior tool-use benchmarks and synthetic datasets improve tool selection and execution but usually assume stationary, cooperative goals. Trajectory2Task extends synthetic generation toward controlled ambiguity, intent drift, infeasibility, executable traces, and verification.

  • Benchmarks: Traditional tool-use benchmarks evaluate tool selection, argument filling, and sometimes multi-step plans, but mostly assume a fixed user goal.They provide limited assessment of goal updates during dialogue.
  • Benchmarks: τ-bench and UserBench move toward multi-turn, user-facing evaluation, yet do not jointly control ambiguity, intent drift, and infeasibility as separate stress factors.These limitations leave systematic investigation of the three complex scenarios incomplete.
  • Training resources: Existing tool-use resources include synthetic instruction data, curated dialogues, runnable environments, and self-supervised signals for tool-calling decisions.Examples include ToolBench, APIBank, and Toolformer.
  • Synthetic data generation: Synthetic tool-use frameworks increasingly execute tools and filter trajectories, but many generated dialogues remain behaviorally simplified with cooperative users and largely stationary goals.Complex real-world interaction patterns remain underrepresented.
  • Synthetic data generation: Trajectory2Task constructs tasks from executable trajectories, adapts user intent across three scenarios, and pairs each task with traces that enable automatic verification.The resulting data support scalable training and evaluation in challenging tool-use settings.

3 Methods

The paper models complex tool use as decision-making under partial observability and non-stationary user intent, then introduces Trajectory2Task to generate executable, verifiable tasks from explored trajectories. It instantiates controlled ambiguous, changing, and infeasible scenarios while retaining executable traces as verification backbones for training and evaluation.

  • Problem Formulation: The interaction is modeled as a POMDP with latent user intent, environment state, agent actions, and observations that update as intent and tool context evolve.The agent chooses actions from an internal belief state, receives environment feedback, and may observe user-intent drift after its outward message.
  • Trajectory2Task: Trajectory2Task reverses the usual task-first process by exploring executable tool-call trajectories before deriving user-facing tasks.This avoids making solver-generated trajectories the ground truth and pairs each valid task with an executable gold trajectory and verified final state or required output.
  • General Synthesis Pipeline: The synthesis pipeline alternates trajectory exploration and task summarization, using seed trajectories, API-DAG tool sampling, filtering, and validity checks for realism, necessity, and correctness.Explorers produce complete trajectories with dependent calls; summarizers convert them into naturally reachable requests, and validity checks assess whether the task and trace align.
  • Complex User Scenarios: Three controlled scenarios stress missing information, mid-dialogue goal changes or additional requests, and infeasible user intents while preserving executable trajectories for verification.Ambiguous tasks reveal omitted information only when explicitly requested; changing tasks require an intent shift or multiple issues, and scenario-specific checks enforce these properties.
  • Trajectory-based Supervised Fine-Tuning: Only automatically verified trajectories are retained for supervised training, with each output containing reasoning tokens and an outward natural-language response or tool call.This training representation exposes how agents should ask, adapt, select tools, and handle infeasible requests.

4 Experiments

Experiments construct and evaluate Retail-3I using exploratory tool trajectories, then test seven LLMs and trajectory-based SFT on complex user-intent scenarios. Existing models degrade notably under ambiguous, changing, and infeasible intents, while fewer than 3k successful trajectories improve smaller models and transfer across domains.

  • Data generation: The benchmark collects exploratory trajectories, filters invalid rollouts, and constructs scenario tasks with verified successful trajectories for evaluation and training.Exploratory rollouts average 12 steps and are capped at 20 steps; the final benchmark contains 1,099 synthesized complex-scenario tasks.
  • Benchmark results: Seven state-of-the-art LLMs perform best in GENERAL, where Claude models reach 0.79–0.81 Pass^1.GENERAL removes the initial user constraint from the ambiguous-intent setup.
  • Benchmark results: Every model loses performance under AMBIGUOUS, with smaller models degrading more sharply than larger models when constraints are missing.Qwen3-4B falls from 0.477 to 0.259 Pass^1, while Claude-3.7 falls from 0.794 to 0.767.
  • Benchmark results: CHANGING reaches only 0.675 Pass^1 at best, while INFEASIBLE falls to ≤0.571 Pass^1 and ≤0.448 Pass^3 even for the strongest models.Invalid cancellation attempts despite order-status policies illustrate imperfect constraint awareness.
  • Trajectory-based SFT: Trajectory-based SFT on 2,872 generated trajectories yields large gains across all complex-intent scenarios, especially CHANGING and INFEASIBLE.With fewer than 3k trajectories, the trained 4B model reaches performance comparable to or exceeding larger baseline models.
  • Trajectory-based SFT: SFT improvements transfer beyond retail to AIRLINE despite training only on retail tools, indicating learned general tool-calling behaviors rather than domain memorization.For example, Qwen3-8B improves from 0.207 to 0.287 Pass^1 on AIRLINE.

5 Conclusions

Trajectory2Task provides a general, verifiable pipeline for generating tasks and executable trajectories under ambiguous, changing, and infeasible intents. Experiments show substantial degradation in dynamic scenarios, while trajectory-based SFT improves lightweight models’ robustness, stability, and transfer to unseen domains.

  • Conclusions: Trajectory2Task converts executed tool trajectories into user-facing tasks with controlled intent adaptations and automatic validation for closed-loop evaluation and training.The pipeline targets ambiguous, changing, and infeasible user intents.
  • Conclusions: Frontier models degrade substantially in dynamic scenarios, with lightweight models especially vulnerable.The conclusion summarizes benchmark findings across the three complex user-intent conditions.
  • Conclusions: Training on fewer than 3k generated successful trajectories improves robustness and stability and generalizes to domains unseen during training.The paper interprets these gains as evidence of improved general tool-use decision behavior.

Limitations

The study relies on an LLM-driven user simulator and instantiates Trajectory2Task only on TAU2-BENCH, limiting behavioral realism and ecosystem coverage.

  • Simulator realism: The LLM-driven user simulator may not reflect real distributions of long-term preferences, adversarial attempts, or nuanced social dynamics.Ambiguity, intent drift, and infeasibility are modeled through prompting rather than direct observation of human conversation logs.
  • Scope: The pipeline is instantiated only on TAU2-BENCH as a case study rather than exhaustive coverage of real-world tool ecosystems.The authors propose extending it to richer user state, longer-horizon memory, flexible interactions, human-grounded validation, and other domains.

B Additional Results and Error Analysis

Fine-tuned models consistently outperform their baselines across complex user scenarios, while failure analysis shows improved execution but persistent challenges in interpreting ambiguous or changing intent.

  • Pass@k Results: Fine-tuned models consistently outperform their baseline across multiple complex user scenarios in Pass@k results.The reported pattern covers representative models and Ambiguous, Changing, and Infeasible scenarios.
  • Reward Breakdown: Fine-tuning reduces both database-related and communication-related failures in Ambiguous and Changing scenarios compared with base models.
  • Failure Taxonomy: Missing required actions are the most common failure category across models and scenarios, including skipped tool calls or incomplete required steps.
  • Failure Taxonomy: Fine-tuning noticeably reduces missing-action failures, but communication failures become relatively more prominent in Ambiguous and Changing scenarios.The pattern suggests that interpreting underspecified or evolving intent remains a major challenge after basic execution improves.

C Human Validation of Generated Tasks

The generated tasks were checked with an LLM judge and a small human validation study, with 80% agreement and a slight tendency toward conservative judgments in borderline cases.

  • Validation Procedure: Task validity was initially assessed with an LLM-based judge, following common practice in synthetic data generation pipelines.
  • Validation Procedure: A human annotator reviewed 20 randomly sampled tasks across scenarios, including instructions, solution traces, and environment or tool schemas.The review judged whether evaluation criteria aligned with each task.
  • Agreement Results: 80% agreement (16/20 tasks) was observed between the human annotator and the LLM judge on overall validity labels.
  • Agreement Results: The LLM judge produced three false negatives and one false positive, suggesting reasonable reliability with slight conservatism in borderline cases.

D Data Generation Prompt

The data-generation prompts guide exploration and trajectory summarization so generated user scenarios are actionable, realistic, policy-aware, and aligned with the observed tool actions.

  • D Data Generation Prompt: The explorer agent constructs plausible customer-service scenarios and explores useful tools while making exactly one tool call per turn.The prompt also requires authentication and obtaining required IDs or parameters before dependent tool calls.
  • D Data Generation Prompt: The exploration prompt instructs the agent to retrieve missing information before dependent calls and try a different approach when needed.
  • D Data Generation Prompt: Trajectory summarization reverse-engineers a uniquely actionable user intention that would produce the same observed agent action sequence.
  • ### Requirements: Generated intents must contain enough information for exact actions while using minimal distinguishing descriptors and avoiding vague phrasing.Explicit IDs may be omitted when natural unique identifiers suffice.
  • ### Requirements: The generated JSON specifies the reason for calling, known and unknown information, task instructions, infeasibility reasons, and prohibited actions when applicable.The prompt also requires evaluating correctness and necessity from the final state and checking required communicated information.
  • ### Requirements: The prompts record known, unknown, and optional direct-communication information, including relevant user details, missing knowledge, and supported numerical results.
  • ### Requirements: Infeasible scenarios are designed to violate tool constraints or domain policies, with prompts requiring infeasibility reasons and actions the agent should or should not take.The task should reflect realistic user behavior even when the request is impossible to handle.
  • ### Requirements: Infeasible-task prompts distinguish policy and tool infeasibility, require policy-aware task specifications, and explicitly reflect every successful write action in the user intent.

# TASK DESCRIPTION

The task description presents a full agent execution trace and evaluates whether the implied final state and user-facing information are correct, necessary, and supported.

  • # TASK DESCRIPTION: The task includes the full execution trace produced by the agent during exploration.
  • # TASK DESCRIPTION: Correctness and necessity are evaluated from the final state implied by the trajectory rather than the exact action path.
  • # TASK DESCRIPTION: Required user-facing information, such as a price difference or refund amount, must be communicated and supported by the execution.

# EVALUATION TASK

The evaluation task requires judging whether a task–trajectory pair is realistic, complete, consistent with instructions, and correctly represented in a JSON response.

  • Realism: The evaluator checks whether a real user would plausibly make the request.
  • Necessity: It assesses whether the trajectory achieves required actions without missing critical steps.
  • Necessity: It checks database writes and communicated information against the task instructions.
  • Correctness: It evaluates whether the task description matches the trajectory, information flow, and tool-call arguments.
  • Output format: The evaluation output is formatted as a JSON object containing scores, explanations, issue lists, and suggestions.
  • Infeasible-task evaluation: For infeasible scenarios, the judge receives the scenario, policy, prohibited actions, required actions, communication requirements, and an exploratory trajectory.

# TASK DESCRIPTION

The task description supplies the scenario, infeasibility rationale, required and forbidden actions, communication requirements, and criteria for evaluating the associated exploratory trajectory.

  • Task inputs: The task specifies the user’s reason for calling, known and unknown information, and task instructions.
  • Infeasibility constraints: It records why the request is infeasible and which actions are forbidden under the scenario.
  • Communication requirements: Required communication must be identified and checked for task relevance and support from execution evidence.
  • Trajectory interpretation: The trajectory is an exploratory execution trace that may contain redundant or noisy actions, so evaluation focuses on the implied final state rather than the exact path.
  • Validity criteria: Validity is judged through realism, necessity, and correctness, including plausible details, required effects, permitted writes, coherent information flow, and reasonable tool arguments.
  • Changing-intent validity: Changing-intent validity additionally checks whether the trajectory follows user-intent revisions rather than treating a stable preference change as a new intent.
Loading 2601.20144v3…