APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay

1 Introduction

• Robust agents must converse across turns, infer intent, gather missing details, invoke APIs accurately, and follow domain policies, but high-quality interaction data is scarce and costly to collect.These demands arise in practical tasks such as financial transactions, appointment scheduling, and customer service.Source: §1 · pp. 1–2
• Prior work addressed single-turn function calling [26] and agent training [41], while multi-turn efforts [51] [50] [11] lacked human-agent interplay and verified, diverse grounded trajectories.The paper identifies verification and synthesis of such multi-turn data as an unresolved barrier to agent capability.Source: §1 · p. 2
• APIGen-MT first creates verified task blueprints with grounded actions, then uses simulated human-agent interplay to generate multi-turn trajectories with dialogue, tool use, and environment feedback.Blueprints draw on sampled APIs, policies, domain data, and personas; validation combines format and execution checks with an LLM committee review.Source: §1 · p. 2
• APIGen-MT-trained xLAM-2-fc-r models across Llama 3.1/3.2 and Qwen 2.5 (1B–70B) outperform many frontier models, including GPT-4o, on τ-bench and BFCL.Figure 1 compares larger xLAM-2-fc-r models (8B–70B) against baselines on BFCL v3 and τ-bench.Source: §1 · p. 2, Figure 1 · p. 2
• The authors open-source 5K synthetic APIGen-MT-5k trajectories and the trained xLAM-2-fc-r models to advance agent research.The released models span Llama 3.1/3.2 and Qwen 2.5 architectures, from 1B to 70B parameters.Source: §1 · p. 2

Tool-Use Agents

• Tool-use research spans reusable tools, workflow composition, and specialized agent training methods, while many function-calling frameworks remain limited to predefined tools [26] [43].Examples include ToolMaker’s use of code-repository tools [44], critique-informed planning [13], selective-step fine-tuning [48], Agent-FLAN [12], and ToRL.Source: §2 · p. 3

Interactive Conversational Benchmarks

• MultiChallenge [40], ToolDial [38], InterCode [47], CRMArena [19], and ToolSandbox [28] evaluate context maintenance, tool-augmented dialogue, iterative problem-solving, customer management, or stateful tool use.User simulations support systematic, realistic interactions [49] [28] [31]; APIGen-MT complements these benchmarks with synthetic multi-turn conversations for training and evaluation.Source: §2 · p. 3

Synthetic Data Generation

• Synthetic-data methods include simulated agent interactions [37, 29, 42, 20], instruction composition [18] [11], intermediate graphs [6], and multi-turn planning [55].BUTTON [11] combines 2–3 atomic tasks but lacks systematic quality control and filtering during task composition; MAGNET [50] transforms generated function-signature paths into queries and calls.Source: §2 · p. 3
• APIGen-MT applies across existing environments, generating domain-constrained tasks and executable ground-truth calls through iterative feedback, then simulated agent-human interplay produces verifiable long trajectories.The framework focuses on reliable tool selection and parameter generation, addressing prior work’s emphasis on reasoning or single-turn scenarios.Source: §2 · p. 3

3.1 Multi-Turn Interaction Problem Formulation

• The interaction is modeled as a POMDP in which the assistant chooses tool calls or human-facing responses from history while the environment state remains latent.Interaction ends with a user termination or turn limit, and reward depends on cumulative environment-state change and the assistant’s responses.Source: §3.1 · pp. 3–4

3.2 APIGen-MT Framework Overview

• APIGen-MT extends APIGen [26] with a two-phase pipeline that builds validated task blueprints before simulating realistic multi-turn interactions.The separation addresses the difficulty of one-shot conversation synthesis, where an intermediate error can derail a task and turns depend on prior calls and outputs.Source: §3.2 · p. 4, Figure 2 · p. 4

3.2.1 Phase 1: Task Configuration and Groundtruth Generation

• Phase 1 creates each blueprint’s instruction, groundtruth actions, and expected outputs, then iteratively applies execution checks, multi-reviewer majority voting, and feedback-driven refinement.Automated checks verify action formats and executability in the simulated environment; reviewers assess coherence, completeness, and task sensibility.Source: §3.2.1 · pp. 4–5

3.2.2 Phase 2: Human-Agent-Environment Interaction Trajectory Collection

• Phase 2 simulates an unaware human user and test agent in an executable environment, then accepts trajectories only when state-based and output-based checks verify task completion.The resulting trajectories capture dialogue, agent actions, and environment responses, combining natural turn-by-turn dynamics with solutions grounded in validated blueprints.Source: §3.2.2 · p. 5

4.1.1 API Dependency Graph and Context Samplers

• APIs are modeled as a dependency graph for random-walk task sequences, while domain-specific samplers ground generation in APIs, policies, data, personas, and examples.The API sampler distinguishes read and write APIs, prioritizing necessary write actions for ground truth while leaving read API use flexible during interaction.Source: §4.1.1 · p. 6

4.1.2 Multi-Stage Validation for τ-bench

• Three-stage validation checks task format, executable actions and policy compliance, then uses committee judgments for semantic alignment and feedback-driven refinement.The committee scores correctness, completeness, satisfaction, and creativity; majority voting determines assessments, and failing tasks are revised using consolidated feedback.Source: §4.1.2 · p. 7

4.1.3 Reverse Task Recombination for Complex Task Construction

• Reverse Task Recombination builds longer tasks from validated simpler tasks, then rechecks policy compliance and validates the combined instruction’s semantic alignment.It concatenates actions and outputs, synthesizes an overarching instruction, and skips Stage 1 for the combined task because constituent actions were already checked; it addresses the difficulty of directly generating complex long-horizon tasks.Source: §4.1.3 · pp. 7–8

4.2 Phase 2: Simulated Human-Agent Interplay and Trajectory Collection

• An LLM-simulated human incrementally reveals persona-guided task details, and rejection sampling retains only successful trajectories based on final state and response checks.Each task is attempted up to three times, with unique successful trajectories collected for downstream use; Best-of-N (N=4) sampling and self-critique help keep the simulated human faithful to the task across turns.Source: §4.2 · p. 8

4.3 Data Collection & Statistics

• 2.5x boost in task-collection success rate yielded 70%, while collected trajectories required gpt-4o to take an average of 12 turns to complete tasks.The collection pipeline used an agentic review committee and iterative refinement; the turn count describes trajectories generated using APIGen-MT.Source: §4.3 · p. 8, Figure 4 · p. 9, Figure 5 · p. 9

5.1 Experimental Setup

• Training uses filtered behavioral cloning on assistant responses, jointly adds function-calling and other agent data, and evaluates on BFCL v3 and τ-bench.The models are full-finetuned for at most three epochs.Source: §5.1 · p. 9

5.2 Experiment Results

• 78.19% overall accuracy for xLAM-2-70b-fc-r and 75.83% for xLAM-2-32b-fc-r place them first and second on BFCL v3, surpassing all proprietary and open-source models.xLAM-2-70b-fc-r also achieves 75.12% multi-turn accuracy; xLAM-2-8b-fc-r scores 69.25%, xLAM-2-3b-fc-r 56.00%, and xLAM-2-1b-fc-r 43.12%, all outperforming o1 (36%) and gpt-4o in function-calling mode (41%).Source: §5.2 · p. 9, Table 1 · p. 10
• 56.2% success rate for xLAM-2-70b-fc-r on τ-bench exceeds GPT-4o (52.9%), DeepSeek v3 (40.6%), and Llama 3.1 70B Instruct (38.2%), while approaching Claude 3.5 Sonnet (60.1%).The smaller xLAM-2-32b-fc-r (54.6%) and xLAM-2-8b-fc-r (46.7%) variants also surpass larger baselines.Source: §5.2 · p. 10, Table 2 · p. 10

5.3 Consistency & Stability Experiments

• On τ-bench, xLAM-2-70b-fc-r has a higher pass^5 score than Claude on the airline domain despite a slightly lower pass^1 score, indicating greater consistency across trials.The pass^k curves measure the chance that all k independent task trials succeed, averaged across tasks.Source: §5.3 · p. 11, Figure 6 · p. 11
• BoN user simulation improves success rate and reduces variance across five τ-bench retail trials for gpt-4o and xLAM-2-70b-fc-r.The paper attributes these gains to a self-critiquing mechanism in the user simulation, which can improve both stability and agent performance.Source: §5.3 · p. 11, Table 3 · p. 11

5.4 In-Depth Analysis of Model Behavior

• For long τ-bench tasks, xLAM-2-70b-fc-r succeeds more often than gpt-4o but less often than Claude 3.5 Sonnet (new); its interaction count is at par with gpt-4o and higher than Claude’s.The paper links xLAM-2-70b-fc-r’s extra interactions to retrieving user details in stages.Source: §5.4 · pp. 11–12, Figure 7 · p. 11

Conclusion

• APIGen-MT separates detailed blueprint creation from dialogue simulation, producing structurally correct and natural trajectories whose models outperform baselines on τ-bench and BFCL v3, including smaller models in multi-turn settings.Stabilization techniques improve behavioral consistency and reliability; the authors open-source the synthetic data and trained models.Source: §6 · p. 12

Limitations and future directions

• Some stochasticity remains in simulated human behavior, and failed second-phase trajectories are discarded despite their potential value as contrastive training signals.The authors suggest more deterministic simulation or refined filtering, and leveraging failed trajectories in model training.Source: §6 · p. 12
• Multi-stage validation incurs computational overhead, while extending APIGen-MT to more domains and adding reinforcement-learning self-improvement remain future directions.The authors suggest more efficient validation or adaptive sampling to improve scalability.Source: §6 · p. 12

• BFCL v3 evaluates single-turn, multi-turn, and multi-step function calling through request understanding, function selection, valid parameters, and output interpretation.It reports overall accuracy as a weighted average across evaluation categories.Source: A Benchmarks Description · p. 16
• τ-bench tests agents’ ability to interact with simulated human users and APIs across dynamic, multi-domain conversations while following domain-specific policies.Its retail and airline customer-service settings require agents to maintain context, understand intent, and follow complex rules.Source: A Benchmarks Description · p. 16

• The τ-bench prompts cover task configuration generation, alignment validation, final semantic review, trajectory collection, and stabilized human simulation.Source: B Prompts · p. 16, Figure 8 · p. 16, Figure 9 · p. 17
• For alignment validation, each review-committee LM scores the prompt, and the pipeline uses majority voting.Source: Figure 9 · p. 17
• The stabilized human simulation prompt uses a Best-of-N (BoN) User LM setting for τ-bench’s retail domain.Source: Figure 12 · p. 18