Source-linked AI summary
APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay
Akshara Prabhakar, Zuxin Liu, Ming Zhu, Jianguo Zhang, Tulika Awalgaonkar, Shiyu Wang, Zhiwei Liu, Haolin Chen, Thai Hoang, Juan Carlos Niebles, Shelby Heinecke, Weiran Yao, Huan Wang, Silvio Savarese, Caiming Xiong
TL;DR
Multi-turn agent training lacks sufficient high-quality data capturing realistic human-agent dynamics, while direct conversation synthesis is vulnerable to intermediate errors and dependency inconsistencies. APIGen-MT separates verified blueprint generation from simulated human-agent trajectory collection, and models trained on its data outperform existing baselines across τ-bench and BFCL v3, including competitive results from smaller models.
Problem
High-quality multi-turn agent interaction data is scarce, and one-shot synthesis struggles with intermediate errors and dependencies across function calls and turns.
Method
APIGen-MT generates validated task blueprints with groundtruth actions, then transforms them into complete trajectories through simulated human-agent interplay and verification.
Results
Models trained on APIGen-MT data outperform existing baselines on τ-bench and BFCL v3, with smaller models showing competitive multi-turn performance.
Takeaways & Limitations
The two-phase blueprint-to-trajectory design produces synthetic data that balances structural correctness with natural conversational dynamics for agent training.
Takeaways & Limitations
Human simulation retains some stochasticity, failed trajectories are discarded, and multi-stage validation incurs computational overhead.
Abstract
from arXiv · showhide
Training effective AI agents for multi-turn interactions requires high-quality data that captures realistic human-agent dynamics, yet such data is scarce and expensive to collect manually. We introduce APIGen-MT, a two-phase framework that generates verifiable and diverse multi-turn agent data. In the first phase, our agentic pipeline produces detailed task blueprints with ground-truth actions, leveraging a committee of LLM reviewers and iterative feedback loops. These blueprints are then transformed into complete interaction trajectories through simulated human-agent interplay. We train a family of models -- the xLAM-2-fc-r series with sizes ranging from 1B to 70B parameters. Our models outperform frontier models such as GPT-4o and Claude 3.5 on $τ$-bench and BFCL benchmarks, with the smaller models surpassing their larger counterparts, particularly in multi-turn settings, while maintaining superior consistency across multiple trials. Comprehensive experiments demonstrate that our verified blueprint-to-details approach yields high-quality training data, enabling the development of more reliable, efficient, and capable agents. We open-source 5K synthetic data trajectories and the trained xLAM-2-fc-r models to advance research in AI agents. Models at https://huggingface.co/collections/Salesforce/xlam-2-67ef5be12949d8dcdae354c4; Dataset at https://huggingface.co/datasets/Salesforce/APIGen-MT-5k and Website at https://apigen-mt.github.io
1 Introduction
APIGen-MT addresses the scarcity and limitations of realistic multi-turn agent data by generating verified task blueprints and simulated human-agent trajectories. The resulting models are reported to outperform strong baselines across agentic benchmarks, with 5K trajectories and trained models released.
- Robust agents require realistic multi-turn data, but public corpora contain scarce high-quality interaction data and advanced models still struggle with complex multi-turn function calling.
- Prior approaches largely target single-turn interactions or omit realistic human-agent interplay, leaving multi-turn trajectory verification and synthesis insufficiently addressed.
- APIGen-MT generates detailed task blueprints with verifiable groundtruth actions, then converts them into realistic multi-turn trajectories through simulated human-agent interaction.
- The authors train xLAM-2-fc-r models from 1B to 70B parameters and report superior performance on τ-bench and BFCL, including results surpassing gpt-4o.
- The work releases 5K synthetic trajectories and the trained xLAM-2-fc-r models for further agent research.
2 Related Work
Prior work explores tool use, interactive benchmarks, and synthetic data generation, but often emphasizes single-turn or reasoning settings. APIGen-MT extends this line by producing verifiable multi-turn trajectories with realistic agent-human interactions across environments.
- Tool-Use Agents: Tool-use research improves LLM functionality through external tools, reusable tool construction, workflow composition, and learning from demonstrations.
- Interactive Conversational Benchmarks: Interactive benchmarks evaluate context maintenance, tool-augmented dialogue, iterative problem-solving, customer management, and stateful tool use with simulated users.
- Synthetic Data Generation: Synthetic-data methods generate realistic datasets through multi-agent simulation, instruction composition, intermediate graphs, and multi-turn planning.
- APIGen-MT generates domain-constrained tasks, executable groundtruth function calls, and verifiable long trajectories through iterative refinement and simulated agent-human interplay.
3 APIGen-MT Method for Synthesizing High-Quality Multi-Turn Data
APIGen-MT frames multi-turn agent interaction as a partially observable process and separates task correctness from conversational realization. Its two-phase pipeline validates blueprints, simulates interactions, and accepts trajectories that satisfy groundtruth state and output checks.
- 3.1 Multi-Turn Interaction Problem Formulation: Multi-turn interaction is modeled as a POMDP in which the assistant infers user intent while acting through tool calls or responses under environment and domain constraints.
- 3.2 APIGen-MT Framework Overview: Direct one-shot conversation synthesis is fragile because intermediate errors can cause failure and later turns depend on earlier function calls and outputs.
- 3.2 APIGen-MT Framework Overview: APIGen-MT uses two phases: it first creates a detailed task blueprint, then uses that blueprint to guide realistic multi-turn interaction generation.
- Phase 1: Task Configuration and Groundtruth Generation: Phase 1 constructs user instructions, groundtruth action sequences, and expected outputs using contextual generation, automated checks, and LLM committee review.
- Phase 1: Task Configuration and Groundtruth Generation: Feedback loops let the agentic design reflect on validation results and progressively improve task configurations.
- Phase 2: Human-Agent-Environment Interaction Trajectory Collection: Phase 2 simulates a human user and test agent in an executable environment, producing trajectories that are checked against groundtruth actions and expected outputs.
- Separating blueprint construction from conversational simulation supports verifiability, natural turn-by-turn dynamics, modular debugging, and scalability across interaction patterns.
4 A Case Study of APIGen-MT on τ-bench
The τ-bench instantiation of APIGen-MT generates validated task blueprints and uses them to produce realistic, verifiable multi-turn trajectories. Its agentic feedback and simulated human-agent interplay support complex task construction and successful data collection.
- Task Generation: Task instances are generated through API-graph random walks and domain-specific sampling of APIs, policies, and data.The API graph represents dependencies between APIs and supports realistic task sequences.
- Framework Instantiation: APIGen-MT first synthesizes task configurations specifying user intent, groundtruth actions, and expected outputs before simulating turn-by-turn interactions.The blueprint establishes a verifiable target for the subsequent executable interaction phase.
- Validation and Refinement: A three-stage validation pipeline checks task structure and execution, evaluates semantic alignment with a committee of LLM judges, and refines failures using aggregated feedback.Majority voting is used to stabilize assessment, while accepted tasks pass a threshold-based final review.
- Complex Task Construction: Reverse Task Recombination constructs longer tasks by concatenating independently validated action and output components, rechecking policy compliance, and synthesizing a unified instruction.The method focuses validation on the semantic coherence and alignment of the combined task.
- Data Collection and Statistics: A 2.5x boost raises task collection success to 70% when the pipeline uses a review committee and iterative reflection.The framework also collects long trajectories requiring a strong model such as GPT-4o to average 12 turns to complete a task.
- Data Collection and Statistics: The resulting two-phase process generates diverse, realistic, and verifiable multi-turn data for complex domains with strict policy constraints.These trajectories are intended for training and evaluating conversational agents.
5 Experiments
Experiments evaluate APIGen-MT-trained models on BFCL v3 and τ-bench, finding strong overall and multi-turn performance, improved consistency, and competitive efficiency against frontier and open-source baselines.
- Benchmark results: BFCL v3 overall accuracies reach 78.19% for xLAM-2-70b-fc-r and 75.83% for xLAM-2-32b-fc-r, ranking first and second.Both models surpass the evaluated proprietary and open-source models.
- Benchmark results: 75.12% multi-turn accuracy for xLAM-2-70b-fc-r exceeds the reported o1 and gpt-4o function-calling results of 36% and 41%.Smaller variants also achieve 69.25% at 8B, 56.00% at 3B, and 43.12% at 1B.
- Benchmark results: 56.2% τ-bench success for xLAM-2-70b-fc-r exceeds Llama 3.1 70B Instruct at 38.2%, DeepSeek v3 at 40.6%, and GPT-4o at 52.9%.The model approaches Claude 3.5 Sonnet at 60.1% under the default naive user setting.
- Benchmark results: Smaller xLAM-2 variants achieve 54.6% at 32B and 46.7% at 8B on τ-bench, surpassing larger baselines.The results are presented as evidence of efficient knowledge transfer with fewer parameters.
- Consistency and stability: As trial count increases, APIGen-MT models show less success-rate decline in pass^k curves, indicating stronger consistency across repeated tasks.On τ-airline, xLAM-2-70b-fc-r has higher pass^5 than Claude despite slightly lower pass^1.
- Consistency and stability: Best-of-N user simulation yields higher average success rates and lower variance for gpt-4o and xLAM-2-70b-fc-r on τ-retail.The setting combines user-response sampling with self-critique to improve stability.
- Efficiency analysis: xLAM-2-70b-fc-r requires interaction counts comparable to gpt-4o but more than Claude, partly because it retrieves user details in stages.The efficiency comparison appears in the performance/efficiency analysis on τ-bench.
6 Discussion
The discussion presents APIGen-MT as a modular two-phase synthesis framework whose validation and stabilization mechanisms support reliable multi-turn data, while identifying remaining scalability and simulation limits.
- Discussion: Decoupling blueprint creation from trajectory simulation combines structural correctness with natural dialogue dynamics.The modular design also isolates task-design issues from conversational-modeling issues for debugging and scalability.
- Discussion: APIGen-MT-trained models outperform existing baselines on τ-bench and BFCL v3, including competitive multi-turn performance from smaller models.The conclusion also reports more consistent and reliable behavior from stabilization techniques.
- Limitations and future directions: Residual stochasticity remains in simulated human behavior despite Best-of-N sampling and self-critique.The authors suggest more deterministic simulation or refined filtering metrics as future remedies.
- Limitations and future directions: The pipeline discards failed second-phase trajectories, incurs computational overhead during multi-stage validation, and has not yet been extended broadly across domains.Future directions include using failures as contrastive signals, adaptive validation, additional domains, and reinforcement learning.
A Benchmarks Description
The paper evaluates agents with BFCL v3 for function calling and τ-bench for realistic simulated-human interactions.
- BFCL v3: BFCL v3 evaluates single-turn, multi-turn, and multi-step function calling using user understanding, function selection, parameter validity, and output interpretation.Its overall accuracy is a weighted average across evaluation categories.
- τ-bench: τ-bench measures an agent’s ability to interact with simulated hum
B Prompts
The appendix documents prompts for task generation, validation, review, trajectory collection, and stabilized human simulation in τ-bench, including policy-grounded action requirements.
- Prompt stages: The τ-bench prompt set covers task configuration generation, alignment validation, final semantic review, trajectory collection, and stabilized human simulation.These stages are represented by Figures 8 through 12.
- Task generation: Task generation prompts combine task and domain guidelines with sampled user, order, and domain information to mimic realistic retail intentions.The generated task includes an instruction, groundtruth actions, and outputs for information requests.
- Groundtruth actions: Groundtruth-action prompts prioritize database-modifying actions, omit separate calls for direct information requests, and include multiple calls when tasks require them.The prompts also require precise parameters and adherence to retail policies and common-sense practices.
- Human simulation: The BoN user prompt asks a judge to score candidate human responses from 0 to 10 for correctness.This supports the stabilized human-simulation stage shown in Figure 12.