Source-linked AI summary
TOUCAN: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP Environments
Zhangchen Xu, Adriana Meza Soria, Shawn Tan, Anurag Roy, Ashish Sunil Agrawal, Radha Poovendran, Rameswar Panda
TL;DR
Open-source agent research lacks large, realistic, permissively licensed tool-agentic data covering diverse and multi-turn interactions. TOUCAN synthesizes 1.5 million trajectories from real-world MCP environments with extensive filtering and validation, and its fine-tuned models outperform larger counterparts on major agent benchmarks while advancing the MCP-Universe Pareto frontier.
Problem
Open-source progress is constrained by limited high-quality, permissively licensed tool-agentic data, especially for diverse, realistic, multi-tool, and multi-turn interactions.
Method
TOUCAN synthesizes 1.5 million trajectories from nearly 500 real-world MCP servers using staged task and trajectory generation, filtering, validation, and diversity extensions.
Results
TOUCAN-fine-tuned models surpass larger closed-source counterparts on BFCL V3 and achieve strong performance across τ-Bench, τ 2-Bench, and MCP-Universe evaluations.
Takeaways & Limitations
TOUCAN provides an open-source alternative for training agentic models across parallel, multi-step, multi-turn, and edge-case tool use.
Takeaways & Limitations
The dataset excludes MCP servers requiring special configurations or credentials, potentially omitting important servers and widely used scenarios.
Abstract
from arXiv · showhide
Large Language Model (LLM) agents are rapidly emerging as powerful systems for automating tasks across domains. Yet progress in the open-source community is constrained by the lack of high quality permissively licensed tool-agentic training data. Existing datasets are often limited in diversity, realism, and complexity, particularly regarding multi-tool and multi-turn interactions. To address this gap, we introduce Toucan, the largest publicly available tool-agentic dataset to date, containing 1.5 million trajectories synthesized from nearly 500 real-world Model Context Protocols (MCPs). Unlike prior work, Toucan leverages authentic MCP environments to generate diverse, realistic, and challenging tasks with trajectories involving real tool execution. Our pipeline first produces a broad spectrum of tool-use queries using five distinct models, applies model-based quality filtering, and then generates agentic trajectories with three teacher models using two agentic frameworks. Rigorous rule-based and model-based validation ensures high-quality outputs. We also introduce three extension mechanisms to further diversify tasks and simulate multi-turn conversations. Models fine-tuned on Toucan outperform larger closed-source counterparts on the BFCL V3 benchmark and push the Pareto frontier forward on MCP-Universe Bench.
1 INTRODUCTION
TOUCAN addresses the shortage of high-quality, permissively licensed tool-agentic data with a large dataset built from real-world MCP environments. Models fine-tuned on it outperform larger closed-source systems on key agentic benchmarks.
- TOUCAN targets limitations in prior datasets, including restricted tool diversity, simulated or missing tool responses, single-turn focus, and insufficient scale.These gaps constrain training data for production-relevant agentic interactions.
- 1.5 million trajectories from nearly 500 real-world MCP servers make TOUCAN a large open-source tool-agentic dataset.The dataset covers parallel and multi-step tool calls, multi-turn dialogues, and edge-case tool use.
- The pipeline uses MCP specifications and multiple models to synthesize tasks and trajectories, then applies model-based, rule-based, and execution-focused quality checks.Its extensions add edge-case, interactive, and multi-turn tasks.
- TOUCAN-fine-tuned models surpass larger closed-source counterparts on BFCL V3 and achieve strong results across multi-turn and other agentic evaluations.The reported benchmark coverage includes BFCL V3, τ-Bench, τ 2-Bench, and MCP-Universe.
2 RELATED WORK
Prior work progressed from tool-calling datasets toward agent-focused benchmarks and MCP-based evaluations. TOUCAN is situated within this shift toward diverse, realistic, and multi-dimensional tool-use assessment.
- Earlier datasets expanded tool-calling coverage across APIs, domains, programming languages, and synthetic data, but differed in quality and scale.Examples include Gorilla, ToolAlpaca, ToolLLM, and API Pack.
- TOUCAN’s construction pipeline is presented as a systematic five-stage process with extensions for increasing data diversity and realism.The stages run from MCP server onboarding through trajectory filtering.
- Recent agent benchmarks evaluate parallel, multi-step, multi-turn, realistic, and edge-case tool-use interactions rather than simple function-call correctness.BFCL, τ-Bench, and ACEBench exemplify this broader evaluation focus.
- Emerging MCP benchmarks assess tool-use behavior across dimensions such as accuracy, tool selection efficiency, resource usage, parameter construction, and execution success.This reflects evaluation beyond simple correctness.
3 TOUCAN: SCALING TOOL-AGENTIC DATA WITH REAL WORLD MCPS
TOUCAN constructs a large tool-agentic dataset through a staged pipeline using real-world MCP servers, model-based synthesis, filtering, and extensions for harder and more realistic interactions.
- 3.1 TOUCAN Generation Pipeline: TOUCAN comprises over 1.5 million tool-agent trajectories constructed from real-world MCP tools and organized through a five-stage construction pipeline.The stages are MCP server onboarding, task synthesis, task filtering, trajectory generation, and trajectory filtering, followed by three extension mechanisms.
- 3.1 TOUCAN Generation Pipeline: The pipeline synthesizes tasks from single servers, multiple servers, and manually selected featured servers to support varied tool combinations and workflows.Single-server tasks require 1 to N tools, while multi-server tasks integrate tools across sampled servers and featured-server tasks explore realistic combinations from metadata.
- 3.1 TOUCAN Generation Pipeline: Tasks are filtered across six quality dimensions, and trajectories are generated with three LLMs and two agent frameworks before rule- and LLM-based post-filtering.Post-filtering excludes failed connections, missing tool calls, tool-response failures, local paths, and incorrect required-tool use; retained trajectories are high-quality, concise, and executable.
- 3.2 TOUCAN Extensions: Three extensions address single-turn and trivial-selection limitations through irrelevant-task generation, persona-based diversification, and multi-turn dialogue simulation.The multi-turn extension splits complex tasks into sequential sub-questions or extends conversations with follow-up queries, after which the core pipeline is repeated.
- 3.3 Data Analysis: TOUCAN analysis covers multi-server and multi-tool instances, non-trivial tool selection, multi-turn interactions, and both single and parallel tool calls.Its analysis also includes subset statistics and LLM-based quality statistics.
4 EXPERIMENTS
Experiments fine-tune Qwen models on a high-quality TOUCAN subset and evaluate them across several agentic tool-use benchmarks. The tuned models improve broadly, surpass larger models in selected comparisons, and achieve better performance-efficiency trade-offs on MCP-Universe.
- 4.2 Experimental Results: TOUCAN fine-tuning improves performance across BFCL V3, τ-Bench, and τ 2-Bench, with BFCL V3 gains over larger production LLMs and top multi-turn performance.The evaluation compares fine-tuned Qwen2.5 models with baseline and larger models across multiple agentic tool-use benchmarks.
- 4.2 Experimental Results: TOUCAN-tuned models push the MCP-Universe Pareto frontier forward, achieving higher overall task success rates at smaller model sizes.Figure 8 compares overall task success rate with model size.
- 4.2 Experimental Results: Ablations on Qwen2.5-14B-Instruct show that all three extension components contribute to improved benchmark scores.The ablation progressively adds the extensions to isolate their contributions.
5 CONCLUSION AND FUTURE WORK
TOUCAN is presented as a 1.5-million-trajectory tool-agentic dataset, with future work focused on expanding server coverage, reducing real-execution costs, and benchmarking web search.
- Conclusion: TOUCAN contains 1.5M trajectories and is positioned as a comprehensive dataset for training better agentic models.The conclusion identifies TOUCAN as a tool-agentic dataset and reports superior performance on BFCL V3 and MCP-Universe.
- Expanding to More MCP Servers: The dataset excludes MCP servers requiring special configurations, potentially omitting widely used scenarios such as Notion and GitHub.The authors identify manual onboarding or automated onboarding agents as future approaches for expanding coverage.
- Expert Models to Simulate Tool-Responses: Because real tool execution is slow and costly, the authors plan expert models that simulate tool responses to reduce trajectory-generation costs.The proposed simulation component is intended as an alternative to real execution while retaining quality.
- MCP Benchmark for Web Search: The authors also plan an MCP benchmark focused on web search capabilities as tool use becomes central to LLM agents.The proposed benchmark targets web-search scenarios for synthesizing complex reasoning tasks.
6 USE OF LARGE LANGUAGE MODELS (LLMS)
The authors used LLMs to improve manuscript readability and generate the pipeline diagram, with all generated content subsequently verified by the authors.
- LLM Assistance: LLMs assisted with grammar, clarity, readability, and generation of the paper’s pipeline diagram.The assistance covered manuscript presentation and one included visual.
- Verification: The authors iteratively verified all LLM-generated content for accuracy, quality, and consistency with the paper’s scientific contributions.Verification was performed by the authors after generation.
7 ETHICS STATEMENT
The ethics statement addresses ownership, sensitive information, data currency, and hallucination risks associated with TOUCAN and its use.
- Data Ownership and Licensing: TOUCAN uses specification files voluntarily published on Smithery and provides a repository-based takedown process for legitimate removal requests.The authors state that the files were collected in June 2025 and that removal requests will be honored.
- Sensitive Information: Rule-based pre-filtering detects common personally identifiable information patterns, although human error could still expose URLs, tokens, or email addresses.The authors characterize the risk as minimal because specifications generally use placeholders rather than real information.
- Data Evolution: TOUCAN reflects real-world tool-use scenarios available in June 2025, while its modular pipeline supports future updates and customization.The pipeline can be used to expand domain coverage and tailor tool representations.
- LLM Hallucinations: Human verification remains necessary because hallucinations remain a general risk in outputs from models fine-tuned with TOUCAN.Trajectories combine LLMs, agent frameworks, and remote MCP servers to reduce tool-call execution and response errors.
A DATASET SCHEMA AND EXAMPLES
The appendix defines TOUCAN’s instance schema and illustrates trajectories containing metadata, tasks, tool calls, responses, annotations, and multi-turn interactions.
- Dataset Schema: Each TOUCAN instance includes metadata, a task question, target tools, messages, quality assessments, and the number of message rounds.The schema also includes a subset field identifying the generation pipeline.
- Dataset Schema: The subset field distinguishes original single-turn processing, server shuffling, question diversification, and multi-turn extensions.These options specify which pipeline transformations were applied to generate a trajectory.
- Trajectory Example: A trajectory example shows a user question, assistant explanation, tool calls, tool outputs, and a final parsed response across multiple messages.The example resolves the ENS name blockscout.eth to an Ethereum address through MCP tools.
- Trajectory Example: The example’s annotations score tool-selection difficulty and uniqueness, question quality, scenario realism, verifiability, stability, and overall quality.Response annotations additionally assess completeness, conciseness, tool-use percentage, and order correctness.
- Trajectory Example: The ENS example demonstrates multi-turn continuation, where a later user request asks for an address already resolved in the preceding response.The assistant repeats the previously resolved Ethereum address in the follow-up answer.
- Trajectory Example: ENS provides human-readable names that map to hexadecimal blockchain addresses, as illustrated by blockscout.eth and its resolved address.The example explains benefits including usability, trust verification, branding, and error prevention.
B MORE ON DATASET ANALYSIS
TOUCAN’s featured servers span diverse domains, enabling combinations for complex tool-use tasks.
- Featured servers cover a wide range of domains, enabling combinations for complex tool-use tasks.
B.2 MORE ON MCP SERVER ANALYSIS IN TOUCAN
TOUCAN includes diverse MCP servers, tool counts, and topical coverage, with extensions further broadening the dataset.
- Most of TOUCAN’s 495 MCP servers expose fewer than 10 tools, while the server distribution highlights diverse domains.
- Embedding visualization shows broad topic coverage, and TOUCAN’s extensions further increase overall dataset coverage.
C MORE ON EXPERIMENTS
The experiments use supervised fine-tuning and compare models across scales, with annotation reliability assessed against human judgments.
- Kimi-K2 achieves the highest overall correlation with human judgments among cost-efficient annotation options and is deployed for the annotation pipeline.
- Fine-tuning uses NVIDIA H100 GPUs, with hyperparameters reported in Table 5.
C.3 MORE ON ABLATION STUDIES
The ablation study reports that TOUCAN’s extensions improve BFCL V3 performance, while the appendix documents the prompts used to generate varied tool-use tasks.
- Ablation results: All TOUCAN extensions meaningfully improve model performance on the BFCL V3 benchmark.
- Server annotation prompts: Server annotation prompts assign primary, secondary, and optional custom labels based on server functionality, tools, and use cases.
- Task generation prompts: The task-generation prompts target realistic single-tool, multi-tool, and multi-server questions requiring tool use and, where needed, sequential tool calls.
- Task generation prompts: Prompts instruct generators to avoid overly simple questions, include relevant constraints, and omit exact tool names from user questions.
- Task generation prompts: Multi-tool prompts require exactly a specified number of complementary tools and ensure that a single tool cannot reasonably solve the question.