PaperlayerView exact revision on arXiv

Source-linked AI summary

SynthTools: A Framework for Scaling Synthetic Tools for Agent Development

Tommaso Castellani, Naimeng Ye, Daksh Mittal, Thomson Yen, Emmanouil Koukoumidis, William Zeng, Hongseok Namkoong

arXiv:2511.09572v2cs.AIcs.LGcs.SE

TL;DR

Tool-use agents need diverse, controllable environments and verifiable tasks, but existing training collections and task-generation approaches remain limited in coverage and scalability. SynthTools provides an LLM-based pipeline that generates tools, validates their simulation, and constructs tasks and trajectories. The authors report consistent Qwen3 midtraining gains across multiple tool-use benchmarks and describe synthetic interaction datasets as useful for continued pretraining.

  • Problem

    Tool-use training needs broad, diverse environments and verifiable multi-step tasks, while current collections and task-generation approaches remain limited in scale, domain coverage, or systematic verification.

    Source: §1 · pp. 1–3

  • Method

    SynthTools uses an LLM-based pipeline to hierarchically generate domain-grounded tools, simulate and validate them, and assemble them into tasks with ground-truth trajectories.

    Source: §1 · p. 3, §2 · p. 3, §3 · pp. 4–5

  • Results

    Midtraining Qwen3 models on SynthTools consistently outperforms corresponding base models across model sizes and task categories, with gains across multiple benchmarks.

    Source: §6.2 · p. 12

  • Takeaways & Limitations

    The results point to synthetic interaction corpora as a useful ingredient for continued pretraining of tool-using AI systems.

    Source: §7 · pp. 12–13

Abstract

from arXiv · show

For agentic systems to use external tools to solve complex, long-horizon tasks, we need a large set of diverse and controllable tool-use environments. We introduce SynthTools, a fully LLM-based pipeline spanning the entire lifecycle: environment generation, simulation, validation and task construction. By operating end-to-end through LLMs, our framework complements other tool-use environments bottlenecked by the complexity of real APIs, and ensures scalability and controllability by design. The framework consists of three components: top-down environment generation, which hierarchically constructs diverse, domain-grounded tool environments; environment simulation and validation, which ensures tools can be reliably emulated and filters out those that cannot; and bottom-up task and trajectory generation, which produces solvable and verifiable tasks together with multi-step trajectories, exposing control over difficulty, length, trajectory composition, and domain focus to guarantee flexibility. As a concrete instantiation, we release the dataset comprising $73{,}883$ validated tools across $6{,}800$ environments and $100$ fields, $79{,}925$ verifiable tasks as well as the pipeline to generate trajectories at scale. Training Qwen3 models of various sizes on a corpus of trajectories generated from these tasks yields gains across multiple tool-use benchmarks, including real APIs, indicating tool-use capabilities trained on synthetic data may transfer to some real environments. Together, these results suggest that SynthTools can serve as a useful infrastructure for large-scale training of tool-use agents.

1 Introduction

Tool-use agents need comprehensive, diverse training environments, but existing collections and task-generation approaches remain constrained in scale, coverage, realism, and verifiability. SynthTools proposes an LLM-based pipeline to address these gaps and releases a large collection of validated tools and verifiable tasks.

1 Introduction

  • Current models remain unreliable with tools, while training collections struggle to provide the scale and domain diversity needed for agent training.The paper also notes that prominent tool-use corpora are closed-source or limited in tools and domains.Source: §1 · pp. 1–2
  • Real APIs complicate scalable interaction, curated suites cover few domains, and most synthetic environments use programmatic rather than LLM-based simulation.The paper contrasts LLM simulation's flexibility with practical API constraints and limited hand-built tool collections.Source: §1 · p. 2, Table 1 · p. 2
  • Rich tool ecosystems are insufficient alone: realistic tasks require coordinated workflows, but naive prompting often yields shallow scenarios and evaluation can depend on LLM judges or manual verification.The paper identifies a scalable, principled way to generate rich ecosystems and verifiable tasks as an unmet need.Source: §1 · p. 3
  • SynthTools releases 73,883 validated tools across 6,800 environments and 79,925 verifiable tasks, with control over task difficulty and domain coverage.The paper describes this concrete dataset as more than twice prior collections in both domains and tools per domain.Source: §1 · p. 3

2 SynthTools Framework and Capabilities

SynthTools combines hierarchical environment generation, tool simulation and validation, and task and trajectory construction. Its task collection spans domains and difficulty levels, and a Qwen3 case study reports consistent gains across multiple benchmarks.

2 SynthTools Framework and Capabilities

  • Top-down generation creates 73,883 domain-grounded tools and can expand by at least an order of magnitude, while allowing practitioners to select fields or subdomains.The framework reports richer domains and more complex tools than existing curated and generated environments.Source: §2 · pp. 3–4
  • Simulation produces flexible tool responses, while validation checks correctness, filters unstable tools, and retries to improve reliability.The framework reports 94% accuracy for tool-response simulation and 97% accuracy identifying incorrect behaviors.Source: §2 · p. 4
  • Bottom-up task construction assembles validated tools into coherent multi-step workflows, pairing each step with a known correct tool call to form explicit ground-truth trajectories.The resulting collection contains 79,925 tasks across 100 fields.Source: §2 · p. 4
  • Midtraining Qwen3 on SynthTools improves agentic tool-use capabilities, with consistent gains across multiple benchmarks.The task collection spans diverse domains and difficulty levels and requires coordinated multi-step tool use and long-horizon reasoning.Source: §2 · p. 4

3 Top-down Environment Generation

Top-down environment generation hierarchically refines domains into subdomains, task families, and concrete tools with domain-specific interfaces. The resulting tools scale across fields while retaining diversity and complex schemas.

3 Top-down Environment Generation

  • The generation procedure hierarchically refines a domain into subdomains, task families, and concrete tools, addressing naive prompts' tendency to produce redundant, overly simple APIs.Tool schema complexity can be tuned through argument number and type, with lists and complex objects described as harder than strings and numbers.Source: §3 · p. 4
  • Tools are designed to be composable through upstream dependencies and downstream affordances, and each includes a name, description, parameter schema, and I/O contract.The contract specifies preconditions, postconditions, and error modes; targeted prompting controls diversity, complexity, and I/O behavior.Source: §3 · p. 5
  • The instantiation generates 100 fields and 6,800 environments, with 85,447 tools before filtering and 73,883 afterward.The paper reports this as more than a 2× increase in scale over existing datasets.Source: §3 · p. 5
  • Across generated tools, only 9% have pairwise embedding similarity above τ = 0.85; in E-commerce and Retail, scaling tool counts within one field produces greater diversity rather than duplicates.Separately, sampling 200 tools from each of 20 fields showed sufficient breadth and diversity.Source: §3 · pp. 5–6, Figure 3 · p. 6
  • Generated tools include rich failure modes and output schemas, which the paper presents as reflecting the diversity and complexity of real-world tool ecosystems.Figure 2 gives a Financial Trading tool as an example.Source: Figure 2 · p. 5, §3 · p. 6

4 Environment Simulation and Validation

SynthTools combines tool simulation with validation to make generated tools reliably executable. Its simulator and LLM judge are evaluated through manual checks, large-scale stress tests, and comparisons with deterministic ACEBench behavior.

4.1 Tool Simulation

  • The simulator checks schema-level errors programmatically, uses an LLM to validate semantic constraints, then generates responses using tool-call metadata when needed.For valid calls, metadata can guide state-based response deduction; otherwise, the simulator generates outputs following predefined schemas and domain-specific patterns.Source: §4.1 · p. 7
  • A four-mode test suite covers schema and constraint failures, valid calls with new inputs, and valid calls whose desired outputs are known.The known-output mode supports automatic verification against metadata.Source: §4.1 · pp. 7–8

4.1.1 Manual Evaluation

  • 97% overall accuracy on 698 SynthTools responses; ACEBench matching was 94% (151/161), with 14 of 20 tools achieving perfect agreement.The remaining ACEBench mismatches primarily reflected implementation-specific ordering differences rather than errors in core simulation logic.Source: §4.1.1 · p. 8

4.1.2 LLM-as-Judge Evaluation

  • At scale, 86% of simulated responses were correct and more than 89% had no more than one error across 8–10 stress-test calls.Audit filtering left the distribution over argument types and number of arguments per tool unchanged.Source: §4.1.2 · p. 9

4.2 Tool Validation

  • The Tool Validation module uses an LLM judge to filter error-prone specifications and support large-scale evaluation and on-the-fly verification with retries.The judge assesses a tool specification, test call, and simulator response, returning a correctness judgment, confidence score, and rationale.Source: Figure 4 · p. 7, §4.2 · p. 9
  • 97% accuracy with a 0% false positive rate across 200 manually inspected and 498 automatically graded cases for the LLM judge.The judge made 3 errors on stress tests and 6 actual errors on the automatically graded set; 9 further judgments there were overly strict rather than misclassifications.Source: §4.2 · p. 9

5 Bottom-up Task and Trajectory Generation

SynthTools rebuilds tasks from validated tools, combining related capabilities into solvable, verifiable workflows with explicit solution trajectories. Task length and difficulty can be tuned through workflow design.

5 Bottom-up Task and Trajectory Generation

  • For each validated toolset, the pipeline proposes tool-use chains, builds mini-tasks step by step, verifies their solvability, and updates environment state from successful tool outputs.Ground-truth calls and state evolution form an explicit trajectory that can be checked against the final environment state and tool-call sequence.Source: §5 · p. 10
  • Task difficulty can be adjusted by varying the number of mini-tasks, balancing explicitness against ambiguity, and adding plausible irrelevant tools to the candidate pool.Orchestrating tool-call dependencies favors complex, multi-step workflows.Source: §5 · p. 11
  • The pipeline constructs 79,925 tasks across 100 fields and 6,800 tool collections, pairing each mini-task with a ground-truth call and final environment state.Each mini-task is successfully executed during generation, ensuring feasibility; chaining the calls yields an explicit solution trajectory.Source: §5 · p. 11

6 Case Study: Midtraining with SynthTools

Midtraining Qwen3 Base models on SynthTools trajectories is associated with consistent gains across tool-use benchmarks and model sizes. The case study frames scalable, structurally diverse interaction data as a fit for continued pretraining.

6.2 Experiments and Results

  • Across model sizes and task categories, midtrained models consistently outperform their corresponding base models on ACEBench, BFCL, and API-Bank.The results are summarized in Table 3.Source: Table 3 · p. 12, §6.2 · p. 12

6.1 Midtraining

  • SynthTools trajectories are designed for midtraining, a continued-pretraining phase intended to narrow the distribution gap between generic text and downstream reasoning, tool use, and structured interaction.The passage describes this phase as using a next-token prediction objective, often with higher-quality instruction-formatted data.Source: §6.1 · p. 12

6.2 Experiments and Results

  • The case study midtrains Qwen3 Base models at 0.6B, 1.7B, and 4B on generated trajectories, then evaluates tool/API-use categories on API-Bank, ACEBench, and BFCL.API-Bank includes executable real-world APIs, while BFCL-v3 Live evaluates against functioning APIs; the authors say these results suggest synthetic-tool capabilities can transfer to realistic API-style interactions.Source: §6 · p. 11, §6.2 · p. 12

7 Conclusion

SynthTools provides a scalable, structured framework for synthetic tool ecosystems and verifiable training and evaluation tasks. The results support continued pretraining on these environments and point to synthetic interaction corpora as a potential ingredient for tool-using AI.

7 Conclusion

  • SynthTools is presented as a scalable, structured framework for building synthetic tool ecosystems and verifiable tasks for agent training and evaluation.The conclusion says its environments serve as an effective dataset for continued pretraining and points toward synthetic interaction corpora as a key ingredient for advancing tool-using AI systems.Source: §7 · pp. 12–13

A Validation Details

The validation audit measures simulator reliability across tool-level stress tests, response accuracy, test modes, and verdict types. It also gives illustrative call outcomes and reports training and computational details.

A Validation Details

  • 57% of tools produced no incorrect responses, 89% had at most one, 10% had two or more, and response-level accuracy was 94%.The incorrect responses are concentrated in a small tail of hard-to-simulate tools.Source: A Validation Details · p. 18, Table 4 · p. 18
  • Mode 1 accuracy was 95.9%, compared with 83.6% for Mode 2 and 82.5% for Mode 3; tools with underspecified schemas were dropped during filtering.Mode 1 uses schema and type checks, while Mode 2 requires reasoning about semantic constraints; underspecified schemas force the simulator to hallucinate missing behavior.Source: A Validation Details · p. 18, Table 5 · p. 18
  • Among incorrect simulator responses, 58.3% were should success got error, 31.5% inappropriate response cases, and 10% should error got success; false rejections trigger regenerations via the retry mechanism.Silent false acceptances would propagate noise into the dataset.Source: A Validation Details · pp. 18–19

A.1 Failure and success models of tool calls

  • Examples include rejecting an insurance update with an empty fields array, rejecting mismatched fields and values, and returning PASS with an invalid-regulation error message.The examples label schema mismatch, parameter inconsistency, and metadata inconsistency failure modes.Source: §A.1 · p. 19

B Midtraining with SynthTools

  • Training used approximately 23,000 trajectories and 200M tokens on 4×NVIDIA A100 GPUs 40 GB; greedy decoding at T = 0 made evaluation deterministic, so no confidence intervals were reported.Table 7 lists Qwen3 training hyperparameters.Source: B Midtraining with SynthTools · p. 19, Table 7 · p. 20

C.1 Computational Detail

  • Tool generation used gpt-oss-120b on 4×NVIDIA A100 GPUs 40 GB, and Table 8 reports token usage per environment and task across pipeline stages.Source: §C.1 · p. 20, Table 8 · p. 20

D Deduplication details

The deduplication details describe exact and semantic filtering, including a similarity threshold chosen to balance compactness and coverage. The section also gives examples of generated tools in Financial Trading and Ecommerce and Retail.

D Deduplication details

  • A multi-stage deduplication pipeline first compares normalized tool names and then tool bodies within each domain or field, reducing redundancy before later stages.The tool body includes its description, parameters, usage, and output schema; the semantic phase adapts SemDeDup [1] to the dataset’s structure.Source: D Deduplication details · p. 20
  • Cross-field duplication is rare, while most residual redundancy lies between adjacent tasks and subdomains; the validation sweep sets τ = 0.85 to balance compactness and coverage.Figure 8 describes similarity scores across fields, fields, subfields, and tasks; Figure 9 shows tool-drop percentage by similarity threshold.Source: D Deduplication details · p. 21, Figure 8 · p. 22, Figure 9 · p. 22

E.1 Example 1 (Field: Financial Trading) - Given in Main Paper

  • Figure 10 presents an example of a pipeline-generated tool from the Financial Trading field.Source: Figure 10 · p. 23

E.2 Example 2 (Field: Ecommerce and Retail)

  • The Ecommerce and Retail example is an Image Quality Analyzer that accepts image URLs and optional resolution and background checks, returning quality analysis, issues, and recommendations.Its listed parameters include a minimum resolution default of 800 pixels, and its stated accepted image formats are JPEG, PNG, and WebP.Source: §E.2 · p. 24

F Example Tasks

The examples show generated tasks as multi-tool sequences with execution logs and ground-truth validation. One follows ecommerce performance reporting; the other demonstrates financial-trade record processing and compliance checking.

F.1 Example 1 (Field: Ecommerce and Retail)

  • The Ecommerce and Retail example chains comparison, reporting, formatting, alerting, dashboard updates, and scheduling; ground-truth validation marks the task solved with confidence 0.98.The agent’s generated call matches the required ground-truth parameters.Source: §F.1 · p. 25

F.2 Example 2 (Field: Financial Trading)

  • The Financial Trading example is an eight-tool compliance pipeline that ingests CSV trade records, validates and normalizes them, encrypts and archives them, then checks retention compliance.The task specifies a 24-hour ingestion window and CFTC retention requirements.Source: §F.2 · pp. 26–27
  • The Financial Trading example’s ground-truth validation marks the task solved, and its final compliance response reports retention-window, encryption-at-rest, and access-control checks passed.The example also shows an AES-256-CBC response reporting that an IV is required but not provided.Source: §F.2 · pp. 27–28

G Tool Distribution

The section reports tool-embedding distributions across 100 fields and states that tools remain diverse at that scale.

G Tool Distribution

Related papers

Loading 2511.09572v2…