PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Laws for Tool-Use Agents via Controllable Environment Synthesis

Andrew Szot, Samira Abnar, Omar Attia, Abbas Kazerouni, Michael Kirchhof, Cheng-Yu Hsieh, Raviteja Vemulapalli, Alexander Toshev

arXiv:2610.37648v1cs.LG

TL;DR

The paper studies how the composition of synthetic training environments affects tool-use agent generalization. SynthAPI varies tool, task, and database-state diversity to derive a predictive scaling law; composition-aware scaling predicts held-out performance, and its optimized recipe outperforms prior large-scale tool-use data.

  • Problem

    It remains unclear which properties of synthetically generated tool-use data drive downstream performance, even as stateful multi-step interactions are scarce and difficult to produce.

    Source: §1 · p. 1

  • Method

    SynthAPI generates executable, database-backed tool environments and controls the diversity of tools, tasks, and initial data profiles to derive a composition-dependent scaling law.

    Source: §1 · pp. 1–2

  • Results

    Across four model sizes, composition-aware scaling predicts held-out normalized downstream scores with 0.0285 MAE versus 0.1706 for task-count-only scaling.

    Source: §1 · p. 2

  • Takeaways & Limitations

    As data budgets grow, increasing tool and database-profile diversity and data complexity is more effective than simply generating more tasks.

    Source: §1 · p. 2

    Baseline comparisons do not isolate synthesis-method effects or relative data efficiency because training regimes differ substantially.

    Source: §4.4 · p. 9

Abstract

from arXiv · show

We explore how to build performant large language model (LLM) tool-use agents that complete complex, multi-step tasks. Specifically, we study how data composition in synthetically generated tool-use environments affects the training of such agents. To do so, we propose the SynthAPI pipeline, which explicitly controls the diversity of generated environments along the axes of tools, tasks, and the databases on which the tools operate. By systematically varying SynthAPI's generation controls in 458,752 tasks across 849 trained models of four different sizes, we derive a scaling law that predicts downstream tool-use agent generalization based on the scale and composition of the synthesized environments. These predictable relationships reveal several lessons for scaling tool-use agent data. As data budgets grow, scaling tool and database diversity becomes more effective than increasing the number of tasks, and increasing data complexity yields larger gains. Scaling according to these findings yields agents that outperform models trained on data from prior work in large-scale tool-use agent and synthetic environment generation across four evaluation benchmarks.

1 Introduction

SynthAPI makes synthetic multi-step tool-use environments controllable across tools, tasks, and starting database states, enabling analysis of how data composition shapes generalization. The resulting scaling law predicts downstream performance, and its optimized recipe outperforms prior large-scale tool-use data on several benchmarks.

1 Introduction

  • Stateful multi-step tool interactions are scarce in internet text and require realistic tasks, diverse tools, and rich environment data.Prior executable-environment work emphasized increasing data quantity, leaving the effects of generated-data properties on downstream performance open.Source: §1 · p. 1
  • SynthAPI generates executable environments with structured databases, tasks with initial database states called data profiles, and success verifiers.Its explicit controls vary tool, task, and data-profile diversity so fixed training budgets can be allocated among them.Source: §1 · pp. 1–2
  • The composition-aware scaling law achieves 0.0285 mean absolute error on normalized downstream scores, versus 0.1706 for a model using task count alone.It extrapolates across four model sizes to held-out runs in the top 25% of training FLOPs, including runs up to 16× the largest fitting budget.Source: §1 · p. 2
  • As task budgets grow, expanding tool diversity, repeating instructions with different data profiles, and increasing profile complexity become more valuable than simply adding tasks.These are the composition patterns SynthAPI’s controls enable the authors to compare.Source: §1 · p. 2
  • SynthAPI-trained agents outperform prior large-scale tool-use data by 30.5 percentage points on average across AppWorld, BFCL, and ASTRA-Bench.The optimized recipe also beats the strongest per-benchmark synthesis baselines by 7.6–11.6 points on AppWorld Test-Challenge and 2.0–3.7 points on BFCL.Source: §1 · p. 2

2 Related Work

Prior tool-use data synthesis spans demonstrations, live APIs, simulated responses, and executable environments, but the effects of environment composition on generalization remain underexplored. SynthAPI distinguishes itself through explicit composition controls and a scaling law for predicting downstream task success.

2 Related Work

  • Prior tool-use synthesis includes predefined-API demonstrations [Schick et al., 2023; Qin et al., 2023; Chen et al., 2024], live APIs and executable functions [Liu et al., 2024b; Prabhakar et al., 2025], and simulated responses [Liu et al., 2024a; Tang et al., 2023].Other approaches combine data sources and synthesis pipelines [Zhang et al., 2025].Source: §2 · p. 3
  • LLM-response simulators have scaled tool interactions, including 335k trajectories in Nemotron-Agentic, but increasingly complex outputs and state transitions are difficult to simulate reliably.SynthAPI instead scales through database-backed executable environments; prior executable-environment studies primarily varied environment counts or domain diversity at fixed task counts.Source: §2 · p. 3
  • Unlike prior executable-environment synthesis, SynthAPI controls diversity and complexity across tools, task instructions, and initial database states to study composition’s effects on generalization.It varies fixed-budget allocations among new tools, additional tasks for existing tools, and alternative database states, then uses a joint scaling law to predict allocations for larger compute budgets.Source: §2 · p. 3
  • Prior tool-use scaling laws predict benchmark loss from agent data, whereas SynthAPI predicts downstream task success from generated-environment scale and diversity.Its prediction factors include tools, tasks, and data profiles.Source: §2 · p. 3

3 SynthAPI: Synthetic Environments for Tool-Use Agents

SynthAPI builds interactive tool-use environments in two phases: it generates database-backed apps and APIs, then constructs feasible, verifiable tasks within them. Its controls vary tool, task, and starting-state diversity, while task execution uses a discovery-based interface.

3.1 API Generation

  • The first phase has an LLM build coherent apps from Python tools, database schemas, and APIs that read or modify instantiated database objects.Apps organize interacting tools around functionalities, and their APIs are exposed as callable tools.Source: §3.1 · pp. 3–4

3.2 Task Generation

  • The second phase creates task instructions, Python verifiers, and starting database states with core data and distractors, then generates an oracle solver and filters infeasible or already-completed tasks.The oracle uses the same APIs as the agent but can access privileged identifiers directly from the database object.Source: §3.2 · p. 4
  • Agents interact by sending Python code blocks and discover tools through meta-tools that list apps, tools, and documentation before task completion.Tasks involve tools from a single app, though the authors report generalization to multi-app downstream tasks.Source: §3.2 · pp. 4–5
  • Figure 2 plots aggregate downstream performance against task budget, with panels comparing app-to-task allocations, database instantiations, and distractor-data levels.For panel (a), the highest curve at a fixed task count indicates the best allocation.Source: Figure 2 · p. 5
  • Apps, tasks, and initial database states are independently varied through controls over app count, tasks per app, database instantiations per instruction, and distractor data.Task diversity scales with A, T, and S, with N = A · T · S; D changes data complexity without changing task count.Source: §3.2 · p. 5

4 Experiments

The experiments test how SynthAPI data composition predicts tool-use generalization, how the best allocation changes with budget, and whether the resulting recipe transfers to established benchmarks. They find that composition-aware scaling predicts held-out performance and that SynthAPI-trained agents outperform comparison baselines on downstream benchmarks.

4.1 Experimental Setup

  • The study trains 849 models on generated task populations and evaluates generalization on benchmarks with tools and tasks distinct from SynthAPI environments.Training uses one verifier-successful expert trajectory per task when available, then supervised fine-tunes Qwen3.5-family students; the diagnostic set uses normalized scores.Source: §4.1 · pp. 5–7

4.2 Task Properties Analysis

  • At 64–512 tasks, 8 apps perform best, while 64 apps are clearly best from 4096 tasks; larger app counts also raise the fitted performance ceiling.The gains are not explained by exhausting app functionality: even at 1024 tasks per app, the median fraction of tools used is 51%.Source: §4.2 · pp. 7–8
  • The best number of database profiles per instruction rises with budget: one at 32–512 instances, two at 512–4096, and four above 4096.Each profile changes the starting database state while keeping the task instruction fixed.Source: §4.2 · p. 8
  • Larger distractor sets (D = 4) yield larger gains at high task budgets, though benchmarks with denser-state retrieval demands may scale differently.Source: §4.2 · p. 8

4.3 Scaling Law

  • Composition-aware scaling predicts held-out performance up to 16× larger compute budgets with MAE 0.0285 versus 0.1706 for quantity-only scaling.The fitted performance ceiling increases with app count, database profiles per task, and profile complexity; tasks per app control progress toward that ceiling.Source: §4.3 · pp. 8–9

4.4 Baseline Comparison

  • SynthAPI-Opt consistently beats SynthAPI-SubOpt at equal task budget, while SynthAPI-Big surpasses prior baselines on AppWorld and BFCL.Comparisons with prior methods do not isolate synthesis-method effects or relative data efficiency because their training regimes differ.Source: Table 1 · p. 7, Table 2 · p. 9, §4.4 · p. 9
  • The benchmark comparison supports SynthAPI’s transfer across AppWorld and BFCL, but does not establish relative data efficiency or isolate synthesis-method effects.Compared methods combine SFT and RL on smaller datasets, whereas SynthAPI-Big uses SFT on a much larger synthetic corpus.Source: §4.4 · p. 9

5 Conclusion

SynthAPI creates executable, database-backed app environments with controllable tool, task, and data-profile diversity, and uses them to study training-data composition. The conclusion reports a scaling law for choosing composition under a compute budget and notes limits in user interaction, document retrieval, and validation scope.

5 Conclusion

  • SynthAPI independently scales tool, data-profile, and task diversity; its joint scaling law predicts downstream performance and identifies data composition settings for a compute budget.Source: §5 · p. 10
  • The scaling law is validated on one data pipeline; SynthAPI also omits user interaction and has limited semantic depth for long-document retrieval tasks.Source: §5 · p. 10

A.1 App Generation Details

  • App APIs are restricted to four database create, read, update, and delete operations, allowing database changes to be tracked in an access log.Source: §A.1 · p. 14

A.2 Task Generation Details

  • Tasks use short, actionable instructions and generated initial states, verifiers, oracle solvers, and distractor records while remaining feasible and unsolved initially.Source: §A.2 · pp. 15–17

A.3 Agent Interface Details

  • Each task is a deterministic Gymnasium environment whose database state changes through API calls; tool documentation and serialized results support agent interaction.Source: §A.3 · pp. 18–19

A.4 SynthAPI App and Task Statistics

SynthAPI’s 64 generated apps expose many APIs, data classes, and functionalities, with API parameters and entity-ID requirements that often call for tool composition.

A.4 SynthAPI App and Task Statistics

  • 10,472 APIs span 64 apps, with a median of 144 APIs per app; 6,266 are write APIs and 4,206 are read APIs.Source: §A.4 · p. 19
  • Apps use a median of 55 data classes and implement a median of 81 functionalities; most APIs take 1–10 arguments.Source: §A.4 · p. 19
  • Most APIs require at most one ID parameter referencing an existing entity, indicating that agents often need to compose APIs to retrieve the appropriate ID.Source: §A.4 · p. 19
  • Across 24,103 API parameters, types extend beyond strings, while most apps have some APIs requiring two or three entity IDs.Source: §A.4 · p. 19

B Training Details

The training pipeline collects verified GLM-5.2 expert trajectories and supervises reasoning and code, with model-size-specific training settings and early stopping.

B Training Details

  • Each task contributes one successful GLM-5.2 trajectory, selected from up to four attempts, with the expert reasoning trace saved for supervision.Source: B Training Details · pp. 19–20
  • 459 tokens per action block and 8 tools per trajectory are the respective medians in the GLM-5.2 expert-trajectory statistics.Source: Figure 5 · p. 20, B Training Details · p. 20
  • Training applies next-token cross-entropy to expert reasoning and code, masking prompts and environment tokens; runs use a held-out validation set and may early-stop on rising loss.Source: B Training Details · pp. 20–21
  • Training uses bf16, a 2 × 10−5 learning rate, and model-size-specific packing and accumulation settings; SynthAPI-Big uses twice the nodes and four epochs.Source: B Training Details · p. 21

C Analysis Details

The scaling-law parameters are fitted independently for each model size, with the highest-compute runs held out to test extrapolation beyond the fitting range.

C Analysis Details

  • The scaling-law coefficients are fitted separately by model size, holding out the top 25% of training FLOPs to test extrapolation beyond the fitted compute range.Source: C Analysis Details · p. 21, Table 3 · p. 21

D Benchmark and Evaluation Details

Evaluation uses the same agent interface and prompt as training, with benchmark tools exposed through documentation meta-tools. Benchmark-specific settings constrain task completion, difficulty, split, and output length.

D Benchmark and Evaluation Details

  • Agents use the same interface and prompt as training, access benchmark APIs through documentation meta-tools, and execute tools by emitting Python code.Called environment tools are dispatched to the environment.Source: D Benchmark and Evaluation Details · p. 21
  • AppWorld is evaluated by task goal completion with a 50-step maximum, while ASTRA-Bench uses only the hardest complexity in each of its three task categories.Source: D Benchmark and Evaluation Details · p. 21
  • BFCL evaluation uses v3's multi-turn split with default benchmark settings, and per-action output is capped at 4096 tokens including reasoning and action.Evaluation uses the default Qwen3.5 sampling parameters in vLLM.Source: D Benchmark and Evaluation Details · p. 21

E.1 Additional Analysis

The appendix reports app-to-task allocation plots by model size and evaluates the scaling law's fit and extrapolation errors for four student sizes.

E.1 Additional Analysis

  • App-to-task allocation experiment plots are shown for the 2B, 4B, 9B, and 27B models.Source: §E.1 · p. 21, Figure 6 · p. 22, Figure 7 · p. 22
  • For the 2B student, in-sample RMSE is 0.043 and MAE is 0.033, while extrapolation RMSE and MAE are both 0.004.Source: §E.1 · p. 21
  • For the 4B student, in-sample RMSE is 0.050 and MAE is 0.039; extrapolation RMSE is 0.066 and MAE is 0.051.Source: §E.1 · pp. 21–22
  • For the 9B student, in-sample RMSE is 0.050 and MAE is 0.038, while extrapolation RMSE is 0.033 and MAE is 0.026.Source: §E.1 · p. 22
  • For the 27B student, in-sample RMSE is 0.076 and MAE is 0.049, while extrapolation RMSE is 0.037 and MAE is 0.028.Source: §E.1 · p. 22

E.2 Benchmark Results Breakdown

The appendix provides full benchmark result breakdowns alongside the aggregate results reported in Table 1.

E.2 Benchmark Results Breakdown

  • Tables 4–6 give the full breakdown behind Table 1's single aggregate number per benchmark and model size.The breakdowns cover both AppWorld splits and per-category scores on ASTRA-Bench and the SynthAPI holdout.Source: §E.2 · p. 23
  • Table 4 reports AppWorld results for the Test-Normal and Test-Challenge splits.Its SimSAT, ToolACE, and ToolAlpaca results are attributed to Lee et al. (2026).Source: Table 4 · p. 23
  • Table 5 breaks ASTRA-Bench results down by task category; because one task can count for multiple categories, the overall score is not the average of category scores.Source: Table 5 · p. 23

E.3 Results on Another Data Synthesis Pipeline

The authors test whether the app-to-task allocation pattern also appears in SimSAT. At higher task budgets, broader app coverage performs better, matching the SynthAPI observation.

E.3 Results on Another Data Synthesis Pipeline

  • The SimSAT comparison samples task sets from A = 8, 16, or 32 apps at fixed budgets N = 128, 256, 512, or 1024.SimSAT contains synthetic tasks and corresponding tool-use trajectories generated from 50 apps.Source: §E.3 · pp. 23–24
  • At N = 1024, SimSAT tasks from A = 32 apps outperform those from A = 16, while at N = 128, 256, and 512, A = 32 generally underperforms A = 8 or 16.Source: Figure 10 · p. 23, §E.3 · p. 24
  • The higher-budget advantage of using more apps aligns with the SynthAPI finding that tool diversity matters more as training-task counts scale.Source: §E.3 · p. 24

Related papers

Loading 2610.37648v1…