PaperlayerView exact revision on arXiv

Source-linked AI summary

ToolSpace: Scaling Tool-Centric Environment Synthesis for Long-Horizon Agents

Guanting Dong, Zhichao Sun, Xinyu Yang, Rui Wang, Xin Deng, Jiajie Jin, Zhixiong Yang, Junhao Huang, Han Han, Mingyu Ma, Xiaotao Wang, Yuyang Hu, Xiaoshuai Song, Xiaoxi Li, Jiejun Tan, Zhecan Wang, Shihan Dou, Honglin Guo, Zhiheng Xi, Tao Gui, Tao Ji, Qi Zhang, Yutao Zhu, Ji-Rong Wen, Zhicheng Dou

arXiv:2611.05036v1cs.CL

TL;DR

Existing data-centric environment synthesis derives tools and tasks after synthesizing databases and scenarios, while preserving real tool orchestration across environments remains an open challenge. ToolSpace builds a tool-composition graph from execution traces; across seven long-horizon benchmarks, ToolSpace-744B-A40B reports a higher average score than GPT 5.6 Sol.

  • Problem

    Data-centric synthesis builds databases and scenarios before deriving tools and tasks, while preserving real tool orchestration across diverse environments remains an open challenge.

    Source: §1 · p. 2

  • Method

    ToolSpace mines data, state, and retry links between adjacent calls in real execution traces, clusters the resulting tool-composition graph, and synthesizes tasks through state-aware walks with replay checks.

    Source: §1 · p. 3, §7 · p. 16

  • Results

    Across seven long-horizon tool-use benchmarks, ToolSpace-744B-A40B scores 56.6 average versus GPT 5.6 Sol’s 48.9 and matches Claude Opus 5.

    Source: §1 · p. 3

  • Takeaways & Limitations

    ToolSpace produces Pro benchmark versions that exceed standard versions in difficulty, tool-call turns, and cross-environment breadth.

    Source: §1 · p. 3

    Environment synthesis cost is estimated by synthesized-file count, which does not directly measure token, wall-clock, API, or compute cost.

    Source: §6.4 · p. 15

Abstract

from arXiv · show

Large language models are deployed as long-horizon agents that compose diverse tools across MCP services into executable workflows. Existing MCP environment-scaling methods are data-centric: they first synthesize a database and scenario, then derive tools and tasks from them. As a result, tool interfaces may not match real ones, tool compositions are not grounded in execution, and tasks rarely span more than one environment. We propose ToolSpace, a tool-centric framework that reverses this order. ToolSpace mines data, state, and retry dependencies from adjacent calls in execution traces of real MCP services and open agent corpora, using deterministic rules rather than LLM judgment, and clusters the resulting weighted tool-composition graph into environments. Databases are synthesized only for state-mutating environments, while query and stateless tools replay recorded returns. ToolSpace then performs state-aware random walks with rollback and decompiles each chain into a natural-language task with assertion-based checks, keeping only samples that replay successfully from the initial state. Applied to AutomationBench and tau3-Bench Banking, the pipeline produces harder Pro versions on which 7 frontier models drop by up to 38.2 points. Across seven long-horizon tool-use benchmarks, ToolSpace raises Qwen3.6-35B-A3B from 23.0 to 41.0 average score (43.2 with RL) and GLM-5.2 from 44.6 to 56.6, on par with Claude Opus 5. Ablations show that execution-witnessed edges clearly outperform LLM-judged ones, and performance keeps improving with data and environment scale.

1 Introduction

Existing environment-scaling methods synthesize databases and scenarios before deriving tools, leaving real interface fidelity, execution-grounded tool links, and cross-environment reachability unresolved. ToolSpace instead starts from real tool execution traces and uses their compositions to synthesize replayable tasks and harder benchmarks.

1 Introduction

  • ToolSpace addresses three limits of data-centric synthesis: mismatched tool interfaces, inauthentic tool links, and tasks confined to single environments.The paper identifies their shared root cause as deriving tool composition from synthesized environments rather than modeling real agent execution.Source: §1 · p. 2
  • The paper frames tool composition—not the data scale of one environment—as the basis for expanding synthesized task complexity.Its motivation is to model real connectivity and state transfer among tools across an extensible environment ecosystem.Source: §1 · p. 2
  • ToolSpace mines data, state, and retry dependencies from real execution traces using deterministic rules, then clusters the weighted tool-composition graph into environments.It synthesizes databases only for operation environments and reuses original returns for other tools.Source: §1 · p. 3
  • State-aware random walks sample executable chains, which are decompiled into natural-language tasks with assertion-based checks and retained only when they replay successfully.Rollback sampling follows state-compatible successors by edge weight and executes chains step by step.Source: §1 · p. 3
  • ToolSpace extends AutomationBench and τ3-Bench Banking into Pro versions with greater task difficulty, more tool-call turns, and broader cross-environment coverage.On AutomationBench Pro, seven frontier models drop by up to 38.2 points relative to the standard version.Source: §1 · p. 3
  • Across seven long-horizon tool-use benchmarks, ToolSpace-744B-A40B scores 56.6 versus GPT 5.6 Sol’s 48.9 and matches Claude Opus 5.The paper also reports scaling relationships between data scale, environment scale, task diversity, and agent performance.Source: §1 · p. 3

2 Related Work

Prior agent-synthesis work scales environments or builds task graphs from structured dependencies, but ToolSpace makes execution-witnessed, cross-environment tool composition its organizing primitive.

Agent Environment Scaling

  • Environment-scaling methods synthesize interaction stacks, but fix which tools may compose before tasks are drawn; ToolSpace instead recovers tool-composition structure witnessed by executions.Simulator-style methods can be inexpensive but are prone to hallucination and state inconsistency, while executable-environment methods synthesize databases, tools, and verifiable tasks.Source: §2 · pp. 3–4

Graph-Based Agentic Data Synthesis

  • ToolSpace mines typed data, state, or retry edges from adjacent real calls across environments, weights them by witnessed frequency, and admits walks only after replay checks pass.Checks include state, trace, and output checks from the initial state.Source: §2 · p. 4
  • Unlike prior graph-based synthesis that walks knowledge, API-dependency, or function-signature graphs, ToolSpace grounds its graph in adjacent real tool calls and verifies replay.The resulting tool composition is the framework’s primitive for cross-environment tasks and benchmarks.Source: §2 · p. 4

3 Preliminaries and Problem Formulation

The formulation contrasts tasks built from synthetic environment data and tools with ToolSpace’s objective: construct real-trace-grounded tool chains that cover long-horizon, cross-environment tasks while passing replay verification.

Tool-use tasks

  • A tool-use task produces an interaction trajectory of tool actions and observations, and a verifier scores the task, trajectory, and final state.The agent chooses actions from the task and interaction history; each tool returns an observation and updates the environment state.Source: §3 · p. 4

Data-centric environment synthesis

  • Data-centric synthesis derives tools and tasks from a fabricated database, so tool arguments may diverge from real interfaces and cross-environment reachability is undefined.Tools from separate database-and-tool pairs have no shared typed relation in that formulation.Source: §3 · p. 4

Tool-centric environment synthesis

  • Tool-centric synthesis seeks a typed graph from real execution traces, environment clusters, on-demand data synthesis, and replay-verified task chains.Every chain must satisfy dependency reachability, state preconditions, and argument provenance from the task, initial state, or earlier observations; the objective is cross-environment coverage subject to replay verification.Source: §3 · p. 4

4 Method

ToolSpace couples exploration of real tool relations with synthesis of executable cross-environment tasks. It retains only chains that replay from the initial state and pass verification.

Overview

  • The two components are presented as a coupled framework that raises the difficulty of existing environment benchmarks.Source: §4 · p. 5

Tool Space Exploration

  • Tool Space Exploration extracts calls from real traces, mines data, state, and retry dependencies by rules, builds a composition graph, clusters tools, and synthesizes databases only for operation-bearing clusters.Source: §4 · p. 5

Cross-Environment Task Synthesis

  • Cross-Environment Task Synthesis uses State-Aware Random Walks to explore long-horizon chains, decompile them into tasks and verification scripts, and retain only chains that replay from the initial state and pass checks.Source: §4 · p. 5

4.1 Tool Space Exploration

ToolSpace builds its tool space from ordered execution traces, models tool relationships as a weighted graph, and clusters tools into environments with state materialized selectively. Its synthesis design distinguishes operation, state-query, and stateless environments to limit database and API requirements.

4.1.1 Agent Trajectory Collection

  • ToolSpace collects real MCP-service calls and web-sourced trajectories, filtering lower-authenticity sources to retain calls matching real tool names and schemas.MCP tasks are kept when their tool definitions exactly match real servers; workspace traces retain explicit tool calls and exclude shell wrappers.Source: §4.1.1 · p. 5

4.1.2 Tool-Space Graph Modeling

  • Adjacent calls form a typed tool-composition graph whose data, state, and retry edges are extracted by deterministic rules rather than LLM judgment.The graph weights candidate edges by observed transition frequency and removes low-frequency links; data edges also require downstream arguments to trace to upstream outputs.Source: §4.1.2 · p. 6

4.1.3 Agent Environment Data Synthesis

  • ToolSpace clusters the weighted graph into 804 environments and materializes databases on demand for operation environments, while state-query and stateless environments use snapshots or cached returns.This design reduces environment-synthesis and API costs.Source: §4.1.3 · p. 7

4.2 Cross-Environment Task Synthesis

ToolSpace samples state-compatible tool chains across environments, using rollback and replay verification to retain executable walks. It then converts retained chains into natural-language tasks with assertion-based checks.

4.2.1 State-Aware Random Walks

  • ToolSpace samples graph neighbors only when their inputs are reachable from earlier returns or the current state, with sampling weighted by edge weight until target chain length and environment coverage are reached.More frequent real-trace connections are more likely to be extended.Source: §4.2.1 · p. 8
  • Failed or edge-inconsistent tool executions trigger rollback to the prior snapshot and resampling; chains that cannot proceed are discarded and retained walks are replayed from their initial state.Replay checks that state continuity holds and every tool remains executable.Source: §4.2.1 · p. 8

4.2.2 Reverse Verifiable Task Synthesis

  • A language model decompiles each retained chain and its state changes into a natural-language task that omits tool names and call order.A verification set is generated for each chain instead of relying on a reference answer.Source: §4.2.2 · p. 8
  • A sample is retained only when replaying its chain from the initial state passes checks of final state, intermediate snapshots, and required returned fields.The dataset records the task, verification script, tool chain, and initial snapshot.Source: §4.2.2 · p. 9

4.3 Detailed Statistics of ToolSpace

ToolSpace’s graph spans many tools and environments, while its synthesized trajectories combine long tool sequences with multiple environments. The reported graph edges are grounded in adjacent calls from real trajectories.

Graph Scale and Dependency Structure

  • 77,760 tool-call-pattern nodes derive from 2,042 canonical tools, with mean outdegree 2.3 and maximum outdegree 56.Data, state, and retry edges account for 69%, 25%, and 6%, respectively, and every edge is witnessed by adjacent calls in real trajectories.Source: §4.3 · p. 9

Environment Diversity and Selective Grounding

  • The reference inventory covers 804 environments, of which 181 are operational and 623 are non-operational.Operational environments contain at least one state-modifying tool; stateless environments dominate, and cached returns avoid database synthesis for them.Source: §4.3 · p. 10

Task Scale and Complexity

  • 34,376 trajectories were synthesized; among the 21,385 with at least five tool calls, average length was 30,466 tokens and average calls per trajectory were 18.86.The maximums were 855,386 tokens and 209 calls per trajectory.Source: §4.3 · p. 10
  • Trajectories with at least five calls averaged 2.29 environments and reached 22, with at least 66.2% involving two or more environments.The resulting tasks require both long-horizon tool orchestration and consistent environment state.Source: §4.3 · p. 10

5 AutomationBench Pro and 𝜏3-Bench Banking Pro

ToolSpace expands AutomationBench and τ3-Bench Banking into cross-environment Pro splits using linked tools and longer walks. Pro tasks require more tool calls than Standard on both benchmarks.

Benchmark Construction

  • ToolSpace builds benchmark tool graphs to link outside tools sharing data or state relations, then restores or synthesizes databases, uses larger walk lengths and breadths, and generates tasks with verification scripts.The resulting Pro splits contain 300 AutomationBench tasks and 98 Banking tasks.Source: §5 · p. 10
  • Three doctoral-level experts reviewed candidate tasks and verification scripts; retained tasks required approval from at least two reviewers, with Fleiss’ κ=0.82 and 93.1% observed agreement.The review took two days.Source: §5 · p. 10

Data Statistics

  • AutomationBench average tool calls rise for every model from 26.8–43.6 under Standard to 56.3–60.6 under Pro; Banking averages rise from 23.1–32.2 to 27.3–36.3.Figure 6 compares average tool calls on both benchmarks.Source: Figure 6 · p. 10, §5 · p. 11

6 Experiments

ToolSpace improves long-horizon tool-use performance across benchmarks and scales, produces more difficult Pro splits, and shows gains with increasing data and environment coverage. Analyses also examine environment relevance, task diversity, synthesis cost, and component contributions.

6.2 Main Results

  • Across seven benchmarks, Qwen3.6-35B-A3B rises from 23.0 to 41.0 with ToolSpace SFT and from 41.0 to 43.2 with RL; ToolSpace-744B-A40B matches Claude Opus 5.ToolSpace-35B-A3B also raises AutomationBench from 8.2 to 45.8; 7–8B environment-scaling baselines remain below 10 on four cross-environment, long-horizon suites.Source: Table 1 · p. 12, §6.2 · p. 13

6.3 Results on AutomationBench Pro and 𝜏3-Bench Banking Pro

  • All seven frontier models score lower on the Pro splits than Standard, with AutomationBench drops reaching 38.2 points.Claude Opus 5.5 and GPT 6.1 Sol drop 32.9 and 26.1 points on AutomationBench, respectively.Source: Table 2 · p. 13, §6.3 · p. 13

6.4 Quantitative Analyses

  • Performance improves overall with environment scale and reaches its full-setting optimum, though the scaling pattern differs by benchmark.AutomationBench rises from 9.2 to 33.8 between 1/16 and 1/8 environments; 𝜏3-Bench Banking's largest jump occurs at the full set, from 19.6 to 34.0.Source: §6.4 · pp. 13–14, Figure 7 · p. 14
  • Performance improves overall with more ToolSpace data, and the full dataset performs best on all five benchmarks.AutomationBench rises from 25.2 at 1/16 data to 45.8 with the full dataset; the step from half to full data adds 4.6 points on AutomationBench.Source: §6.4 · pp. 14–15, Figure 7 · p. 14
  • Random-50% and Top-50% environments differ by at most 1.7 points on four benchmarks, and both approach full-data performance.The comparison uses equal numbers of environments, with Random-50% excluding many of the most benchmark-relevant environments.Source: §6.4 · p. 15, Figure 9 · p. 15
  • ToolSpace produces 42 task-question clusters, compared with 25 for EnvScaler, 31 for Agent-World, and 34 for Agent-Reflex.The visualization uses embeddings reduced with t-SNE and clustered with DBSCAN.Source: Figure 8 · p. 14, §6.4 · p. 15
  • ToolSpace synthesizes 1,218 files versus Agent-World's 2,724, a reduction of 1,506 files or 55.3%.File counts are a cost proxy, not a direct measure of token, time, API, or compute cost.Source: §6.4 · p. 15, Figure 9 · p. 15
  • Removing any component degrades all five benchmarks; state edges matter most on banking and AutomationBench, while data edges matter most on BFCL V4 and OfficeQA Pro.LLM-judged edges degrade every benchmark almost as much as removing state edges; rollback and edge-weighted sampling contribute in between, while retry edges have the smallest effect.Source: §6.4 · p. 16, Table 3 · p. 16

7 Conclusion

ToolSpace builds a tool-centric framework for cross-environment agent-data synthesis from execution traces. Its results include stronger benchmark performance, harder Pro splits, and improvements from execution-witnessed edges and increased data and environment scale.

7 Conclusion

  • ToolSpace mines dependencies between adjacent calls in real-service traces and open agent corpora, then samples and verifies state-compatible tool chains to synthesize cross-environment tasks.It synthesizes databases only for state-mutating environments and reuses recorded returns for query and stateless tools.Source: §7 · p. 16

Related papers

Loading 2611.05036v1…