PaperlayerView exact revision on arXiv

Source-linked AI summary

EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RL

Minrui Xu, Zilin Wang, Mengyi DENG, Zhiwei Li, Zhicheng Yang, Xiao Zhu, Yinhong Liu, Boyu Zhu, Baiyu Huang, Chao Chen, Heyuan Deng, Fei Mi, Lifeng Shang, Xingshan Zeng, Zhijiang Guo

arXiv:2605.18703v1cs.CLcs.LG

TL;DR

Agentic RL lacks scalable, robust environments and realistic tool-use data reflecting implicit human reasoning. EnvFactory automates both environment construction and trajectory synthesis, improving Qwen3-series performance by up to 15% on BFCLv3 and across other benchmarks.

  • Problem

    Agentic RL lacks scalable, executable environments and realistic tool-use trajectories that capture contextual ambiguity and implicit reasoning.

  • Method

    EnvFactory autonomously constructs and verifies stateful tool environments while using topology-aware sampling and calibrated refinement to synthesize natural multi-turn trajectories.

  • Results

    Up to 15% on BFCLv3, 8.6% on MCP-Atlas, and 6% on conversational benchmarks, with higher training efficiency than concurrent work using more environments.

  • Takeaways & Limitations

    EnvFactory provides a unified, data-efficient approach for training agents in complex tool-use environments.

  • Takeaways & Limitations

    Existing synthetic environments may be stateless or depend on pre-collected documents, limiting coverage of unseen tool ecosystems.

Abstract

from arXiv · show

Equipping LLMs with tool-use capabilities via Agentic Reinforcement Learning (Agentic RL) is bottlenecked by two challenges: the lack of scalable, robust execution environments and the scarcity of realistic training data that captures implicit human reasoning. Existing approaches depend on costly real-world APIs, hallucination-prone LLM simulators, or synthetic environments that are often single-turn or depend on pre-collected documents. Moreover, synthetic trajectories are frequently over-specified, resembling instruction sequences rather than natural human intents, reducing their effectiveness for RL training. We introduce EnvFactory, a fully automated framework that addresses both challenges. EnvFactory autonomously explores and verifies stateful, executable tool environments from authentic resources, and synthesizes natural multi-turn trajectories through topology-aware sampling and calibrated refinement, producing grounded queries with implicit intents. Using only 85 verified environments across 7 domains, EnvFactory generates 2,575 SFT and RL trajectories. Despite using significantly fewer environments than prior work, which are often 5 times more, EnvFactory achieves superior training efficiency and downstream performance, improving Qwen3-series models by up to +15% on BFCLv3, +8.6% on MCP-Atlas, and +6% on conversational benchmarks including $τ^2$-Bench and VitaBench. By fully automating both environment construction and trajectory synthesis, EnvFactory provides a scalable, extensible, and robust foundation for Agentic RL.

1. Introduction

Agentic RL depends on scalable, executable environments and realistic, verified tool-use data, but existing approaches face scalability, reliability, and realism limitations. EnvFactory addresses both through autonomous environment construction and topology-aware trajectory synthesis for SFT and RL.

  • Motivation: Agentic RL requires environments that faithfully capture real-world interactions with low-latency, stable execution, alongside data reflecting contextual ambiguity and implicit reasoning.These factors support generalization, reliable reward signals, and stable policy optimization.
  • Limitations: Existing environments trade off authenticity, scalability, and reliability: production systems are costly and latency-prone, simulators hallucinate, and synthetic methods remain limited.The introduction identifies production, simulated, and synthetic environments as three major categories with distinct shortcomings.
  • Limitations: Synthetic trajectories are often over-specified, whereas realistic user requests are concise and implicit, requiring contextual reasoning and logical inference.Over-specification explicitly enumerates task requirements and reasoning steps to improve pass rates, reducing fidelity to real-world interaction patterns.
  • EnvFactory: EnvFactory unifies autonomous environment construction with topology-aware, graph-guided trajectory generation to scale executable tool ecosystems and produce realistic, verified data.Its refining stage injects implicit intents and ambiguity, transforming rigid instruction lists into more natural queries.
  • Results: 85 verified environments comprising 842 tools yielded 1,622 SFT and 953 RL multi-turn, multi-step trajectories across commerce, finance, travel, office, lifestyle, research, and utilities.The introduction states that concurrent work often uses 5 times more environments, while EnvFactory emphasizes data efficiency and training effectiveness.

2. Related Work

Related work highlights scalability and realism challenges in tool-agent environments, as well as difficulties modeling dependencies among sequential tool calls. Existing methods trade off authenticity, scalability, computational cost, and dependency coverage, motivating EnvFactory’s combined graph-construction strategy.

  • Environment Scaling for Tool Agents: Production environments use real-world APIs and MCP servers for authentic execution but are expensive to scale, network-latency prone, and destabilizing for RL training.Simulated environments instead use LLMs to emulate tool behavior and state dynamics.
  • Dependency Tool Graph: Dependency graphs model strong relationships among tools, but realistic trajectory generation remains challenging when sequential queries require multiple preceding outputs.Naive random walks often fail to fully resolve such dependencies.
  • Dependency Tool Graph: Semantic matching efficiently constructs dependency graphs but may miss implicit logical relationships, whereas LLM-based reasoning is flexible but computationally expensive and potentially inconsistent.Existing graph construction commonly relies on either semantic similarity matching or LLM-based reasoning.
  • Dependency Tool Graph: EnvFactory combines semantic matching with LLM-augmented refinement for dependency-graph construction and introduces a topological traversal approach.The passage contrasts this approach with commonly used naive random walks.

3. Method

EnvFactory constructs executable, stateful tool environments from authentic resources, validates them through iterative testing, and uses refined dependency graphs to synthesize realistic multi-turn tool-use trajectories. Its training reward combines trajectory quality, state correctness, and a length penalty.

  • Environment Construction: Each environment comprises metadata, a stateful database schema, executable Python implementation, and an MCP-based tool interface.The interface exposes tool names, descriptions, and parameter specifications to agents.
  • Environment Construction: EnvGen proposes environments from authentic external sources, models their stateful databases, implements executable tools, and adds each verified environment to the pool.Search identifies coverage gaps and retrieves API documentation, technical reports, and usage examples to guide metadata design.
  • Environment Construction: State schemas formalize entities, relationships, mutable states, parameters, intermediate states, and persistent records for isolated, reproducible rollouts.Pydantic schemas provide standardized serialization for loading and dumping environment states.
  • Environment Construction: A Test Agent iteratively checks interface consistency, execution success, expected results, and correct database transitions, then revises failures using structured error reports.The error reports localize failure sources and provide revision suggestions.
  • Tool Graph and Query Synthesis: The dependency graph models tools and parameters, combining semantic parameter matching with LLM-based logical refinement to capture missing dependencies and remove spurious edges.Refinement prevents parameter-less tools from remaining disconnected, including tools such as Notion’s delete_all_notes.
  • Tool Graph and Query Synthesis: Topology-aware sampling recursively resolves optional, external, and internally satisfiable parameters, then branches across neighbors to generate nonlinear tool-use chains.This strategy addresses the limitations of sequential random walks and supports more complex trajectory synthesis.
  • Training Objective: The reward is R = α·Rtraj + (1 − α)·Rstate − γ·Plength, combining trajectory-based reward, state-based reward, and a length penalty.α and γ ≥ 0 weight the reward components and penalty.

4. Experiments and Analysis

Experiments show that EnvFactory’s compact, verified training setup improves tool-use performance across model scales and benchmark types, with SFT providing the strongest foundation and RL adding further gains. Ablations further show that SFT initialization, refined trajectories, and balanced reward weighting are important for stable and effective training.

  • Experimental Setup: 85 environments yield 1,622 SFT and 953 RL conversations averaging 4.82 turns and 3.29 steps per turn.The environments span seven domains: commerce, finance, travel, office, lifestyle, research, and utilities.
  • Experimental Setup: EnvFactory evaluates Qwen3-1.7B, Qwen3-4B, and Qwen3-8B on BFCL v3, 𝜏2-Bench, VitaBench, and MCP-Atlas against AWM and EnvScaler.Training uses SFT followed by tool-call-only RL, implemented with LlamaFactory, VeRL, and GRPO.
  • Main Results: SFT improves BFCL multi-turn scores from 16.75 to 23.25 for Qwen3-1.7B and from 33.50 to 44.25 for Qwen3-4B.SFT also improves Qwen3-1.7B on 𝜏2-Bench from 14.61 to 15.57 and Qwen3-4B on retail from 38.60 to 47.37.
  • Main Results: RL after SFT raises overall scores from 18.60 to 19.74, 27.29 to 30.77, and 30.82 to 33.40 for Qwen3-1.7B, 4B, and 8B, respectively.On BFCL multi-turn, Qwen3-4B improves from 44.25 to 48.50 and Qwen3-8B from 46.50 to 49.00.
  • Generalization and Scaling: EnvFactory improves conversational and non-conversational benchmarks while achieving stronger BFCL-v3 performance with 85 environments and 2,575 training tasks.Increasing the environment pool consistently improves BFCL-v3 multi-turn performance, but later additions from 75 to 85 environments may add overlapping logic or task structures.
  • Ablation Studies: Refined trajectories outperform unrefined ones, direct RL is smaller and less stable than RL after SFT, and α=0.5 achieves the best BFCL peak accuracy of 41.38%.These ablations support SFT initialization, refinement for ambiguity calibration, and balanced state-based plus trajectory-matching rewards.

5. Conclusion · A. Broader Impact

EnvFactory addresses Agentic RL’s environment and trajectory bottlenecks by automatically constructing verified, stateful tool environments and realistic training trajectories. The framework broadens access to complex tool-use research while requiring safeguards against misuse and risks from online-resource and LLM-guided synthesis.

  • 5. Conclusion: EnvFactory targets two Agentic RL bottlenecks: scalable, verifiable environments and realistic trajectories with implicit reasoning.It is presented as a fully automated framework addressing both limitations.
  • 5. Conclusion: The framework autonomously constructs verified, stateful environments by exploring real-world online resources and recursively resolving tool dependencies.This contrasts with approaches relying on production APIs, hallucination-prone simulators, or static synthetic environments.
  • 5. Conclusion: EnvFactory transforms over-specified synthetic interactions into more realistic training trajectories, helping bridge the realism gap in tool-use learning.The supplied passage introduces this trajectory transformation as the framework’s second response to existing limitations.
  • A. Broader Impact: The framework automates executable-environment and trajectory construction, lowering the barrier to developing robust agents for complex tool-use.This broader contribution is described as making agent development more accessible.
  • A. Broader Impact: By offering an alternative to costly APIs and hallucination-prone simulations, EnvFactory supports broader participation in Agentic RL research across diverse, high-fidelity domains.The passage specifically mentions domains such as finance and research, although its list is truncated.
  • A. Broader Impact: Rapid generation of executable tool-use ecosystems could enable malicious automation, including large-scale financial fraud or sophisticated phishing, without safeguards.The passage calls for responsible oversight when applying the framework to sensitive domains.
  • A. Broader Impact: Reliance on online resources and LLM-guided proposals introduces additional risks that require responsible oversight.The supplied passage begins identifying these risks but is truncated before specifying them fully.

B. Limitations · C. LLM Usage Declaration

EnvFactory’s stateful MCP interface requires isolated sessions and dedicated connections, limiting parallelism and creating a synthesis-throughput bottleneck. The authors state that LLMs were used only for language editing and polishing, with all AI-modified content reviewed and validated.

  • B. Limitations: Stateful MCP servers require strict session isolation to prevent cross-contamination.Write-capable tools can modify a shared environment database.
  • B. Limitations: Each conversation requires a dedicated transport connection to its target servers.This requirement follows from the stateful MCP design.
  • B. Limitations: Dedicated connections constrain the degree of parallel tool invocation.The limitation arises during large-scale data synthesis.
  • B. Limitations: The MCP architecture creates a throughput bottleneck during large-scale data synthesis.The manuscript identifies limited parallel invocation as the source of this bottleneck.
  • C. LLM Usage Declaration: LLMs were used strictly for language editing and textual polishing.Their stated purpose was to enhance presentation quality.
  • C. LLM Usage Declaration: The authors declare that the ideas, framework, experiments, and data analysis are their original work.The declaration covers the novel ideas, methodological framework, experimental execution, and data analysis.
  • C. LLM Usage Declaration: The authors carefully reviewed and validated all content modified by AI tools.The stated purpose of review was to ensure accuracy.

D. Compute Usage · E. Additional Related Work

EnvFactory’s compute profile covers GPU requirements for trajectory synthesis and model training, while its related-work context traces RL advances, optimization granularity, and challenges surrounding RLVR exploration and capability gains.

  • D. Compute Usage: 20 GPU hours synthesizes 1,000 multi-turn, multi-step trajectories with Qwen3-30B-A3B-Thinking-2507 on 2 × 80GB GPUs.The model generates data and distills reasoning processes during SFT data synthesis.
  • D. Compute Usage: 10 GPU hours fine-tunes Qwen3-4B for 3 epochs using LlamaFactory on 8 × 80GB GPUs.This is the reported SFT training configuration.
  • D. Compute Usage: 20 GPU hours trains Qwen3-4B for 10 epochs using VeRL on 8 × 80GB GPUs.This is the reported RL training configuration.
  • D. Compute Usage: Asynchronous trajectory generation supports scaling, with 1,000 multi-turn, multi-step trajectories taking roughly 20 hours or approximately 1.2 minutes per conversation.The section also summarizes token consumption across environment construction and query synthesis.
  • E. Additional Related Work: RL post-training progressed from reward-model pipelines and DPO to RLVR, while GRPO aggregates group-level outputs and TreeRPO supplies tree-sampled, step-level dense rewards.These methods target downstream performance, generalization, and finer-grained guidance of intermediate reasoning steps.
  • E. Additional Related Work: RLVR’s intrinsic-capability gains remain contested because Yue et al. report no Pass@k improvement, alongside concerns about rapid output-entropy decline during training.Pass@k is described as closely coupled to an LLM’s reasoning upper bound.
  • E. Additional Related Work: SvS stabilizes entropy and boosts Pass@k through self-play-style problem augmentation, while DARS improves Pass@1 and Pass@k using difficulty-adaptive sampling with large-batch training.Both approaches address exploration collapse or training biases through altered data or rollout strategies.

F. Implementation Details

EnvFactory’s implementation combines specialized models for environment and trajectory synthesis with GRPO- and SFT-based training pipelines. Evaluation and simulation settings specify computational budgets, tool-use sampling constraints, and limited MCP-Atlas coverage.

  • Data Synthesis Setup: Kimi-K2-Thinking proposes, drafts, constructs, and verifies MCP environments in EnvGen.DeepSeek-V3.2-Chat generates RL tool-use trajectories, while Qwen3-30B-A3B-Thinking-2507 synthesizes SFT trajectories to distill reasoning.
  • Reinforcement Learning Setup: GRPO with Verl trains RL models on 8 × 80 GB GPUs for 10 epochs, using learning rate 1 × 10^-6, rollout size 8, and batch size 256.Maximum trajectory and generation lengths are 16k and 4k tokens; each interaction turn is an individual training sample.
  • Supervised Fine-Tuning Setup: SFT with LlamaFactory runs on 8 × 80 GB GPUs for 3 epochs at learning rate 1 × 10^-6 and batch size 256.RL initializes from the checkpoint after the first SFT epoch; tool-call or user-interaction steps become separate samples, and failed calls are filtered.
  • MCP-Atlas Setup: 30 of 36 MCP-Atlas servers and 291 of 500 tasks are evaluated because of network connectivity constraints.Excluded servers are mongodb, oxylabs, brave-search, wikipedia, slack, and google-workspace.
  • Simulated User Details: Simulated users are restricted to user-authorizable or physically constrained tools and receive only scenario, dialogue-history, and current-database-state context.Their knowledge is limited to externally available parameters, excluding internal identifiers and backend state.

G. Data Statistic

This section compares environments and training samples across baselines, marking higher efficiency with ¥.

  • Table 5 compares baselines by the number of environments and training samples, with ¥ indicating higher efficiency.

H. Algorithms · I. Prompts

The paper combines topology-aware graph sampling with executable MCP-generation and validation prompts. Together, these components enforce feasible tool dependencies, structured scenario coverage, systematic diagnosis, and justified graph refinement.

  • H. Algorithms: Topology-aware sampling guarantees that every sampled tool has its required inputs satisfied before inclusion, avoiding infeasible naive random walks.The method operates on a directed dependency graph and processes each node through backward dependency resolution followed by forward expansion.
  • H. Algorithms: Backward resolution recursively satisfies invalid inputs by sampling producer tools, with recursion capped at D_max = 3 and stochastic override p = 0.1.An input is valid when optional, user-providable, or already produced by a visited tool.
  • H. Algorithms: Forward expansion adds a dependency-resolved node to the visited set and samples one outgoing neighbor for subsequent processing.Outgoing neighbors are defined by directed edges from the current tool.
  • I.1. Prompts for EnvGen: Generated scenarios specify unique IDs, complexity levels, expected behavior, and complete scenario data, including boundary cases that should trigger validation errors.Allowed complexity levels are simple, medium, complex, and boundary; expected_behavior defaults to pass unless invalid input is tested.
  • I.1. Prompts for EnvGen: The validator executes all available tools except load_scenario and save_scenario, reuses one client_id, continues after failures, and classifies expected validation errors.Its structured result records scenario status, loading and saving outcomes, per-tool results, and detailed unexpected-error diagnoses.
  • I.2. Prompts for ToolGraph: The ToolGraph refinement prompt adds only missing, justified directed edges by evaluating semantic complementarity, data-flow feasibility, workflow plausibility, and parameter/context alignment.It receives tool descriptions and the current adjacency map, where an edge indicates that a successor may depend on or follow its predecessor.

Implementation Architecture · 4. Error Diagnosis

The error-diagnosis architecture classifies failures and records where they occurred, what happened, what was expected, and why the failure arose.

  • 4. Error Diagnosis: Failures are classified by error type, including tool execution error, state inconsistency, and schema mismatch.These categories provide concrete labels for diagnosing execution failures.
  • 4. Error Diagnosis: A tool execution error is one example of the failure types surfaced by the architecture.The framework distinguishes this from state inconsistency and schema mismatch.
  • 4. Error Diagnosis: Diagnosis identifies the specific tool or method where the failure occurred.The architecture records the error location rather than only the failure category.
  • 4. Error Diagnosis: The diagnostic record includes the actual error message and a stack trace when available.This preserves the concrete execution evidence needed for investigation.
  • 4. Error Diagnosis: Diagnosis compares the expected behavior with what actually happened.This comparison makes the discrepancy explicit for each failure.
  • 4. Error Diagnosis: Root-cause analysis explains why the failure occurred.Examples include load_scenario failing on empty lists or a tool returning the wrong field name.

3. Fix Implementation Guidelines · Background Analysis · Target Tool Analysis

The guidelines emphasize root-cause fixes that preserve existing functionality and satisfy MCP generation requirements, while background and target-tool analyses use prior context, user constraints, and tool-call subgoals to construct coherent queries.

  • 3. Fix Implementation Guidelines: Fix root causes rather than symptoms while preserving passing scenarios, original functionality, and structure.The guidelines also require adherence to MCP tool generation requirements and mental testing of edge cases.
  • 3. Fix Implementation Guidelines: Verify complete Pydantic models, full state loading and saving, correct self.xxx access, exact schemas, error handling, and initialized reference data.These checks cover both state management and tool-output correctness.
  • Background Analysis: Analyze previous turns to determine what has already been accomplished.This analysis informs how subsequent actions should be framed.
  • Background Analysis: Use the scenario and user profile to shape tone, preferences, and constraints such as budget-conscious or eco-friendly requirements.The user profile provides contextual constraints for the response.
  • Target Tool Analysis: Analyze target tool calls to identify their underlying subgoals and determine whether their relationships are sequential, parallel, or conditional.This establishes the execution logic that the natural-language query must motivate.
  • Target Tool Analysis: Weave all tool subgoals into one cohesive natural-language query with smooth transitions and logical flow, concatenating independent subgoals when appropriate.The query should naturally motivate every specified tool execution.
Loading 2605.18703v1…