Source-linked AI summary
ScaleEnv: Scaling Environment Synthesis from Scratch for Generalist Interactive Tool-Use Agent Training
Dunwei Tu, Hongyan Hao, Hansi Yang, Yihao Chen, Yi-Kai Zhang, Zhikang Xia, Yu Yang, Yueqing Sun, Xingchen Liu, Furao Shen, Qi Gu, Hui Su, Xunliang Cai
TL;DR
Interactive training environments are scarce, while existing synthesis methods face limitations in diversity and scalability. SCALEENV generates validated executable environments and dependency-aware verifiable tasks from scratch, and training on them improves performance on unseen multi-turn benchmarks while showing that environmental diversity matters for generalization. Its domain-agnostic synthesis also raises safety concerns because it could model harmful or unethical behaviors without oversight.
Problem
Interactive environments for agent self-exploration remain scarce, and existing synthesis methods have limited diversity and scalability.
Method
SCALEENV synthesizes executable tools, databases, and verifiable tasks using procedural testing, tool dependency graphs, and controlled graph expansion.
Results
Training on SCALEENV substantially improves performance on unseen τ 2-Bench and VitaBench benchmarks and reveals an environmental scaling curve for generalization.
Takeaways & Limitations
The results support scaling environmental diversity as important for developing robust generalist agent capabilities.
Takeaways & Limitations
Because SCALEENV can synthesize arbitrary interactive environments, it could be misused to model harmful or unethical behaviors without proper oversight.
Abstract
from arXiv · showhide
Training generalist agents capable of adapting to diverse scenarios requires interactive environments for self-exploration. However, interactive environments remain critically scarce, and existing synthesis methods suffer from significant limitations regarding environmental diversity and scalability. To address these challenges, we introduce ScaleEnv, a framework that constructs fully interactive environments and verifiable tasks entirely from scratch. Specifically, ScaleEnv ensures environment reliability through procedural testing, and guarantees task completeness and solvability via tool dependency graph expansion and executable action verification. By enabling agents to learn through exploration within ScaleEnv, we demonstrate significant performance improvements on unseen, multi-turn tool-use benchmarks such as $τ^2$-Bench and VitaBench, highlighting strong generalization capabilities. Furthermore, we investigate the relationship between increasing number of domains and model generalization performance, providing empirical evidence that scaling environmental diversity is critical for robust agent learning.
1. Introduction
SCALEENV addresses the scarcity and unreliability of interactive training environments by automating their construction from scratch. It combines validated executable components with dependency-aware task synthesis and reports improved out-of-distribution generalization.
- Interactive environments are needed for agents to act, receive feedback, and iteratively refine behavior, but synthesized tools and simulators can be unreliable.The introduction identifies realism as a core challenge alongside the need for executable tools and dynamic feedback.
- SCALEENV builds a domain foundation from LLM-generated tool and database schemas, functional code, procedural testing, and a global Tool Dependency Graph.The framework separates environment construction from task construction while validating generated components before graph consolidation.
- SCALEENV constructs a complete training ecosystem of executable toolkits, high-fidelity environment states, and verifiable user intents.The resulting ecosystem is used for agent training rather than static data interpolation.
- Training Qwen-3 models on SCALEENV yields substantial gains on unseen τ 2-Bench and VitaBench benchmarks under strictly out-of-distribution evaluation.Training domains are disjoint from evaluation domains, which also use data formats absent from training.
- The paper reports an Environment Scaling Curve and identifies environmental diversity as a critical determinant of robust generalization.The contribution frames environment scaling as a data-centric approach to agent training.
2. Related Works
Tool-learning research has moved from static demonstrations toward autonomous exploration, increasing the need for scalable interactive environments. Existing real-world and simulated alternatives trade off realism, diversity, safety, cost, or scalability.
- Earlier tool-learning methods showed that language models could master function calls from static demonstrations.The field subsequently shifted toward reinforcement learning to reduce reliance on expert trajectories and support self-exploration.
- Large-scale reinforcement learning requires scalable interaction environments for agent exploration.The paper positions environment synthesis as a response to this infrastructure requirement.
- The literature organizes agent environments around diversity, realism, and scalability.These dimensions motivate comparison among real-world and LLM-simulated environments.
- Real-world environments offer realism but suffer from limited domain availability, restricted action spaces, latency, costs, and insufficient state-altering tasks.Safety policies and remote-tool constraints further limit their use for scalable training.
3. Preliminaries
The paper formalizes agent interaction as a tool-using POMDP grounded in executable databases and tools. Tasks bind an initial environment to a user intent and profile, with rewards evaluating the final state against that intent.
- A domain foundation B = ⟨Σ, T⟩ contains databases defining valid environment states and executable tools defining available functions or APIs.The database set Σ specifies valid states, while T contains the domain’s executable functions.
- An environment E is initialized with database values conforming to the domain schema and executable tools inherited from the domain foundation.Its initial hidden state represents the external world at the start of an episode.
- A task ψ = ⟨E, u, Puser⟩ combines an environment, hidden user goals, and a user profile containing information such as permissions, location, and history.The task specifies the setting in which the agent interacts with the simulated user.
- The interaction is modeled as a POMDP whose state includes environment state, interaction history, and user intent.Actions comprise natural-language responses and tool commands, while observations comprise user feedback and tool results.
- Tool actions update the environment and history, whereas response actions update only the history with the simulated user’s reply.The final reward evaluates whether the resulting environment state satisfies the user intent.
4. Method
SCALEENV separates executable graph construction from task instantiation to synthesize reliable, interactive environments and solvable multi-step tasks. Procedural testing validates code, while dependency graphs and controlled expansion provide structured complexity for exploration.
- Framework overview: SCALEENV decouples synthesis into Executable Graph Construction and Task Instantiation, producing tasks for scalable agentic reinforcement learning.The two phases provide a modular framework for constructing interactive and verifiable environments.
- Executable Graph Construction: Tool and database schemas formalize operational logic before executable code is generated and validated as a functional sandbox.Tool schemas specify interfaces and pre/post-conditions, while database schemas derive tables and integrity constraints from tool definitions.
- Verification: A rule-based evaluator compares the final database state with the ground-truth state using field-specific matching policies.The policies distinguish exempt fields, hard constraints, and semantic matching requirements.
- Executable Graph Construction: Procedural testing executes generated tools on matched database instances and checks successful state transitions, anticipated rejections, and unexpected failures.Unexpected failures trigger debugging and iterative correction of tool code or database instances.
- Task Instantiation: Task instantiation requires entity consistency and interaction completeness so valid actions produce meaningful observations across the feasible action space.These requirements support trial-and-error exploration rather than only an optimal trajectory.
- Task Instantiation: Graph Expansion samples executable seed chains, initializes constraint-satisfying environments, and expands them through dependency-aware controlled growth.The process uses a gating policy conditioned on available tools and graph complexity to balance diversity and solvability.
5. Experiments
SCALEENV improves zero-shot generalization on unseen domains, reasoning challenges, interaction formats, and complex tasks. Experiments further show that broader environmental diversity and executable verification are associated with stronger and more reliable performance.
- Main Results: Generalization to Unseen Domains: Qwen3-SE consistently outperforms baselines across diverse domains and unseen benchmarks.The evaluation includes τ^2-Bench and VitaBench, with training domains disjoint from evaluation domains.
- Reasoning Generalization: Qwen3-SE-32B doubles base-model performance on VitaBench’s challenging cross-domain subset.The subset tests ambiguous user needs, proactive information retrieval, and complex multi-step planning.
- Domain and Format Generalization: SCALEENV improves transfer across all 7 evaluation domains and to τ^2-Bench’s novel textual-policy format.τ^2-Bench requires adherence to lengthy policies and handling complex state transitions absent from SCALEENV training.
- Performance Upper Bound Analysis: Pass@4 nearly doubles on VitaBench’s complex cross-domain subset, raising the model’s success potential across four attempts.Pass@4 measures the probability of generating at least one correct solution within four attempts.
- Domain Scaling Analysis: Zero-shot generalization rises monotonically as training domains increase from N = 2 to 16, without fully plateauing at N = 16.The domain-scaling experiment keeps the task count fixed while varying the number of unique training domains.
- Ablation Studies: Removing execution verification consistently degrades τ^2-Bench performance because runtime-invalid tool calls introduce conflicting reward signals.The ablation omits actual parameter-driven execution and iterative state patching for solvability.
6. Conclusion
SCALEENV synthesizes high-fidelity interactive environments and verifiable tasks, improving baseline performance on unseen benchmarks and supporting zero-shot generalization. The paper argues that scaling environmental diversity matters more than task quantity for developing generalist agent capabilities.
- SCALEENV synthesizes high-fidelity interactive environments and verifiable tasks through a complete generation pipeline.
- SCALEENV training significantly boosts baseline-model performance on unseen benchmarks, evidencing robust zero-shot generalization.
- Scaling environmental diversity is more critical than task quantity for cultivating generalist agent capabilities.
Impact Statement
SCALEENV addresses data scarcity in generalist tool-use-agent training with a virtual sandbox for learning complex capabilities. Its generative flexibility also creates risks of constructing harmful or unethical environments without oversight.
- SCALEENV provides a safe virtual sandbox for agents to learn complex tool-use capabilities.
- The framework is intended to mitigate accidental-harm risks associated with training directly on real-world APIs.
- Because SCALEENV can synthesize arbitrary interactive environments, inadequate oversight could enable harmful or unethical behavior modeling.
- Future work should investigate safety-alignment mechanisms that prevent malicious-domain synthesis while preserving training diversity.
A. Tool Semantic Diversity and OOD Verification
SCALEENV's synthesized domains occupy a broad semantic spectrum, while evaluation domains lie in distinct regions from training clusters. This separation provides evidence that evaluation is genuinely out of distribution rather than template memorization.
- Semantic diversity: A pretrained encoder and t-SNE projection are used to visualize tool-description embeddings across training and evaluation domains.
- Semantic diversity: The 16 synthesized domains form widely distributed clusters spanning a broad semantic spectrum from Smart Home to Healthcare Telemedicine.
- OOD verification: Evaluation domains from τ 2-Bench and VitaBench occupy distinct regions with significant semantic separation from training clusters.
- OOD verification: The visualization provides empirical evidence that the evaluation setup is out of distribution.
- OOD verification: The reported gains therefore are not attributed to simple memorization of tool templates or domain-specific logic interpolation.
B. Detailed Statistics of Synthesized Domains and Tasks
The 16 synthesized domains vary substantially in action-space scale, database-state complexity, and tool-dependency density. These dimensions characterize the structural diversity of the generated training environments.
- The 16 synthesized domains exhibit structural diversity across action-space scale, state-space complexity, and dependency density.
- Action Space Scale: Executable tools range from approximately 25 in Online Learning to over 70 in Entertainment Media Query.
- State Space Complexity: Database tables range from 5 in Job Seeking to 22 in Agriculture Environment, representing different state-tracking difficulties.
- Dependency Density: Figure 5 plots tool count against database-table count, with color intensity and bubble size encoding Tool Dependency Graph density.
- Dependency Density: Job Seeking and Knowledge Management have denser tool-dependency graphs, whereas Agriculture Environment is relatively sparser.
B.2. Domain and Task Synthesis Cost
ScaleEnv pairs synthesized domain environments with verifiable tasks and trains agents through multi-turn tool-use interaction. The synthesis process requires substantially more tokens for a complete domain foundation than for an individual task.
- Token consumption: 546k tokens synthesize one complete domain foundation, compared with 93.2k tokens for one verifiable task.Table 9 reports average token usage for domain foundations including schemas and code, and for individual verifiable tasks.
- Training universe: A complete domain environment pairs a synthesized environment with a verifiable task in the unified training universe.The framework represents each instance as (B_k, ψ_j), where ψ_j belongs to the tasks of environment B_k.
- Policy optimization: GRPO samples groups of trajectories and optimizes the agent policy over tool-use time steps.The objective uses step-wise importance sampling and group-derived advantages, with trajectory rewards normalized by intra-group statistics.
D. Examples of Tool and Database
The Job Seeking example implements a structured database and executable tools for managing applications, stages, interviews, feedback, and related metadata. Its tools validate inputs, update records, and return operation-specific results.
- Database schema: The Job Seeking domain uses structured entities for applications, notes, stages, interview schedules, and interview feedback.The database schema includes linked records such as application notes, application stages, interview schedules, and interview feedback.
- Application records: Job applications store identifiers, applicant details, job information, status, dates, referral sources, and related application materials.Fields include application_id, applicant_name, email, phone, job_title, company_name, application_date, status, referral_source, and timestamps.
- Executable tools: The delete tool validates an application identifier, checks record existence, removes the application, and returns deletion status with a timestamp.Missing tables or applications raise errors, while successful deletion returns the application ID, status, and deletion time.
- Executable tools: The batch-update tool validates a list of application IDs, status, and timestamp, then reports updated_count and failed_updates.Nonexistent applications are added to the failed list while valid records receive the new status and formatted timestamp.
- Executable tools: The archive tool updates applications older than a cutoff date to an archive status.The cutoff date is specified in yyyy-mm-dd format, and the default archive status is archived.
E. Agent-User Interaction Trajectory in ScaleEnv
ScaleEnv supports multi-turn job-application interactions in which users request searches, scheduling, status histories, and feedback recording. Tool calls produce structured results that are then summarized conversationally.
- Application search: A user request for energy-analyst applications leads to a keyword search over job titles.The system returns matching applications including PetroChina Group, CNOOC, Envision Energy, Tencent Technology, and others.
- Status tracking: Stage-history queries expose application progress, including passed screening and waiting for a technical interview.For APP001, the history records resume screening passed and a later waiting-for-technical-interview stage.
- Interview scheduling: A scheduling request triggers tools that add an interview schedule, application note, and a preparation-time calculation.The recorded interview is for Senior Energy Analyst at PetroChina Group, with 3 days or about 76 hours before the interview.
- Feedback recording: The system records interview feedback with a self-rating of 4/5 and links the resulting feedback record to the interview.The feedback covers industry-trend discussion and identifies model-detail responses as an area for improvement.
- Interview tracking: The system retrieves interview details for an application, including interview type, date, interviewer, and location.For APP002, the returned record identifies a phone screening on 2024-03-08 with Liu HR by phone.