PaperlayerView exact revision on arXiv

Source-linked AI summary

EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesis

Xiaoshuai Song, Haofei Chang, Guanting Dong, Yutao Zhu, Ji-Rong Wen, Zhicheng Dou

arXiv:2601.05808v2cs.CLcs.AIcs.LG

TL;DR

LLM agent training needs rich, reliable tool-interaction sandboxes, yet real systems are restricted and simulated or manually built environments have important scalability and consistency limits. EnvScaler programmatically synthesizes executable environments and validated scenarios, producing 191 environments and about 7K scenarios; experiments show improved performance on complex multi-turn, multi-tool tasks across three benchmarks.

  • Problem

    LLM agent training lacks scalable access to rich, diverse tool-interactive environments because real systems are restricted and simulated environments can be inconsistent.

  • Method

    EnvScaler uses SkelBuilder to synthesize executable environment skeletons and ScenGenerator to generate states, tasks, and rule-based trajectory validation functions.

  • Results

    EnvScaler synthesizes 191 environments and about 7K scenarios, and significantly improves LLM performance on complex multi-turn, multi-tool tasks across three benchmarks.

  • Takeaways & Limitations

    Automated synthesis provides a scalable source of interactive training environments for improving tool learning and generalization in LLM agents.

  • Takeaways & Limitations

    The construction process relies on LLM synthesis, which may introduce biases compared with real systems, and coverage is limited for open environments and real-system characteristics.

Abstract

from arXiv · show

Large language models (LLMs) are expected to be trained to act as agents in various real-world environments, but this process relies on rich and varied tool-interaction sandboxes. However, access to real systems is often restricted; LLM-simulated environments are prone to hallucinations and inconsistencies; and manually built sandboxes are hard to scale. In this paper, we propose EnvScaler, an automated framework for scalable tool-interaction environments via programmatic synthesis. EnvScaler comprises two components. First, SkelBuilder constructs diverse environment skeletons through topic mining, logic modeling, and quality evaluation. Then, ScenGenerator generates multiple task scenarios and rule-based trajectory validation functions for each environment. With EnvScaler, we synthesize 191 environments and about 7K scenarios, and apply them to Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) for Qwen3 series models. Results on three benchmarks show that EnvScaler significantly improves LLMs' ability to solve tasks in complex environments involving multi-turn, multi-tool interactions. We release our code and data at https://github.com/RUC-NLPIR/EnvScaler.

1 Introduction

EnvScaler addresses the difficulty of scaling reliable, diverse tool-interactive environments for training LLM agents. It combines automated environment construction and scenario validation, improving performance on complex multi-turn, multi-tool tasks.

  • Motivation: Training capable LLM agents requires diverse environments with states, tools, and interaction logic, but real systems are restricted and simulated environments can be inconsistent.These constraints make scalable environment creation a central challenge for tool-use training.
  • Motivation: Prior approaches either model only tool interfaces, depend on preexisting environments, or lack complete interaction logic and automatic assessment.These limitations constrain the scalability and coverage of programmatically built sandboxes.
  • EnvScaler: SkelBuilder mines environment themes, constructs executable states and tools, and evaluates quality through iterative testing and checking agents.The pass rate from repeated testing indicates environment quality.
  • EnvScaler: ScenGenerator creates initial state data, challenging tasks, and terminal-state validation functions for multiple scenarios within each environment.The validation functions check whether final states satisfy expected task conditions and provide reward signals.
  • Results: 191 environments and about 7K scenarios were synthesized, and evaluations show improved performance on complex multi-turn, multi-tool interactions.The synthesized environments were applied to SFT and RL for Qwen3 series models and evaluated on multiple benchmarks.

2 Related Work

Related work studies LLM tool use through simulated environments, programmed sandboxes, and synthetic trajectories. EnvScaler targets scalable, stateful, general tool interaction without requiring access to preexisting environments.

  • Synthetic Training: Synthetic static trajectories support training data construction but generally cannot support LLM agents’ self-exploration.EnvScaler focuses on interactive environments for general tool use across domain-specific settings.
  • Tool-Interactive Environments: Tool-interactive environments let agents query information or change environmental state through tools while receiving action feedback and rewards.This interaction setting supports policy optimization for tool use.
  • Environment Simulation: Some work uses LLM reasoning and world knowledge to simulate environments, but these environments are prone to hallucinations and inconsistencies.They also lack transparency and persistent state management.
  • Programmatic Sandboxes: Programmed sandbox approaches may model isolated stateless functions, rely on environmental priors, or lack automatic assessment.These constraints limit scalability and environment coverage.

3 Automated Env Skeleton Synthesis

SkelBuilder automates the synthesis of executable, stateful tool-interaction environments from mined task contexts, then tests and filters them for quality. Its pipeline produces environment programs, documentation, tool interfaces, and validated state-transition logic.

  • Environment representation: SkelBuilder represents each environment with executable program files, documentation, and a tool interface set.Executable files implement states, tools, and rules; documentation describes the environment; interfaces expose tool names, parameters, and descriptions.
  • Task-guided discovery: It mines environment themes from existing task resources, filters for domain-specific stateful settings, and deduplicates inferred descriptions by embedding similarity.This replaces manual pre-setting or API-collection derivation with reverse inference from retained tasks.
  • Logic planning: An LLM enriches each description with state definitions, domain rules, and tool operations, forming a structured blueprint and environment documentation.The rules are concatenated with the environment description to create the documentation supplied to agents.
  • Program modeling: The program-modeling stage converts planned states into class attributes, generates rule-consistent tool methods, and assembles them into a complete Python environment class.Attributes represent environment states, while methods implement supported tool operations and state transitions.
  • Quality inspection: A dual-agent loop evaluates actual tool execution by testing positive and negative calls, checking source code and state changes, and averaging pass rates across rounds.Environments scoring below a predefined threshold are discarded; the reported construction used 100 rounds and a 0.85 threshold.
  • Synthesis outcome: 191 environments were retained, averaging 18.58 tools and 21.38 state categories per environment.These statistics indicate that the synthesized environments contain multiple tools and state dimensions.

4 Automated Env Scenario Synthesis

ScenGenerator turns each environment skeleton into multiple state-grounded task scenarios with rule-based trajectory validation. It generates tasks from initial state and environment rules, then scores trajectories by the proportion of satisfied terminal conditions.

  • Scenario construction: ScenGenerator adds initial state data, tasks, and an evaluation mechanism because an environment skeleton alone cannot support complete agent interactions.It automatically constructs multiple task scenarios for each environment.
  • State-grounded task generation: It first synthesizes an environment’s initial database state and derives challenging tasks from that state to maintain task solvability and consistency.State grounding prevents tasks from requiring unavailable entities or actions, such as canceling a nonexistent order.
  • Task generation: Rather than reversing tool-call sequences, ScenGenerator directly prompts an LLM to generate scenario-specific tasks from the initial state, tools, and rules.The paper notes that reversed sequences can produce low-quality tasks and cannot uniquely represent multiple valid solution paths.
  • Trajectory validation: For each task, an LLM decomposes requirements into verifiable conditions and generates terminal-state validation functions that return True or False.Each function receives the final environment state after the trajectory.
  • Reward design: The trajectory reward equals the proportion of validation functions that pass, capturing partial completion rather than only a single boolean judgment.Final-state evaluation is agnostic to the action sequence and accommodates multiple valid solution paths.
  • Scenario evaluation: Table 3 reports average scores and step counts for LLMs evaluated on a random sample of 50 scenarios.The table caption identifies Qwen3-30B as Qwen3-30B-A3B-Thinking-2307 and NT as Non-Think.

We consider two interaction settings:3

EnvScaler supports Non-Conversation and Conversation interaction settings, with trajectories converted into supervised targets or validation-based rewards for training.

  • In Non-Conversation settings, agents receive complete task information initially and interact through tool calls until completion or a step limit.
  • In Conversation settings, agents progressively acquire task information through dialogue with an LLM-simulated user while also invoking tools.
  • For SFT, teacher trajectories serve as student learning targets, whereas RL converts trajectories into rewards using validation functions.
  • Under Non-Conv, average trajectories contain about 15 steps, increasing to over 25 steps under Conv.

5 Experiments

Experiments evaluate EnvScaler across training strategies, model sizes, benchmarks, environment scaling, interaction patterns, direct RL, environment quality, and synthesis cost.

  • 5.2 Main Results: SFT improves performance across all three benchmarks, and adding RL further increases performance.
  • 5.2 Main Results: SFT gains average 8.67 points on BFCL-MT, 4.29 points on Tau-Bench, and 11.57 points on ACEBench-Agent across three models.
  • 5.2 Main Results: Qwen3-8B gains across all datasets with RL, while Qwen3-1.7B improves on BFCL-MT and ACEBench-Agent but slightly declines on Tau-Bench.
  • 5.2 Main Results: EnvScaler produces larger gains on BFCL-MT and ACEBench-Agent than on Tau-Bench, where Qwen3-1.7B slightly declines on the Airline task.
  • 5.3 Train-Test Env Similarity Analysis: Training on either the most similar or least similar environments significantly outperforms the baseline, with only minor differences between subsets.
  • 5.4 Effect of Scaling Environment: Qwen3-4B performance on BFCL-MT and ACEBench-Agent rises as the number of training environments increases, with the largest gain from 0 to 20 environments.
  • 5.5 Effect of Interaction Patterns: Full interaction-pattern training achieves the highest overall score, while Conv helps Miss-Parm and Non-Conv performs better on Base and Long-Context.
  • 5.6 Results of Training with Direct RL: Direct RL improves all models to varying degrees without SFT, but its gains remain limited compared with SFT plus RL and depend strongly on model size.

6 Conclusion

EnvScaler synthesizes executable tool-interactive environments and task scenarios for agent training, while its current scope excludes several real-system characteristics and modalities.

  • 6 Conclusion: EnvScaler combines SkelBuilder for environment skeletons with ScenGenerator for states, tasks, and rule-based trajectory verification.
  • Limitations: The framework mainly targets domain-specific, stateful environments and has limited support for open environments such as web search or information access.
  • Limitations: EnvScaler lacks explicit simulation of latency, network fluctuations, error patterns, and multimodal tool inputs and outputs.
  • Limitations: The current study is limited to around 200 synthesized environments and 7K scenarios because of experimental resources.

A.6 Example of Environment’s Task Scenario

A task scenario specifies an initial environment state, a task, and state-check functions, with example trajectories shown for both interaction settings.

  • Initial State Configuration: The initial state configuration initializes the environment’s state data for a task scenario.
  • Task: The task is defined under the configured initial state.
  • State Check Functions: State-check functions evaluate task checkpoints and compute the trajectory’s reward score.
  • Non-Conversation: A Non-Conversation trajectory example uses Qwen3-30B-A3B-Thinking-2507 with the environment.
  • Conversation: Conversation and Non-Conversation trajectory examples correspond to the same task scenario.

B.1 Details of Evaluation Benchmarks

The evaluation uses three multi-turn or multi-step tool-use benchmarks spanning diverse, stateful application domains and interaction challenges.

  • BFCL-v3 Multi-Turn: BFCL-v3 Multi-Turn evaluates consistent function calling across eight environments, including vehicle control, transactions, ticketing, and file systems.Its subsets test standard tasks, incomplete information, unavailable functions, and long-context interactions.
  • Tau-Bench: Tau-Bench evaluates multi-turn human–AI interactions, domain-specific API use, and compliance with complex retail and airline business rules.The airline subset involves booking, rescheduling, baggage, and insurance handling with greater reasoning challenges.
  • ACEBench-Agent: ACEBench-Agent evaluates multi-turn and multi-step tool calls in dynamic environments such as mobile apps, food delivery, financial services, and travel booking.Its subsets distinguish tasks where users provide information across turns from tasks specified once at the start.

B.2 Details of Implementation

The implementation trains Qwen3 models with SFT and RL on synthesized environments, then evaluates them on three multi-turn tool-use benchmarks using standardized inference procedures.

  • SFT: SFT uses Qwen3-30B-A3B-Thinking-2507 as a teacher across 140 environments and produces about 9K final trajectories after filtering.The teacher interacts under two settings, and invalid-format or impossible trajectories are removed.
  • SFT: Training supervises only each sub-sample’s final-round reasoning and action, using three epochs, a 1e-6 learning rate, and a 32K-token maximum sequence length.Samples are split by interaction round so reasoning can be learned separately for each round.
  • RL: RL uses Reinforce++ in the ROLL framework with a 0.1 KL coefficient, a 1.0×10^-6 learning rate, and up to 200 training steps.Each step samples 64 tasks and eight trajectories per task in the Non-Conversation setting.
  • Evaluation: Evaluation covers BFCL-v3 Multi-Turn, Tau-Bench, and ACEBench-Agent, averaging three runs at temperature 0.7 with native function-calling interfaces.Historical actions are retained while historical reasoning is removed for evaluation.

C.1 Results of Training in Non-thinking Mode

In non-thinking mode, single-turn supervision does not improve multi-turn tool use, while EnvScaler’s action-based SFT transfers across some benchmarks but can degrade reasoning-heavy performance.

  • Dataset comparison: Single-turn APIGen supervision degrades performance on all evaluated benchmarks, indicating that it cannot improve multi-turn tool-use capabilities.APIGen provides 60K single-turn tool-use examples.
  • Dataset comparison: APIGen-MT significantly improves Tau-Bench but fails to transfer to BFCL-MT or ACEBench-Agent despite containing 5K multi-turn examples.Its training environments are limited to the two environments identical to Tau-Bench.
  • Interaction settings: Training only in the Conversation setting encourages user-facing dialogue before tool use, conflicting with BFCL-MT’s interaction style.The comparison highlights the importance of training in both Conversation and Non-Conversation settings.
  • EnvScaler SFT: EnvScaler SFT significantly improves BFCL-MT and ACEBench-Agent but slightly degrades Tau-Bench in non-thinking mode.The authors attribute the Tau-Bench degradation to action-only learning when that benchmark relies more on reasoning.

C.2 Analysis of Discarded Envs

The environment pipeline filters synthesized environments through cyclic dual-agent testing, retaining those above a success threshold, and constructs stateful tasks, tools, and verification functions.

  • Environment filtering: 191 of 266 synthetic environments are retained after 100 cyclic test rounds, requiring a test success rate above 0.85 and yielding a 28.2% rejection rate.The assessment also analyzes failure cases from testing logs.
  • Task filtering: Task filtering selects actionable requests that depend on current state within persistent, domain-specific systems supporting both information queries and state-changing operations.Examples include file management and calendar systems with data that persists over time.
  • Skeleton construction: SkelBuilder infers environment descriptions, state variables, constraint rules, and information or state-modification operations from task and environment inputs.These planned operations are programmatically converted into executable class methods.
  • Scenario generation: ScenGenerator creates initial state data, generates tasks under specific environments and states, and converts verification checkpoints into functions over final states.The resulting checks assess whether agent modifications satisfy task conditions.
  • Executable environments: The synthesized environments expose structured application state and tool operations, with example schemas covering users, contacts, messages, conversations, and messaging constraints.Testing compares method behavior against pre- and post-call internal states and returned observations.
Loading 2601.05808v2…