AutoForge: Automated Environment Synthesis for Agentic Reinforcement Learning

Motivation

• Prior simulated-environment work was limited to semi-automated synthesis or insufficiently challenging tasks, while simulated-user instability remained neglected.These limitations constrained the breadth and depth of agent training and could undermine agent feedback.Source: §1 · p. 1

Approach

• The framework monitors simulated-user accuracy during rollouts and masks trajectories whose failures result from user errors, promoting fairer advantage estimation.This mechanism is designed to reduce bias from unstable simulated-user feedback.Source: §1 · pp. 1–2
• AutoForge combines an automated pipeline for synthesizing environments and challenging tasks with Environment-level Relative Policy Optimization (ERPO).The pipeline begins from tool documentation; tool dependencies and reasoning-enriched sequences are used to form task blueprints, while ERPO targets multi-environment agentic RL.Source: §1 · pp. 1–2

Evaluation

• AutoForge was evaluated on τ-bench, τ2-Bench, and VitaBench, with findings indicating improved task-solving after extended multi-turn interaction.The reported analyses also examine out-of-domain generalization and training stability.Source: §1 · p. 2

Findings

• The study reports strong out-of-domain generalization and robust training stability, supporting simulated environments as a setting for agentic reinforcement learning.The conclusion presents these findings as evidence for the promise of mock environments in agentic RL.Source: §1 · p. 2

Agentic Reinforcement Learning

• Agentic reinforcement learning trains agents through sequences of interactions with environments to maximize cumulative reward.Related work extends language models with external tools as action spaces or interactive services.Source: §2 · p. 2
• Search, code-generation, and general tool-manipulation agents use retrieval, execution feedback, or API invocation to train tool-using language models.Examples include search agents, CodeRL, DataMind, ToolRL, and ToolN1.Source: §2 · p. 2

Environment Emulation

• Environment emulation addresses the high wall-clock cost of interacting with real-world tools, where idle waiting can dominate experiments.Prior work therefore explores simulated environments to provide more efficient feedback.Source: §2 · p. 2
• AutoForge’s framework pairs scalable synthesis of simulated environments and difficult verifiable tasks with an ERPO algorithm for multi-environment agentic RL.The framework figure presents these components together as its core design.Source: Figure 1 · p. 3

Environment and Task Synthesis

• AutoForge’s synthesis pipeline converts tool documentation into environment states, executable functions, complex tool sequences, and task blueprints for RL training.The environment is represented by state structure and operations, while task difficulty comes from tool sequences augmented with reasoning and dependency structure.Source: §3.1 · p. 3, §3.2 · p. 4, §3.3 · p. 4

Environment Synthesis

• Synthetic environments store deterministic, extensible state structures as key–value attributes, while an LLM generates compatible Python implementations from tool descriptions and those structures.Attribute names are generated during environment construction; specific values are instantiated later to maintain consistency between environments and tasks.Source: §3.1 · p. 3

Tool-Sequence Generation

• Tool sequences are sampled from a dependency graph, merged to combine requirements, and enriched with reasoning nodes and edges to form complex directed acyclic graphs.LLM-based redundancy removal makes merged sequences more coherent, while reasoning nodes derive higher-level information from preceding outputs.Source: §3.2 · p. 4

Task Generation

• Tasks are generated from instantiated environments and DAGs, with completion verified by comparing the final environment state against a gold state rather than requiring one exact tool sequence.This allows multiple valid tool-call sequences to solve the same task while retaining verifiable outcomes.Source: §3.3 · p. 4

Environment-Level Agentic RL

• The RL stage uses user-centered multi-turn rollouts, preserves prior reasoning across user queries, and compares the resulting state with the gold state to compute rewards.The rollout ends when the simulated user judges that the task requirements are satisfied.Source: §3.4 · pp. 4–5
• An LLM identifies simulated-user errors and masks affected trajectories so they do not influence advantage or loss computation, reducing distortion from erroneous feedback.The method targets failures such as hallucinated or omitted user information during rollouts.Source: §3.4 · p. 5

In-domain Performance

• AutoForge improves performance on 𝜏-bench, 𝜏2-Bench, and VitaBench, outperforming its Qwen3-Thinking-30B-A3B backbone and approaching advanced closed-source models.The comparison uses main results across the three agentic benchmarks.Source: Table 1 · p. 6, §4.2 · p. 7

Out-of-Domain Performance

• Both SFT and RL variants improve ACEBench-zh performance over Qwen3-Thinking-30B-A3B, with the RL variant achieving larger gains.ACEBench-zh differs in multi-turn and tool-invocation formats and includes unseen tools, making it an out-of-domain evaluation.Source: §4.2 · p. 7, Figure 2 · p. 7

Impact of the User Agent

• Stronger simulated users improve 𝜏2-Bench scores, with the largest gains in telecom tasks where users can invoke tools.These findings indicate that simulated-user reliability affects fair and accurate agent evaluation.Source: Table 2 · p. 7, §4.3 · p. 8

Environment-Level Advantage Estimation

• Environment-level advantage estimation produces higher and more stable rewards than group-level estimation by improving standard-deviation estimates within each environment.Figure 3a compares the corresponding reward curves.Source: §4.3 · p. 8, Figure 3 · p. 8

Masking Out Erroneous User Behaviors

• Masking trajectories caused by simulated-user errors stabilizes training by preventing correct agent actions from being penalized for user-induced task failures.Without masking, the training curve declines during later stages.Source: §4.3 · p. 8

Effectiveness of Interleaved Thinking

• Interleaved thinking enhances agent capabilities by preserving prior-turn analyses and plans for reflection during subsequent decisions.The comparison is presented in Figure 4.Source: §4.3 · p. 8, Figure 4 · p. 8

• AutoForge automatically generates simulated environments and high-difficulty, verifiable tasks, expanding agentic RL across breadth and depth.The conclusion presents this unified pipeline as a central contribution.Source: §5 · p. 8
• MEU mitigates unfairness in advantage estimation caused by simulated-user instability, while environment-level estimation improves training stability.These mechanisms address distinct sources of instability in agentic RL.Source: §5 · p. 8
• Extensive experiments demonstrate strong agent capabilities and out-of-domain generalization, highlighting the potential of simulated environments for agentic reinforcement learning.Source: §5 · p. 8

• The pipeline requires tool description documents, limiting the inputs from which it can generate mock environments.Future work targets construction from task topics or general text.Source: Limitations · p. 9
• The current RL setup trains on only a few environments, leaving the effects of scaling environment count on training and generalization unresolved.The authors identify this scaling question as future work.Source: Limitations · p. 9
• ERPO is limited to outcome-based reward supervision rather than turn-level value supervision for step-by-step decisions.The authors propose exploring turn-level value supervision.Source: Limitations · p. 9

A.1 Masking Out Erroneous User Behaviors

• An LLM judge checks whether simulated-user actions align with the user’s overall intent and whether returned information is incorrect or out of scope.This mechanism is used to identify user mistakes that lead to task failure.Source: §A.1 · p. 11
• The detailed appendix includes a dedicated prompt for masking erroneous user behaviors and a separate prompt for the optimized user.The optimized prompt is implemented in GLM-4.5 (Zeng et al., 2025).Source: §A.1 · p. 11, §A.2 · p. 11

A.2 Optimized User Prompt

• The optimized user prompt instructs the simulator to generate one message at a time and reveal only information necessary for the current step.It also prohibits hallucinating information absent from the instruction.Source: §A.2 · p. 11
• The simulator must keep requests strictly grounded in the instruction and must not alter, relax, extend, or substitute stated constraints.The prompt explicitly preserves time, date, budget, and specific-term constraints.Source: §A.2 · p. 11
• The conversation cannot end until all requirements are expressed, all tasks are completed, and the results are correct and complete.The prompt separately states that technical inability may justify accepting a transfer to a human.Source: §A.2 · pp. 11–12
• After successful completion, the simulated user ends the conversation with the standalone marker “###STOP###”.The prompt requires checking that every instructed task is complete before emitting the marker.Source: §A.2 · p. 12
• Table 3 compares time consumption using the environment’s average execution time as the unit.Its scope is to compare the timing components discussed in the time-efficiency analysis.Source: Table 3 · p. 12

• The environment’s execution time is significantly lower than LLM-related time, demonstrating the synthetic environment’s time efficiency.The comparison covers rollout and parameter-update costs.Source: B Time Consumption Analysis · p. 12
• The main wall-clock bottleneck is waiting for the LLM-based user simulator’s reply, while function execution and return are virtually free.Source: B Time Consumption Analysis · p. 12