PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning

Zhaoyang Wang, Canwen Xu, Boyi Liu, Yite Wang, Siwei Han, Zhewei Yao, Huaxiu Yao, Yuxiong He

arXiv:2602.10090v3cs.AIcs.CLcs.LG

TL;DR

Agentic RL is limited by the scarcity, cost, and unreliability of diverse training environments. AWM provides a fully synthetic pipeline for generating executable, database-backed tool-use environments at scale, and agents trained on them generalize to unseen environments across three benchmarks.

  • Problem

    Training multi-turn tool-use agents requires diverse environments, but real-world, human-created, and existing synthetic environments are limited, expensive, or difficult to scale reliably.

  • Method

    AWM generates scenarios, tasks, database-backed state, MCP interfaces, executable transitions, and verification code to create fully executable synthetic environments for multi-turn RL.

  • Results

    Agents trained on AWM generalize to out-of-distribution unseen environments across three tool-use benchmarks.

  • Takeaways & Limitations

    AWM supplies 1,000 diverse environments and 10,000 tasks as an open-source resource for large-scale agentic RL.

  • Takeaways & Limitations

    Experiments train on only 526 of the 1,000 environments and focus mainly on Qwen3 models at 4B, 8B, and 14B scales.

Abstract

from arXiv · show

Recent advances in large language model (LLM) have empowered autonomous agents to perform multi-turn interactions with tools and environments. However, scaling such agent training is limited by the lack of diverse and reliable environments. In this paper, we propose Agent World Model (AWM), a fully synthetic environment generation pipeline. Using this pipeline, we scale to 1,000 environments covering everyday scenarios, in which agents can interact with rich toolsets and obtain high-quality observations. Notably, these environments are code-driven and backed by databases, providing more reliable and consistent state transitions than environments simulated by LLMs. Moreover, they enable more efficient agent interaction compared with collecting trajectories from realistic environments. To demonstrate the effectiveness of this resource, we perform large-scale reinforcement learning for multi-turn tool-use agents. Thanks to the fully executable environments and accessible database states, we can also design reliable reward functions. Experiments on three benchmarks show that training exclusively in synthetic environments, rather than benchmark-specific ones, yields strong out-of-distribution generalization. The code is available at https://github.com/Snowflake-Labs/agent-world-model.

1. Introduction

Training multi-turn tool-use agents is constrained by environments that are expensive, scarce, difficult to scale, or unreliable. AWM addresses this gap with 1,000 executable synthetic environments and reports out-of-distribution generalization across three benchmarks.

  • Motivation: Large-scale reinforcement learning for tool-use agents requires diverse environments, but real-world and human-created environments are expensive, resource-scarce, and difficult to scale.Many real-world scenarios lack public APIs, while existing human-created sets contain only a few environments.
  • Motivation: Scalable environment synthesis is needed to provide executable, reliable environments that support replicable agent interaction and learning.Code-based construction is presented as a way to control state transitions and maintain consistency.
  • AWM: AWM synthesizes executable tool-use environments by generating tasks, database schemas, interfaces, backend code, and verification from high-level scenarios.The interface is exposed through MCP, while verification compares database states before and after execution.
  • AWM: 1,000 unique environments span everyday scenarios, provide dozens of tools, support parallel isolated instances, and can be reset or restarted for efficient online RL.The environments are used for large-scale multi-turn RL with 1,024 environment instances per step.
  • Results: Agents trained on AWM generalize to out-of-distribution unseen environments across three tool-use benchmarks.The training environments and tasks are not tailored to specific benchmarks or scenarios.

2. Related Work

Prior agent training relies on static data, real APIs, small environment sets, or synthetic simulations that can be costly and inconsistent. AWM contributes a programming- and database-based alternative that synthesizes large-scale executable environments without predefined task sets or API documentation.

  • Tool-use Agents: Tool-use agents have been trained with supervised data, API documentation, generated trajectories, or interactive environments, but existing training grounds remain static, real-world, or small-scale.These settings are difficult to use for RL requiring extensive interactions, fast execution, and reliable state transitions.
  • Agent Data Synthesis: Synthetic agent data methods generate tasks, tool specifications, and trajectories to scale training, but they do not by themselves provide complete executable environments.The related work describes synthetic data primarily as a source of training examples and interaction traces.
  • Environment Synthesis: Environment synthesis follows LLM-based simulation and programming-based synthesis.LLM-based simulation generates transitions and observations through prompting, whereas programming-based synthesis uses code and databases.
  • Environment Synthesis: LLM-based simulation can hallucinate state transitions and is inefficient for RL because each environment step may require an LLM call.Programming-based synthesis instead drives transitions and observations through code.
  • AWM: AWM synthesizes from scratch, uses database-backed state management and code-augmented verification, and provides 1,000 environments, 35,062 tools, and 10,000 verified tasks.The pipeline avoids assuming predefined task sets or API documentation and is described as the largest open-source environment set to date.

3. Agent World Model

AWM constructs partially observable, executable environments by turning scenarios and tasks into database-backed state, MCP tools, code-driven transitions, and task-specific verification. The pipeline produces large-scale artifacts with automated correction, while verification combines database checks with trajectory-level judgment to address imperfect synthetic execution.

  • Environment: AWM models each environment as a POMDP with state, actions, observations, transitions, and task-specific rewards.The database defines state, the interface defines actions, observations, and transitions, and verification provides rewards.
  • Scenario and Task Synthesis: Scenario synthesis generates diverse stateful applications, and task synthesis creates 10 API-solvable tasks per scenario under a post-authentication assumption.The design avoids purely UI-dependent actions and focuses on deeper functionality rather than access control.
  • Database and Interface: The database layer constructs SQLite schemas and synthetic data, while the MCP interface exposes Python tools whose operations read and write database state.The schema guides the minimal toolset needed to make the generated tasks executable.
  • Verification: Verification compares database states before and after execution to derive task rewards, then combines structured signals with the agent trajectory through an LLM-as-a-Judge.The verifier returns Completed, Partially Completed, Agent Error, or Environment Error, while accounting for incomplete updates, execution failures, and timeouts.
  • Verification: Code-driven verification is complemented by LLM-as-a-Judge because state-only success criteria can be fragile under transient failures, partial executions, and infrastructure issues.The paper reports that code-augmented LLM-as-a-Judge is more robust in its analysis and failure-case study.
  • Self-Correction: Execution-based self-correction runs and tests each generated component, feeding errors back to the LLM for regeneration when failures occur.This mechanism is applied across synthesis stages to handle runtime generation errors.
  • Pipeline Results: 1,000 environments and 10,000 tasks were synthesized, with over 85% first-attempt success and 1.13 average repair iterations; AWM has 5× more environments than EnvScaler.The reported statistics support large-scale synthesis with minimal human participation beyond 100 scenario names.
  • Pipeline Results: 62.6% of tasks completed under the 20-step budget, while 13.7% exceeded that budget.These complexity statistics were measured with Claude-4.5-Sonnet as the agent backbone.

4. Agentic Reinforcement Learning

AWM supports online RL for tool-use agents with hybrid rewards and history-aware optimization. The training procedure combines executable task verification with inference-consistent truncated histories.

  • Training setup: AWM performs online reinforcement learning for tool-use agents using Group Relative Policy Optimization (GRPO).Agentic interaction involves long-horizon trajectories with interleaved observations and tool calls.
  • Reward design: Hybrid rewards combine step-level format correctness with task-level outcome verification.Invalid tool calls trigger early termination and an immediate negative reward; normally terminated rollouts receive task-level rewards.
  • Reward design: Task-level rewards are 1.0 for completed tasks, 0.1 for partially completed tasks, and 0.0 otherwise.
  • History-aware training: Training with full histories creates a distribution mismatch because deployment frameworks may truncate long interaction histories.The optimization therefore applies the same truncation used during inference.
  • History-aware training: The objective splits trajectories into subtrajectories conditioned on truncated histories, aligning optimization with inference-time execution.Group-relative advantages are computed from rollout rewards under GRPO.

5. Experiments

AWM is evaluated for out-of-distribution tool-use generalization against base agents and alternative environment-training methods. Across three benchmarks, it shows broad gains and strongest overall generalization, while the benchmark suite also exposes important scope differences.

  • Evaluation: Agents are evaluated on τ 2-bench, BFCLv3, and MCP-Universe, which cover conversational tasks, function calling, and real-world MCP workflows.The benchmarks differ substantially from the training distribution and use different tool-calling formats.
  • Baselines: Baselines include untrained base models, RL in LLM-simulated environments, and EnvScaler’s synthesized programming-based environments.
  • Implementation: AWM training uses Qwen3 thinking models at 4B, 8B, and 14B scales, with 526 environments and 3,315 tasks due to computation limits.
  • Main results: 12.11 points: the 8B BFCLv3 overall score rises from 53.83 to 65.94 with AWM, surpassing Simulator and EnvScaler.AWM improves performance across all models, with broadly distributed gains and modest weakness on hallucination.
  • Main results: AWM is competitive with EnvScaler on τ 2-bench, exceeds Simulator there, and achieves the best overall results on MCP-Universe.The comparison with Simulator suggests programming-based state consistency provides a more stable learning signal while reducing RL latency.
  • Complexity analysis: AWM improves over Base across all complexity buckets on BFCLv3 and τ 2-bench, although absolute gains shrink on harder tasks.Relative gains remain substantial on harder tasks.

6. Analysis

AWM environments are executable, diverse, challenging, and sufficiently reliable for large-scale agentic RL. Its code-augmented verification and scalable training setup support robust evaluation and continued performance gains as environment count increases.

  • AWM evaluates environments on quality, difficulty, and diversity, the three properties identified as crucial for agentic RL training.
  • Quality: The environment error rate remains around 4% during RL training, while AWM yields fewer blocked tasks than EnvScaler.Manual inspection attributes 44% of bugs to unhandled edge cases and 14% to database-constraint conflicts.
  • Quality and diversity: Claude-4.5 achieves 99% environment-code success, Qwen3.5 reaches 77%, and diversity remains essentially constant across generator models.
  • Difficulty: 36.1% and 62.6% Pass@1 rates for GPT-5.1 and Claude-4.5-Sonnet respectively show that AWM tasks are non-trivial, with pass rates decreasing as complexity rises.Additionally, 69% of Very-Hard tasks are unsolved by either model.
  • Diversity: Embedding diversity remains stable while category coverage increases, and cross-environment AST function duplication is 0.0% with endpoint-name Jaccard of 0.004.
  • Verification: Code-augmented verification achieves the best results across model scales and benchmarks by combining structured state differences with reasoning-LLM judgments.GPT-5 judging costs about $1.80 per training step, while asynchronous execution adds negligible latency.
  • Judge reliability: GPT-5.1 achieves 95.5% pairwise agreement, Fleiss’ κ of 0.891, and a 9.2% reward-flip rate across repeated trajectory judgments.

7. Conclusion

AWM is a scalable, code-driven pipeline that produces executable, database-backed environments for tool-use agents. Across three benchmarks, agents trained on these synthetic environments generalize well to out-of-distribution domains and outperform comparison methods.

  • AWM mirrors software development to synthesize 1,000 executable environments with 10,000 tasks, SQL-backed state, a unified MCP interface, and parallel isolated instances.

Impact Statement

The paper releases an open-source pipeline and synthesized environments for training tool-use agents. It aims to lower the barrier for research on agentic systems, while warning that synthetic environments may not fully reflect reality.

  • The released pipeline and environments lower the barrier for research on agentic systems, but synthetic environments may not fully reflect real-world scenarios.

Limitations

The limitations concern fixed, imperfect synthesis, restricted training coverage, and incomplete robustness and safety evaluation. These boundaries motivate self-evolving generation, stronger semantic validation, broader training, and deployment safeguards.

  • Opportunities for Self-Evolving: AWM’s fixed generation process limits autonomous improvement beyond its initial capabilities.The authors identify self-evolving environment synthesis as future work.
  • Synthesis Pipeline Optimization: Runtime-error-driven self-correction may miss logical inconsistencies or subtle bugs, and current tasks do not span multiple scenarios or environments.The authors suggest proactive semantic checking, human inspection, and cross-environment tasks.
  • Training Scale and Model Coverage: Experiments train on 526 of 1,000 environments and mainly use Qwen3 models at 4B, 8B, and 14B scales.
  • Robustness & Safety: AWM exposes agents to synthesis noise but does not evaluate intentionally adversarial perturbations such as corrupted database rows, malicious tool outputs, or prompt injection.
  • Robustness & Safety: Post-authentication task generation leaves access-control behavior untrained, creating a deployment boundary for unauthorized access and irreversible mutations.The authors call for adversarial-robustness evaluation, access-control training, and domain-specific human oversight before production deployment.

Use of AI Assistants

The appendix details AWM’s synthesis components, shared self-correction, tool abstraction, validation, and verification-code generation for executable agent environments.

  • Environment synthesis: The pipeline synthesizes POMDP components spanning database state, tool interfaces, and task-specific verification.The database defines state, the interface defines actions and observations, and verification supplies rewards.
  • Database synthesis: Generated database schemas are executed and self-corrected through error summaries, with acceptance when fewer than 10% of tables fail.The loop retries schema generation using captured execution errors and summarized fixes.
  • Data synthesis: Sample data is generated from task preconditions so existing records satisfy operations such as inventory updates.The generator outputs table-specific INSERT statements for task-executable states.
  • Interface synthesis: A separate interface specification precedes code generation because direct generation often creates inconsistent interfaces for environments with 30+ tools.The specification defines the minimal endpoint set before implementation.
  • Verification: Each task receives verification code that compares initial and final database states and returns structured completion signals for judging.The verifier extracts changed records, expected outcomes, and diagnostics from SQL queries.
  • Agent interaction: Agents use two meta-tools to discover environment-specific tools and invoke them with JSON arguments, while rule-based checks detect malformed interactions.Format errors and server errors can trigger immediate trajectory termination.

A.5. Training and Evaluation Details

The training setup uses parallel isolated environments, history-aware optimization, structured reward computation, and cross-format benchmark evaluation. Ablation and reliability studies assess training efficiency and judge consistency.

  • Training setup: Each training step launches 1,024 isolated MCP environment instances backed by independent SQLite copies.Instances are reset after rollouts to prevent concurrent state interference.
  • History management: History-aware training splits each rollout into turn-specific samples so optimization mirrors truncated inference context.This removes the training–inference distribution shift but increases forward passes by a factor of T.
  • Reward computation: The code-augmented judge compares initial and final database states, combines structured verification with trajectories, and assigns four outcome labels.The classifications are Completed, Partially Completed, Agent Error, and Environment Error.
  • Evaluation: Evaluation covers three benchmarks with different tool-calling formats, bridged by format converters.The benchmarks use direct tool names, function-calling syntax, and the MCP protocol.
  • Reward analysis: 27% lower average rollout time is achieved with step-level format correctness rewards.Without this reward, format errors remain above 20% after 50 optimization steps and task completion saturates below 40%.
  • Judge reliability: 91.2% pairwise agreement is achieved by GPT-5.1 under strict four-class judging, compared with 82.7% for Qwen3.5.The study samples 100 trajectories and evaluates each five times with three judges.

B.3. Code-Level Diversity Analysis

AWM’s code-level analysis finds no function-level structural duplicates across its 1,000 environments at the stated TSED threshold, supporting substantial implementation diversity.

  • Structural similarity: No function-level structural duplicates occur across any environment pair at TSED threshold 0.5.The analysis applies AST-based Tree-Structure Edit Distance to all 1,000 environments.
  • Lexical similarity: Maximum lexical similarity remains low, with token-, endpoint-, and class-name comparisons indicating limited overlap.The reported maximum reaches 0.324, with residual similarity potentially coming from shared Python and FastAPI keywords.
  • Overall finding: The code-level findings corroborate the embedding-level diversity analysis and support AWM’s suitability for large-scale environment synthesis.The conclusion concerns the synthesized environment collection rather than a single scenario.

B.4. Case Study for Verification

The verification case study illustrates how combining structured database evidence with trajectory reasoning handles imperfect, idempotent, and ambiguous tool interactions more robustly than either rigid code-only or trajectory-only judging.

  • Case studies: A clean database-grounded query lets the verifier deterministically confirm returned records, illustrating the strongest regime for structured evidence.The judge and verifier agree when the database state and trajectory clearly support completion.
  • Case studies: For an idempotent routine-creation task, code-only verification would flag failure despite unchanged snapshots, while the code-augmented judge marks completion.Trajectory context resolves the mismatch between identical states and a successful execution path.
  • Design rationale: The design addresses transient failures, idempotent tasks, and ambiguous tool behavior by combining database snapshots, rule-based checks, and trajectory context.This combination is intended to reduce both false negatives and false positives.
  • Case studies: In a duplicate-event case, a judge without verification decides Completed even though the agent failed, whereas the verifier-augmented judge correctly decides Failed.The comparison demonstrates the value of structured verification for detecting misleading trajectories.
Loading 2602.10090v3…