PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning

Zhaoyang Wang, Canwen Xu, Boyi Liu, Yite Wang, Siwei Han, Zhewei Yao, Huaxiu Yao, Yuxiong He

arXiv:2602.10090v3cs.AIcs.CLcs.LG

TL;DR

Training tool-use agents at scale requires diverse environments that remain scarce and expensive to scale. Agent World Model (AWM) synthesizes 1,000 executable, database-backed environments and 10,000 tasks for reinforcement learning. Across three benchmarks, AWM-trained agents generalize to unseen domains, outperforming LLM-simulated training and concurrent synthesis methods.

  • Problem

    Large-scale reinforcement learning for tool-use agents requires diverse environments, but available environments are scarce, expensive to scale, and often lack diversity.

    Source: §1 · p. 1

  • Method

    1,000 code-driven, database-backed environments and 10,000 tasks are synthesized by the Agent World Model (AWM) pipeline for large-scale agentic reinforcement learning.

    Source: §7 · p. 9

  • Results

    Across three benchmarks, AWM-trained agents generalize to unseen domains, outperforming LLM-simulated training and concurrent synthesis methods.

    Source: §7 · p. 9

  • Takeaways & Limitations

    Synthetic-environment training can build tool-use capabilities that transfer to real-world scenarios.

    Source: §5.2 · p. 7

    Task generation assumes post-authentication access, so access-control behavior is not directly trained and requires additional training before production deployment.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Recent advances in large language model (LLM) have empowered autonomous agents to perform multi-turn interactions with tools and environments. However, scaling such agent training is limited by the lack of diverse and reliable environments. In this paper, we propose Agent World Model (AWM), a fully synthetic environment generation pipeline. Using this pipeline, we scale to 1,000 environments covering everyday scenarios, in which agents can interact with rich toolsets and obtain high-quality observations. Notably, these environments are code-driven and backed by databases, providing more reliable and consistent state transitions than environments simulated by LLMs. Moreover, they enable more efficient agent interaction compared with collecting trajectories from realistic environments. To demonstrate the effectiveness of this resource, we perform large-scale reinforcement learning for multi-turn tool-use agents. Thanks to the fully executable environments and accessible database states, we can also design reliable reward functions. Experiments on three benchmarks show that training exclusively in synthetic environments, rather than benchmark-specific ones, yields strong out-of-distribution generalization. The code is available at https://github.com/Snowflake-Labs/agent-world-model.

1. Introduction

AWM is an open-source pipeline that synthesizes code-driven, database-backed tool-use environments for large-scale agent RL. It produces 1,000 environments, and agents trained on AWM generalize to unseen environments across three tool-use benchmarks.

1. Introduction

  • Agent RL faces scarce, expensive environments: real-world interaction is difficult to scale, human-built environments lack diversity, and LLM-simulated transitions incur hallucinations and inference costs.These limitations motivate scalable synthesis of executable, reliable environments rather than relying primarily on task synthesis or trajectory collection.Source: §1 · p. 1
  • Earlier code-based environment synthesis leaves gaps in openness, RL suitability, domain coverage, scale, and state-consistency guarantees.DeepSeek-V3.2 (DeepSeek-AI et al., 2025) and Qwen Tongyi (Fang et al., 2025) release neither pipelines nor environments; Qwen targets SFT rather than RL. Community efforts (Feng et al., 2025; Sullivan et al., 2025; Cai et al., 2025; Zhang et al., 2025a; Song et al., 2026) face limitations involving game-like scope, human priors, state consistency, or scale.Source: §1 · p. 2
  • AWM generates user tasks, database schemas, and backend tool implementations, exposing tools through Model Context Protocol (MCP).Scenario-derived requirements guide schema creation, and the schema guides interface design and implementation so tools operate on a consistent data model.Source: §1 · p. 2
  • Database-state verification augments an LLM judge to supply RL rewards, while execution-based self-correction repairs generated code at every synthesis stage.Verification compares database states before and after agent execution; execution errors are fed back to the LLM for correction.Source: §1 · p. 2
  • 1,000 unique environments provide executable sandboxes with dozens of tools, supporting isolated parallel instances and easy resets for online RL.The environments span scenarios including shopping, social media, finance, and travel; training uses 1,024 environment instances per step.Source: §1 · p. 2
  • Agents trained on AWM generalize to unseen environments across three tool-use benchmarks without benchmark-tailored training environments or tasks.The introduction reports out-of-distribution generalization but provides no numerical benchmark scores.Source: §1 · p. 2

2. Related Work

AWM addresses the shortage of diverse, efficient, executable environments for agent reinforcement learning. It distinguishes itself through from-scratch synthesis, database-backed state consistency, and a large collection of environments, tools, and verified tasks.

Tool-use Agents

  • Tool-use training largely relies on static data or small-scale environments, leaving insufficient diverse, executable environments with fast interactions and reliable state transitions for RL.ToolLLM (Qin et al., 2024), for example, trains on LLM-generated trajectories using simulated responses rather than actual tool execution.Source: §2 · p. 2

Agent Data Synthesis

  • Synthetic agent-data methods generate tasks, tool specifications, and trajectories without providing executable environments, limiting alternative-action exploration and grounded feedback from state changes for RL.These methods treat environments as given or simulate tool responses with LLMs.Source: §2 · p. 2

Environment Synthesis

  • Environment synthesis divides into LLM-based simulation, which risks hallucinated transitions and costly per-step model calls, and programming-based synthesis, which drives transitions and observations through code.AutoEnv creates 36 game-like environments; concurrent EnvScaler (Song et al., 2026) generates 191 interactive environments from existing task sets. Other approaches extract tool graphs from documentation (Fang et al., 2025) or use human-designed type systems (Sullivan et al., 2025).Source: §2 · pp. 2–3
  • 1,000 environments, 35,062 tools, and 10,000 tasks paired with verification code distinguish AWM, which synthesizes environments without predefined task sets or API documentation.Database-backed state management enforces consistency and enables code-augmented verification for RL. To their knowledge, the authors describe AWM as the largest open-source environment set.Source: §2 · p. 3

3. Agent World Model

AWM synthesizes task-driven, database-backed POMDP environments with tool interfaces and state-grounded verification for agentic reinforcement learning. Its pipeline produces 1,000 executable environments and 10,000 tasks with minimal human input.

3.1. Scenario Generation

  • 100 popular domain-name seeds guide scenario generation, with CRUD filtering, embedding deduplication, and category caps yielding 1,000 diverse stateful-application scenarios.Generation targets applications requiring database interactions rather than static information retrieval, spanning domains including finance, travel, retail, and workflow.Source: §3.1 · p. 3

3.2. Task Generation

  • 10 API-solvable, post-authentication tasks per scenario yield 10,000 executable tasks that determine required database entities and API endpoints.Tasks cover diverse scenario functionalities while excluding purely UI-dependent actions and assuming login is already complete.Source: §3.2 · p. 4

3.3. Environment Synthesis

  • Database schemas and initial data instantiate state; Python tools exposed through MCP define actions, observations, and transitions, while verification defines rewards.These components instantiate the POMDP specification, and agents interact with each environment only through the unified MCP interface.Source: §3 · p. 3, §3.3 · p. 4

3.3.1. ENVIRONMENT

  • Task-derived SQLite schemas constrain transitions, while precondition-satisfying sample data makes every task initially executable.Only task-required tables and endpoints are generated, with tool schemas preceding Python implementation to guide generation of environments potentially exceeding 3,000 code lines. Schemas also document tool summaries, typed parameters, and responses.Source: §3.3.1 · p. 4
  • State-based checks and trajectory-aware LLM-as-a-Judge (Zheng et al., 2023) determine task rewards, improving robustness to imperfect environment execution.Checks compare task-relevant database states before and after execution; the judge returns Completed, Partially Completed, Agent Error, or Environment Error. Code-only verification is also available, although the reported analysis finds the combined approach more robust.Source: §3.3.1 · pp. 4–5
  • Execution-based self-correction tests generated components in isolation and feeds runtime errors and problematic code back to the LLM for regeneration.Correction repeats for up to five iterations or until successful execution. The pipeline tolerates up to 10% errors per stage to reduce costs, trading stricter correction for additional expense.Source: §3.3.1 · p. 5

3.3.2. PIPELINE RESULTS AND ANALYSIS

  • 1,000 executable environments and 10,000 tasks give AWM 5× more environments than EnvScaler (Song et al., 2026), with human input limited to 100 scenario-name seeds.First-attempt synthesis success exceeds 85%, and repairing generation failures takes 1.13 iterations on average. With Claude-4.5-Sonnet (Anthropic, 2025a) and a 20-step budget, task completion is 62.6%, while 13.7% of tasks exceed that budget.Source: Table 3 · p. 5, §3.3.2 · p. 5, Table 2 · p. 5

4. Agentic Reinforcement Learning

AWM trains tool-use agents with online Group Relative Policy Optimization (GRPO) (Shao et al., 2024), combining tool-call format checks with task-outcome rewards. History-aware optimization applies the same history truncation used during inference.

4. Agentic Reinforcement Learning

  • Online reinforcement learning uses Group Relative Policy Optimization (GRPO) (Shao et al., 2024) to train tool-use agents in synthesized environments.Long-horizon trajectories interleave observations and tool calls, requiring careful reward design and alignment between training and inference.Source: §4 · p. 5

4.1. Reward Design

  • Hybrid rewards combine step-level format checks with code-augmented LLM task verification, terminating invalid tool-call rollouts early to discourage malformed actions and save computation.Task rewards are 1.0 for Completed, 0.1 for Partially Completed, and 0.0 otherwise. Early termination gives r_t = −1.0; normal termination broadcasts the task reward to all action steps; otherwise r_t = 0.Source: §4.1 · pp. 5–6

4.2. History-Aware Training

  • History-aware training splits trajectories into subtrajectories conditioned on truncated histories, aligning optimization with inference rather than training exclusively on full histories.Many RL frameworks (Sheng et al., 2024; Hu et al., 2025) optimize completed rollouts in one forward pass, selecting action tokens while masking observations. Full-history training creates a distribution mismatch when inference uses truncated histories.Source: §4.2 · p. 6

5. Experiments

AWM improves over Base across all three out-of-distribution benchmarks, remains competitive with EnvScaler on τ 2-bench, and achieves the best overall MCP-Universe results. Gains extend across task-complexity levels despite differences between synthetic training scenarios and benchmark requirements.

5.1. Experimental Setup

  • Evaluation tests out-of-distribution transfer on τ 2-bench, BFCLv3, and MCP-Universe (Luo et al., 2025b).Coverage includes airline, retail, and telecom dialogue; function-calling categories covering non-live, live, multi-turn, and hallucination tests; and real-world MCP workflows. MCP-Universe excludes GUI-dependent 3D design and repository-management tasks requiring authenticated GitHub or Notion access.Source: §5.1 · p. 6
  • 526 environments and 3,315 tasks support training Qwen3 thinking models (Yang et al., 2025) at 4B, 8B, and 14B scales.Training runs for up to 96 optimization steps with 1,024 isolated environment instances per step; evaluation standardizes access through list tools and call tool. Baselines include untrained models, task-and-tool-matched Simulator (Chen et al., 2025), and EnvScaler (Song et al., 2026), which synthesizes 191 programming-based environments.Source: §5.1 · p. 6

5.2. Main Results

  • Training omits conversational interaction, refusal scenarios, browser automation, and information retrieval, despite their importance to the evaluated benchmarks.τ 2-bench requires multi-turn dialogue, BFCLv3 stresses hallucination resistance, and MCP-Universe centrally involves browser automation and information retrieval.Source: §5.2 · p. 6
  • AWM improves over Base across all benchmarks, consistently exceeds Simulator on τ 2-bench, and achieves the best overall MCP-Universe results.AWM remains competitive with EnvScaler on τ 2-bench, with large Financial and Location gains on MCP-Universe. The authors suggest executable state consistency provides a steadier learning signal than Simulator, while avoiding per-interaction LLM calls reduces RL latency.Source: Table 4 · p. 7, §5.2 · p. 7
  • 65.94, up from 53.83, is AWM’s 8B BFCLv3 overall score, surpassing Simulator and EnvScaler.BFCLv3 performance improves across all model sizes, although hallucination resistance remains a modest weakness. The authors attribute this weakness to a format-correctness reward that encourages tool use and penalizes refusals.Source: §5.2 · p. 6, Table 4 · p. 7
  • AWM improves over Base in every BFCLv3 and τ 2-bench complexity bucket, although absolute gains shrink on harder tasks.Difficulty is measured by required tool calls or ground-truth actions; relative gains remain substantial. MCP-Universe is excluded because it lacks reference trajectories or other complexity indicators.Source: §5.2 · p. 7, Table 5 · p. 7

6. Analysis

AWM environments combine strong synthesis quality, challenging tasks, and sustained diversity. Code-augmented verification, history-aware training, and larger environment pools improve agent performance.

6.1. Quality of Synthesized Environments

  • AWM outperforms EnvScaler on Task Feasibility, Data Alignment, and Toolset Completeness under both judges, while producing fewer blocked tasks.Roughly 3x more code brings only a moderate bug increase; 44% of inspected bugs involve unhandled edge inputs and 14% involve database-constraint conflicts. Environment errors remain around 4% during RL, mostly internal errors triggered by specific agent behaviors; the Environment Error reward mitigates imperfect-environment issues.Source: §6.1 · pp. 7–8
  • 99% environment-code success for Claude-4.5 and 77% for Qwen3.5 support largely model-agnostic synthesis, with diversity essentially constant across generators.Claude-4.5 matches GPT-5 on quality, while Qwen3.5 produces usable but bug-heavier environments. The generator comparison covers 100 environments and 1,000 tasks.Source: Table 7 · p. 8, §6.1 · p. 8
  • 36.1% and 62.6% overall Pass@1 for GPT-5.1 and Claude-4.5-Sonnet, respectively, show non-trivial tasks; pass rates decrease monotonically with complexity.69% of Very-Hard tasks remain unsolved by either model; GPT-5.1’s occasional system-prompt refusals may partly explain its lower rate. Qwen3-4B, Qwen3-8B, and Qwen3-14B never solve 27.1%, 23.7%, and 28.2% of tasks during RL, respectively, with 14B optimizing for only 32 steps.Source: Table 8 · p. 8, §6.1 · p. 8
  • 0.0% cross-environment AST function duplicates and 0.004 endpoint name Jaccard complement stable embedding diversity and expanding category coverage as the environment pool grows.The semantic and topical analysis covers 1,000 environments, supporting diversity beyond implementation-level differences.Source: §6.1 · p. 8, Figure 3 · p. 8

6.2. Analysis on Verification Design

  • Code-augmented verification outperforms LLM-only and code-only verification across all model scales and benchmarks by combining structured state-diff signals with LLM reasoning.LLM-only verification lacks grounding in state changes, while code-only verification can produce false negatives under environment imperfections. GPT-5 judging costs about $1.80 per training step with at most 1,024 samples, with negligible asynchronous latency.Source: §6.2 · pp. 8–9, Table 9 · p. 9
  • 95.5% pairwise agreement, Fleiss’ κ of 0.891, and a 9.2% reward flip rate demonstrate GPT-5.1 judge consistency.The study evaluates 100 trajectories five times each using three LLMs. Qwen3.5-122B-A10B reaches 88.1% agreement, while high cross-model agreement supports verification signals that capture completion rather than model-specific judgments.Source: Table 10 · p. 9, §6.2 · p. 9

6.4. Environment Scaling Curve

  • 526 training environments improve performance further over 100, while AWM outperforms EnvScaler on BFCLv3 and MCP-Universe at matched environment counts.Training on 10 environments severely degrades performance, likely from overfitting to the limited distribution; scaling to 100 yields substantial gains. The scaling analysis uses the 4B model.Source: Figure 4 · p. 9, §6.4 · p. 9

6.3. Analysis on History-Aware Training

  • History-aware training achieves the best results when training and inference history limits align, supporting optimization directly under truncated histories.The 4B full-history variant is relatively insensitive to inference misalignment and improves slightly on τ 2 when truncated, consistent with reduced interference from irrelevant turns.Source: Table 11 · p. 9, §6.3 · p. 9

7. Conclusion

Agent World Model (AWM) synthesizes executable agentic environments through a scalable pipeline that mirrors software development. Agents trained in these synthetic environments generalize to out-of-distribution domains, outperforming LLM-simulated training and concurrent synthesis methods across three benchmarks.

7. Conclusion

  • Agents trained in AWM’s synthetic environments generalize to out-of-distribution domains, outperforming both LLM-simulated training and concurrent synthesis methods across three benchmarks.Source: §7 · p. 9
  • 1,000 environments with 10,000 tasks were synthesized using AWM’s scalable pipeline, which mirrors software development to produce executable agentic environments.Source: §7 · p. 9
  • Code-driven environments backed by SQL databases use a unified MCP interface and support parallel isolated instances for large-scale agentic reinforcement learning.Source: §7 · p. 9

Impact Statement

AWM presents an open-source pipeline and synthesized executable environments intended to support research on tool-use agents. Synthetic environments may not fully reflect real-world scenarios, motivating safeguards for deployment.

Impact Statement

  • AWM presents an open-source pipeline for synthesizing executable environments to train tool-use agents.Source: Impact Statement · p. 10
  • Releasing both the generation pipeline and synthesized environments aims to lower barriers to studying agentic systems.Source: Impact Statement · p. 10
  • Synthetic environments may not fully reflect real-world scenarios, so the authors encourage safeguards when deploying agents trained on AWM.Source: Impact Statement · p. 10

Limitations

AWM’s limitations include fixed environment generation, incomplete semantic validation, restricted training coverage, and untested adversarial robustness and access control. The authors identify further validation and safety measures as prerequisites for production deployment.

Opportunities for Self-Evolving

  • AWM’s fixed generation process limits autonomous improvement beyond initial capabilities; agent-assisted synthesis of new environments remains future work.The proposed self-evolving approach would let trained agents contribute to environment synthesis to support continued adaptation and capability improvement.Source: Limitations · p. 10

Synthesis Pipeline Optimization

  • AWM’s self-correction primarily fixes runtime errors through trial-and-error rather than deeper semantic validation.Proposed improvements include LLM detection of logical inconsistencies and subtle bugs, plus human inspection when resources permit. Tasks spanning multiple scenarios or environments are another proposed extension compatible with the architecture.Source: Limitations · p. 10

Training Scale and Model Coverage

  • 526 of 1,000 synthesized environments supported training, primarily using the Qwen3 model family (Yang et al., 2025) at 4B, 8B, and 14B scales.Compute constraints limited coverage, although experiments showed consistent out-of-distribution generalization across three benchmarks. The synthesis pipeline is the core contribution, and both the pipeline and released environments are model-agnostic.Source: Limitations · p. 10

Robustness & Safety

  • 74% of environments contain at least one bug, and ∼4% of RL rollouts hit runtime errors; intentionally injected adversarial perturbations remain unevaluated.Agents encounter imperfect-synthesis noise, but evaluation does not cover corrupted database rows, malicious tool outputs, or prompt-injection content in observations.Source: Limitations · p. 10
  • Post-authentication task generation leaves access-control behavior untrained, risking unauthorized data access, irreversible state mutations, and unintended call chains.Some scenarios involve finance and healthcare, where errors have asymmetric costs, although AWM contains no real personal data. Production deployment requires adversarial robustness evaluation, access-control training, and domain-specific human-in-the-loop oversight.Source: Limitations · p. 10

Use of AI Assistants … A.4. Tool Calling Format and Validation

AWM expands seed websites into diverse scenarios and concrete tasks, then synthesizes executable database-backed environments with state-based verification. Shared error-correction loops and tool-call validation support environment generation and reinforcement-learning rollouts.

Use of AI Assistants

  • AI assistants primarily polished the paper’s text and created figure icons, with all generated content reviewed and corrected as necessary by the authors.Source: Use of AI Assistants · p. 10

A.1. Scenario Generation

  • 100 seed websites expand into 1,000 scenarios through Self-Instruct-style generation (Wang et al., 2023), with filtering to preserve diversity and stateful interactions.An LLM classifier rejects scenarios unsuitable for CRUD operations, while embedding deduplication uses a cosine-similarity threshold of 0.85. Category caps prevent dominant types from becoming overrepresented; the reported scenario distribution spans diverse domains.Source: §A.1 · p. 15

A.2. Task Generation

  • 10 diverse tasks per scenario define functional requirements while keeping downstream database and tool synthesis tractable.Tasks must be API-solvable without UI interactions, assume post-authentication context, and provide concrete parameters sufficient for execution without clarification. These requirements align schemas and toolsets with user needs rather than over-specifying or omitting functionality.Source: §A.2 · p. 15

A.3. Environment Synthesis

  • Task-derived SQLite schemas and precondition-satisfying sample data undergo execution testing and shared LLM error-feedback correction.Schemas omit authentication fields, and inserts respect foreign-key dependencies; schema/data failures use 10% error thresholds, while environment startup requires 0% errors. Separate LLM calls summarize errors in typically 200-500 tokens for regeneration, with five retries per component before selecting the lowest-error attempt; first-attempt success exceeds 85%, and failed cases average 1.13 correction iterations.Source: §A.3 · pp. 15–16
  • 35 tools and about 2,000 lines of code characterize the average generated environment, implemented from a separately synthesized interface specification.Task-derived specifications define minimal endpoints with typed inputs and response schemas, reducing interface inconsistencies observed with direct generation of 30+ tools. Python MCP servers combine SQLAlchemy models, Pydantic models, and FastAPI handlers, then undergo startup, health-check, and list-tools validation with correction for failures.Source: §A.3 · pp. 15–16
  • Python verification functions compare initial and final database states, grounding LLM-as-a-Judge reward assignment in task-relevant evidence.SQL queries produce changed records, expected outcomes, and diagnostic signals, accompanied by natural-language success and failure criteria to reduce reward hallucination. This verification completes synthesis of the POMDP’s database state, tool-based action and observation spaces, and reward function.Source: §A.3 · pp. 15–16

A.4. Tool Calling Format and Validation

  • Two meta-tools, list tools and call tool, let agents discover and invoke environment-specific capabilities without hardcoded tool information.List tools retrieves descriptions and input/output schemas; call tool invokes a named tool with arguments passed as a JSON string. Invocations follow the Qwen3 tool-calling format (Yang et al., 2025), wrapped in XML tags.Source: §A.4 · p. 16
  • −1 rewards penalize format errors, whereas environment/server errors receive 0; either error immediately terminates the rollout.Stepwise validation requires non-empty reasoning inside <think>...</think>, valid discovered tool names, schema-conforming JSON arguments, exactly one list tools call occurring first, and at least one subsequent successful tool call in multi-turn interactions. Non-error server responses are also required; early termination avoids further computation and reward signals on broken trajectories.Source: §A.4 · p. 16

A.5. Training and Evaluation Details … B.2. Detailed LLM-Judge Reliability

Training combines isolated database-backed environments, history-aware optimization, and asynchronous code-augmented reward judging, with format converters for out-of-distribution evaluation. Analyses show that format rewards improve training efficiency and task completion, while repeated judge assessments show high agreement.

A.5. Training and Evaluation Details

  • 1,024 isolated environment instances run in parallel per training step, each using an independent MCP server and SQLite database copy to prevent rollout interference.Databases reset after each rollout, while background prefetching overlaps next-batch environment preparation with gradient updates to reduce wall-clock time. GRPO training uses KL coefficient 0.001, clip ratio 0.28 for exploration following DAPO, and sequence-level importance sampling to mitigate rollout/training-engine distribution shift.Source: §A.5 · pp. 16–17
  • History-aware sample splitting trains each assistant turn against truncated context, avoiding the distribution shift caused by full-history training followed by truncated-context deployment.Each sample retains the system prompt, initial user message, first assistant-tool exchange, and w = 3 most recent preceding turns; only the target turn contributes loss. Splitting a T-turn rollout into T samples increases forward passes per rollout by a factor of T.Source: §A.5 · p. 17
  • Code-augmented GPT-5 (OpenAI, 2025) judging assigns rollout rewards using executable database-state verification and agent trajectories.Verification compares final and initial database states and supplies changed records and success criteria; classifications are Completed, Partially Completed, Agent Error, or Environment Error. Batched asynchronous verification overlaps collection of the next rollout batch, adding negligible training overhead.Source: §A.5 · p. 17
  • Format converters adapt the unified call tool interface to three out-of-distribution benchmarks, separating task-solving ability from native tool-format compliance.τ 2-bench uses wrapped direct tool calls; BFCLv3 uses aggregated function calls without list tools meta-calls; MCP-Universe (Luo et al., 2025b) uses MCP. Evaluation uses temperature 0.6, top-k=20, top-p=0.95, a 131,072-token context via RoPE scaling, and w = 10 for long-context tasks.Source: §A.5 · p. 17

B.1. Analysis on Step-level Format Correctness Reward

  • About 27% lower average rollout time accompanies step-level format correctness rewards, which also reduce format errors and improve task completion.Without this reward, format errors remain above 20% after 50 optimization steps, and task completion saturates below 40%.Source: §B.1 · p. 17, Figure 5 · p. 18

B.2. Detailed LLM-Judge Reliability

  • 91.2% pairwise agreement for GPT-5.1 and 82.7% for Qwen3.5 under the strict four-class setting support code-augmented judge reliability.GPT-5.1, Claude-4.5-Sonnet, and Qwen3.5-122B-A10B each judged 100 trajectories from the 8B agent five times, using default or recommended temperatures. Metrics included self-consistency across all five trials, pairwise agreement, Fleiss’ κ, and reward flip rate, defined as any trial disagreeing with the majority.Source: §B.2 · p. 18

B.3. Code-Level Diversity Analysis

Code-level analysis of all 1,000 AWM environments finds no function-level structural duplicates at TSED threshold 0.5 and low lexical similarity, corroborating the embedding-based diversity findings.

B.3. Code-Level Diversity Analysis

  • Well below 0.01, endpoint-name and class-name cross-environment Jaccard scores indicate little naming overlap.These lexical comparisons complement the structural duplicate analysis.Source: §B.3 · p. 18, Table 14 · p. 19
  • 1,000 AWM environments contain no cross-environment function-level structural duplicates at TSED threshold 0.5.The analysis uses AST-based Tree-Structure Edit Distance (TSED) to detect duplicated function bodies across files.Source: §B.3 · p. 18, Table 14 · p. 19
  • 0.324 is the maximum token Jaccard, with residual overlap potentially attributable to shared Python/FastAPI keywords.The authors interpret the code-level evidence as corroborating embedding-level diversity and supporting large-scale environment synthesis with AWM.Source: §B.3 · pp. 18–19, Table 14 · p. 19

B.4. Case Study for Verification

Code-augmented LLM judging combines structured database evidence with trajectory context to reduce false negatives and false positives in imperfect synthetic environments. Three cases illustrate when database checks are decisive and when trajectory context resolves ambiguity.

B.4. Case Study for Verification

  • Combining database verification with trajectory-aware judging reduces false negatives and false positives caused by imperfect synthetic environments.The verifier extracts structured evidence from database snapshots and rule-based checks, while the judge uses the trajectory to resolve ambiguity.Source: §B.4 · p. 19
  • The verifier and judge agree on a clean bid-history query because the retrieved records and underlying database deterministically support the reported statistics.This case illustrates the regime where structured verification evidence is decisive.Source: §B.4 · p. 19, Figure 28 · p. 42
  • Trajectory context recovers an idempotent routine-creation success that code-only verification would incorrectly reject because initial and final database snapshots appear identical.Despite an environment imperfection, the trajectory supports a Completed judgment rather than the false negative implied by requiring a state change.Source: §B.4 · p. 19, Figure 29 · p. 42
  • Database grounding prevents a false positive when successful tool calls create a duplicate event and add a session under the wrong event ID.The verifier reveals that the intended target event remains unchanged, allowing the code-augmented judge to reject the apparent success.Source: §B.4 · p. 19, Figure 30 · p. 43

Related papers

Loading 2602.10090v3…