PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Agent Learning via Experience Synthesis

Zhaorun Chen, Zhuokai Zhao, Kai Zhang, Bo Liu, Qi Qi, Yifan Wu, Tarun Kalluri, Sara Cao, Yuanhao Xiong, Haibo Tong, Huaxiu Yao, Hengduo Li, Jiacheng Zhu, Xian Li, Dawn Song, Bo Li, Jason Weston, Dat Huynh

arXiv:2511.03773v2cs.AI

TL;DR

LLM-agent RL needs scalable, diverse, and reliable interaction data, but real-environment rollouts are costly and task and reward coverage is limited. DreamGym synthesizes reasoning-driven experiences with replay-buffer grounding and adaptive curriculum generation, achieving strong synthetic and sim-to-real results while reducing external-data requirements.

  • Problem

    LLM-agent RL is constrained by costly rollouts, limited task diversity, sparse or unreliable rewards, and difficult interactive environments.

  • Method

    DreamGym uses a reasoning-based experience model, replay-buffer grounding, and adaptive task generation to synthesize diverse online RL experiences.

  • Results

    Across environments and backbones, DreamGym improves RL training, including over 30% improvement on WebArena and over 40% sim-to-real improvement using less than 10% of external data.

  • Takeaways & Limitations

    DreamGym provides a scalable warm-start strategy for general-purpose RL by enabling synthetic training and reducing reliance on costly real-world interactions.

  • Takeaways & Limitations

    The work primarily investigates single-environment learning setups rather than a universal model spanning multiple environments.

Abstract

from arXiv · show

While reinforcement learning (RL) can empower autonomous agents by enabling self-improvement through interaction, its practical adoption remains challenging due to costly rollouts, limited task diversity, unreliable reward signals, and infrastructure complexity, all of which obstruct the collection of scalable experience data. To address these challenges, we introduce DreamGym, the first unified framework designed to synthesize diverse experiences with scalability in mind to enable effective online RL training for autonomous agents. Rather than relying on expensive real-environment rollouts, DreamGym distills environment dynamics into a reasoning-based experience model that derives consistent state transitions and feedback signals through step-by-step reasoning, enabling scalable agent rollout collection for RL. To improve the stability and quality of transitions, DreamGym leverages an experience replay buffer initialized with offline real-world data and continuously enriched with fresh interactions to actively support agent training. To improve knowledge acquisition, DreamGym adaptively generates new tasks that challenge the current agent policy, enabling more effective online curriculum learning. Experiments across diverse environments and agent backbones demonstrate that DreamGym substantially improves RL training, both in fully synthetic settings and in sim-to-real transfer scenarios. On non-RL-ready tasks like WebArena, DreamGym outperforms all baselines by over 30%. And in RL-ready but costly settings, it matches GRPO and PPO performance using only synthetic interactions. When transferring a policy trained purely on synthetic experiences to real-environment RL, DreamGym yields significant additional performance gains while requiring far fewer real-world interactions, providing a scalable warm-start strategy for general-purpose RL.

1 Introduction

LLM-agent RL is limited by costly, sparse, and unstable interaction data, as well as insufficient task diversity. DreamGym addresses these barriers by synthesizing reasoning-grounded experiences, replay-buffer guidance, and adaptive tasks, improving synthetic and sim-to-real training.

  • Introduction: RL training for LLM agents is hindered by expensive long-horizon rollouts, sparse rewards, limited task diversity, and unstable dynamic environments.These barriers make collecting large-scale, informative online data difficult and costly.
  • Introduction: DreamGym synthesizes online experience data with a reasoning-based model that produces consistent transitions and feedback without requiring perfectly realistic environments.The framework targets diverse, informative, and causally grounded interaction data for agent training.
  • Introduction: An experience replay buffer supplies similar and diverse trajectories, begins with offline knowledge, and is continuously enriched with fresh interactions.This design keeps synthetic rollouts aligned with the agent’s evolving policy and supports stable training.
  • Introduction: Over 40% performance improvement versus training from scratch is achieved with less than 10% of external data through DreamGym-S2R.The approach first trains on diverse, curriculum-driven synthetic experiences before transferring to external environments.
  • Introduction: Over 30% improvement over all baselines and SOTA methods is reported for WebArena, where large-scale RL infrastructure is unavailable.Agents trained purely in DreamGym exceed 30% success rates across all backbones.

2 Related Work

Prior work uses RL, synthetic trajectories, task synthesis, and synthetic environments to improve agent learning, but scalability and adaptivity remain constrained by real-environment data collection and engineering demands.

  • LLM Agents Reinforcement Learning: RL has been applied to LLM-agent alignment, reasoning, mathematics, coding, robotics, games, and control, motivating its broader use for adaptive decision making.The related work positions RL as a route beyond static language generation.
  • LLM Agents Reinforcement Learning: Interactive LLM-agent RL requires long-horizon exploration with sparse rewards and diverse, verifiable tasks that are costly to design and annotate.These constraints can make policy improvement challenging, costly, and prone to training collapse.
  • Synthetic Data for Agent Learning: Static oracle or teacher-generated trajectories support imitation learning but require substantial labeling and lack diversity and adaptivity.These approaches address expert-demonstration scarcity without fully solving scalable exploration.
  • Synthetic Data for Agent Learning: Task-synthesis methods expand RL exploration but still depend on real-environment data collection and inherit its scalability limitations.The cited methods reduce manual effort or broaden tasks without removing costly external interactions.
  • Synthetic Environments: Synthetic environments can generate unlimited on-policy experiences, while recent world-model systems provide environment feedback for planning and training.UI-Simulator is closest to this setting but requires substantial expert engineering to adapt across environments.

3 Preliminaries

The paper formalizes agent learning as an MDP in which a policy maps observable states to actions and is optimized from experience using policy-gradient methods. DreamGym supplies the synthetic interaction framework that instantiates these elements with generated states, rewards, and tasks.

  • 3.1 Notations: An MDP is defined by M = (S, A, T, R, γ, ρ0), covering states, actions, transitions, rewards, discounting, and initial-state distribution.The initial distribution includes the natural-language task instruction τ0.
  • 3.1 Notations: LLM-agent states encode visible configurations such as webpage content or tool outputs, while actions include UI clicks, tool calls, and textual responses.The policy maps states to distributions over discrete operations.
  • 3.1 Notations: DreamGym alternates agent actions with experience-model predictions of next states and rewards, using seed tasks, replay information, and curriculum generation.The framework proposes high-reward-entropy tasks to expose the policy to increasingly informative scenarios.
  • 3.2 Agent Learning from Experience: RL trains πθ from online experiences to maximize expected cumulative reward, typically using policy-gradient updates based on an advantage estimate.An experience contains a task and a state-action rollout.
  • 3.2 Agent Learning from Experience: PPO uses Generalized Advantage Estimation for stability, whereas GRPO normalizes rewards within response groups and removes the value function.DreamGym is orthogonal to the specific RL algorithm and focuses on scaling informative experience synthesis.

4 Scaling Agent Learning via Experience Synthesis

DreamGym synthesizes controllable, reasoning-based experiences for RL by combining an abstract experience model, replay-buffer retrieval, and curriculum task generation. These components support consistent transitions, informative feedback, policy-aligned rollouts, and synthetic-to-real transfer.

  • DreamGym combines a reasoning-based experience model, replay buffer, and curriculum task generator to synthesize diverse RL experiences.The experience model generates trajectories, the replay buffer integrates offline and online knowledge, and task generation uses reward entropy to select challenging variations.
  • 4.1.1 Inference for Experience Rollout Collection: The experience model predicts next states and rewards from the current interaction, history, task instruction, and semantically retrieved demonstrations.Chain-of-thought reasoning guides transitions, while replay-buffer retrieval supplies similar past experiences to improve factuality and reduce hallucinations.
  • 4.1.2 Training Experience Models to Reason: DreamGym trains its experience model with annotated reasoning traces and a joint objective for reasoning generation and next-state prediction.This training is intended to produce faithful action-effect explanations and consistent rollouts that generalize beyond expert trajectories.
  • 4.2 Curriculum-based Task Generation: Reward entropy selects tasks that are feasible yet challenging, while a sampling bound preserves coverage of the original task distribution.Tasks with balanced successes and failures provide the greatest information gain for credit assignment, and λ limits synthetic-task usage per iteration.
  • 4.3 Learning from Synthetic Experiences: DreamGym alternates policy actions with experience-model predictions, then trains the policy on synthetic rollouts before optional real-environment transfer.Table 1 compares offline imitation, real-environment RL, purely synthetic DreamGym training, and DreamGym-S2R using real-transition counts as the data measure.

5 Experiments

DreamGym is evaluated across diverse agent benchmarks, backbones, training strategies, and transfer settings. It enables RL where environments are not RL-ready, matches real-interaction baselines synthetically, reduces training cost, and supports cross-domain transfer with limits under large domain gaps.

  • 5.1 Experimental Setup: The experiments cover WebShop, ALFWorld, and WebArena-Lite across Llama-3.2-3B, Llama-3.1-8B, and Qwen-2.5-7B backbones.Comparisons include offline SFT and DPO, plus online GRPO and PPO; main results use an experience model trained from Llama-3.1-8B-Instruct.
  • 5.2 Main Results: Over 30% success rates across all backbones make DreamGym the only viable RL approach for the non-RL-ready WebArena setting.Agents are trained purely in DreamGym, while zero-shot RL baselines suffer from limited exploration diversity and sparse rewards.
  • 5.2 Main Results: DreamGym matches GRPO and PPO trained on 80K real interactions using only synthetic rollouts on WebShop and ALFWorld.With 5k real rollouts after synthetic mid-training, DreamGym-S2R consistently outperforms real-environment baselines trained from scratch.
  • 5.2 Main Results: Roughly one-third to one-fifth of baseline training effort is sufficient on WebArena, including rollout sampling time and GPU hours.Dense curriculum feedback and lightweight abstract transitions reduce sampling cost and avoid heterogeneous environment bottlenecks.
  • 5.2 Main Results: DreamGym policies transfer between WebShop and WebArena with performance superior to directly trained SFT models, but performance drops substantially on ALFWorld.The results indicate cross-domain transfer within related web environments but expose limits when the domain gap becomes large.

6 Performance Analysis and Ablation Studies

DreamGym’s synthetic experiences improve learning speed and stability, while curriculum generation and reasoning-based modeling provide important contributions to performance. The experience model remains effective with limited offline data and smaller backbones.

  • 6.1 Training Curve Analysis: Success rates improve more rapidly within the first 40 steps, while DreamGym training curves remain smoother than baselines across runs.The synthesized trajectories provide denser and more consistent feedback than sparse real rollouts.
  • 6.2 Ablation on Task Generator: Removing the task generator reduces success by 6.6% on WebShop and 6.0% on WebArena relative to the full DreamGym configuration.Without adaptive task generation, the replay buffer can become repetitive and exploration can stall.
  • 6.3 Ablation on Experience Model: Explicit reasoning and trajectory history support informative, causally coherent experiences, while removing either component harms consistency or increases hallucination.Removing experience reasoning also substantially reduces overall performance.
  • 6.4 Ablation on Experience Model Backbones and Offline Training Data: With only 2k–10k offline samples, the experience model reaches competitive performance; Llama-3.1-8B exceeds 50% on WebShop with 10k samples.Llama-3.2-3B reaches about 55% on WebShop with 20k samples, while WebDreamer achieves around 13% on WebArena in the extreme low-data regime.

7 Conclusion

DreamGym reduces the cost of RL for language agents by generating scalable, reasoning-driven synthetic experiences and adaptive curricula. Across environments and backbones, it improves synthetic and sim-to-real training through reasoning-based modeling, replay-buffer grounding, and curriculum generation.

  • 7 Conclusion: DreamGym generates scalable, reasoning-driven synthetic experiences, state transitions, and adaptive curricula to reduce reliance on costly real-environment rollouts.The framework creates challenging yet solvable tasks tailored to the agent’s evolving policy.
  • 7 Conclusion: Experiments across diverse environments and model backbones show consistent gains in fully synthetic and sim-to-real settings.The reported gains are attributed to the synergy of reasoning-based modeling, replay-buffer grounding, and curriculum generation.

Limitations and Future Work

The study primarily evaluates DreamGym in single-environment learning setups. Future work could extend it toward a universal world model spanning multiple environments.

  • Limitations and Future Work: DreamGym is primarily investigated in single-environment learning setups rather than across unified multi-environment models.The authors suggest extending the framework to unify multiple environment models and enable knowledge transfer.

A Detailed Experiment Settings

This section provides implementation details for each environment.

  • A Detailed Experiment Settings: Implementation details are provided separately for each environment.

A.1 WebShop

WebShop evaluates agents in a large, realistic e-commerce environment with diverse products and natural-language shopping instructions. DreamGym trains its experience model from demonstrations and additional oracle/random trajectories under substantial compute resources.

  • A.1 WebShop: WebShop contains 1.18M products and 12,087 crowd-sourced instructions requiring search, customization, and purchasing.The benchmark also includes noisy webpage text, compositional requirements, and diverse page types.
  • A.1 WebShop: The environment tests language grounding through query reformulation, noisy text handling, and strategic exploration across page types.
  • A.1 WebShop: WebShop experiments fine-tune all three agent backbones with the standard Verl-Agent setup.
  • A.1 WebShop: DreamGym’s WebShop experience model uses 1,600 human demonstrations plus 2,000 oracle and random trajectories for training.The test set is fixed, and training trajectories come from remaining tasks to avoid contamination.
  • A.1 WebShop: All WebShop experiments use 8 A100 nodes and 4 H100 nodes, including baselines and DreamGym.

A.3 WebArena

WebArena is a realistic, long-horizon web benchmark evaluated by functional correctness across multiple sites and tools. Its RL setup is constrained by limited parallel infrastructure and known trajectory and evaluator reliability issues.

  • A.3 WebArena: WebArena spans e-commerce, forums, GitLab, and content-management sites with tools, knowledge bases, multi-tab browsing, and diverse actions.
  • A.3 WebArena: WebArena-Lite evaluates agents on 165 challenging tasks selected from 812 original tasks, while 647 remaining tasks support training.
  • A.3 WebArena: The baseline supports PPO and GRPO but operates only four AWS servers, limiting parallel interaction collection.
  • A.3 WebArena: Some trajectories fail to execute properly, and WebArena’s original evaluator incorrectly judges certain tasks; collected trajectories are nevertheless retained for RL training.
  • A.3 WebArena: DreamGym obtains offline WebArena trajectories from successful demonstrations by high-performing public leaderboard agents using accessibility-tree observations.

B Theoretical Analysis

The theoretical analysis studies whether synthetic-environment training can improve performance after transfer to real environments. It focuses on learning-relevant signals rather than strict state-reconstruction fidelity.

  • B Theoretical Analysis: Under mild assumptions, DreamGym establishes performance guarantees for policies trained synthetically and evaluated in real environments.
  • B Theoretical Analysis: The analysis optimizes reward accuracy and domain consistency instead of requiring strict state-reconstruction fidelity.

B.1 Provably Policy Improvement in Real Environments Trained with Synthetic Experiences

DreamGym models real-environment training through a reasoning-based synthetic MDP and analyzes when synthetic policy updates transfer to real policy improvement. The theory attributes the gap to reward and transition errors, with trust-region updates controlling optimization.

  • B.1 Provably Policy Improvement in Real Environments Trained with Synthetic Experiences: DreamGym induces a synthetic MDP from its experience model and evaluates the learned policy in the real environment under a shared abstract state space.
  • B.1 Provably Policy Improvement in Real Environments Trained with Synthetic Experiences: Theorem 1 states that a trust-region update optimized in the synthetic environment guarantees real-environment policy improvement when synthetic gain exceeds model and trust-region penalties.
  • B.1 Provably Policy Improvement in Real Environments Trained with Synthetic Experiences: The experience-model error combines feedback faithfulness εR with domain consistency of state transitions εP.
  • B.1 Provably Policy Improvement in Real Environments Trained with Synthetic Experiences: The performance gap depends on reward accuracy and transition domain consistency rather than strict state-reconstruction error.
  • B.1 Provably Policy Improvement in Real Environments Trained with Synthetic Experiences: The experience-model judge scores whether predicted states are causally consistent with actions and exhibit meaningful state variation.
Loading 2511.03773v2…