DynaWeb: Model-Based Reinforcement Learning of Web Agents

• Live-web online RL is expensive, risky, and difficult to scale, while prior web world models had not made imagined rollouts first-class experience for on-policy optimization.Source: §1 · pp. 1–2
• The learned web server predicts naturalistic accessibility-tree page transitions and task feedback from page and action inputs, enabling imagined rollouts as direct on-policy RL experience.Source: §1 · pp. 2–3
• DynaWeb randomly interleaves world-model rollouts with fully real expert trajectories and real on-policy interaction, preserving the on-policy signal while reducing costly live exploration.The expert trajectories are independent of the world model and represent ground-truth web interactions.Source: §1 · pp. 2–3
• DynaWeb demonstrates consistent performance improvements on WebArena and WebVoyager and provides empirical insights into world-model-based imagination for web-agent training.Source: §1 · p. 3

Web Agent

• Web agents use LLM backbones and structured interaction frameworks, with progress also coming from data scaling, stronger training pipelines, and inference-time search or self-reflection.Interactive evaluation benchmarks include WebShop, Mind2Web, WebArena, VisualWebArena, WebVoyager, WebWalker, and MMInA.Source: §2 · p. 3

World Models

• Prior LLM world models support hypothetical-trajectory planning, while WebDreamer [Gu et al., 2024] and WMA [Chae et al., 2025] simulate web-action outcomes mainly to guide inference-time decisions.RAP [Hao et al., 2023] combines LLM-based world modeling with Monte Carlo Tree Search, and WKM [Qiao et al., 2024] distills world knowledge from agent trajectories.Source: §2 · p. 3
• DynaWeb trains a dedicated web world model to generate multi-step imagined trajectories for policy optimization, moving beyond inference-time simulation toward model-based RL for web agents.The approach targets the data inefficiency and cost of real-environment interaction.Source: §2 · pp. 3–4

Reinforcement Learning of Agents

• Web-agent RL includes outcome-reward online policy optimization and self-evolving curricula, while other approaches combine structured supervision or behavior cloning with policy optimization.Examples include WebAgent-R1 [Wei et al., 2025], WebRL [Qi et al., 2025], WebDancer [Wu et al., 2025b], and WorkForceAgent-R1 [Zhuang et al., 2025].Source: §2 · p. 4
• RL-based web agents typically require extensive real-environment interaction, which can be expensive, unstable, and risky at scale.This limitation motivates imagination-driven training.Source: §2 · p. 4

3.1 Problem Formulation

• The task is a partially observable Markov decision process: the agent observes visible page information in an accessibility tree and an LLM policy conditions on the query, history, and instructions to produce reasoning and atomic browser actions.The accessibility tree captures the structured layout of visible web elements [Zhang et al., 2024a].Source: §3.1 · pp. 4–5
• An episode is an observation-reasoning-action trajectory that receives a scalar task-completion reward through self-assessment or environment-specific evaluation.Source: §3.1 · p. 5

3.2 Web World Model

• The world model predicts naturalistic next accessibility-tree observations directly, but trains on free-form state-change descriptions because whole-page prediction has low information gain when a transition changes only part of a page.At inference, its instruction-following capability applies the predicted changes to the current observation.Source: §3.2 · pp. 5–6
• The world model is trained on cleaned StanfordNLP/NNetNav transitions, using GPT-oss-120b-generated reasoning traces and state changes as targets to create a reusable simulator.The data pipeline filters missing observations, invalid actions, and inconsistent transitions.Source: §3.2 · p. 6

3.3 DynaWeb: Model-based RL of Web Agents

• DynaWeb generates multi-step policy trajectories in a learned synthetic web environment, assigns terminal task-completion rewards, and uses the imagined rollouts for policy-gradient training without live web interaction.The return is the task-level reward obtained through model-based self-assessment conditioned on the query.Source: §3.3 · p. 6
• Real expert trajectories independent of the world model are randomly interleaved with imagined rollouts, while GSPO [Zheng et al., 2025] applies one sequence-level importance ratio uniformly to each trajectory’s tokens.The expert demonstrations are included to improve training stability and sample efficiency.Source: §3.3 · pp. 6–7

4.1 Setups

• Evaluation covers simulated WebArena and live WebVoyager tasks, using Success Rate (SR) as the primary metric.WebArena contains 812 tasks across five websites and four domains; WebVoyager has 643 open-ended tasks from 15 websites, with inaccessible tasks filtered and results averaged across two evaluation runs.Source: §4.1 · p. 7
• DynaWeb trains NNetNav-initialized agents in task-specific web world models, interleaving dreamed rollouts with expert trajectories.Half of dreamed trajectories are randomly replaced with gold-standard trajectories; world models are fine-tuned for one epoch, and synthetic rollouts run for up to five steps from sampled initial or intermediate states.Source: §4.1 · pp. 7–8

4.2 Main Results

• 31.0% average success rate on WebArena improves on Offline-RL’s 26.7%, a relative gain of 16.1%, with DynaWeb best on Reddit, GitLab, CMS, and Shopping.The gains span multiple website domains rather than being limited to one interface or interaction pattern.Source: Table 1 · p. 8, §4.2 · p. 9

4.1 Setups

• Baselines include base and proprietary agents, supervised fine-tuning, offline reinforcement learning, and inference-time lookahead.The lookahead baseline uses a web world model to simulate candidate future states and select actions by expected advantage; offline RL trains on online exploration trajectories scored by an outcome-based reward model.Source: §4.1 · p. 8

4.2 Main Results

• DynaWeb achieves the best overall success rate on WebVoyager and outperforms competing methods across many individual live websites.It underperforms the strongest baselines on a few sites, including ArXiv and GitHub, which typically require longer-horizon planning and handle rapidly evolving page states.Source: Table 2 · p. 9, §4.2 · p. 9

5.1 WM Dream Length Matters

• Average dream lengths of 4–5 steps perform best, as shorter rollouts under-complete tasks and longer rollouts compound world-model errors.Dream length trades off sufficient interaction depth for multi-step tasks against degraded training signals from accumulated model inaccuracies.Source: §5.1 · p. 10

5.2 Error Analysis of Local vs. Full-Page Transition Prediction

• The world model achieves a 97% sequence-level match rate for non-full-page transitions, which make up the majority of interaction steps.This metric compares predicted state-change descriptions with ground truth.Source: §5.2 · p. 10
• For full page transition actions, which account for approximately 36% of the dataset, the world model averages 3.4/4 under GPT-5 judging, summed across four prediction-quality criteria.Source: §5.2 · pp. 10–11
• About 12% of full-page transition predictions are judged completely incorrect, while the rest typically retain high-level structure and semantics despite minor element-level inconsistencies.The analysis links greater full-page transition difficulty to errors propagating and compounding during multi-step imagination.Source: §5.2 · p. 11

5.3 Effect of Real Interaction Data Percentage

• Adding approximately 40% real trajectories substantially improves success over the SFT baseline, while more real data yields diminishing returns.Agents trained exclusively on simulated trajectories tend to underperform SFT; moderate real data stabilizes learning while most training signal can still come from simulated rollouts.Source: §5.1 · p. 10, §5.3 · p. 11

5.4 Essential Role of WM Training

• Agents trained with the supervised world model achieve substantially higher success rates than those using frozen GPT-oss-120b on both WebArena and WebVoyager.The comparison holds agent architecture, RL algorithm, and training protocol fixed, indicating that general-purpose LLM priors alone do not provide a reliable simulator without learned web transition dynamics.Source: §5.4 · p. 12

6 Conclusion

• DynaWeb consistently improves strong open-source web agents across WebArena and WebVoyager.The framework trains agents through imagination rather than live web interaction.Source: §6 · p. 12
• DynaWeb combines policy-driven imagined rollouts from a learned web world model with real expert trajectories for on-policy optimization without live-environment costs and risks.Source: §6 · p. 12
• The analysis highlights rollout length, the regularizing role of real expert data, and explicitly trained world models as key principles for scalable imagination-driven web-agent training.These findings point toward world-model-centric learning for long-horizon decision making in complex environments.Source: §6 · p. 12

A.1 WebArena Training Prompt

• Figure 5 shows the system prompt used for training and evaluation of the WebArena agent.Source: Figure 5 · p. 19

A.2 World Model System Prompt

• The system prompt specifies the user objective, current webpage state, and executed actions as inputs, and requires predictions of web-state changes and resulting next-step accessibility.Figure 6 shows the prompt used to train the model to predict next-step accessibility trees from actions and current observations.Source: §A.2 · p. 19, Figure 6 · p. 20

A.3 Training Recipe and Hyperparameters

• DynaWeb trains agents online in a synthetic web environment powered by a learned world model.Source: §A.3 · p. 21
• Agents are initialized from a supervised web-navigation model and optimized with Group Sequence Policy Optimization (GSPO) using sparse terminal rewards.Source: §A.3 · p. 21
• World-model imagined rollouts are interleaved with real expert trajectories to stabilize learning.Source: §A.3 · p. 21