Source-linked AI summary
DynaWeb: Model-Based Reinforcement Learning of Web Agents
Hang Ding, Peidong Liu, Junqiao Wang, Ziwei Ji, Meng Cao, Rongzhao Zhang, Lynn Ai, Eric Yang, Tianyu Shi, Lei Yu
TL;DR
Web-agent reinforcement learning is limited by the cost, risk, and difficulty of collecting live-web experience. DynaWeb learns a web world model for imagined on-policy rollouts and mixes them with real expert trajectories, consistently improving open-source agents across WebArena and WebVoyager.
Problem
Online reinforcement learning for web agents requires live-web interaction that is inefficient, expensive, difficult to control, and risky at scale.
Method
DynaWeb trains web agents with a learned world model that predicts web-page transitions, generating imagined on-policy rollouts interleaved with real expert trajectories.
Results
DynaWeb consistently improves strong open-source web agents across the WebArena and WebVoyager benchmarks.
Takeaways & Limitations
Model-based imagination offers a practical and scalable path for training web agents while reducing reliance on costly real-environment interaction.
Abstract
from arXiv · showhide
The development of autonomous web agents, powered by Large Language Models (LLMs) and reinforcement learning (RL), represents a significant step towards general-purpose AI assistants. However, training these agents is severely hampered by the challenges of interacting with the live internet, which is inefficient, costly, and fraught with risks. Model-based reinforcement learning (MBRL) offers a promising solution by learning a world model of the environment to enable simulated interaction. This paper introduces DynaWeb, a novel MBRL framework that trains web agents through interacting with a web world model trained to predict naturalistic web page representations given agent actions. This model serves as a synthetic web environment where an agent policy can dream by generating vast quantities of rollout action trajectories for efficient online reinforcement learning. Beyond free policy rollouts, DynaWeb incorporates real expert trajectories from training data, which are randomly interleaved with on-policy rollouts during training to improve stability and sample efficiency. Experiments conducted on the challenging WebArena and WebVoyager benchmarks demonstrate that DynaWeb consistently and significantly improves the performance of state-of-the-art open-source web agent models. Our findings establish the viability of training web agents through imagination, offering a scalable and efficient way to scale up online agentic RL.
1 Introduction
DynaWeb addresses the cost, risk, and control problems of live-web interaction by making a learned web world model central to online reinforcement learning. It combines imagined rollouts with real expert trajectories to train web agents without relying entirely on live interaction.
- Motivation: Large-scale online RL for web agents is constrained because live-web interaction is inefficient, expensive, difficult to control, and potentially hazardous.Agents may trigger irreversible purchases, account modifications, or data submissions while facing nondeterministic page dynamics and external interference.
- DynaWeb: DynaWeb elevates a learned web world model from a planning aid to a synthetic environment for online policy optimization.Conditioned on page representations and agent actions, the model predicts next-state web representations and provides task-level feedback signals.
- Training experience: DynaWeb randomly interleaves imagined rollouts with fully real expert trajectories from existing training data.This mixture preserves interactive learning signals while reducing dependence on costly live-web exploration.
- DynaWeb: The framework trains agents with policy-driven imagined rollouts that can serve directly as on-policy reinforcement-learning experience.The world model predicts naturalistic web page states, enabling training without live web interaction.
- Results: Experiments on WebArena and WebVoyager show consistent performance improvements, with analysis of world-model-based imagination for web-agent training.The paper presents these experiments as evidence for the framework’s effectiveness and examines how imagination should be designed and used.
2 Related work
Prior web-agent work uses LLMs, data scaling, prompting, planning, and reinforcement learning, but many approaches still depend on costly real-environment interaction. DynaWeb instead makes a dedicated web world model part of the learning process, using imagined trajectories for policy optimization rather than only inference-time guidance.
- Web agents: Web agents commonly use LLMs as decision-making backbones with reasoning and interaction frameworks for structured multi-step web actions.Related systems include ReAct, MCP, and Cognitive Kernel, alongside interactive benchmarks such as WebShop.
- Web agents: Data-centric and agent-training methods improve web agents by collecting or synthesizing interaction data and scaling broader agent pipelines.Examples include Explorer, NNetNav, InSTA, WebThinker, WebDancer, WebSailor, WebShaper, Cognitive Kernel-Pro, and MiroFlow.
- World models: World-model approaches in web environments have primarily used LLMs to simulate action outcomes for inference-time planning or prompting.WebDreamer and WMA use simulation to guide immediate action selection rather than to improve the policy through learned trajectories.
- World models: DynaWeb trains a dedicated web world model and uses multi-step imagined trajectories directly for policy optimization.This positions the world model as a core learning component rather than an auxiliary decision-time tool.
- Reinforcement learning: RL-based web agents improve policies from interactive feedback but typically rely on large amounts of real-environment interaction.That reliance can make training expensive, unstable, and risky at scale.
3 Method
DynaWeb formulates web interaction as a partially observable task and trains agents in a learned synthetic web environment. It combines imagined rollouts with real expert trajectories and sequence-level policy optimization to support long-horizon learning without live-web interaction.
- 3.1 Problem Formulation: Web agents operate in a partially observable environment, receiving accessibility-tree observations and selecting atomic browser actions over multi-step trajectories.The policy conditions on the task query, interaction history, and system instructions, while terminal rewards measure task completion.
- 3.2 Web World Model: DynaWeb trains a learned web world model to predict naturalistic next observations from accessibility-tree states and agent actions.The model operates directly in observation space and generates web-page representations that the policy can consume during imagined rollouts.
- 3.2 Web World Model: The world model predicts state changes before applying them to the current observation, improving information gain when website transitions modify only part of a page.It is trained on cleaned real web interaction trajectories and becomes a reusable simulator for multi-step imagined trajectories.
- 3.3 DynaWeb: Model-based RL of Web Agents: The agent interacts with the world model as a synthetic environment to generate multi-step trajectories without interacting with the live web.Each imagined trajectory begins from the initial observation, repeatedly samples actions, and receives a model-based task-completion reward after termination.
- 3.3 DynaWeb: Model-based RL of Web Agents: DynaWeb mixes imagined rollouts with fully real expert trajectories and optimizes them using Group Sequence Policy Optimization.GSPO assigns each trajectory a sequence-level likelihood ratio and uses trajectory-level advantages from terminal returns for policy optimization.
4 Experiments
DynaWeb is evaluated against supervised, reinforcement-learning, and inference-time lookahead baselines on WebArena and WebVoyager. It achieves the strongest overall results across heterogeneous self-hosted and live websites, while remaining weaker on some long-horizon, highly dynamic tasks.
- Evaluation setup: The experiments use WebArena’s 812 tasks across five websites and four application domains, and WebVoyager’s 643 open-ended tasks from 15 live websites.Success Rate is the percentage of user instructions successfully completed.
- WebArena: DynaWeb achieves the highest average success rate across WebArena’s Reddit, GitLab, Maps, CMS, and Shopping domains.The evaluation compares supervised, RL, and inference-time lookahead baselines.
- WebArena: DynaWeb achieves the strongest WebArena performance, improving average success rate from 26.7 (Offline-RL) to 31.0, a relative gain of 16.1%.Its gains are reported across Reddit, Gitlab, CMS, and Shopping.
- WebVoyager: DynaWeb’s WebVoyager advantage extends across many live websites and the All Rec aggregate.Reported strengths include Amazon, Apple, BBC News, Cambridge Dict, Coursera, Google Map, Google Search, HuggingFace, and Wolframα.
- WebVoyager: DynaWeb underperforms the strongest baselines on some sites, including ArXiv and GitHub, where tasks require longer-horizon planning and highly branching, rapidly changing page states.These sites identify remaining challenges for world modeling and dynamic UI handling.
5 Analysis
The analysis identifies rollout depth, real trajectory data, transition difficulty, and explicit world-model training as central determinants of DynaWeb’s effectiveness. Moderate dreams and a substantial but limited amount of real data improve learning, while model errors and weak simulators constrain it.
- Dream length: Dream lengths of 4–5 steps produce the best agent success rate.Shorter trajectories under-complete tasks, while longer ones compound hallucinations and degrade training signals.
- Transition difficulty: The world model achieves 97% sequence-level match rate on localized state updates, which constitute most interaction steps.Full page transitions are more difficult and represent approximately 36% of the dataset.
- Transition difficulty: On full page transitions, the world model achieves an average score of 3.4/4 across action semantics, causal reasoning, page correctness, and faithfulness.Only about 12% of cases are judged completely incorrect, while most preserve high-level transition structure and semantics.
- Real trajectory data: Approximately 40% real trajectories substantially outperform the SFT baseline, while higher proportions yield diminishing returns.Agents trained exclusively on simulated trajectories tend to underperform SFT.
- Real trajectory data: Real interaction data acts as a critical regularizer by anchoring learning to reliable transitions and reward signals.This balance reduces reliance on costly real-environment interaction while retaining simulated-rollout training.
- World-model training: A supervised task-specific world model substantially outperforms a frozen GPT-oss-120b on downstream WebArena and WebVoyager success rates.The comparison keeps the agent architecture, RL algorithm, and training protocol fixed, isolating world-model training.
6 Conclusion
DynaWeb trains web agents through imagined interaction with a learned world model, mixing policy-driven rollouts with real expert trajectories instead of relying on live web interaction. Across WebArena and WebVoyager, it consistently improves strong open-source agents and highlights rollout length, real data, and explicit world-model training as key design principles.
- Conclusion: DynaWeb trains web agents through imagination rather than live web interaction.It uses a learned web world model to generate policy-driven imagined rollouts mixed with real expert trajectories.
- Conclusion: DynaWeb consistently improves strong open-source web agents across WebArena and WebVoyager.The conclusion presents model-based imagination as a practical and scalable path for web-agent training.
- Conclusion: The analysis identifies rollout length, real expert data, and explicitly trained world models as key principles for imagination-driven training.These findings motivate world-model-centric learning for long-horizon decision making in complex environments.
A.1 WebArena Training Prompt
The WebArena training prompt specifies the agent’s role, inputs, available actions, observation format, and completion criteria. It is fixed across real and imagined interaction during training and evaluation to support reproducibility.
- Prompt contents: The system prompt instructs an AI assistant to issue a browser action from the task objective, current step, observations, interaction history, and previous notes.It also specifies an interaction-history summary format.
- Prompt usage: The same prompt is used across real-environment interaction and imagined world-model rollouts.It remains fixed throughout training and evaluation.
- Prompt usage: The prompt is provided as the WebArena agent’s system prompt for training and evaluation.Figure 5 documents this prompt artifact.
A.2 World Model System Prompt
The web world model’s system prompt defines the information provided for predicting the next accessibility-tree state after an action.
- The prompt includes the user objective, current webpage state, and executed actions as model inputs.
- It specifies the required output format for predicting web state changes and the resulting next-step accessibility tree.
A.3 Training Recipe and Hyperparameters
DynaWeb optimizes an initialized web agent with online reinforcement learning in a learned synthetic web environment, combining imagined rollouts with real expert trajectories.
- Training uses online reinforcement learning in a synthetic web environment powered by a learned world model.
- The agent is initialized from a supervised web navigation model and optimized with Group Sequence Policy Optimization using sparse terminal rewards.
- Imagined world-model rollouts are interleaved with real expert trajectories to stabilize learning.