PaperlayerView exact revision on arXiv

Source-linked AI summary

Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents

Yu Gu, Kai Zhang, Yuting Ning, Boyuan Zheng, Boyu Gou, Tianci Xue, Cheng Chang, Sanjari Srivastava, Yanan Xie, Peng Qi, Huan Sun, Yu Su

arXiv:2411.06559v2cs.AI

TL;DR

Irreversible web actions make backtracking difficult, and test-time search can hurt efficiency. WEBDREAMER uses LLMs to simulate and score candidate-action outcomes, alongside training Dreamer-7B from synthesized web interactions. It improves on reactive baselines across benchmarks and is competitive with tree search while being 4–5 times more efficient on VisualWebArena.

  • Problem

    Real-world web actions can be irreversible, making backtracking difficult, while excessive test-time search reduces efficiency.

    Source: §1 · p. 1

  • Method

    WEBDREAMER uses LLMs to simulate action outcomes and score future states, while a synthesis pipeline trains Dreamer-7B as a specialized world model.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    WEBDREAMER improves over reactive baselines across three benchmarks and is competitive with tree search while 4–5 times more efficient on VisualWebArena.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Dreamer-7B performs comparably to GPT-4o on two online benchmarks, and in-domain fine-tuning can surpass GPT-4o on VisualWebArena.

    Source: §1 · p. 2

    Effectiveness diminishes at planning horizons of two and three steps, primarily because simulated action proposals can hallucinate unavailable actions.

    Source: §5.1 · p. 8

Abstract

from arXiv · show

Language agents based on large language models (LLMs) have demonstrated great promise in automating web-based tasks. Recent work has shown that incorporating advanced planning algorithms, e.g., tree search, is advantageous over reactive planning for web agents. However, unlike simulated sandbox environments, real-world environments such as the web are rife with irreversible actions. This undermines the feasibility of backtracking, a cornerstone of (tree) search. Overly relying on test-time search also hurts efficiency. We advocate model-based planning for web agents that employs a world model to simulate and deliberate over the outcome of each candidate action before committing to one. We systematically explore this paradigm by (1) Proposing a model-based planning framework, WebDreamer, which employs LLMs to serve as both world models and value functions; (2) Training specialized LLMs as world models with a scalable data synthesis pipeline. Empirical results demonstrate that WebDreamer achieves substantial performance improvements over reactive baselines. It is competitive, while being 4-5 times more efficient, with tree search in sandbox environments (VisualWebArena) and also works effectively on real-world websites (Online-Mind2Web and Mind2Web-Live). Furthermore, our trained world model, Dreamer-7B, performs comparable to GPT-4o, highlighting the potential of specialized world models for efficient and effective planning in complex web environments.

1 Introduction

Web planning must contend with irreversible real-world actions and the difficulty of backtracking, while world models for the open-ended Internet remain under-explored. WEBDREAMER uses LLMs to simulate and score possible future states, and the paper reports gains over reactive agents and efficiency competitive with tree search.

1 Introduction

  • Real-world web actions can be irreversible, making backtracking difficult or infeasible and excessive test-time search costly.Examples include submitting orders and changing privacy settings.Source: §1 · p. 1
  • WEBDREAMER uses LLMs to simulate and score possible future states before action, and Dreamer-7B is trained on over 3.1 million synthesized interaction instances.The approach addresses the open-ended, diverse web setting where building world models remains under-explored.Source: §1 · p. 2
  • WEBDREAMER improves over reactive baselines across three benchmarks and is 4–5 times more efficient than tree search on VisualWebArena.It also performs effectively on real-world websites, where tree search can be difficult to implement and deploy.Source: §1 · p. 2

2 Related Work

Prior web-agent work spans reactive systems and tree search, while LLM-based world models extend model-based planning to complex web environments. The paper distinguishes its focus on advanced LLMs and self-trained general world models from a concurrent effort centered on small-scale in-domain training.

2.1 Web Agents

  • Reactive agents remain shortsighted, while tree-search web agents improve multi-step planning but add inference time and risk irreversible actions on live websites.Prior approaches include best-first search and MCTS, as well as GPT-4V-based reward modeling.Source: §2.1 · p. 3

2.2 World Models

  • Unlike traditional RL world models focused on data efficiency, LLM-based world models emphasize decision-making, which this work explores in complex web environments.A concurrent work, Chae et al. (2025), uses small-scale data for in-domain models, whereas this work examines advanced LLMs and self-trained general world models.Source: §2.2 · p. 3

3 WEBDREAMER: Model-Based Planning for Web Agents

WEBDREAMER plans by simulating candidate-action outcomes and scoring trajectories before execution, then repeating decisions as new observations arrive. The paper also describes a pipeline for synthesizing web interaction data and training a specialized state-transition model.

3.1 Preliminary

  • Model predictive control simulates candidate actions over a finite horizon, scores their trajectories, executes the highest-scoring action, and replans after new observations.The formulation addresses the cost and irreversibility of tree search using real interactions.Source: §3.1 · p. 3

3.2 Core Design

  • WEBDREAMER uses LLMs for both outcome simulation and trajectory scoring, then executes the initial action from the highest-scoring simulated trajectory.The illustrated example simulates two-step trajectories for three candidate actions.Source: §3.2 · p. 4, Figure 2 · p. 4
  • Its simulation predicts action effects and imagines a possible next action, generating trajectories to a configurable depth H.State changes are described in concise natural language.Source: §3.2 · p. 4
  • Candidate generation samples top-k actions and uses LLM self-refinement to remove unnecessary options before simulation.The scoring function rates trajectories as complete, on track, or incorrect, and averages scores across multiple simulated trajectories.Source: §3.2 · pp. 4–5

3.3 World Model Data Synthesis and Model Training

  • Web Random Walking samples URLs from the October 2024 Common Crawl Index and uses heuristic-guided actions, adjusting action probabilities and prioritizing newly emerged elements.GPT-3.5-turbo generates contextually relevant queries for search inputs.Source: §3.3 · p. 5
  • Before-and-after visual snapshots are used to synthesize state-change descriptions, producing a filtered dataset of over 3.1M interaction instances.The pipeline filters failed interactions, automation-blocked content, and potentially harmful data.Source: §3.3 · p. 5, Figure 3 · p. 5
  • Dreamer-7B is initialized from Qwen2-VL-7B and trained to describe the next state after an action, with the paper focusing on H=1.The authors report H=1 as the most effective and efficient configuration and use an intrinsic evaluation set for checkpoint selection.Source: §3.3 · p. 6

4 Experiments

Experiments compare WEBDREAMER with reactive agents across three web benchmarks and with tree search on VWA. Results show gains across benchmarks and an efficiency advantage over tree search on VWA.

4.1 Setup

  • Evaluation covered 233 VWA tasks, a stratified 100-task Online-Mind2Web sample, and 104 Mind2Web-Live tasks; all used screenshots, with Set-of-Mark added in VWA.The planning horizon was set to H=1.Source: §4.1 · p. 6
  • Tree search comparisons are limited to VWA because real-site resets cannot undo effects, making backtracking infeasible on live websites.Source: §4.1 · p. 6

4.2 Main Results

  • 34.1% relative gain on VWA, 42.3% on Online-Mind2Web, and 23.8% on Mind2Web-Live over the reactive baseline; WEBDREAMER trails tree search slightly on VWA.Table 1 reports success rates across the three benchmarks.Source: §4.2 · p. 6, Table 1 · p. 7
  • 4.7% on VisualWebArena, 4.0% on Online-Mind2Web, and 4.8% on Mind2Web-Live absolute success-rate improvements for Dreamer-7B over vanilla Qwen2-VL-7B.Dreamer-7B also outperforms Qwen2-VL-72B.Source: §4.2 · pp. 6–7
  • Dreamer-7B performs comparably to GPT-4o on Online-Mind2Web and Mind2Web-Live.Source: §4.2 · p. 7
  • 4–5 times more efficient than tree search on VWA, WEBDREAMER has comparable action-step counts to the reactive baseline while tree search requires about 3 times as many steps.Tree search also introduces about 10 times greater wall-clock latency than reactive baselines; Table 2 compares efficiency using GPT-4o for all methods.Source: §4.2 · p. 7, Table 2 · p. 7

5 Discussions

The discussion finds that simulation and self-refinement contribute to planning performance, while longer horizons can introduce errors. Training-data scaling and in-domain adaptation show promise, with outcomes varying across benchmarks and environments.

5.1 Planning Framework

  • Removing simulation leaves only a small gain over the reactive baseline, while removing self-refinement also lowers performance.The authors attribute the latter decline to filtering less relevant candidate actions when the next optimal action is clear.Source: Figure 4 · p. 7, §5.1 · pp. 7–8
  • Performance beats the reactive baseline at horizons 1, 2, and 3, but begins to diminish at horizons 2 and 3 as simulated proposals hallucinate unavailable actions.Longer simulations make trajectories less distinguishable and accumulate errors; Figure 5 reports the horizon comparison on Online-Mind2Web.Source: Figure 5 · p. 8, §5.1 · p. 8

5.2 Training World Models

  • Larger training datasets improve Mind2Web-Live performance with tapering gains, while Online-Mind2Web shows a small-scale dip followed by more consistent gains.The scaling analysis covers these two online benchmarks, not the sandbox environments.Source: Figure 6 · p. 8, §5.2 · p. 8
  • In-domain continual training improves over Dreamer-7B and reaches or exceeds GPT-4o in some VWA environments, with the strongest benefits on Classifieds and Shopping.The setup trains separate environment-specific models using 25K synthesized interactions per environment; Reddit performance remains unchanged, which the authors associate with dense pages and limited interactions.Source: §5.2 · pp. 8–9, Table 3 · p. 9
  • GPT-4o gives detailed predictions that can include irrelevant URL and interface changes, while Dreamer-7B focuses on concise, action-relevant outcomes.The case study also notes that simulation inaccuracies can lead to incorrect predictions.Source: Figure 7 · p. 9, §5.2 · pp. 9–10

6 Conclusion

The paper explores LLM-based world models for model-based planning in web environments, including a framework and a specialized trained model. The reported results show improvements over reactive baselines, competitive efficiency against tree search, and Dreamer-7B performance comparable to GPT-4o, with further gains from in-domain fine-tuning.

6 Conclusion

  • WEBDREAMER improves substantially over reactive baselines across three benchmarks and is 4–5 times more efficient yet competitive with tree search.The comparison with tree search is on VisualWebArena; the paper notes that tree search is often impractical on real-world websites.Source: §6 · p. 10
  • Dreamer-7B, trained on 3.1 million synthesized web interaction examples, achieves performance comparable to GPT-4o.The paper uses a scalable data synthesis pipeline to train the world model.Source: §6 · p. 10
  • Continual in-domain fine-tuning further improves Dreamer-7B's simulation quality and downstream performance, surpassing GPT-4o.The authors describe the work as groundwork for scaling world models to general web tasks.Source: §6 · p. 10

Overview

The appendix collects implementation details for WebDreamer and data synthesis and world-model training, intrinsic evaluation, and a case study.

A Contribution Statement

  • The contribution statement credits Gu with the planning framework and core experiments, Zhang with data synthesis and world-model training, and Ning with intrinsic and online evaluation.The statement also describes contributions to development, experimentation, and manuscript refinement.Source: A Contribution Statement · p. 14

B.1 Action Proposal

  • WebDreamer’s browser prompt supplies the task, screenshot, page context, and element IDs, while requiring one valid action at a time.The prompt also specifies an action format and a stop action for completed tasks.Source: §B.1 · pp. 14–15

B.2 Self-Refinement

  • The self-refinement prompt filters candidate actions for possible task relevance, including indirect actions that may help over a longer horizon.Its role is not to select the best action at the current step.Source: §B.2 · pp. 15–16

B.3 World Model

  • The world-model prompt predicts webpage state changes from the initial screenshot, prior changes, and an operation to perform.It requests changes such as elements appearing, disappearing, or changing content.Source: §B.3 · p. 16

B.4 Reward Model

  • The reward-model prompt judges whether simulated steps reach the user’s goal and discounts predicted changes that seem unrealistic or unavailable.It can also recognize simulated failure that appears on track toward success.Source: §B.4 · pp. 16–17

C.1 Data Synthesis Prompt

  • Data-synthesis examples combine an element’s referring expression with a description of the resulting webpage change, using a randomly selected template.The pipeline uses before-and-after screenshots and Qwen2-VL-72B to describe the target element and state change.Source: §C.1 · pp. 17–18
  • The results report generalization to unseen benchmark prompts and Set-of-Mark images despite training on few prompt templates and natural images.The cited passage names Online-Mind2Web, Mind2Web-Live, and VisualWebArena.Source: §C.1 · p. 17

C.2 Training Data Statistics

The training corpus comprises millions of interactions across varied websites and trajectory depths, with click, hover, type, and select as its primary interaction types.

C.2 Training Data Statistics

  • Over 3.1 million interaction instances from 1.2 million unique URLs form the synthesized training data.The pipeline explores web pages across a wide range of domains.Source: §C.2 · p. 18
  • Clicks account for 84% of the data; click, hover, type, and select are the four primary interaction types, while scroll was excluded for little downstream benefit.Source: §C.2 · p. 18
  • Training actions span multiple positions in multi-step trajectories, with an average position of 4.4 rather than being limited to initial steps.The authors say this distribution may help the world model learn web dynamics relevant to long-term planning.Source: §C.2 · p. 18

C.3 World Model Training

The world models are fine-tuned from a shared vision-language backbone, while checkpoint selection uses an intrinsic evaluation intended to isolate world-model performance.

C.3 World Model Training

  • World models use the Qwen2-VL-7B-Instruct backbone and structured prompts asking for the webpage state after an action on an element.Dreamer-7B is trained for up to two epochs on the full training dataset.Source: §C.3 · p. 18
  • Actions used for training span different trajectory depths rather than only initial steps.Source: Figure C.1 · p. 19
  • Classifieds, Reddit, and Shopping variants continue training from Dreamer-7B for one epoch on their respective in-domain datasets.They use a reduced learning rate and shorter warmup for adaptation on smaller datasets.Source: §C.3 · p. 19

D Intrinsic Evaluation of World Models

  • Intrinsic evaluation provides a more controlled, independent assessment of world models because downstream tasks also depend on factors such as the policy model.The authors use this evaluation for model development and checkpoint selection.Source: D Intrinsic Evaluation of World Models · p. 19

D.1 Dataset Construction

The intrinsic evaluation tests whether world models rank ground-truth actions above deviations, using task trajectories and three levels of accuracy.

D.1 Dataset Construction

  • The evaluation set contains 44 tasks, 141 states with deviations, and 279 deviation actions derived from Mind2Web trajectories.Failed trajectories supply negative actions; deviations on successful trajectories are filtered to reduce false negatives.Source: §D.1 · p. 19

D.2 Evaluation Metrics

  • State-level accuracy is the checkpoint-selection metric, requiring the ground-truth action to score at least as highly as every deviation at a state.The authors choose it because it closely aligns with downstream task form.Source: §D.2 · p. 20
  • Pair-wise accuracy checks ground-truth actions against deviations individually, while task-level accuracy requires correct state-level rankings at every state in a task.Source: §D.2 · p. 20

D.3 Results

Intrinsic evaluation finds the fine-tuned 7B model comparable to GPT-4o on task-level accuracy and stronger on pair-wise and state-level accuracy. Across evaluated world models, task-level accuracy also correlates with downstream task success.

D.3 Results

  • The fine-tuned 7B model matches GPT-4o on task-level accuracy and outperforms it on pair-wise and state-level accuracy.Table D.1 reports intrinsic evaluation results for various world models.Source: Table D.1 · p. 20, §D.3 · p. 20
  • 0.8455 Pearson correlation connects task-level accuracy with task success rate across the evaluated world models.The evaluation compares task-level accuracy on the intrinsic set with task success rate on Mind2Web-Live.Source: §D.3 · p. 20
  • The authors hope intrinsic evaluation can help advance web world model development.Source: §D.3 · p. 21

E.1 Error Caused by Imperfect World Model Simulation

The section presents one error case attributed to imperfect world model simulation and one positive case in which simulation predicts the correct action.

E.1 Error Caused by Imperfect World Model Simulation

E.2 Positive Case Benefiting from World Model Simulation

Related papers

Loading 2411.06559v2…