WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Model

1 Introduction

• Autonomous self-improvement can plateau, and declining exploration diversity and costly real-world trials constrain broader trajectory discovery.The paper also notes that simulation-based approaches typically provide only one- or two-step look-ahead.Source: §1 · pp. 1–2
• WebEvolver co-trains a world model that predicts the next web observation from the current state and an attempted action, evolving alongside the agent.The model is fine-tuned on trajectories collected during agent-environment interactions.Source: §1 · p. 2
• The world model generates self-instructed trajectories from simulated unseen web environments and performs multi-step look-ahead at inference.The paper says hallucinated states are acceptable during training because the agent learns flexible action prediction under noisy circumstances.Source: §1 · p. 2
• 10% performance improvement over the self-evolving baseline OpenWebVoyager across real-world, open-domain web environments.The reported evaluation includes Mind2Web-Live, WebVoyager, GAIA-web, and SimpleQA.Source: §1 · p. 2
• The authors report gains in performance and adaptability with minimal human supervision and no distillation from stronger LLMs.They argue that integrating dynamic world models addresses limitations of purely data-driven self-training.Source: §1 · p. 2

2 Related Work

• Prior self-improvement work iteratively refines agent policies using selected trajectories, while this paradigm optimizes LLM loss rather than reward functions directly.The paper characterizes its approach as iterative supervised fine-tuning with binary trajectory selection.Source: §2 · p. 3
• World-model approaches use LLMs to simulate and plan, including web methods such as WebDreamer and WMA.The cited discussion situates these methods within broader work on world models for agent reasoning.Source: §2 · p. 3
• WebDreamer and WMA use world models at inference, but the described approaches do not use them to train the agent itself.WMA's multi-step simulations are used for inference-time action selection, whereas WebEvolver generates rollouts up to depth seven for training.Source: §2 · p. 4
• WebEvolver co-learns a dedicated world model during self-improvement to support multi-step trajectory synthesis and look-ahead planning.The paper presents this as a more robust foundation for interactive decision-making than current prompt-based approaches.Source: §2 · p. 4

3.1 Problem Formulation

• The task is a POMDP: the agent acts on visible webpage observations represented as accessibility trees through browser operations, producing trajectories scored by self-assessment.Source: §3.1 · p. 4

3.2 Agent Self-Improvement

• The policy samples query trajectories, retains recent observations plus thoughts and actions, rejection-samples successful runs, then adds newly successful runs to the next iteration’s training set.Source: §3.2 · pp. 4–5

3.3 WebEvolver

• The world model learns next-observation prediction from self-improvement trajectories, conditioning on webpages and actions while retaining action and thought context.Source: §3.3 · p. 5
• The world model replaces real web transitions to synthesize policy-training trajectories, while World Model Look-Ahead simulates d-step rollouts scored by an LLM evaluator to select actions.Source: §3.3 · pp. 5–6

4.1 Setup

• The setup used Llama-3.3-70B for self-improvement on WebVoyager and Mind2Web-Live, with experiments repeated in roughly the same time window and averaged.Some websites were filtered out because they were inaccessible from the experimental environment.Source: §4.1 · p. 6

4.2 Self-Improvement

• For self-improvement, the agent sampled and filtered trajectories, fine-tuned through two iterations, and trained a co-evolving world model to predict subsequent observations.The final WebEvolver model combined world-model-synthesized data with self-improvement SFT data; the self-improvement loop used Llama rather than GPT-4o for evaluation.Source: §4.2 · pp. 6–7

4.3 Inference-time World Model Look-ahead (WMLA)

• WMLA sampled up to three candidate actions, simulated look-ahead trajectories at depths 1–3, and selected an action using GPT-4o scoring.The original policy generated the first action at temperature 0; subsequent candidates used temperature 0.7 and a prompt to differ from earlier actions.Source: §4.3 · p. 7

4.4 Results and Analysis

• A further 4% improvement on WebVoyager came from adding world-model-synthesized data to iteration 1 SFT data, outperforming synthetic trajectories without world modeling.The initial self-improvement iteration increased success rate by 6% over the zero-shot baseline, while performance plateaued at iteration 2.Source: §4.4 · p. 8
• WMLA performed best at depth 2, while depth 3 brought diminishing returns and generation depths above 2 had intrinsic-evaluation scores below 0.50.The evaluation used structural correctness, similarity, and overall assessment to assess generated webpages.Source: §4.4 · p. 8
• GAIA-web Level 1 and SimpleQA improved with self-improvement and world-model augmentation, but Level 2 showed limited gains because deeper reasoning and extended interactions were outside the training scope.The evaluation used GPT-4o to decompose queries and generate or calculate results, and Bing because of CAPTCHA challenges on Google.Source: §4.4 · pp. 8–9
• The branching-factor ablation found performance plateaued around k=3–5, with k=3 remaining the best choice.The authors attribute the plateau to states rarely offering more than five substantially different actions.Source: §4.4 · p. 9
• The world model generated plausible GitHub Search sort-menu options without fine-tuning on trajectories involving that control, illustrating pretrained web knowledge that can diversify interactions with unseen websites.The reported example includes options such as best match and most stars.Source: §4.4 · p. 9

• WebEvolver co-learns a world model with the agent policy to improve the self-improvement cycle.Source: §5 · p. 9
• The co-learned world model supports inference-time look-ahead to help select among sampled actions.Source: §5 · p. 9
• Experiments on WebVoyager, Mind2Web-Live, and GAIA-web demonstrate performance gains for self-improving agents.Source: §5 · p. 9

• The agent system includes only action generation; a standalone planning module could further enhance performance.Source: Limitations · p. 9
• Changing websites make exact replication of future web conditions difficult; experiments were run within approximately the same timeframe for fair comparisons.Source: Limitations · p. 9
• GAIA-web and SimpleQA provide supplementary evaluations focused on factual questions that are less susceptible to substantial changes over time.Source: Limitations · p. 9

A Details of Agent Implementation

• Available actions include clicking, typing, waiting, scrolling, going back, restarting at Google, and stopping with an answer or N/A.The prompt says to stop when the objective is achieved.Source: A Details of Agent Implementation · p. 13
• The agent must issue one action at a time, choose actions valid for the current observation, and avoid repeating unchanged actions or continuously waiting.Its response format pairs brief thoughts with the next action.Source: A Details of Agent Implementation · p. 13
• The world-model prompt predicts the next observation from the current accessibility tree and action, using either a free-text description or a structured accessibility tree.The two objectives are named Abstract Description and Accessibility Tree.Source: A Details of Agent Implementation · pp. 13–14
• Automatic task evaluation compares the instruction with the resulting accessibility tree and response, then gives a definitive success or not-success verdict.The evaluator is told not to interact with pages or assume information absent from them.Source: A Details of Agent Implementation · pp. 14–15
• The evaluation prompt treats screenshots as authentic and says they prevail when they contradict the generated response.A response claim absent from the screenshot may still be accepted.Source: A Details of Agent Implementation · p. 15
• Evaluation compares a generated accessibility tree with a ground-truth tree from an actual web server for structure correctness, element correctness, similarity, and functional coherence; scores are selected from 0, 1, 2, 3, with 3 meaning exactly the same and 0 total generation failure.Source: A Details of Agent Implementation · pp. 15–16

B Additional Details on Mind2web-live and WebVoyager Dataset

• Evaluation uses subsets of the Mind2Web-Live and WebVoyager test portions, with website exclusions specified separately for each benchmark.The appendix lists the excluded sites for both datasets.Source: B Additional Details on Mind2web-live… · p. 16
• The example asks for the most-starred machine-learning repositories created after 2023-01-01 on www.github.com; the entered search terms are machine-learning created:>2023-01-01 stars:>1.Source: B Additional Details on Mind2web-live… · p. 17
• The synthesized trajectory example includes a Google search accessibility tree and GitHub results trees showing repository-search sorting controls.The shown GitHub controls include sort options such as Most stars and Fewest forks.Source: Figure 4 · p. 17, Table 0 · p. 17