PaperlayerView exact revision on arXiv

Source-linked AI summary

Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation

Hyungjoo Chae, Namyoung Kim, Kai Tzu-iunn Ong, Minju Gwak, Gwanwoo Song, Jihoon Kim, Sunghwan Kim, Dongha Lee, Jinyoung Yeo

arXiv:2410.13232v2cs.CL

TL;DR

LLM web agents perform poorly on long-horizon tasks, and the paper investigates whether lacking awareness of action outcomes contributes to their decision difficulties. It trains a world model to describe state transitions and uses simulated outcomes to guide action selection. The WMA agent improves WebArena policy selection over vanilla CoT, achieves new SOTA performance on Mind2Web, and is more cost- and time-efficient than a tree-search agent.

  • Problem

    LLM-based web agents lag humans on long-horizon tasks, while existing agents lack world models that let them foresee action outcomes without performing them.

    Source: §1 · p. 1

  • Method

    WMA simulates candidate action outcomes with a world model trained on natural-language descriptions of state transitions, then uses a value function to select an action.

    Source: §1 · p. 2, §4.1.2 · p. 5

  • Results

    WMA improves WebArena success rate over vanilla CoT (16.6% vs 13.1%), achieves new SOTA performance on Mind2Web, and reports 6.8x cost- and 5.3x time-efficiency versus tree search.

    Source: §1 · p. 2, §5.2 · p. 8

  • Takeaways & Limitations

    The results establish a foundation for studying world-model-augmented LLM web agents and simulated environment feedback for policy adaptation.

    Source: §1 · p. 2, §7 · p. 10

    The agent observes only the viewport, so the paper models web navigation as a partially observable Markov decision process.

    Source: §4 · p. 4

Abstract

from arXiv · show

Large language models (LLMs) have recently gained much attention in building autonomous agents. However, the performance of current LLM-based web agents in long-horizon tasks is far from optimal, often yielding errors such as repeatedly buying a non-refundable flight ticket. By contrast, humans can avoid such an irreversible mistake, as we have an awareness of the potential outcomes (e.g., losing money) of our actions, also known as the "world model". Motivated by this, our study first starts with preliminary analyses, confirming the absence of world models in current LLMs (e.g., GPT-4o, Claude-3.5-Sonnet, etc.). Then, we present a World-model-augmented (WMA) web agent, which simulates the outcomes of its actions for better decision-making. To overcome the challenges in training LLMs as world models predicting next observations, such as repeated elements across observations and long HTML inputs, we propose a transition-focused observation abstraction, where the prediction objectives are free-form natural language descriptions exclusively highlighting important state differences between time steps. Experiments on WebArena and Mind2Web show that our world models improve agents' policy selection without training and demonstrate our agents' cost- and time-efficiency compared to recent tree-search-based agents.

1 INTRODUCTION

LLM-based web agents lag humans on long-horizon tasks, and the paper links their decision difficulties to lacking a world model of action outcomes. It introduces a world-model-augmented agent that predicts state transitions using concise descriptions and reports improved policy selection and efficiency.

Motivation

  • On WebArena, GPT-4 achieves a 14.41% task success rate, compared with 78.24% for humans.The paper frames this gap as a question about why LLMs perform worse in web navigation.Source: §1 · p. 1

Approach

  • The WMA agent simulates candidate actions’ outcomes and uses a value function to select the action with the highest estimated reward.Its world model is trained on free-form natural-language descriptions that highlight important state differences, rather than entire next-page observations.Source: §1 · p. 2

Contribution

  • Compared with recent tree-search-based agents, the paper reports 6.8x cost-efficiency and 5.3x time-efficiency.The authors also describe their work as pioneering world models in LLM-based web agents.Source: §1 · p. 2

2 RELATED WORK

Prior web-agent research spans benchmarks for navigation, training-free methods, and world models used in autonomous agents. This paper situates its approach in that landscape by evaluating on WebArena and Mind2Web and representing web-state changes with free-form descriptions.

Web navigation benchmarks

  • WebArena and Mind2Web cover diverse web tasks, with WebArena emphasizing functional correctness and realistic scenarios.The paper adopts both for their generalizability and resemblance to real-world web interactions.Source: §2 · p. 2

LLM-based web agents

World models in autonomous agents

  • World models can support learning or planning without environment interaction, using raw observations or latent representations.Prior LLM-based approaches also convert visual observations to natural language or transform observations into structural formats [Zhang et al., 2024; Wang et al., 2024a].Source: §2 · pp. 2–3

3 PRELIMINARY ANALYSES: ARE CURRENT LLMS AWARE OF ENVIRONMENT DYNAMICS IN WEB NAVIGATION?

The preliminary analyses test whether current LLMs predict web-action outcomes and whether seeing those outcomes helps them choose actions aligned with user goals. The reported results indicate weak next-state prediction but better action selection when candidate outcomes are available.

3 PRELIMINARY ANALYSES: ARE CURRENT LLMS AWARE OF ENVIRONMENT DYNAMICS IN WEB NAVIGATION?

  • The preliminary analyses address both outcome prediction and goal-aligned action choice when candidate outcomes are provided.They use human-annotated WebArena trajectories as the basis for these questions.Source: §3 · p. 3

3.1 PRELIMINARY ANALYSIS I - LLMS STRUGGLE WITH PREDICTING THE NEXT STATES CAUSED BY THEIR ACTIONS

  • Across four closed-source LLMs tested on 100 WebArena instructions, next-state prediction averages 54.75%, and Claude-3.5-Sonnet performs almost as badly as random guessing.The task asks each model to choose between the golden next state and a lexically similar incorrect state; humans perform significantly better.Source: §3.1 · p. 3, §3 · p. 3, Figure 1 · p. 3

3.2 PRELIMINARY ANALYSIS II - LLMS MAKE BETTER ACTION SELECTION WHEN ACCESSING THE OUTCOME OF EACH ACTION CANDIDATE

  • With candidate next states available, GPT-4o action-selection accuracy rises from 53% to 73%, while LLMs’ average accuracy improves by up to 38%.The comparison supplies the next state for each candidate action versus relying on the current state and user objective alone.Source: Figure 2 · p. 3, §3.2 · p. 4

3.3 INSIGHTS FROM PRELIMINARY ANALYSES

  • The analyses conclude that LLMs struggle to predict how actions affect next states, but can make better decisions when given those effects.The authors identify world models as a promising direction for complex, long-horizon web navigation.Source: §3.3 · p. 4

4 WORLD-MODEL-AUGMENTED WEB AGENTS

The paper models web navigation as a partially observable process and equips web agents with world models that simulate plausible next observations to support better decisions. It trains those models on abstracted state changes and uses their predictions to select actions at inference time without updating the policy model.

4 WORLD-MODEL-AUGMENTED WEB AGENTS

  • Web navigation is modeled as partially observable, with hidden states, language-guided actions, and accessibility-tree observations.The agent receives a partial observation after each action.Source: §4 · p. 4

4.1.1 STEP I: HARVESTING AGENT-ENVIRONMENT INTERACTION DATA

  • Training data comes from instruction-conditioned agent-environment trajectories, recorded as instructions, observations, actions, and next observations.An LLM agent iteratively acts from current observations to pursue each user instruction.Source: §4.1.1 · p. 5

4.1.2 STEP II: TRANSITION-FOCUSED OBSERVATION ABSTRACTION

  • Full textual next observations repeat much of the current state and remain long, reducing training information gain and increasing computational costs.Accessibility-tree observations still average 4K tokens.Source: §4.1.2 · p. 5
  • The abstraction matches elements across consecutive observations, extracts updated, deleted, and added elements, then prompts an LLM to describe the changes in free-form natural language.These transition-focused descriptions replace full next observations as world-model training targets.Source: §4.1.2 · p. 5

4.1.3 STEP III: LEARNING ENVIRONMENT DYNAMICS

  • The world model is trained to predict the abstracted next observation from the instruction, current observation, and action.This prediction lets the model simulate potential action outcomes.Source: §4.1.3 · p. 5

4.2 INFERENCE-TIME POLICY OPTIMIZATION WITH THE WORLD MODEL

  • At inference time, the agent samples candidate actions, predicts their next observations, and selects the action with the highest value-function reward estimate.The policy model remains frozen, so the world model augments action selection without policy training.Source: §4.2 · p. 6

5 EXPERIMENTS

Across WebArena and Mind2Web, WMA improves agent performance, while using less time and API cost than tree search. Ablations support fine-tuning world models, transition-focused observations, and simulated next states for reward estimation.

5.1 SETUPS AND IMPLEMENTATION DETAILS

  • WebArena and Mind2Web provide the evaluation settings, with 870 synthetic instructions and 14K WebArena instances used to train world models.Mind2Web uses offline training trajectories; WebArena tasks span five websites and four domains.Source: §5.1 · pp. 6–7

5.2 MAIN RESULTS

  • WMA reaches 16.6% success rate versus vanilla CoT’s 13.1% on WebArena, while tree search reaches 19.2%.WMA’s relative gain over CoT is 29.7%, compared with tree search’s 28.0%.Source: Table 1 · p. 7, §5.2 · p. 8
  • WMA significantly outperforms AWM and achieves new state-of-the-art performance on Mind2Web.The authors report that training on Mind2Web data also shows strong generalization, valuable when collecting data for new web environments is difficult.Source: §5.2 · p. 8, Table 3 · p. 8
  • WMA works with both accessibility-tree observations in WebArena and HTML observations in Mind2Web, without additional policy-model training.The policy model is frozen during the approach.Source: §5.2 · p. 8

5.3 ANALYSES OF TIME AND COST EFFICIENCY

  • WMA takes 140.3 seconds per instance versus tree search’s 748.3 seconds, and tree search requires 6.8 times more API cost.The authors attribute the efficiency difference to simulating candidate actions rather than executing them in the environment.Source: §5.3 · pp. 8–9, Table 4 · p. 9

5.4 ABLATION STUDIES

  • Using simulated next states in reward estimation improves reward prediction and task performance over estimating reward from only the current observation and action.The comparison is against a Q-value function using only (o_t, a_t).Source: §5.4 · p. 9, Table 5 · p. 9
  • Fine-tuning yields better world models than two-shot prompting, while predicting full accessibility trees produces the worst success rate among the ablations.The authors attribute the full-tree result to redundant and repeated information obscuring critical state changes.Source: §5.4 · p. 9, Table 5 · p. 9
  • Agent performance trends upward with more sampled actions, and the fine-tuned value model performs slightly better than prompted GPT-4o-mini.The authors describe fine-tuning the value function as a reasonable alternative when API budgets are limited.Source: §5.4 · pp. 9–10, Figure 6 · p. 9, Table 6 · p. 9

6 FURTHER ANALYSES

The further analyses test simulated-feedback self-refinement and categorize errors in world-model predictions. Self-refinement improves accuracy over CoT, though simulate-score-select performs better in the reported Map experiment.

6.1 COMBINING SELF-REFINE WITH OUR WORLD MODELS

  • In the Map experiment, simulated-feedback self-refinement improves accuracy by 1.8 percentage points over CoT, while simulate-score-select achieves almost twice the accuracy.The policy model refines its action after the world model simulates the next observation.Source: §6.1 · p. 10, Table 7 · p. 10

6.2 TYPES OF ERRORS IN WORLD MODELS’ PREDICTIONS

  • Among 50 sampled erroneous predicted states, 42% involve counterfactual elements, 26% web-element or function errors, 24% overly generic statements, and 8% other errors.A CS major manually judged errors by comparing the viewport with the predicted observation.Source: §6.2 · p. 10
  • The error categories include invented elements, misuse of web components, and predictions that skip ahead beyond the next observation.The sampled states come from WebArena world-model predictions.Source: §6.2 · p. 10

7 CONCLUSIONS

The paper presents world models as a way to address LLM web agents’ difficulty understanding environment dynamics. Its experiments report policy-selection efficacy, improved cost and time relative to tree search, and new state-of-the-art performance on Mind2Web.

7 CONCLUSIONS

  • The paper’s central contribution is incorporating world models into LLM-based web agents to address their limitations in understanding environment dynamics.The conclusion frames this as groundwork for future research in web navigation.Source: §7 · p. 10
  • WMA simulates action outcomes to support policy selection and outperforms tree search in cost and time while achieving new state-of-the-art performance on Mind2Web.The conclusion summarizes results across WebArena and Mind2Web.Source: §7 · p. 10
  • The conclusion identifies transition-focused observation abstraction as the approach used to train the world models.Figure 7 reports statistics for error types in predicted observations.Source: §7 · p. 10, Figure 7 · p. 10

A FURTHER ANALYSES

The paper explores multimodal world models, multi-step planning, and prediction coverage. Visual input improves reported metrics, while deeper search helps but width performs slightly better at equal exploration budget.

A FURTHER ANALYSES

  • Multimodal input leads to improvements across all reported metrics on Mind2Web cross-task, despite using a smaller model than Llama-3.1-8B.The authors conclude that the framework adapts to multimodal settings and that visual input provides benefits.Source: A FURTHER ANALYSES · p. 14, Table 8 · p. 14
  • 17.1 → 19.6: increasing search depth from (w = 3, d = 1) to (w = 3, d = 3) improves performance, but width is slightly better at equal budget.The comparison at equal exploration budget is (w = 9, d = 1) versus (w = 3, d = 3); action-element IDs can change between search and execution after reload.Source: A FURTHER ANALYSES · p. 14, Table 9 · p. 14
  • Coverage measures the proportion of ground-truth observation sentences covered by the predicted observation, assessed with an LLM-as-judge procedure.The evaluation uses GPT-4o as judge and compares the world model with GPT-4o-mini and GPT-4o on 100 preliminary-analysis samples.Source: A FURTHER ANALYSES · pp. 14–15

B EXPERIMENTAL DETAILS OF PRELIMINARY ANALYSES

The preliminary analyses use binary classification to evaluate next-state prediction and construct negative examples from lexically similar incorrect states. The second analysis samples negative actions from GPT-4o-mini using a two-shot demonstration.

B.1 PRELIMINARY ANALYSIS I

  • Next-state prediction is evaluated as binary classification rather than generation, avoiding the difficulty of evaluating generated HTML or accessibility trees and potential judge bias.The authors note there is no consensus that LLMs can judge this capability.Source: §B.1 · p. 15
  • Negative next-state examples are the top lexically similar incorrect states selected with difflib; answer choices are then randomly shuffled.The lexical similarity is calculated between the golden next state and similar but incorrect states.Source: §B.1 · p. 15

B.2 PRELIMINARY ANALYSIS II

  • GPT-4o-mini greedily generates nine negative action candidates using a two-shot demonstration.The candidates are used in preliminary analysis II.Source: §B.2 · p. 15

C.1.1 DATASET CONSTRUCTION

The appendix describes dataset construction, training, inference, and benchmark setups for the world models and agents. It reports synthetic WebArena training data, a transition-focused abstraction, and distinct evaluation procedures for WebArena and Mind2Web.

C.1.1 DATASET CONSTRUCTION

  • 14,200 WebArena instances were generated with GPT-4o-mini to collect world-model training data, while transition abstraction matches observation elements and prompts an LLM to describe state changes.The matching uses the Hungarian algorithm; the resulting free-form description captures the transitions.Source: §C.1.1 · pp. 15–16

C.1.2 TRAINING

  • World models and value functions were trained for two epochs on eight RTX 4090 GPUs, using a 1e-5 learning rate and around three GPU hours.Source: §C.1.2 · p. 16

C.2 INFERENCE

  • At inference, the system samples 20 actions, selects the three most frequent, predicts their next states, scores them with a value function, and chooses the highest-reward action.Top-p decoding uses p = 1.0.Source: §C.2 · p. 16

C.3 DETAILS ON WEBARENA

  • WebArena experiments use the standardized Docker environment; its tasks span Shopping, CMS, Reddit, Gitlab, and Map, with Map searches using online OpenStreetMap.The supplied setup notes that the Docker route-search button does not work properly.Source: §C.3 · pp. 16–17

C.4 DETAILS ON MIND2WEB

  • Mind2Web evaluation reports element accuracy, action F1, step success rate, and task success rate, using ground-truth history at each step.A step requires both the selected element and action to be correct; a task succeeds only if all steps succeed.Source: §C.4 · p. 17

C.5 IMPLEMENTATION DETAILS OF BASELINES

The appendix details baseline implementations and supplemental procedures. It notes that element filtering can remove relevant elements and that the tree-search cost analysis covers only a subset of WebArena instances.

C.5 IMPLEMENTATION DETAILS OF BASELINES

  • Vanilla CoT samples 20 actions and selects the most frequent, while the Tree Search Agent uses the official repository and is evaluated on only 10% of WebArena instances for cost and time analysis.Agent Workflow Memory uses repository code and is tested with and without MindAct candidate generation; baseline results marked with an asterisk come from original papers.Source: §C.5 · p. 17

C.6 ISSUE REGARDING THE ELEMENT FILTERING MODULE OF MINDACT

  • The MindAct element-selection filter can remove relevant elements as well as irrelevant ones, so AWM was reimplemented with and without filtering.Source: §C.6 · pp. 17–18

C.7 INSTANCE IDS OF ADAPTED TASKS FOR WEBARENA

  • The adapted WebArena evaluation randomly samples 200 instances: 50 Shopping, 50 Gitlab, and 100 Map tasks.Map contributes more instances because its inference is shorter and therefore more cost- and time-efficient.Source: §C.7 · p. 18

D.1 SELF-REFINE

  • Self-refine generates an initial CoT action, obtains feedback from the method's value model, then revises the action accordingly.Source: §D.1 · p. 18

D.2 ERROR TYPE ANALYSIS AND EXAMPLES

Appendix D.2 reports a 50-error WebArena analysis and illustrates four error types; the following sections show successful inferences and catalog prompts used in the study.

D.2 ERROR TYPE ANALYSIS AND EXAMPLES

  • The WebArena error analysis samples 50 inference errors across Shopping (8), CMS (11), Gitlab (11), Reddit (10), and Map (10).The examples are presented as illustrations of four error types.Source: §D.2 · p. 18

F PROMPTS

  • The prompt appendix covers prediction, action selection, TaO refinement, transition-focused abstraction, reward calculation, accessibility-tree CoT, and self-refinement.It includes prompts for next-state prediction and reward calculation in the world-model pipeline, alongside preliminary-analysis prompts.Source: F PROMPTS · pp. 18–19

D.2 ERROR TYPE ANALYSIS AND EXAMPLES

  • Illustrated errors include retaining old search text, imagining unlisted products, using vague descriptions, and predicting a state several steps ahead.The examples correspond to low competence in web elements/functions, counterfactual imagination, correct-yet-overly-generic statements, and other errors, respectively.Source: Figure 9 · p. 21, Figure 10 · p. 21, Figure 11 · p. 22

E EXAMPLES OF SUCCESSFUL INFERENCE

Loading 2410.13232v2…