PaperlayerView exact revision on arXiv

Source-linked AI summary

Discriminative World Models for Web Agents

Kelvin Li, Dhruv Pendharkar, Anish Pahilajani, Chuyi Shang, Leon Oks, Leonid Karlinsky, Rogerio Feris, Trevor Darrell, Roei Herzig

arXiv:2609.02885v1cs.AIcs.LG

TL;DR

Existing web world models generate fixed-format next states, although downstream ranking requires representations that distinguish competing action outcomes. The paper introduces predicted-state matching with branching decision-point data, and reports improvements in matching accuracy, PRM-style action ranking, and WebArena-Lite task success. These findings support evaluating world models by whether they preserve action-relevant differences for decision making.

  • Problem

    Fixed-format supervised next-state prediction is misaligned with action selection because rankers need predicted states that distinguish competing outcomes.

  • Method

    Predicted-state matching trains representations to distinguish the true resulting state from alternatives, using branching web-agent data with multiple actions and resulting states.

  • Results

    The approach improves held-out matching accuracy, WebPRMBench action ranking, and WebArena-Lite end-to-end task success over supervised-next-state and action-ranking baselines.

  • Takeaways & Limitations

    Web world models should preserve action-relevant differences for downstream decision making rather than only reproduce predefined next-state representations.

  • Takeaways & Limitations

    Generality beyond WebArena environments and GPT-4o remains to be established, and matching accuracy is a model-based proxy subject to judge-model limitations or biases.

Abstract

from arXiv · show

Recent web agents use world models for test-time action selection by sampling candidate actions, predicting the resulting web states, and ranking them with a ranker model or a Process Reward Model (PRM). These world models are typically trained via supervised next-state prediction to generate fixed representations like HTML or AXTree snapshots. However, this objective is misaligned with the downstream ranker, which relies on predicted states being discriminative across candidates to accurately score them. To address this, we introduce predicted-state matching, a training objective where the predicted representation must distinguish the true resulting state from those reached by alternative actions. We train these models using a branching web-agent dataset derived from WebArena Go-Browse trajectories, where every decision point contains multiple alternative actions and their resulting states. Experiments on our held-out predicted-state matching benchmark show that our approach outperforms world models trained with supervised next-state prediction. We further show that our approach improves PRM-style action ranking on WebPRMBench compared with action-only PRMs and PRMs augmented with supervised-next-state world models. Finally, on WebArena-Lite, using our world model for test-time action selection improves end-to-end task success. Our project page is available at: https://dhruvpendharkar.github.io/dwm/.

1 Introduction

Web-agent world models should represent action-relevant differences between competing outcomes, not merely reproduce fixed-format next-state targets. Predicted-state matching and branching data support this objective and improve evaluation and downstream action selection.

  • Web navigation is a partially observed, multi-step problem requiring agents to predict how actions affect future steps.
  • Fixed-format next-state prediction can miss action-relevant differences between competing outcomes.Textual summaries may omit specific changes, while structured representations can bury relevant changes among unchanged page structure.
  • Predicted-state matching trains representations to distinguish the true resulting state from alternatives reached by other actions.A fixed judge evaluates whether the generated representation identifies the queried action’s resulting state among alternatives.
  • The branching dataset contains local decision points with multiple executable actions and their resulting states.This structure supports both training and held-out evaluation of whether predicted representations distinguish alternative outcomes.
  • The approach improves held-out predicted-state matching, PRM-style action ranking, and end-to-end task success.The reported evaluations cover predicted-state matching, WebPRMBench action ranking, and WebArena-Lite test-time action selection.

2 Related Work

Prior web-agent work uses reactive action prediction, PRMs, and world models, but commonly lacks explicit comparison of alternative action outcomes. This work introduces branching data and evaluates predicted states by their ability to distinguish those outcomes and support ranking.

  • Reactive web agents repeatedly select one action from the task, history, and current observation without explicitly comparing alternatives.
  • Test-time action selection lets agents propose multiple actions and use a ranker or PRM to select among them.
  • World-model planning predicts next states for proposed actions so rankers can compare outcomes before execution.
  • Existing world models usually generate fixed state formats through supervised next-state prediction, which does not train or evaluate discriminativeness among competing outcomes.
  • Linear-trajectory datasets provide limited supervision because they record only the action taken along a single path.
  • The branching dataset records multiple actions and resulting states at each local decision point for predicted-state matching and held-out evaluation.

3 Method

The method trains web world models to generate next-state representations that distinguish competing action outcomes rather than reproduce fixed-format targets. It constructs branching decision points from repeated states in Go-Browse trajectories and trains a matching-based representation, then supplies these predictions to downstream rankers.

  • Problem setup and branching data: Each decision point contains multiple candidate actions from one current state, with each action mapped to its observed resulting next state.This branching structure exposes how different actions from the same state lead to different next states.
  • Problem setup and branching data: Repeated browser states from linear Go-Browse trajectories are merged into a state-action graph whose outgoing edges define branching decision points.Each graph node is a browser state represented by its accessibility tree, and each directed edge records an executed action with its observed next state.
  • Predicted-state matching: The world model generates a textual predicted next-state representation for a queried action, without supervision to match a predefined state string.The textual format supports interpretability and compatibility with language-model-based ranking, while the representation is optimized for matching.
  • Predicted-state matching: A matching judge selects which of two candidate next states corresponds to the generated representation, using only the representation and candidate states.The judge excludes the instruction, history, current state, and queried action, so the representation must itself identify the correct outcome.
  • Predicted-state matching: Predicted-state matching trains representations to distinguish the true next state from alternatives reached by other actions at the same decision point.The objective is representation-agnostic: ground-truth next states serve as matching candidates rather than fixed strings to reproduce.
  • Using predicted next-state representations for action ranking: Predicted next-state representations are optionally added to downstream PRMs or rankers to compare candidate actions, alongside action-only and supervised-prediction baselines.The ranking interface keeps the preference-training setup fixed while varying the source of next-state information.

4 Evaluation

The evaluation tests predicted-state matching for world-model quality, downstream action ranking, and end-to-end web-agent success. Across these settings, predicted-state representations outperform supervised-next-state alternatives and improve candidate-action selection.

  • 4.1 Predicted-state matching: The held-out benchmark evaluates whether a predicted representation identifies the queried action’s true next state among alternatives from the same decision point.The primary setup uses a fixed Qwen3-32B judge and covers five WebArena domains.
  • 4.1 Predicted-state matching: Our model achieves the best overall predicted-state matching accuracy, outperforming existing world models and a data-matched supervised AXTree-prediction baseline.The controlled baseline uses the same branching data and Qwen3-8B initialization but a fixed-format generation objective.
  • 4.1 Predicted-state matching: Predicted-state matching representations remain superior across Qwen3-32B, GPT-4o, and Llama-3.1-70B matching judges.This tests whether the representations are discriminative beyond the judge model used during training.
  • 4.2 WebPRMBench with trained reward models: In trained-ranker comparisons, state-matching representations outperform WebWorld-8B states under the same reward-model backbone and answer-only SFT setup.The controlled comparison changes only the predicted next-state information supplied to the ranker.
  • 4.3 WebPRMBench with frozen rankers: With frozen Qwen2.5 rankers, predicted states improve over no-state ranking and outperform WebWorld-8B states across both Qwen2.5-3B and Qwen2.5-7B.The result shows downstream usefulness without additional reward-model training.
  • 4.4 End-to-end agent evaluation: 28.48% task success is achieved by Bo5 plus state matching, compared with 21.82% for Bo5 and 13.94% for ReAct-style selection.The WebArena-Lite result indicates that predicted next-state information helps GPT-4o rank sampled candidate actions.

5 Conclusion

The paper concludes that predicted-state matching and branching data train web world models to preserve action-relevant differences. These representations improve matching accuracy, action ranking, and end-to-end task success relative to fixed-format next-state prediction.

  • 5 Conclusion: Predicted-state matching trains next-state representations to distinguish true outcomes from alternatives instead of reproducing a fixed state format.The objective is paired with a branching dataset containing competing actions and resulting states.
  • 5 Conclusion: The model improves held-out matching accuracy, WebPRMBench ranking with trained and frozen rankers, and WebArena-Lite success with Best-of-5 selection.Together, these evaluations connect discriminative representations to both intermediate ranking and end-to-end performance.

6 Limitations

The evaluation is bounded by benchmark and model choices, judge-based measurement, and incomplete action branching. Broader environments, models, and denser branches remain open directions.

  • Benchmark and model scope: Findings may not generalize beyond WebArena-derived benchmarks or GPT-4o policy evaluation.The matching benchmark uses Go-Browse trajectories in WebArena, while end-to-end evaluation uses WebArena-Lite with GPT-4o.
  • Judge dependence: Matching accuracy is a model-based proxy that may inherit limitations or biases shared by language-model judges.The study evaluates with Qwen3-32B, GPT-4o, and Llama-3.1-70B-Instruct and observes consistent improvements across judges.
  • Branch coverage: The branching data does not enumerate every possible action at each decision point.It contains executable alternative-action branches observed in Go-Browse trajectories, leaving denser exploration and broader environments for future work.

7 Ethical Considerations

The work relies on previously released benchmark artifacts and trajectories for research evaluation and model development. Experiments remain limited to benchmark environments, with no new human-subject data or live personal-account actions.

  • Research artifacts: The project uses existing artifacts such as Go-Browse, WebArena, WebPRMBench, WebWorld, WebDreamer, and Qwen model families.Derived data, code, and checkpoints should follow the original artifacts’ licenses and terms.
  • Data provenance: No new human-subject data are collected and no human annotators are recruited.The data come from benchmark environments and previously released trajectories.
  • Deployment boundary: Experiments do not execute actions on real user accounts or live personal data.The work is restricted to benchmark environments intended for research on web-agent world modeling and action ranking.
  • Potential risks: Unrestricted deployment could increase autonomous web-agent capabilities and enable unintended or harmful actions on real websites or sensitive information.The paper does not evaluate or advocate unrestricted autonomous deployment.

B Output Length Analysis

The proposed model produces shorter predicted-state representations than WebWorld-8B while achieving stronger predicted-state matching and downstream action-ranking performance.

  • Representation length: 91.6 tokens per state versus 412.7 tokens for WebWorld-8B accompanies stronger benchmark performance.The comparison excludes WebWorld-8B’s additional reasoning tokens.
  • Matching performance: 80.80% predicted-state matching accuracy versus 70.17% for WebWorld-8B.This result is reported on the predicted-state matching benchmark.
  • Action ranking: 72.70% average Best-of-N accuracy versus 67.63% for WebWorld-8B on WebPRMBench.The predicted states augment the controlled Qwen2.5-7B reward model.
  • Interpretation: The results suggest gains arise from more effective state representations rather than greater output length.The model is shorter than WebWorld-8B while outperforming it on both evaluated tasks.

C Training Details

This section supplies additional implementation and training details for the predicted-state-matching world model.

  • Training details: The section provides additional implementation and training details for the predicted-state-matching world model.

C.1 Implementation Details

The world model is optimized with Group Relative Policy Optimization (GRPO), using sampled completions, group-relative advantages, and KL regularization.

  • Optimization: GRPO samples 8 completions per input and applies each completion’s sequence-level advantage across its generated tokens.The policy objective uses group-relative advantages derived from total sequence-level rewards.
  • Optimization: Training uses KL regularization with coefficient 0.04.
  • Decoding: Completions are sampled at temperature 1.0 during training, while inference uses greedy decoding.

C.2 Training and Data Efficiency

The model is trained with relatively limited compute and data compared with prior web world models, yet predicted-state matching remains stronger on held-out matching accuracy. Accuracy is also relatively robust to the format-reward weight.

  • Training Cost: 48.75 hours on eight A100 GPUs totals 390 GPU-hours for 4,830 optimization steps with effective batch size 32.Six GPUs are used for optimization and two for generation during GRPO training.
  • Data Efficiency: The training set contains 30,920 pairwise examples from 7,730 branching decision points.This is smaller than the datasets reported for WebDreamer-7B and WebWorld-8B.
  • Data Efficiency: Despite smaller scale, predicted-state matching achieves stronger held-out matching accuracy than the compared world-model approaches.The comparison is made against existing web world models and supervised-next-state prediction baselines.
  • Format Reward Ablation: Predicted-state matching accuracy is relatively robust to the choice of λfmt.Table 5 reports the effect of the format-reward weight on matching accuracy.
  • Prediction Interface: The prediction prompt supplies the current accessibility-tree state, optional action history, and performed action, requiring a specific next-state prediction.
  • Matching Evaluation: The evaluation prompt asks a judge to choose which of two candidate actual next states better matches the predicted state.
Loading 2609.02885v1…