PaperlayerView exact revision on arXiv

Source-linked AI summary

WebArbiter: A Principle-Guided Reasoning Process Reward Model for Web Agents

Yao Zhang, Shijie Tang, Zeyu Li, Zhen Han, Volker Tresp

arXiv:2601.21872v2cs.AI

TL;DR

Web agents need reliable step-level supervision because long-horizon interactions involve sequential decisions and potentially irreversible actions, while existing WebPRMs provide coarse or brittle judgments. WebArbiter models rewards as structured, principle-guided text generation trained through reasoning distillation and reinforcement learning. It achieves SOTA benchmark performance and improves reward-guided trajectory search, while remaining limited by text-only, English-only evaluation and inference-time trajectory-search scope.

  • Problem

    Existing WebPRMs provide coarse scalar signals or brittle checklist-based judgments for long-horizon web interactions requiring reliable step-level decisions.

  • Method

    WebArbiter generates principle-guided reasoning and preference verdicts, using reasoning distillation followed by reinforcement learning to align judgments with correctness.

  • Results

    WebArbiter achieves SOTA performance on WEBPRMBENCH and improves reward-guided trajectory search on WebArena-Lite by up to 6.4 points over the best prior WebPRM.

  • Takeaways & Limitations

    Principle-guided reasoning WebPRMs provide a robust and interpretable foundation for scalable web agents within the evaluated settings.

  • Takeaways & Limitations

    WebArbiter uses text-only accessibility trees and is evaluated exclusively in English-language web environments, limiting coverage of visual and multilingual settings.

Abstract

from arXiv · show

Web agents hold great potential for automating complex computer tasks, yet their interactions involve long-horizon, sequential decision-making with irreversible actions. In such settings, outcome-based supervision is sparse and delayed, often rewarding incorrect trajectories and failing to support inference-time scaling. This motivates the use of Process Reward Models (WebPRMs) for web navigation, but existing approaches remain limited: scalar WebPRMs collapse progress into coarse, weakly grounded signals, while checklist-based WebPRMs rely on brittle template matching that fails under layout or semantic changes and often mislabels superficially correct actions as successful, providing little insight or interpretability. To address these challenges, we introduce WebArbiter, a reasoning-first, principle-inducing WebPRM that formulates reward modeling as text generation, producing structured justifications that conclude with a preference verdict and identify the action most conducive to task completion under the current context. Training follows a two-stage pipeline: reasoning distillation equips the model with coherent principle-guided reasoning, and reinforcement learning corrects teacher biases by directly aligning verdicts with correctness, enabling stronger generalization. To support systematic evaluation, we release WebPRMBench, a comprehensive benchmark spanning four diverse web environments with rich tasks and high-quality preference annotations. On WebPRMBench, WebArbiter-7B outperforms the strongest baseline, GPT-5, by 9.1 points. In reward-guided trajectory search on WebArena-Lite, it surpasses the best prior WebPRM by up to 6.4 points, underscoring its robustness and practical value in complex web tasks.

1 INTRODUCTION

WebArbiter addresses unreliable step-level supervision for long-horizon web tasks with principle-guided reasoning and structured preference judgments. The paper also introduces WEBPRMBENCH and reports strong benchmark and trajectory-search performance.

  • Motivation: Web agents require reliable decisions throughout long, multi-step interactions because actions such as incorrect form submissions may be irreversible.Outcome-only supervision is poorly suited to these settings.
  • Motivation: Existing WebPRMs provide either coarse, weakly grounded scalar scores or checklist-based judgments that are brittle under dynamic layouts and state-dependent action semantics.These limitations motivate dedicated, more interpretable process supervision.
  • WebArbiter: WebArbiter generates structured justifications and preference verdicts by dynamically deriving principles from user intent and the current web state.Its two-stage training uses reasoning distillation followed by reinforcement learning to align verdicts with correctness.
  • Evaluation: WEBPRMBENCH provides 1,150 step-level preference instances across 4 web environments, with Pairwise and Best-of-N Accuracy as primary metrics.The benchmark covers everyday and enterprise web tasks with one correct action and four rejected alternatives per instance.
  • Results: 9.1 points separate WebArbiter from GPT-5 on Avg. BoN Acc, while WebArbiter also surpasses WebShepherd by up to 6.4 points in WebArena-Lite trajectory search.The benchmark results also report consistent superiority over WebShepherd across environments.

2 RELATED WORK

Prior web-agent research spans demonstration-based state–action distillation, open-ended reflexive exploration and search, and reinforcement learning. Reward models likewise divide into scalar and generative families with different scalability and reliability trade-offs.

  • Web-Agent Research: One web-agent direction distills environment-specific state–action pairs from demonstrations, performing strongly on seen states but remaining brittle on novel ones.SteP is identified as a leading example on WebArena.
  • Web-Agent Research: A second direction uses reflexive evaluation and search to support open-ended exploration.
  • Web-Agent Research: A third direction applies reinforcement learning to web-agent development.
  • Reward Models: Scalar reward models assign one numeric score, whereas generative reward models produce natural-language feedback from which rewards are extracted.Generative models support single-instance evaluation and multi-response comparison but raise reliability concerns from bias and hallucination.

3 METHODOLOGY

WebArbiter frames web process reward modeling as structured text generation over candidate actions and trains it through principle-guided distillation followed by correctness-aligned reinforcement learning. Its design combines explicit action reasoning, preference verdicts, and auditable step-level rewards for long-horizon web navigation.

  • Problem formulation: Web navigation is formalized as a POMDP in which partial observations and action histories support generating trajectories toward task completion.The environment is represented with state, action, and observation spaces, and task completion is determined from the final state.
  • Data construction: Training data converts expert positive actions and rejected actions, each paired with reasoning traces, into pairwise preference samples.These samples form the training preference dataset used by WebArbiter.
  • Model formulation: WebArbiter evaluates candidate actions with associated reasoning traces and autoregressively generates a structured justification ending in a preference verdict.The framework uses the instruction, current observation, and prior action-reasoning history to select the most appropriate candidate.
  • Principle-guided reasoning: The model dynamically derives principles from user intent and the current state, then checks whether each candidate action truly advances task completion.This reasoning-first design replaces opaque scalar scores and fixed checklist templates with auditable analyses intended to improve credit assignment under environment changes.
  • Training pipeline: Training uses two stages: reasoning distillation teaches coherent principle-guided justifications, while reinforcement learning aligns verdicts with correctness signals.Distillation uses teacher-generated justifications; RL treats the model as a judgment policy and assigns a binary reward based on whether its verdict matches the ground-truth preference.
  • Training pipeline: The reinforcement-learning objective maximizes expected correctness reward while using KL regularization to stabilize the distilled reasoning style.GRPO is used to optimize the objective under binary verifiable rewards, producing interpretable step-level reward signals.

4 WEBPRMBENCH

WEBPRMBENCH evaluates WebPRMs across four web environments using preference-based metrics that test both pairwise correctness and robust ranking among distractors.

  • Benchmark construction: The benchmark expands prior coverage by enriching WebArena and adding AssistantBench and WorkArena alongside Mind2Web.This broadens coverage of real-world tasks across heterogeneous environments.
  • Evaluation metrics: Pairwise Accuracy measures correctness on individual preference pairs, whereas Best-of-N Accuracy tests robustness when ranking multiple candidates.The two metrics capture local preference fidelity and multi-candidate decision reliability.
  • Benchmark results: WebArbiter-7B achieves the highest Avg BoN Acc on WEBPRMBENCH, outperforming GPT-5 by 9.1 points.Table 2 compares Pairwise and BoN Accuracy across web environments.
  • Evaluation metrics: BoN Accuracy is stricter than Pairwise Accuracy because the correct action must outrank every distractor simultaneously.The benchmark uses four distractors per instance.

5 EXPERIMENTS

Experiments compare WebArbiter with proprietary and open-source judges, evaluate its benchmark performance and downstream trajectory search, and analyze its training components. WebArbiter achieves the strongest reported results, while ablations show that principle-guided reasoning and reasoning distillation support stability and generalization.

  • Experimental setup: WebArbiter is evaluated on WEBPRMBENCH and practical applications using Pairwise Accuracy and Best-of-N Accuracy.The experiments also include reward-guided trajectory search.
  • Experimental setup: The comparison includes proprietary LLM-as-judge models, open-source LLM-as-judge models, and prior WebPRMs.GPT-5 is among the proprietary baselines, while WebShepherd is the previous SOTA WebPRM.
  • Main results: WebArbiter achieves the highest Avg. Pairwise Acc and Avg. BoN Acc, while LLM-as-judge performance drops sharply on BoN Acc.WebArbiter sustains strong results under realistic multi-candidate settings.
  • Main results: WebArbiter-7B reaches 74.60% Avg. BoN Acc and surpasses WebShepherd-8B by an absolute gain of 31%.Both models are trained on the same WEBPRM Collection drawn from Mind2Web.
  • Main results: WebArbiter attains SOTA BoN Acc on Mind2Web and WorkArena while remaining competitive with proprietary LLMs on WebArena and AssistantBench.The reported pattern spans heterogeneous, noisy, and enterprise-scale environments.
  • Training design: Cold Start RL performs well on in-domain Mind2Web but collapses on out-of-domain benchmarks, whereas adding principles improves cross-environment performance.Principle augmentation is especially beneficial on AssistantBench and WorkArena.
  • Training design: Reasoning distillation improves judgment stability, particularly in multi-candidate settings, and remains advantageous under the limited-data 10K regime.Under identical data budgets, distilled reasoning outperforms answer-only SFT across environments.
  • Downstream analysis: WebArbiter achieves the highest gains in WebArena-Lite trajectory search across both GPT-4o-mini and GPT-4o policy models.The analysis attributes gains to reasoning over spurious correlations, equivalent paths, and semantic query precision across domains.

6 CONCLUSION

WebArbiter is presented as a reasoning-first, principle-inducing process reward model that produces auditable step-level judgments, while WebPRMBench enables systematic evaluation across diverse web environments. Experiments show state-of-the-art benchmark performance and improved reward-guided trajectory search on WebArena-Lite.

  • WebArbiter frames reward modeling as structured text generation and produces auditable step-level judgments with rationales.
  • Reasoning distillation and reinforcement learning yield progress-aware signals that verify task advancement and generalize across dynamic web environments.
  • WEBPRMBENCH provides systematic WebPRM evaluation across four diverse environments with step-level preference annotations.
  • Experiments demonstrate state-of-the-art performance on WEBPRMBENCH and improved reward-guided trajectory search on WebArena-Lite.

LIMITATIONS AND FUTURE WORK

The paper identifies three scope boundaries: WebArbiter uses text-only accessibility trees, is trained and evaluated only in English web environments, and currently supports inference-time trajectory search rather than policy post-training.

  • Text-only accessibility trees may miss layout, spatial, or visual cues needed for accurate progress assessment.The authors identify multimodal observations as a natural extension and provide failure cases in Appendix J.
  • Training and evaluation are limited to English-language web environments, where multilingual settings may introduce divergent page structures and interaction conventions.
  • Current WebPRM use focuses on inference-time trajectory search rather than dense reward supervision during agent post-training.Whether step-level supervision can improve the policy itself remains an open direction.

B EXAMPLE OF PREFERENCE DATASET

The preference dataset represents a web task state with its instruction, current observation, action history, reasoning traces, and candidate actions, then labels the preferred action. The ICLR example illustrates this construction using two competing links.

  • Each preference instance combines a task instruction, current observation, prior actions, and prior reasoning traces.The example asks the agent to find the 2026 ICLR conference submission page.
  • Candidate actions are paired with reasoning traces that explain how each link might advance the task.The candidates are the “Call for Papers” and “About” links on the ICLR homepage.
  • The dataset labels the “Call for Papers” action as preferred over the “About” action.
  • The example is serialized as a dataset representation after presenting the instruction, observation, history, candidates, and label.

C TRAINING DETAILS

The paper describes implementation, prompting, and data-construction details for training and evaluating WebArbiter, including staged distillation and RL, structured judging prompts, and preference-pair assembly.

  • Training setup: Training uses Qwen2.5 and Qwen3 backbones with distillation and RL stages implemented using LLama-Factory and VERL.Experiments use eight NVIDIA A100-80GB GPUs with fixed random seeds.
  • Training setup: The distillation stage uses LoRA, cosine learning-rate scheduling, 0.1 warmup, 8,192-token sequences, and five training epochs.
  • Training setup: The RLVR stage uses GRPO with fixed batch and mini-batch sizes, FSDP, and vLLM tensor-parallel rollouts.
  • Prompt repository: The judging prompt asks the model to summarize state, identify criteria, compare two responses, analyze them, and output a preference verdict.The prompt conditions evaluation on the URL, accessibility tree, trajectory, and current web page context.
  • Preference data: WEBPRMBENCH preference pairs use successful human-verified trajectories as positives and alternative actions from diverse open-source and proprietary policy models as negatives.Rationales are truncated to a fixed length, and the benchmark records website visits across a diverse long-tailed environment distribution.

E.2 DATASET COMPOSITION AND STATISTICS

WebPRMBench contains diverse, environment-verified preference data spanning four web environments and common interaction types. Its evaluation design emphasizes whether actions advance task progress, while BoN provides more discriminative model separation than Pairwise Acc.

  • Dataset composition: WebPRMBench contains 1,150 step-level preference instances across four web environments, each with one positive action and four negative alternatives.The benchmark covers everyday and enterprise web tasks.
  • Dataset composition: The benchmark covers a diverse, long-tailed distribution of visited websites.This distribution is intended to reflect real-world web environments.
  • Action statistics: Click and fill actions dominate both positive and negative examples, whose similar distributions prevent action type from reliably signaling correctness.Correctness therefore depends on state- and task-conditioned progress rather than global action-frequency heuristics.
  • Evaluation statistics: BoN Acc and Pairwise Acc are strongly positively correlated across all environments, indicating broadly aligned model quality rankings despite differing correlation strengths.Their relationship varies with interaction structure and distractor difficulty.

F ANALYSIS OF BoN Acc VS. Pairwise Acc EVALUATION

The analysis compares Pairwise Acc with BoN Acc for WebPRM evaluation. Although the metrics are strongly aligned, BoN is stricter and better separates models in multi-candidate decision settings.

  • Metric design: BoN Acc requires the correct action to outperform all distractors simultaneously, making it more representative of realistic multi-candidate ranking than isolated Pairwise Acc.WebPRMs are commonly used to rank multiple candidate actions during agent execution.
  • Discriminative power: BoN Acc consistently has higher score variance than Pairwise Acc across all four environments, indicating less compression and larger model separation.This makes BoN particularly useful when robust multi-candidate judgment is required.
  • Metric relationship: BoN Acc and Pairwise Acc are strongly positively correlated across all environments, so they capture broadly aligned notions of WebPRM quality.Correlation strength varies with interaction structure and distractor difficulty.

G GENERALIZATION ACROSS BACKBONE FAMILIES

WebArbiter generalizes across backbone families and supports inference-time scaling, while case studies show reasoning-based preferences handling path ambiguity and task preconditions more reliably than checklist supervision. The analyses also expose limitations of text-only observations in judging interaction effects and element references.

  • Backbone generalization: WebArbiter’s two-stage training pipeline is evaluated across Qwen2.5 and Qwen3 backbones using the same data, distillation strategy, and RL procedure.Only backbone-specific hyperparameters differ.
  • Backbone generalization: 72.55% Avg. BoN Acc is achieved by WebArbiter-4BQwen3 versus 59.06% for WebArbiter-3BQwen2.5, approaching the 74.60% of WebArbiter-7BQwen2.5 with roughly half the parameters.The result suggests stronger base models amplify the benefits of principle-guided reasoning distillation.
  • Backbone generalization: The two-stage training pipeline generalizes across model families and benefits from stronger base models without pipeline modifications.Additional backbone results support this conclusion.
  • Inference-time scaling: Both Pairwise and BoN Acc improve as the sampled reward-evaluation budget increases for WebArbiter-3B and WebArbiter-7B.The improvement is moderate for Pairwise Acc but more pronounced under the stricter BoN Acc.
  • Path ambiguity: Under multiple equivalent GitLab paths, checklist supervision may prefer a procedurally typical detour, whereas WebArbiter selects the action that directly advances milestone creation.The case illustrates brittleness under structural variability.
  • Task preconditions: When a merge request is ambiguous, WebArbiter prioritizes search because identifying the correct request is a prerequisite for subsequent review or response.Checklist supervision may instead reward opening an arbitrary visible request.
  • Text-only limitations: In the safe-action case, WebArbiter incorrectly prefers hover because the accessibility tree omits the interaction effect, causing it to overestimate safety.The text-only observation cannot reveal that hovering produces no state change and adds a redundant step.
  • Text-only limitations: In the element-reference case, WebArbiter favors a semantically aligned candidate without independently verifying its bid against the accessibility tree.The analysis identifies visual observations or explicit element-level verification as possible mitigations.
Loading 2601.21872v2…