PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories

Xing Han Lù, Amirhossein Kazemnejad, Nicholas Meade, Arkil Patel, Dongchan Shin, Alejandra Zambrano, Karolina Stańczak, Peter Shaw, Christopher J. Pal, Siva Reddy

arXiv:2504.08942v2cs.LGcs.AIcs.CL

TL;DR

It remains unclear how effectively LLM judges evaluate web-agent trajectories, although evaluation is needed to determine whether requested tasks were completed. AgentRewardBench uses expert-annotated trajectories to assess 12 judges, finding that no judge consistently stands out across benchmarks and simpler input representations can achieve higher expert agreement.

  • Problem

    The effectiveness of LLM judges for evaluating web-agent trajectories remains unclear, although trajectory evaluation is needed to determine whether requested tasks were completed.

    Source: §1 · pp. 1–2

  • Method

    AgentRewardBench comprises over 1,300 expert-labeled trajectories spanning task success, unintended side effects, and repetitive actions, and is used to evaluate 12 LLM judges.

    Source: §1 · p. 2, §7 · p. 9

  • Results

    No judge consistently stands out across benchmarks, and simpler input representations can achieve higher agreement with expert annotators than prior approaches.

    Source: §4.3 · p. 7, §7 · p. 10

  • Takeaways & Limitations

    The benchmark provides a basis for designing automatic evaluators and reward models that better reflect expert judgments.

    Source: §7 · p. 10

    Current LLM judges struggle to detect nuanced trajectory issues and may agree with an agent’s reasoning even when it is wrong.

    Source: §6 · p. 9

Abstract

from arXiv · show

Web agents enable users to perform tasks on web browsers through natural language interaction. Evaluating web agents trajectories is an important problem, since it helps us determine whether the agent successfully completed the tasks. Rule-based methods are widely used for this purpose, but they are challenging to extend to new tasks and may not always recognize successful trajectories. We may achieve higher accuracy through human evaluation, but the process would be substantially slower and more expensive. Automatic evaluations with LLMs may avoid the challenges of designing new rules and manually annotating trajectories, enabling faster and cost-effective evaluation. However, it is unclear how effective they are at evaluating web agents. To this end, we propose AgentRewardBench, the first benchmark to assess the effectiveness of LLM judges for evaluating web agents. AgentRewardBench contains 1302 trajectories across 5 benchmarks and 4 LLMs. Each trajectory in AgentRewardBench is reviewed by an expert, who answers questions pertaining to the success, side effects, and repetitiveness of the agent. Using our benchmark, we evaluate 12 LLM judges and find that no single LLM excels across all benchmarks. We also find that the rule-based evaluation used by common benchmarks tends to underreport the success rate of web agents, highlighting a key weakness of rule-based evaluation and the need to develop more flexible automatic evaluations. We release the benchmark at: https://agent-reward-bench.github.io

1 Introduction

AGENTREWARDBENCH evaluates whether LLM judges can assess web-agent trajectories against expert judgments, addressing the limits of task-specific rules and costly human annotation.

  • Web agents can use browsers to complete online tasks, but task-specific rules are costly to design and may miss valid successful trajectories.For example, an agent might find a different but valid listing than the one anticipated by the rules.Source: §1 · pp. 1–2
  • Human annotation is slow and expensive to scale, leaving open whether LLMs can judge trajectory success and which models best match expert evaluations.Source: §1 · p. 2
  • Accurate automatic evaluation could measure agent progress on new tasks, synthesize finetuning trajectories, and support reward models for reinforcement learning.The paper notes that it remains unclear whether current rule-based or LLM-based evaluators reflect expert judgments of success.Source: §1 · p. 2
  • Rule-based evaluations used by environment benchmarks severely underestimate agent capabilities and fail to reflect expert-defined success.The paper also analyzes weaknesses of existing LLM judges.Source: §1 · p. 2
  • AGENTREWARDBENCH contains 1300 trajectories from 4 popular LLM agents across 5 web environments, with expert labels for success, side effects, and repetitive actions.The environments range from online shopping and forum posting to specialized professional tasks.Source: §1 · p. 2
  • The benchmark evaluates existing and novel LLM judges alongside rule-based methods, then examines weaknesses of current LLMs as judges.Source: §1 · p. 2

2 Related Works

Prior work develops web agents and benchmarks, applies LLM judges to agent trajectories, and synthesizes trajectories for agent training. AGENTREWARDBENCH evaluates those judges against expert annotations across diverse web-agent scenarios.

Web Agents and Environments

LLM Judges

  • Prior LLM judges cover dialogue preferences (Zheng et al., 2023), coding agents (Zhuge et al., 2024), and web trajectories (He et al., 2024); AGENTREWARDBENCH tests judgments against expert annotations across agents, benchmarks, and dimensions.Zheng et al. report over 80% agreement with human votes; Pan et al. report that the best judge achieves 80.6% accuracy against WebArena’s rule-based evaluator.Source: §2 · p. 3

Trajectory Synthesis

  • Trajectory synthesis supports judge-training loops, contrastive preference pairs, and model fine-tuning, but these methods validate judge quality through downstream improvement rather than direct assessment.AGENTREWARDBENCH enables researchers to select LLM judges for web-task categories based on their effectiveness at evaluating web agents.Source: §2 · p. 3

3 AGENTREWARDBENCH

AgentRewardBench evaluates LLM judges by comparing their assessments of web-agent trajectories with expert annotations. It collects trajectories across five web benchmarks and labels success, side effects, and repetition.

3.1 Assessment Framework

  • Expert annotators assess browser observations, actions, and reasoning, labeling goal success, side effects, and repetition; success is primary, while auxiliary labels can inform training or inference.A judge estimates the expert annotations from the goal, trajectory, and questions; its trajectory representation can range from an action list to a processed observation history.Source: §3.1 · pp. 3–4

3.2 Tasks and Environments

  • The five benchmarks cover general-purpose, visual-reasoning, real-world, and enterprise tasks, totaling 351 tasks across 8 environments and 66 websites, split into 51 development and 300 test tasks.The suite includes text-based WebArena, visual VisualWebArena, real-world AssistantBench, and WorkArena and WorkArena++ tasks of differing difficulty.Source: §3.2 · p. 5

3.3 Web Agents Design

  • Four agents—GPT-4o, Claude 3.7 Sonnet, Llama-3.3-70B, and Qwen2.5-VL—generated 1,302 trajectories across five benchmarks, with 196 in development and 1,106 in test.Llama-3.3-70B was excluded from VisualWebArena because it is text-only. Annotators produced 3,906 binary labels, with 89.3% inter-annotator agreement on success for the double-annotated WebArena trajectories.Source: §3.3 · pp. 5–6

4 LLM judges for web tasks

The section compares three LLM judge designs and evaluates their performance for web tasks. Simplified judges can outperform prior approaches in precision, but no judge consistently leads across benchmarks, and both LLM and rule-based evaluation have important limitations.

4.1 Judge implementations

  • AER uses agent thoughts, actions, and final browser state, while NNetNav rates summarized observation changes; the simplified judge directly predicts success, side effects, and repetition.AER uses a screenshot with a vision-enabled judge or a caption with GPT-4o; NNetNav uses Llama 3.3 70B and thresholds ratings at 4. The simplified design can use accessibility trees or screenshots and does not require a separate captioning or change-summarization model. AER; NNetNav [Murty et al., 2025].Source: §4.1 · pp. 6–7

4.2 Evaluation

  • Precision is the primary metric because true positives matter for rejection finetuning and reward modeling, while recall and F1 are auxiliary measures that capture false negatives.Reducing false negatives can preserve valid trajectories and improve sample efficiency.Source: §4.2 · p. 7

4.3 Judge Performance

  • GPT-4o- and Claude 3.7 Sonnet-based simplified judges achieve higher precision than prior approaches, but no judge consistently leads across benchmarks.The authors interpret the higher precision as evidence that removing separate captioning or change-summarization models does not hinder judge capability.Source: §4.3 · p. 7
  • Across evaluated judges, none exceeds 70% precision, meaning 30% of trajectories are erroneously marked successful; this limits usefulness for downstream finetuning and methods such as RFT and RL.Source: §4.3 · p. 7
  • Rule-based evaluation has 55.9% recall, rejecting some successful trajectories over inconsequential differences and thereby underestimating agent performance.For example, WebArena can reject an answer that identifies Acadia National Park if it does not exactly match the configured string.Source: §4.3 · p. 7
  • With the simplified GPT-4o-mini judge, screenshots yield higher precision for success and repetition, accessibility trees yield higher recall, and using both performs worse than screenshots alone.The authors suggest that additional information can distract rather than assist the judge.Source: Table 2 · p. 7, §4.3 · p. 8

5 Revisiting how we evaluate task success rate

The section finds that rule-based success rates diverge from expert judgments, while LLM judges tend to overestimate success. These discrepancies motivate more expert-aligned methods for estimating task success.

Revisiting how we evaluate task success rate

  • LLM judges can estimate benchmark success where dedicated rule-based success functions are unavailable; the comparison uses expert annotations, rule-based evaluation, and GPT-4o with accessibility trees.Rule-based evaluation is the standard for many web-agent benchmarks but may not agree with experts.Source: §5 · p. 8

Rule-based evaluation does not reflect expert-defined success rates

  • Rule-based evaluation underestimated GPT-4o’s expert-annotated success rate by 16.7% on WebArena and 18.5% on VWA, while LLM judges generally overestimated success.The overestimation had two exceptions in WorkArena++; rule-based methods consistently underestimated expert-defined success rates.Source: §5 · p. 8
  • Rule-based evaluation ranked Qwen2.5-VL above GPT-4o on WebArena and WorkArena++, while experts preferred GPT-4o on every benchmark, with over 14% higher success on VWA.On VWA, rule-based evaluation ranked them equally.Source: §5 · p. 8

6 Error Analysis

LLM judges can incorrectly mark web-agent trajectories successful when they trust the agent’s reasoning over webpage evidence, overlook incomplete instructions, or fail to recognize misused actions. These examples show their difficulty detecting nuanced errors in trajectories.

Grounding mismatch

  • A judge without screenshots may accept an agent’s mistaken screen interpretation and wrongly claim it reached the requested product page.The Qwen2.5-VL agent selected the second item in the first row while claiming it selected the second-row, second-column item.Source: §6 · p. 8

Misleading agent reasoning

  • Misleading agent reasoning can convince a judge that a task succeeded even when the required action was never completed.In a WorkArena++ example, the agent claimed it applied a unique-ID search filter after unsuccessful attempts, and the judge echoed that claim.Source: §6 · pp. 8–9

Missed instruction details

  • A judge may mistake finding the requested product for buying it, overlooking the purchase instruction and falsely labeling the trajectory successful.Figure 3 illustrates the cereal task: the agent added the product to the cart but stopped before completing the purchase.Source: §6 · p. 9, Figure 3 · p. 9

Misunderstanding action intents

  • A judge may overlook a misused action near task completion, such as an agent reporting a travel-time task as unfeasible instead of giving its answer.The GPT-4o judge recognized that the travel time was visible on screen but did not flag the agent’s incorrect final response.Source: §6 · p. 9

Overall finding

  • LLM judges struggle to detect nuanced trajectory errors and may agree with an agent’s reasoning even when it is wrong.The authors identify missed details and misunderstood actions as examples of this limitation.Source: §6 · p. 9

7 Conclusion

AGENTREWARDBENCH evaluates LLM judges against expert annotations of web-agent trajectories, finding that simpler inputs can improve agreement and rule-based evaluation can reject valid trajectories.

7 Conclusion

  • The benchmark contains over 1,300 expert-annotated trajectories, assessing agent success, unintended side effects, and unnecessary repeated actions.Source: §7 · p. 9
  • The authors evaluate 12 LLM judges and present the benchmark as a resource for developing judges, automatic evaluators, and reward models that better reflect expert judgments.Source: §7 · pp. 9–10
  • Simpler input representations can achieve higher agreement with expert annotators than prior approaches, while rule-based evaluation tends to reject valid trajectories and understate some agents’ success.The findings motivate more flexible automatic evaluators that better reflect expert judgments.Source: §7 · pp. 9–10

A Benchmark · A.1 Environment and Experiments Details

The experiments sample tasks across WebArena, VisualWebArena, and WorkArena, and use a unified browser-agent platform with specified environment and token-limit settings.

A.1 Environment and Experiments Details

  • AssistantBench agents visited 66 unique domains, each 1–129 times, and used DuckDuckGo because the original homepage blocked browser automation.Source: §A.1 · p. 14
  • WebArena tasks were sampled by domain–evaluation-method groups, with up to 8 tasks per group and up to 9 for VisualWebArena.Groups combine web domain with the original evaluation method: string matching, HTML-based programs, webpage image querying, or final URL matching; some groups had very few tasks.Source: §A.1 · p. 14
  • WorkArena task categories were distributed as evenly as possible: sophisticated memory (15), information retrieval (20), contextual understanding infeasible tasks (21), planning and problem solving (22), and data driven decision making and reasoning (22).Source: §A.1 · p. 14
  • WebArena distributions were Wikipedia (8), Map (18), Reddit (18), Shopping Admin (18), Shopping (19), and Gitlab (19).Source: §A.1 · p. 14
  • VisualWebArena distributions were Wikipedia (17), Reddit (27), Classifieds (28), and Shopping (28).Source: §A.1 · p. 14
  • BrowserGym parses LLM outputs into browser actions, while AgentLab manages experiments and a unified representation to reduce unintended prompt and input differences across LLMs (Drouin et al., 2024; Chezelles et al., 2025).AgentLab binary flags are listed in Table 5; the setup limits inputs to 40K tokens and outputs to 8192 tokens.Source: §A.1 · p. 14

A.2 Annotations

The annotations were built from 1,302 trajectories and gathered through a custom interface using questions aligned with those posed to LLM judges. Expert annotators could flag uncertainty and deliberate toward consensus, while sharing documented knowledge about agents, web environments, and automatic evaluation.

Trajectory filtering

  • 1,302 trajectories were collected across five benchmarks, with 196 assigned to development and 1,106 to test.The collection covered 351 tasks; Llama 3.3 trajectories excluded VisualWebArena because it is not multimodal, and two WebArena tasks timed out despite restart attempts.Source: §A.2 · p. 14

Interface

  • A custom Gradio interface showed each task goal, trajectory steps, model reasoning and actions, and screenshots with the action element overlaid.Annotators answered questions about success, side effects, and repetitiveness that matched those posed to LLM judges.Source: §A.2 · p. 14

Shared Knowledge

  • Annotators shared knowledge about agent capabilities and limitations, benchmark websites, and the automatic evaluation methods used by web environments.They knew agents could hallucinate or repeat text; websites could be task-specific or real-world, including those associated with WebArena (Zhou et al., 2024), VisualWebArena (Koh et al., 2024), and WorkArena (Drouin et al., 2024; Boisvert et al., 2025); and string or URL matching may not award positive reward even after task termination.Source: §A.2 · pp. 14–15

Annotator agreements and disagreements resolution

  • Uncertain annotations were reviewed, and annotators resolved disagreements through deliberation, generally choosing the most lenient interpretation of ambiguous instructions.The first half of annotation was conducted in person with annotators present concurrently, helping them reach consensus and align subsequent judgments.Source: §A.2 · p. 15

A.3 LLM Judges

This section reports extended results for 10 LLM judges across benchmarks and describes model-agnostic prompt templates designed to make outputs parseable.

Prompts

  • The prompts use model-agnostic system and user templates with distinct tags such as <success> and <reasoning> to facilitate output parsing.The templates avoid model-specific commands so they can transfer across LLMs.Source: §A.3 · p. 15

Results

  • The extended results report overall and fine-grained performance for 10 LLM judges, with precision primary and F1 and recall auxiliary metrics.Overall results are in Table 7, and benchmark-level results across all agents are in Table 8.Source: §A.3 · p. 15, Table 7 · p. 17, Table 8 · p. 18

Related papers

This paper cites

Loading 2504.08942v2…