Source-linked AI summary
AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories
Xing Han Lù, Amirhossein Kazemnejad, Nicholas Meade, Arkil Patel, Dongchan Shin, Alejandra Zambrano, Karolina Stańczak, Peter Shaw, Christopher J. Pal, Siva Reddy
TL;DR
Evaluating web-agent trajectories is difficult because rule-based methods are hard to extend and can reject successful behavior, while human evaluation is slower and more expensive. AgentRewardBench benchmarks LLM judges against expert annotations and finds that no single judge works best across all benchmarks, whereas rule-based evaluation consistently underestimates success.
Problem
Web-agent evaluation needs automatic methods that are more extensible and accurate than task-specific rules without incurring the cost and delay of human annotation.
Method
AgentRewardBench evaluates 12 LLM judges using over 1300 expert-annotated trajectories from 4 LLM agents across 5 diverse web environments, measuring success, side effects, and repetitive actions.
Results
No single LLM judge consistently excels across benchmarks, while rule-based evaluation tends to reject valid trajectories and underreport expert-defined success rates.
Takeaways & Limitations
The benchmark supports developing more flexible automatic evaluators and reward models that better reflect expert judgments of web-agent trajectories.
Takeaways & Limitations
Current LLM judges remain limited in detecting nuanced trajectory issues and may accept misleading agent reasoning even when it is wrong.
Abstract
from arXiv · showhide
Web agents enable users to perform tasks on web browsers through natural language interaction. Evaluating web agents trajectories is an important problem, since it helps us determine whether the agent successfully completed the tasks. Rule-based methods are widely used for this purpose, but they are challenging to extend to new tasks and may not always recognize successful trajectories. We may achieve higher accuracy through human evaluation, but the process would be substantially slower and more expensive. Automatic evaluations with LLMs may avoid the challenges of designing new rules and manually annotating trajectories, enabling faster and cost-effective evaluation. However, it is unclear how effective they are at evaluating web agents. To this end, we propose AgentRewardBench, the first benchmark to assess the effectiveness of LLM judges for evaluating web agents. AgentRewardBench contains 1302 trajectories across 5 benchmarks and 4 LLMs. Each trajectory in AgentRewardBench is reviewed by an expert, who answers questions pertaining to the success, side effects, and repetitiveness of the agent. Using our benchmark, we evaluate 12 LLM judges and find that no single LLM excels across all benchmarks. We also find that the rule-based evaluation used by common benchmarks tends to underreport the success rate of web agents, highlighting a key weakness of rule-based evaluation and the need to develop more flexible automatic evaluations. We release the benchmark at: https://agent-reward-bench.github.io
1 Introduction
Web agents can perform realistic browser tasks, but evaluating whether their trajectories succeed is difficult to scale reliably. This motivates studying whether LLM judges can provide flexible automatic evaluation.
- Web agents extend language-model interaction from chat interfaces to completing tasks through web browsers.
- Benchmarks need realistic tasks across diverse websites, yet task-specific rules are time-consuming to design and may reject valid alternative solutions.For example, a rule may require one exact listing even when another listing also satisfies the request.
- Human annotation can evaluate trajectories more flexibly, but scaling it across many web agents is slow and expensive.
- The central questions are whether an LLM can determine trajectory success and which LLM is most capable as a web-agent evaluator.
- AGENTREWARDBENCH evaluates LLM judges against expert-labeled trajectories spanning 5 web environments and 4 LLM agents.The benchmark labels success, unintended side effects, and repetitive action cycles across 1300 trajectories.
2 Related Works
Prior work established web-agent benchmarks and LLM-based judging, while trajectory synthesis uses judges to generate or curate training data. AGENTREWARDBENCH builds on these directions to assess judge capability.
- Web Agents and Environments: Web-agent research progressed from program-based heuristics to reinforcement-learning, language-model, and multimodal approaches.
- Web Agents and Environments: Web-agent benchmarks evolved from simplified environments toward realistic websites and specialized professional environments.
- LLM Judges: LLM judges were first applied to chatbot preferences and later extended to modalities, metrics, coding agents, and web-agent trajectories.
- Trajectory Synthesis: Trajectory synthesis uses LLM judges to bootstrap judge-training loops, create contrastive pairs, and produce data for fine-tuning base models.
3 AGENTREWARDBENCH
AGENTREWARDBENCH collects web-agent trajectories, obtains expert annotations, and evaluates LLM judges against those annotations. Its framework represents trajectories through browser observations, actions, and reasoning, while testing diverse tasks, agents, and judge inputs.
- Dataset and Agents: AGENTREWARDBENCH curates trajectories from 5 diverse web environments and tasks using 4 LLM-based web agents.
- Annotation Design: Experts review each trajectory’s screenshots, actions, and reasoning before labeling success and auxiliary issues.
- Assessment Framework: A trajectory is a sequence of browser observations, reasoning-action pairs, and a final observation; observations include screenshots, DOM trees, and accessibility trees.
- Annotation Design: The annotation questions cover task success, unnecessary actions that could cause side effects, and repetition cycles that make no progress.
- Annotation Design: Success agreement is the primary judge criterion, while side-effect and cycle labels can help penalize agents during training or redirect them at inference.
- Assessment Framework: A judge maps a goal, trajectory, and questions to predicted answers by applying a judge-specific trajectory representation before LLM evaluation.
- Dataset and Agents: The evaluated agent backbones are GPT-4o, Claude 3.7 Sonnet, Llama-3.3-70B, and Qwen2.5-VL, with Llama excluded from visual tasks.
- Dataset and Agents: The benchmark contains 1302 trajectories from 4 agents across 5 benchmarks, with development and test splits based on the task partition.
4 LLM judges for web tasks
The paper compares existing and simplified LLM judges for web-agent trajectories, including their input representations and evaluation metrics. Across benchmarks, no judge consistently leads, precision remains limited, and screenshots can outperform combined representations.
- Judge implementations: The study compares AER, NNetNav, and a simplified judge that predicts success, side effects, and repetition in one completion.AER uses agent thoughts, actions, and the final browser state; NNetNav summarizes changes across observations before rating trajectories.
- Judge implementations: The simplified judge accepts either accessibility trees or screenshots and does not require separate captioning or change-summarization models.This design supports both multimodal and text-only LLMs.
- Evaluation: Precision is the ratio of true positives to all predicted positives, while recall and F1 are reported as auxiliary metrics.Precision is emphasized because false positives can add failed trajectories to finetuning or reward-modeling data.
- Results: No judge consistently stands out across benchmarks, although GPT-4o- and Claude 3.7 Sonnet-based simplified judges achieve higher precision than prior approaches.The comparison indicates that judge-backbone choice should depend on the task.
- Results: No judge exceeds 70% precision, so 30% of trajectories marked successful are erroneous.The paper argues that this limits current judges’ usefulness for automatic evaluation, RFT, and RL.
- Input representation: Screenshots alone achieve high precision for success and repetition, while accessibility trees alone provide higher recall; combining both performs worse than screenshots alone.The ablation suggests that additional representation information can distract rather than assist the judge.
5 Revisiting how we evaluate task success rate
The paper revisits benchmark success-rate evaluation by comparing expert annotations, LLM judgments, and rule-based methods. Rule-based evaluation consistently underestimates success, while LLM judges tend to overestimate it, motivating more faithful automatic measures.
- Comparing evaluation methods: Expert annotations, GPT-4o judgments with accessibility trees, and rule-based evaluation provide contrasting estimates of web-agent success rates.The comparison is made across multiple benchmarks.
- Evaluation discrepancies: Rule-based methods consistently underestimate success, whereas the LLM judge overestimates most agents except for two cases in WorkArena++.The direction and size of the discrepancy vary across benchmarks.
- Evaluation discrepancies: Rule-based evaluation reports GPT-4o performance 16.7% lower on WebArena and 18.5% lower on VWA than expert annotations.These differences illustrate the gap between official rule-based rates and expert-defined success.
- Evaluation discrepancies: Rule-based rankings can reverse expert preferences, ranking Qwen2.5-VL above GPT-4o on WebArena and WorkArena++ despite experts favoring GPT-4o on all benchmarks.On VWA, experts assign GPT-4o a success rate over 14% higher than Qwen2.5-VL.
6 Error Analysis
The error analysis identifies recurring ways LLM judges incorrectly accept unsuccessful web-agent trajectories, including grounding failures, misleading reasoning, missed instruction details, and misunderstood action intents.
- LLM judges can accept wrong webpages when they trust an agent’s reasoning without verifying the screenshot-grounded state.In one example, the agent selected the second item in the first row while claiming it selected the second-row, second-column item.
- Misleading agent reasoning can cause judges to mark tasks successful even when the agent’s claimed action was never performed.In WorkArena++, the agent claimed to apply a unique-ID filter after unsuccessful attempts, and the judge accepted that claim.
- Judges may overlook crucial instruction details, such as treating adding a cereal to the cart as completing a requested purchase.The cereal example shows the agent found the correct product but stopped before buying it.
- Judges may also miss incorrect final action intents when an agent completes intermediate steps but reports the result incorrectly.For a travel-time task, the judge noticed the travel time on screen but failed to flag the agent’s incorrect claim that the task was unfeasible.
- Overall, current LLM judges remain limited in detecting nuanced trajectory issues and readily agreeing with incorrect agent reasoning.The authors identify missed details, misunderstood actions, and agreement with erroneous reasoning as targets for future improvement.
7 Conclusion
The conclusion presents AgentRewardBench as a benchmark for expert-grounded evaluation of web-agent judges. Across its evaluation, simpler input representations agreed more with experts, while rule-based methods rejected valid trajectories and underreported success.
- AgentRewardBench contains over 1300 expert-annotated trajectories evaluated across success, unintended side effects, and unnecessary repetition.
- The benchmark evaluates 12 LLM judges and finds that simpler input representations achieve higher agreement with expert annotators than prior approaches.
- Rule-based evaluation tends to reject many valid trajectories, making some web agents’ success rates lower than experts would perceive.
- The benchmark is intended to support automatic evaluators and reward models that better reflect expert judgments of web-agent trajectories.
A.1 Environment and Experiments Details
The experiments span diverse web-agent environments and task subgroups, with standardized sampling, implementation, and input limits. The setup covers multiple evaluation methods and browser-agent infrastructure.
- Environment and implementation: AssistantBench uses DuckDuckGo because the original search homepage blocks browser automation and makes tasks unachievable.
- Task subgroups: The study samples tasks from WebArena and VisualWebArena by combining web domain with the original evaluation method.Methods include string matching, HTML-based programs, webpage image querying, and final URL matching.
- Task subgroups: The sampled tasks cover Wikipedia, maps, Reddit, shopping, classifieds, GitLab, and professional WorkArena categories.
- Environment and implementation: Agent inputs are capped at 40K tokens and outputs at 8192 tokens.
- Environment and implementation: BrowserGym provides Gym-compatible web environments, preprocessing for DOM and accessibility trees, and automatic execution of parsed browser actions.
A.2 Annotations
The annotation process collects trajectories from four LLM agents across five benchmarks and presents experts with goals, reasoning, actions, and screenshots. Uncertainty and disagreement are handled through review and consensus procedures.
- Trajectory collection: Trajectories come from Claude 3.7 Sonnet, GPT-4o, Qwen2.5-VL, and Llama 3.3 across five benchmarks.Llama 3.3 is excluded from VisualWebArena because it is not multimodal.
- Annotation interface: Annotators inspect each task goal, reasoning trace, action, and screenshot before answering questions about success, side effects, and repetitiveness.
- Shared knowledge: Expert annotators may share knowledge about web-agent capabilities, reasoning traces, hallucinations, repetition, and dedicated web environments.
- Agreement resolution: Most tasks receive binary annotations, while uncertain cases are reviewed by other annotators.
- Agreement resolution: Annotators resolve ambiguity and disagreement through deliberation, generally choosing the most lenient interpretation of ambiguous instructions.
A.3 LLM Judges
The paper uses portable prompt templates and evaluates LLM judges against expert-reviewed web-agent trajectories using multiple benchmark-level and fine-grained result tables.
- Prompt design: Model-agnostic system and user prompt templates use distinct tags to facilitate parsing judge outputs.The templates avoid model-specific commands, supporting transfer across LLMs.
- Results: Results are reported across all benchmarks and in fine-grained breakdowns by benchmark, judge, and agent.The tables use precision as the primary metric, with F1 and recall as auxiliary metrics.
- Judge inputs: The judge receives the task goal, webpage state, agent actions, reasoning, and selected trajectory observations to assess performance.Inputs can include the current URL, first and last screenshots, and the last accessibility tree.