Source-linked AI summary
An Illusion of Progress? Assessing the Current State of Web Agents
Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, Yu Su
TL;DR
Web agents’ capabilities are important to measure accurately, yet reported benchmark success may be overly optimistic. The paper introduces a realistic online benchmark and scalable evaluator, finding that frontier agents still struggle with online tasks while WebJudge closely aligns with human judgments.
Problem
Accurate capability measurement matters because web agents may automate work and affect employment, while existing results and evaluations may be insufficient or unreliable.
Method
The paper introduces Online-Mind2Web with 300 tasks across 136 websites and develops WebJudge, an LLM-as-a-judge method using critical intermediate screenshots.
Results
Most existing frontier agents successfully complete only 30% of online tasks, while WebJudge achieves 85.7% agreement with human judgment and a 3.8% success-rate gap.
Takeaways & Limitations
Online-Mind2Web exposes substantial room for web-agent improvement, while WebJudge supports scalable evaluation and downstream uses such as RFT, RL, and reflection.
Takeaways & Limitations
Existing online benchmarks may be limited by small task coverage, time-insensitive tasks, or evaluation criteria vulnerable to website changes.
Abstract
from arXiv · showhide
As digitalization and cloud technologies evolve, the web is becoming increasingly important in the modern society. Autonomous web agents based on large language models (LLMs) hold a great potential in work automation. It is therefore important to accurately measure and monitor the progression of their capabilities. In this work, we conduct a comprehensive and rigorous assessment of the current state of web agents. Our results depict a very different picture of the competency of current agents, suggesting over-optimism in previously reported results. This gap can be attributed to shortcomings in existing benchmarks. We introduce Online-Mind2Web, an online evaluation benchmark consisting of 300 diverse and realistic tasks spanning 136 websites. It enables us to evaluate web agents under a setting that approximates how real users use these agents. To facilitate more scalable evaluation and development, we also develop a novel LLM-as-a-Judge automatic evaluation method and show that it can achieve around 85% agreement with human judgment, substantially higher than existing methods. Finally, we present the first comprehensive comparative analysis of current web agents, highlighting both their strengths and limitations to inspire future research.
1 Introduction
The paper argues that web-agent capability claims require rigorous measurement because existing enthusiasm may rest on insufficient or biased evidence. It introduces a realistic benchmark, a scalable evaluator, and a comparative analysis to reassess current agents.
- Accurately measuring web-agent capabilities is critical because these agents may enable disruptive automation and job displacement.
- Claims of nearly 90% WebVoyager success have generated enthusiasm, but the authors caution that insufficient or biased data can foster unrealistic expectations and irrational decisions.
- Online-Mind2Web contains 300 diverse, realistic tasks across 136 websites and supports manual evaluation of six frontier web agents.
- Most evaluated agents underperform the simple SeeAct agent, while Operator reaches only 61% success, indicating substantial room for improvement.
- WebJudge achieves around 85% agreement with human judgment and a 3.8% average success-rate gap, outperforming existing automatic evaluation approaches.
- A comprehensive comparison of current web agents identifies their respective advantages and limitations to inform further improvement.
2 New Online-Mind2Web Benchmark
The paper develops Online-Mind2Web to address shortcomings in existing benchmarks, including limited realism, coverage, diversity, and reliable evaluation. The resulting benchmark contains 300 tasks across 136 websites and is substantially harder for a simple search agent than WebVoyager.
- Why Introduce a New Benchmark?: Realistic online evaluation requires diverse tasks across real-world websites, whereas existing benchmarks use cached snapshots, sandboxes, or constrained online settings.
- Why Introduce a New Benchmark?: WebVoyager covers only 15 websites, and a simple Google Search agent solves 51% of sampled tasks, revealing limited diversity and shortcut-prone task design.
- Dataset Construction: 47% of 650 sampled original Mind2Web tasks were invalid or had outdated ground-truth trajectories, motivating systematic task verification and filtering.
- Dataset Construction: Online-Mind2Web provides 300 realistic tasks spanning 136 popular websites and maintains validity by replacing tasks that become outdated or infeasible.
- Dataset Construction: The benchmark includes 83 easy, 143 medium, and 74 hard tasks, with difficulty defined by the human reference length.
- Dataset Construction: 22% of tasks were solved by the simple search agent, with success rates of nearly 50%, 18%, and 3% on easy, medium, and hard tasks.
3 WebJudge
WebJudge addresses unreliable online evaluation by selecting informative intermediate screenshots rather than relying only on final outputs or processing every frame. Its pipeline identifies task requirements, filters screenshots, and judges completion using the retained evidence and action history.
- Online web-agent evaluation is difficult because human assessment is labor-intensive and existing rule-based or LLM-based methods remain unreliable.
- WebJudge identifies key task points, selects relevant screenshots using a relevance threshold, and judges completion from the task, retained screenshots, and action sequence.
- The method avoids requiring final responses or intermediate thoughts, improving applicability across agents with different output formats.
- WebJudge primarily judges environment screenshots rather than LLM-generated text or images, making evaluator bias less pronounced in this setting.
- WebJudge’s API-call count scales with trajectory screenshots, so long trajectories can make evaluation costly and slow.
4 Experiments and Results
Experiments evaluate six web agents with human annotations and compare automatic evaluators. Results show substantially lower agent success on Online-Mind2Web than prior WebVoyager reports, while WebJudge closely aligns with human judgments.
- Evaluation Setup: Six web agents are evaluated from tasks initialized at start URLs, with Google Search disallowed to prevent shortcuts.Human annotation of trajectories uses task descriptions, action histories, and screenshots, with at least two annotators per task and a third resolving conflicts.
- Main Results: 61% success rate for Operator and 56.3% for Claude Computer Use 3.7 contrast with approximately 30% for the other agents.These results differ sharply from previously reported WebVoyager performance.
- Automatic Evaluation: 85.7% agreement and a 3.8% success-rate gap are achieved by WebJudge powered by o4-mini, while WebJudge-7B reaches 87% agreement and a 3.9% gap.WebJudge-7B also reduces evaluation cost to two API calls per trajectory.
- Difficulty Analysis: 31.6% average success-rate decrease from easy to medium tasks is followed by a further 15.4% decrease from medium to hard tasks.Claude Computer Use 3.7 and Operator reach 90.4% and 83.1% on easy tasks but still struggle with harder tasks.
- Automatic Evaluation: WebJudge achieves average agreement rates of 83.6% with GPT-4o and 85.7% with o4-mini across automatic evaluation comparisons.Its agent rankings closely align with human evaluation and it supports different web-agent output formats.
- Evaluation Limitations: WebVoyager and Autonomous Evaluation agreement with humans declines significantly as Operator trajectories become longer.WebVoyager is limited by screenshot-related token overload, while Autonomous Evaluation overlooks intermediate steps by focusing on the final screenshot.
5 Analysis
The analysis examines efficiency and failure patterns across agents, with particular attention to Operator’s behavior. It identifies longer failed trajectories, an exploration–exploitation trade-off, and recurring task-specific limitations.
- 5.1 Agent Efficiency: Efficiency score E is the average ratio of agent steps S_i to reference length Ŝ_i across successful tasks, with lower values indicating greater efficiency.The metric accounts for agents requiring different numbers of steps for the same task.
- 5.1 Agent Efficiency: Failed tasks use notably longer trajectories than successful tasks, often because agents repeat actions or encounter unexpected pop-up windows.For Browser Use, Claude Computer Use 3.7, and Operator, failed tasks involve nearly twice as many steps as successful ones.
- 5.1 Agent Efficiency: Operator heavily favors exploration by extensively exploring websites on the fly, illustrating an exploration–exploitation trade-off.The supplied analysis identifies this behavior alongside trajectory-length differences between successful and failed tasks.
- 5.2 Error Analysis and More Discussion: Operator’s error analysis categorizes failures into Filter & Sorting, Incomplete Steps, Navigation, Misunderstanding, and Other errors.Examples include incorrect filters, omitted submission clicks, deviations from intended navigation, and failure to grasp the task goal.
- 5.2 Error Analysis and More Discussion: Operator frequently fails numerical and temporal constraints and sometimes misses niche website features required for task completion.The authors relate sensitivity to numerical inputs to prior work and report these as two notable limitations.
- 5.2 Error Analysis and More Discussion: Other agents often neglect task requirements, hallucinate unmet constraints, explore less, repeat actions, and rely excessively on keyword-based search.These behaviors are presented as failure modes contrasting with Operator’s profile.
6 Conclusions
The paper finds that frontier web agents still struggle with realistic online tasks, while WebJudge enables scalable evaluation and reveals recurring agent limitations.
- Most frontier agents successfully complete only 30% of realistic online tasks.
- WebJudge identifies critical intermediate screenshots and achieves the highest agreement with human judgments among existing methods.
- The analysis highlights sensitivity to numerical or temporal constraints, limited exploration ability, and over-reliance on keyword-based search.
A.1 Web Agents and Benchmarks
Web-agent research has progressed from simulated environments toward real-world applications, alongside efforts to improve agent capabilities and evaluation methods.
- Autonomous web agents have evolved from simple simulated settings to real-world applications.
- The field includes numerous studies targeting improved agent capabilities and online evaluation methods.
- Online evaluation methods include rule-based heuristics and LLM-as-a-judge approaches, complementing human evaluation on live websites.
B.1 The impact of evaluation granularity: WebJudge (CoT) vs. WebJudge (Keypoints-wise)
WebJudge compares evaluation granularity, search constraints, screenshot selection, robustness, and model variants to balance reliability, fairness, cost, and scalability.
- Evaluation granularity: WebJudge (CoT) produces a binary task outcome, whereas Keypoint-wise evaluation checks each key point and requires all key points to be completed.
- Evaluation granularity: Keypoint-wise evaluation can lower agreement because generated key points may be inaccurate or unnecessary, making assessments overly strict.
- Evaluation granularity: WebJudge (CoT) costs approximately half as much as Keypoint-wise evaluation because it uses a single outcome-judgment call.
- Search constraint: Allowing Browser-Use to search improves performance modestly from 26% to 31%, while Online-Mind2Web tasks are less shortcut-friendly than WebVoyager tasks.
- Screenshot selection: A threshold of θ=3 achieves the highest agreement; larger thresholds lose information, while smaller thresholds increase screenshot overload.
- Robustness: WebJudge has an average standard deviation of 1.1% across three runs, indicating low evaluation variance.
- Model comparison: o4-mini offers high precision and scalability but lower recall, whereas WebJudge-7B balances precision and recall with fixed two-call costs per trajectory.
C.2 Task Distribution
Online-Mind2Web organizes realistic tasks across popular websites, diverse domains, and difficulty levels, while its prompts operationalize key-point and outcome evaluation.
- Task Distribution: Tasks come from 136 popular websites spanning domains such as shopping, finance, health, travel, entertainment, and government services.
- Task Distribution: Task difficulty is categorized into three levels based on reference length, with popularity and reference-length distributions reported in Figure C.1.
- Task Distribution: The benchmark categorizes tasks into 12 domains, and Figure C.2 reports the number of websites in each domain.
- Evaluation Prompts: WebJudge prompts evaluators to extract key points explicitly stated in the task description without adding unstated requirements.
- Evaluation Prompts: Key-screenshot prompts assess whether images contain essential task evidence, including actions, progress indicators, filters, or instructions.
- Evaluation Prompts: Outcome judgment combines the task, action history, key points, and potentially important web-page screenshots to assess task completion.
E Implementation Details
The evaluation uses agent-specific browser tools, screenshot trajectories, and automatic evaluators, while Operator case studies illustrate complex web-task behavior.
- Software Tools and Libraries: Open-source agents are evaluated online with Playwright under a 25-step limit, while Claude Computer Use 3.7 uses 50 steps and Operator runs in a remote browser.These settings control repeated-action costs across agent implementations.
- Agent Trajectories: SeeAct and Agent-E capture extended full-page screenshots, whereas Browser Use, Claude Computer Use, and Operator capture only the visible screen portion.The agents therefore provide different visual trajectories for evaluation.
- Automatic Evaluation: WebJudge-7B summarizes screenshots and assigns relevance scores, using held-out agents for evaluation after training on trajectories from other agents.Key screenshot identification is learned separately from later key-point and outcome judgment.
- Operator Case Studies: Operator case studies include filter application and Ctrl+F search to locate a page’s “Compare Side by Side” feature.These examples demonstrate distinct interaction strategies for complex web tasks.
F.1.3 Self-Verification and Error Correction
The examples show that agents can detect and correct some interaction errors, but failures persist when applying numeric, temporal, geographic, and sorting constraints.
- F.1.3 Self-Verification and Error Correction: Operator autonomously corrects selecting BLUE instead of BLACK after self-verifying the requested Uniqlo filter.The correction follows a grounding error during filter selection.
- F.1.4 Failure Cases about Numeric and Temporal Constraints: Operator applies the broader 2001–2012 range instead of the specified 2004–2012 range in a used-car search.This violates the task’s numeric year constraint.
- F.1.4 Failure Cases about Numeric and Temporal Constraints: Operator fails to adjust the time slider correctly when searching for flights with a specified arrival window.The failure concerns temporal control rather than the existence of the flight-search task.
- F.2 Case Study: Other Agents: All three agents issue a single loosely structured keyword query, illustrating over-reliance on hasty search.The example concerns the “Compare Audi A7 with Audi A6” task.
- F.1.4 Failure Cases about Numeric and Temporal Constraints: Browser Use reports cars as near Kentwood, Michigan, although the displayed vehicles are elsewhere, producing hallucinated content.The final response conflicts with the locations shown in the figure.
- F.1.4 Failure Cases about Numeric and Temporal Constraints: Browser Use claims a job is most recent without using the available sort function.The agent asserts the temporal ranking instead of verifying it through sorting.
G Task Examples
Online-Mind2Web combines tasks from existing and newly constructed datasets, spanning diverse domains and requiring agents to satisfy explicit search criteria.
- G Task Examples: Table G.1 samples 20 tasks: 5 from Mind2Web-Live, 5 from original Mind2Web, and 10 newly constructed tasks.The newly constructed examples cover academia, entertainment, transportation, and finance.
- G Task Examples: The task “Find a male senior boxer near zip code 90028” specifies breed, gender, age category, and location as key points.These criteria define the required attributes for the search result.
- G Task Examples: The evaluated trajectory shows Boxer and Senior filters applied with matching results, but the Sex filter remains unchecked.The screenshot is highly relevant but not fully comprehensive for the task.
- G Task Examples: The outcome judgment accepts the selected male senior Boxer because the agent viewed the complete six-item result set and identified a matching listing.The selection satisfies location, breed, age, and gender through the displayed results despite no explicit gender toggle.