PaperlayerView exact revision on arXiv

Source-linked AI summary

WebCanvas: Benchmarking Web Agents in Online Environments

Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, Zhengyang Wu

arXiv:2406.12373v3cs.CLcs.AIcs.LG

TL;DR

Static web benchmarks provide limited evidence about agent performance in an evolving online environment, where tasks and websites change over time. WebCanvas introduces an online evaluation framework built around key nodes, a community-maintained benchmark, and annotation and evaluation tools; its best-performing agent achieved a 23.1% task success rate on Mind2Web-Live.

  • Problem

    Most existing web-agent benchmarks capture static web interactions, while live websites and tasks change over time, making objective, quantitative, and reproducible online evaluation challenging.

    Source: §1 · p. 2

  • Method

    WebCanvas evaluates agents online using path-independent key nodes and provides a collaborative annotation platform, dataset maintenance tools, and an open-source agent reasoning framework.

    Source: §1 · p. 2

  • Results

    GPT-4-turbo with memory and ReAct achieved the best Mind2Web-Live task success rate of 23.1%, and online evaluations showed that competitive static performance does not necessarily carry over to dynamic settings.

    Source: §1 · p. 3

  • Takeaways & Limitations

    WebCanvas provides a platform for researchers and developers to build datasets and evaluate web agents online while collecting feedback on annotation quality and data validity.

    Source: §9 · p. 10

    Online evaluation remains affected by network instability, dynamic task pathways, and limitations of static evaluation functions.

    Source: §8 · p. 9

Abstract

from arXiv · show

For web agents to be practically useful, they must adapt to the continuously evolving web environment characterized by frequent updates to user interfaces and content. However, most existing benchmarks only capture the static aspects of the web. To bridge this gap, we introduce WebCanvas, an innovative online evaluation framework for web agents that effectively addresses the dynamic nature of web interactions. WebCanvas contains three main components to facilitate realistic assessments: (1) A novel evaluation metric which reliably capture critical intermediate actions or states necessary for task completions while disregarding noise caused by insignificant events or changed web-elements. (2) A benchmark dataset called Mind2Web-Live, a refined version of original Mind2Web static dataset containing 542 tasks with 2439 intermediate evaluation states; (3) Lightweight and generalizable annotation tools and testing pipelines that enables the community to collect and maintain the high-quality, up-to-date dataset. Building on WebCanvas, we open-source an agent framework with extensible modules for reasoning, providing a foundation for the community to conduct online inference and evaluations. Our best-performing agent achieves a task success rate of 23.1% and a task completion rate of 48.8% on the Mind2Web-Live test set. Additionally, we analyze the performance discrepancies across various websites, domains, and experimental environments. We encourage the community to contribute further insights on online agent evaluation, thereby advancing this field of research.

1 Introduction

WebCanvas addresses the gap between static web benchmarks and the changing online environment with a framework for live evaluation, community annotation, and dataset maintenance. On Mind2Web-Live, the best-performing agent achieved a 23.1% task success rate, while online evaluation revealed discrepancies with offline performance.

Motivation

  • Mind2Web’s archived interactions no longer fully represented live websites: 96 of 780 tasks (12%) were completely expired, complicating fair and reproducible online evaluation.The authors also report that MindAct, trained in 2023, outperformed closely held models on the static test set but lagged in 2024 online evaluations.Source: §1 · p. 2

Contributions

  • WebCanvas evaluates progress through key nodes—essential task milestones independent of the path taken—and supports collaborative annotation and scheduled data maintenance.Its maintenance strategy uses monitoring and automated alerts to identify invalid action sequences or key nodes and guide corrections as web content changes.Source: §1 · pp. 2–3

Results

  • 23.1% was the best task success rate, achieved by GPT-4-turbo with memory and ReAct reasoning on Mind2Web-Live.Online evaluation also showed that models performing competitively offline do not necessarily retain their competency in dynamic online environments.Source: §1 · p. 3

Dataset and findings

  • Mind2Web-Live comprises 542 tasks sampled from Mind2Web, annotated with key-node verification for online evaluation.The paper also finds that human-provided key-node annotations can benefit agents, while advanced models may generate inaccurate progress indicators that impair execution.Source: §1 · p. 3

2 WebCanvas: An Online Evaluation Framework for Web Agents

WebCanvas models interactive web tasks as agents acting on observations in a changing environment, then evaluates progress using path-independent key nodes. Its scores assess both reaching those milestones and completing tasks efficiently.

2.1 Problem Formulation of Interactive Web-Based Task

  • An agent receives a task instruction and current observation, chooses an action, and receives an updated observation after the environment transitions.The formulation represents the environment with state and action spaces, a transition function, and an observation space.Source: §2.1 · p. 3

2.2 Definition of Key Nodes

  • Key nodes are indispensable steps every successful task path must traverse, such as reaching a webpage or performing a required page action.Because page layouts and element selectors can change, WebCanvas prefers URL states to identify key nodes and uses element-class methods only when URLs cannot represent them.Source: §2.2 · pp. 3–4

2.3 Evaluation Metrics

  • WebCanvas scores each key node using URL, element-path, or element-value targets with exact, include, or semantic matching, then scores task completion and efficiency.Task Finish Score depends on completing all designated key nodes; Efficiency Score is the average number of trajectory steps per step score, L/P.Source: §2.3 · p. 4, Table 1 · p. 4

3 Mind2Web-Live: a Real-time Online Benchmark for Web Agents

Mind2Web-Live adapts Mind2Web tasks into an online benchmark with key-node annotations, supported by maintenance for changing websites.

3.1 Dataset Construction

  • The benchmark samples Mind2Web tasks, excluding time-sensitive descriptions, and re-annotates the selected tasks online.The initial sample comprised 601 training tasks and all 179 cross-task test tasks; 96 obsolete and 142 ambiguous tasks were removed, and 51 instructions were revised.Source: §3.1 · pp. 4–5
  • Table 2 presents the dataset distribution, while Figure 4 presents the distribution of evaluation functions.The dataset split is 438 training tasks and 104 test tasks.Source: Table 2 · p. 5, §3.1 · p. 5, Figure 4 · p. 5
  • 542 tasks—438 for training and 104 for testing—with 2,439 key nodes and 4,550 annotation steps make up Mind2Web-Live.The tasks span webpage types and operations for evaluation in a dynamic online environment.Source: §3.1 · p. 5

3.2 Dataset Maintenance

  • Website and content changes can make previously defined tasks or key nodes obsolete, motivating regular benchmark maintenance.Maintenance records workflow details, replays workflows to identify invalid annotations, and periodically reassesses key nodes with human checks; 18 data items were fixed in three months.Source: §3.2 · p. 5

4 Agent Framework

The agent framework organizes web-agent execution into four modules, while online evaluation assesses intermediate states rather than offline action-prediction accuracy.

4 Agent Framework

  • The framework combines Planning, Observation, Memory, and Reward modules for complex tasks in real-world online web environments.Planning uses task instructions, observations, and action history to produce thoughts and actions; Observation processes webpage source and screenshots into accessibility and visual observations.Source: §4 · p. 6

4.1 Discrepancy between Offline and Online Evaluation

  • Offline evaluation scores single-step action prediction and requires all predicted actions to be correct for task success, unlike online evaluation of intermediate states.Table 5 is identified as a complete comparison of model performance on the Mind2Web-Live test set, sorted by completion rate.Source: §4.1 · pp. 6–7, Table 5 · p. 7

5 Main Result

In live evaluations, GPT-4 outperforms other models in effectiveness and efficiency, while offline-to-online generalization and model rankings remain inconsistent.

5 Main Result

  • GPT-4 outperforms other models in live-environment effectiveness and efficiency, while Qwen is the best-performing open-source model.The Mind2Web-trained model does not generalize well to the online environment one year later, and the offline ranking relationship among MindAct-Large, GPT-3.5, and GPT-4 is reversed online.Source: §5 · p. 7
  • Offline action-prediction metrics omit real-world decision-space complexity and can penalize valid alternatives, making offline and online Task Success Rates inconsistent.Offline testing rewards task success only when all single-step actions are correctly predicted, unlike online evaluation of intermediate states.Source: §5 · p. 7

6 Analysis

Agent performance varies with task and environment; key-node analysis questions final-state-only evaluation, and human-labeled rewards improve performance.

6.1 Factors Influencing Agent Performance

  • Greater task complexity correlates with lower agent performance, while entertainment tasks are handled better than shopping or travel tasks.The authors also report that experimental environment matters, recommending Windows with Chrome or Firefox and servers in the United States.Source: §6.1 · pp. 7–8

6.2 Analysis of Key Node Evaluation

  • Only 46 of 104 test tasks had a final key node sufficient to establish completion, showing that final-state-only evaluation is inadequate in non-reproducible web environments.The analysis uses agent and human-demonstration trajectories for tasks where the agent reached all designated key nodes.Source: §6.2 · p. 8

6.3 Planning with Human-Labeled Reward

  • Generic reward modules do not improve performance and may lower task success and completion rates, whereas human-labeled rewards improve web-agent performance.For human-labeled reward reasoning, GPT-4-generated guidance based on task metadata is integrated with the task reference, history, and current observation.Source: §6.3 · pp. 8–9

7 Related Works

Prior web-agent benchmarks span simulations, saved datasets, stable answers, and dynamic evaluation, while agent frameworks have separately advanced reasoning and multimodal web interaction. WebCanvas evaluates real-world web capability using key nodes and corresponding evaluation functions.

7 Related Works

  • Earlier web benchmarks used simple navigation simulations [15], realistic simulated environments [35] [38] [13] [4], offline datasets [3] [19], stable answers [21], and dynamic methods [11] [20] [9].The paper also notes benchmark efforts on Android, operating systems, and databases [17] [33].Source: §7 · p. 9
  • WebCanvas aims to assess agents in real-world settings through key nodes and their evaluation functions.The paper positions this as a more comprehensive assessment of agent capability.Source: §7 · p. 9
  • Web-agent reasoning frameworks seek to improve agent capabilities [22] [6] [7] [12] [14], while multimodal modules combine visual and semantic information [37] [5] [8].Related work also includes successful reasoning frameworks for logical reasoning challenges [32] [36] [34] [28] [29].Source: §7 · p. 9

8 Discussion & Limitations

The paper identifies open challenges in live web-agent evaluation and argues that online evaluation remains more complex than controlled offline settings.

8 Discussion & Limitations

  • Online evaluation still requires defining key nodes in open environments, unifying agent inference processes, and reducing maintenance costs for real-time data and evaluation functions.The authors report progress toward a robust and accurate online evaluation system.Source: §8 · p. 9
  • Live evaluation adds network instability, dynamic and complex task pathways, and limitations of static evaluation functions; the authors call for continued research and community refinement.These complexities are absent from controlled offline settings.Source: §8 · p. 9

9 Conclusion

The conclusion presents WebCanvas as an online evaluation framework and community platform, with appendices describing data recording, annotation, evaluation rules, metrics, and agent components.

9 Conclusion

  • The authors developed an online web-agent evaluation framework, investigated online evaluation challenges and inference difficulties, and built a community platform for online dataset creation and agent evaluation.The platform collects feedback on dataset design, annotation quality, and data validity.Source: §9 · p. 10

A.1 Recording process

  • iMean Builder records browser operations, execution parameters, target selectors and content, element coordinates, and a screenshot at each step for replication and review.Recorded task steps form the basis of Mind2Web-Live and are annotated with key nodes and evaluation functions.Source: §A.1 · p. 13

A.2 Annotation process

  • Annotations were independently created, reviewed by two team members, and illustrated with examples spanning Yelp parking and Steam DLC tasks.Annotators included paper authors and five trained computer-science undergraduates.Source: §A.2 · p. 13, Table 8 · p. 13, Figure 5 · p. 14

B Comparison of the Mind2Web-Live and Mind2Web Datasets

  • Table 9 compares Mind2Web-Live with Mind2Web, including element, option, and success-rate measures.The supplied caption defines “Ele.” as “Element,” “Op.” as “Option,” and “SR” as “success rate.”Source: Table 9 · p. 14

C How to define evaluation functions

  • For necessary input and click operations, URL changes can define a key-node state; otherwise, evaluation can match the interacted element path or value.Input values may use exact, included, or semantic matching, while non-URL click operations use element-path or element-value matching.Source: C How to define evaluation functions · p. 15

D Additional Evaluation Metrics

  • Human Alignment Score evaluates workflow alignment using completion signals and step scores, with different scoring for completed, incomplete, and step-limit-terminated tasks.The formula uses achieved step scores P and maximum task step scores Pmax.Source: D Additional Evaluation Metrics · p. 16

E.1 Agent Framework

E.2 Action Space

E.3 Additional Experiment Settings

The additional settings describe experiment sampling and compute configurations, then detail the textual and visual components used to observe web pages.

E.3 Additional Experiment Settings

  • Main experiments used the Mind2Web-Live test set; self-reward experiments sampled 130 cases, and most experiments used standard compute instances.Foundation models used a 500-token maximum and temperature 0.7; MindAct used two T4 GPUs, and inference also used Mixtral-8x22B services from Together.ai.Source: §E.3 · p. 17

E.4 Observation Space

  • The accessibility-tree observation filters interactive elements by tag, visibility, usability, and textual or image content, then annotates them for selector-path generation.The tree provides a textual representation of page structure, but contains redundant information requiring filtering.Source: §E.4 · p. 17
  • Screenshots provide visual context to visual-language models and complement accessibility trees by exposing layout, appearance, positioning, and information those trees may miss.The default screenshot resolution is 1080 × 720, adjustable by users to balance inference cost and recognition effectiveness.Source: §E.4 · p. 17

F.1.1 Results on Mind2Web-Live Training Set

The appendix reports model and website analyses, examines common failure modes, and documents data maintenance and additional annotated examples.

F.1.1 Results on Mind2Web-Live Training Set

  • Table 11 compares GPT-3.5 and GPT-4 performance on the Mind2Web-Live training set without the reward module.The supplied caption does not report the table’s values.Source: Table 11 · p. 17

F.1.2 Ablation Study

  • Table 12 reports that GPT-3.5 and Mistral-8x22B do not benefit from memory and ReAct reasoning in online web tasks.The caption encourages more comprehensive evaluation of these modules.Source: Table 12 · p. 18

F.2 Additional Analysis

  • Figures 9–11 examine evaluation-function counts and accuracy across steps, plus trajectories from 50 successful GPT-4-agent tasks sampled across training and test sets.The heatmaps use logarithmic scales; Figure 11 compares human demonstrations with agent trajectories.Source: Figure 9 · p. 19, Figure 10 · p. 19, Figure 11 · p. 19
  • Website completion and success rates are analyzed alongside task complexity, but the website figures combine training and test data because the test set has few tasks.Figure 8 uses action-sequence length and key-node count as complexity references.Source: Figure 8 · p. 18, Figure 12 · p. 20, Figure 13 · p. 20

G.1 Local Optima

  • Local optima arise when confusing similar elements or lacking domain knowledge leads to a wrong path, while limited recovery and memory can preserve prior errors.The authors identify local optima as one main reason for low task success.Source: §G.1 · p. 21, G Qualitative Analysis of Experiments · p. 21

G.2 Premature Termination of Tasks

  • Premature termination can follow hallucinated task simplification, mistaken beliefs that an action executed, or lowered completion standards when progress stalls.An example reaches a rebate page but stops before completing the form.Source: §G.2 · p. 21, G Qualitative Analysis of Experiments · p. 21

G.3 Information Loss in Observation

  • The observation mapping recursively transfers selected child-element information to parents, but discards elements when no interactive parent is found or recursion reaches its limit.The current approach focuses on parent-child relationships; Figures 14–15 illustrate its strategy and a failure case.Source: §G.3 · p. 21, Figure 14 · p. 22, Figure 15 · p. 22

H Details of Data Maintenance

J Limitations & Future works

The authors identify network, annotation, and evaluation-function limits in online assessment, and describe potential societal impacts and the agent’s prompt-based evaluation procedure.

J Limitations & Future works

  • Network variability, including CAPTCHAs, outages, and IP differences, can cause discrepancies between online results and closed-environment results.Execution logs document performance under specific network and website conditions.Source: J Limitations & Future works · p. 26
  • Human annotators may miss valid task pathways, misaligning key nodes and potentially penalizing correct processes; model-based evaluation may instead be unstable.The proposed model-based alternative depends on the model’s capabilities.Source: J Limitations & Future works · p. 26
  • Static evaluation functions cannot adjust task criteria to environmental changes such as time, location, or weather.The authors propose dynamically adapting success criteria using ongoing feedback and environmental data.Source: J Limitations & Future works · p. 26
  • The authors call for continued research and community efforts to improve the reliability and validity of assessments in complex real-world environments.They characterize several issues as unresolved.Source: J Limitations & Future works · p. 26

K Impact Statement

  • The impact statement notes potential privacy and data-security risks, possible labor-market changes, and a risk that benefits could widen the digital divide.The benchmark excludes sensitive sites, login processes, personal information, and irreversible actions.Source: K Impact Statement · p. 26

L Prompts of Planning and Reward Module

  • The reward prompt classifies tasks as doing, finished, or loop and assigns scores from 1, 3, 7, 9, or 10 based on progress and completion.With a reference guide, completion requires satisfying all key conditions.Source: L Prompts of Planning and Reward Module · p. 31

Related papers

Loading 2406.12373v3…