PaperlayerView exact revision on arXiv

Source-linked AI summary

WebSuite: Systematically Evaluating Why Web Agents Fail

Eric Li, Jim Waldo

arXiv:2406.01623v1cs.SEcs.AI

TL;DR

Web-agent benchmarks often measure task completion without revealing why agents fail. WebSuite combines a web-action taxonomy with individual and end-to-end tasks, and its evaluation identifies distinct weaknesses in natbot and SeeAct. The authors conclude that more fine-grained analysis can surface specific improvement opportunities.

  • Problem

    Existing benchmarks often measure whether web agents complete tasks without explaining why they fail, despite agents remaining far below human performance on end-to-end tasks.

    Source: §1 · p. 1

  • Method

    WebSuite uses a taxonomy of web actions and an extensible suite of individual and end-to-end tasks to evaluate agent failures at specific actions.

    Source: §1 · p. 2

  • Results

    WebSuite identifies distinct weaknesses: natbot struggles to fill forms, while SeeAct has a 0% success rate at fill/complexform because it cannot complete select interactions.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The authors conclude that fine-grained analysis of agent failure points highlights specific improvement opportunities and the need for more diagnostic tools.

    Source: §8 · p. 9

    WebSuite’s success rates may not transfer to other websites because its benchmark is rooted in particular web-design decisions and real-world sites vary widely.

    Source: §7 · p. 9

Abstract

from arXiv · show

We describe WebSuite, the first diagnostic benchmark for generalist web agents, designed to systematically evaluate why agents fail. Advances in AI have led to the rise of numerous web agents that autonomously operate a browser to complete tasks. However, most existing benchmarks focus on strictly measuring whether an agent can or cannot complete a task, without giving insight on why. In this paper, we 1) develop a taxonomy of web actions to facilitate identifying common failure patterns, and 2) create an extensible benchmark suite to assess agents' performance on our taxonomized actions. This benchmark suite consists of both individual tasks, such as clicking a button, and end-to-end tasks, such as adding an item to a cart, and is designed such that any failure of a task can be attributed directly to a failure of a specific web action. We evaluate two popular generalist web agents, one text-based and one multimodal, and identify unique weaknesses for each agent. Because WebSuite can disaggregate task failures into specific action failures, this enables granular identification of which UX flows an individual agent has trouble with and immediately highlights promising avenues for improvement. These findings highlight the need for more focused benchmarking on where web agents go wrong to effectively improve agents beyond their weaker performance today.

1 Introduction

Web agents remain far below human performance, while existing benchmarks reveal little about why they fail. WebSuite introduces a taxonomy and extensible tasks to diagnose failures at specific web actions, then uses it to identify distinct weaknesses in natbot and SeeAct.

1 Introduction

  • WebArena reports 14% end-to-end task success for agents versus 78% for humans, while existing benchmarks leave failure causes underexplored.The paper argues that diagnosing failure patterns is needed to identify ways to improve agents.Source: §1 · p. 1
  • WebSuite pairs a taxonomy of web actions with individual and end-to-end tasks designed to attribute task failures to specific actions.The taxonomy organizes operations by high-level category, action, and specific interaction.Source: §1 · p. 2
  • WebSuite evaluations identify natbot’s difficulty filling forms and SeeAct’s difficulty clicking links in search results during end-to-end tasks.The authors say these patterns would otherwise require repeated manual observation to identify.Source: §1 · p. 2
  • The paper situates WebSuite alongside diagnostic benchmarks such as CheckList, CLEVR, and CARETS, and says it can complement existing benchmarks.The authors report that their code is available online.Source: §1 · p. 2

2 Related Work

Prior web-agent benchmarks evaluate either individual low-level actions or complex goals, but generally provide only task success as a signal. WebSuite aims to bridge these benchmark types with contained tasks and end-to-end instantiations.

2 Related Work

  • Higher-level benchmarks assess multi-action goals but usually report only whether the whole task succeeded, leaving failure diagnosis to manual observation.Examples include GAIA, WebShop, Mind2Web, and WebArena.Source: §2 · p. 2
  • MiniWoB and MiniWoB++ test specific low-level tasks, but completing an action alone differs from completing it within a longer sequence.MiniWoB++ adds patterns such as soft reasoning and layouts that vary stochastically.Source: §2 · p. 2
  • WebSuite bridges these benchmark types with contained individual tasks and end-to-end versions that reveal how task success changes across longer sequences.Source: §2 · p. 3

3 Taxonomy of Web Actions

The paper decomposes web operations into categories, actions, and specific interactions so failures can be traced to finer-grained capabilities. Its taxonomy adapts digital-skills categories and draws on web-component patterns, while noting that compound interactions depend on primitive ones.

3 Taxonomy of Web Actions

  • Web actions are analyzed through intermediate steps in an agent’s task, allowing recurring failure modes to be traced to specific actions.Source: §3 · p. 3

3.1 Categories and Actions

  • The taxonomy adapts digital-skills categories into operational, navigational, and informational actions, excluding strategic skills as decomposable into those categories.Operational actions involve basic UX patterns; navigational actions involve structures such as menus; informational actions include retrieval, submission, and search.Source: §3.1 · p. 3, Table 1 · p. 3

3.2 Interactions

  • Interactions distinguish specific forms of an action, such as clicking a link versus a button, and the operational list draws on Material Design components absent from MiniWoB.Examples include snackbars, switches, tooltips, and datagrids; navigational and informational interactions vary more in the information represented than in component type.Source: §3.2 · p. 4
  • Compound interactions such as filling a form or filtering a datagrid depend on their primitive interactions, limiting how independently their success can be assessed.Source: §3.2 · p. 4

4 WebSuite

WebSuite combines individual interaction tasks with end-to-end tasks in a logged web environment. Its task criteria, checkpoints, and logs support evaluating completion and locating which taxonomized interactions failed.

4 WebSuite

  • WebSuite aims to evaluate interactions both in individual tasks and across end-to-end tasks, using a React frontend and Flask backend.Source: §4 · p. 4

4.1.1 Creating tasks for each interaction

  • The extensible individual-task suite covers a core subset of interactions, with some interactions tested in multiple states or uses.Tasks have a starting path, goal, and success criteria based on logs or submitted material.Source: §4.1.1 · p. 4, §4.1.2 · p. 4

4.1.2 Evaluating agents

  • Individual evaluations use 90-second limits for shorter tasks and 300-second limits for longer tasks, with optional stopping conditions to assess task completion rather than appropriate stopping.The authors report agents sometimes continued clicking a single element dozens of times.Source: §4.1.2 · p. 4

4.2.1 Creating the environment

  • The shopping playground includes search, customizable item pages, and checkout; URL checkpoints encode prior state, and the site uses generated shopping-item data.The initial end-to-end tasks are ordering an item and adding one to the cart.Source: §4.2.1 · pp. 5–6, Figure 2 · p. 5

4.2.2 Evaluating agents

  • End-to-end tasks pair goals and checkpoints with golden logs, then use final-page verification and log comparison to assess completion and identify failed interactions.Checkpoint-level overrides handle compound interactions to avoid noisy primitive-level failure patterns; evaluations have a 300-second limit.Source: §4.2.2 · p. 6

5 Experiments

The experiments evaluate natbot and SeeAct across individual and end-to-end tasks, repeating each task eight times. Results are organized to show success by action, task checkpoints, and interactions during end-to-end runs.

5.1 Setup

  • Each agent ran on every individual and E2E task eight times to account for variability.The code was designed to support testing and analysis of additional agents.Source: §5.1 · p. 6

5.2 Results

  • Table 2 reports individual-task success rates aggregated by action.The passage points to Appendix A.2 for more detailed individual-task results.Source: §5.2 · p. 6, Table 2 · p. 7
  • Table 3 reports E2E task completion rates, including success at specific checkpoints and trial counts.Checkpoint trial counts can be lower when an agent inconsistently reaches those checkpoints.Source: Table 3 · p. 7
  • Table 4 reports interaction success rates during E2E tasks, with counts of encountered instances.Source: Table 4 · p. 7

6 Discussion

WebSuite’s granular results point to different improvement targets for natbot and SeeAct, while the discussion identifies ways to extend the analysis to more tasks and realistic settings.

6 Discussion

  • Natbot’s datagrid filter matches USA in names rather than countries, suggesting stronger reasoning or specific policies for complex filtering as possible improvements.The discussion also reports relative strengths in operational and navigational tasks and difficulty with informational tasks.Source: §6 · p. 8
  • SeeAct cannot complete select interactions, contributing to 0% success on fill/complexform; the authors identify select support as an important first improvement.Source: §6 · p. 8
  • Natbot’s E2E results localize its core problem to fill/complexform, with perfect performance on its other tasks.The authors recommend focusing improvement efforts on form-filling.Source: §6 · p. 8
  • SeeAct has 0% E2E success on click/link despite high individual-task performance, and behavior observation revealed clicks on a container div instead of the a link.WebSuite makes patterns across tasks easier to see but does not replace observing agent behavior.Source: §6 · p. 8

6.1 Looking forward

  • As E2E tasks grow, analyzing which interaction failures contribute to task failures becomes a more meaningful data-analysis problem.Source: §6.1 · p. 8
  • The authors call for testing web actions in more diverse, real-world environments, including datasets of individual interactions on popular sites.They describe snapshotting popular online sites as one possible extension.Source: §6.1 · p. 8
  • Existing benchmarks could be augmented by tagging tasks or individual steps with actions from the WebSuite taxonomy.Source: §6.1 · p. 8
  • LLM evaluators could apply the taxonomy to failed-task trajectories to identify failure types, potentially replacing extensive E2E benchmark tagging with post-trajectory analysis.This proposal builds on prior work using LLMs to evaluate agent trajectories.Source: §6.1 · p. 9

7 Limitations and Potential Societal Impact

The authors caution that WebSuite’s results are shaped by its web environment and simplified pages, and note dual-use concerns as agents become more capable.

Specific to our web environment

  • WebSuite’s interaction, action, and category success rates might not transfer to websites with different design patterns, styling, or web practices.The benchmark is rooted in the authors’ web design decisions, including the semantic information available to agents.Source: §7 · p. 9

Simplified web environment

  • The simplified pages use minimal CSS and little extraneous information, so success on a single-button page may not reflect difficulty on a highly styled page.Source: §7 · p. 9

Generalist web agents face dual-use concerns

  • More capable generalist agents raise dual-use concerns, including both desk-work assistance and harmful uses such as manipulation or scamming.Source: §7 · p. 9

8 Conclusion

The conclusion presents WebSuite as a taxonomy and benchmark for diagnosing web-agent failures across lower- and higher-level tasks, with initial analyses of natbot and SeeAct illustrating how it can identify improvement opportunities.

8 Conclusion

  • WebSuite combines a web-action taxonomy with a benchmark spanning lower- and higher-level tasks, and its analysis of natbot and SeeAct identifies specific improvement opportunities.The authors call for finer-grained analysis of agent failure points and further work.Source: §8 · p. 9

A.1 Taxonomized interactions

Tables 5–7 list interactions across operational, navigational, and informational actions; a check mark indicates that at least one task has been implemented for an interaction.

A.1 Taxonomized interactions

  • A check mark indicates that at least one task has been implemented for an interaction.Source: §A.1 · p. 11
  • Table 5 lists operational actions and their interactions.Source: Table 5 · p. 11
  • Table 6 lists navigational actions and their interactions.Source: Table 6 · p. 11
  • Table 7 lists informational actions and their interactions.Source: Table 7 · p. 12

A.2 Detailed individual results

Table 8 reports individual task success rates and explains how trials are weighted when aggregating action-level results.

A.2 Detailed individual results

  • Table 8 reports individual task success rates.Source: Table 8 · p. 13
  • For interactions with multiple tasks, each trial is weighted by the inverse of the number of tasks during action-level aggregation.Source: Table 8 · p. 13
  • The weighting ensures each interaction contributes equally to action performance.Source: Table 8 · p. 13
Loading 2406.01623v1…