PaperlayerView exact revision on arXiv

Source-linked AI summary

WebArena: A Realistic Web Environment for Building Autonomous Agents

Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, Graham Neubig

arXiv:2307.13854v4cs.AIcs.CLcs.LG

TL;DR

WebArena targets the gap between simplified agent environments and authentic, reproducible web-task evaluation. The paper builds a realistic web environment and benchmark with functional outcome validation, then evaluates baseline agents on long-horizon tasks. The best GPT-4-based agent achieves 14.41% end-to-end success versus 78.24% for humans, highlighting the difficulty of these tasks.

  • Problem

    Existing agent environments simplify real-world functionality, task complexity, exploration, and evaluation, limiting fair measurement on tasks that users care about.

  • Method

    The paper builds a self-hosted environment with four operational web applications, human-oriented tools and knowledge resources, and 812 natural-language tasks evaluated by functional correctness.

  • Results

    14.41% was the best GPT-4-based agent’s end-to-end task success rate, compared with 78.24% human performance.

  • Takeaways & Limitations

    The findings underscore the need to improve the robustness and efficacy of autonomous agents for complex web tasks.

Abstract

from arXiv · show

With advances in generative AI, there is now potential for autonomous agents to manage daily tasks via natural language commands. However, current agents are primarily created and tested in simplified synthetic environments, leading to a disconnect with real-world scenarios. In this paper, we build an environment for language-guided agents that is highly realistic and reproducible. Specifically, we focus on agents that perform tasks on the web, and create an environment with fully functional websites from four common domains: e-commerce, social forum discussions, collaborative software development, and content management. Our environment is enriched with tools (e.g., a map) and external knowledge bases (e.g., user manuals) to encourage human-like task-solving. Building upon our environment, we release a set of benchmark tasks focusing on evaluating the functional correctness of task completions. The tasks in our benchmark are diverse, long-horizon, and designed to emulate tasks that humans routinely perform on the internet. We experiment with several baseline agents, integrating recent techniques such as reasoning before acting. The results demonstrate that solving complex tasks is challenging: our best GPT-4-based agent only achieves an end-to-end task success rate of 14.41%, significantly lower than the human performance of 78.24%. These results highlight the need for further development of robust agents, that current state-of-the-art large language models are far from perfect performance in these real-life tasks, and that WebArena can be used to measure such progress.

1 INTRODUCTION

WebArena addresses the gap between simplified agent environments and real-world web tasks with a realistic, reproducible platform and benchmark. It evaluates autonomous agents on functional completion of diverse, long-horizon natural-language tasks.

  • Motivation: Current agent environments simplify real-world functionality, task complexity, exploration, and evaluation of alternative valid solutions.These limitations can create discrepancies between simulated environments and real-world operation.
  • Environment: WebArena provides four fully operational, self-hosted web applications spanning online shopping, discussion forums, collaborative development, and business content management.It also incorporates utility tools, domain documentation, and real-world-derived website content.
  • Environment: WebArena includes tools, knowledge bases, Docker hosting, and gym-APIs to support human-like task execution while preserving usability and reproducibility.Its resources range from maps and calculators to Wikipedia and software manuals.
  • Benchmark: 812 long-horizon tasks translate high-level natural-language intents into concrete web interactions and evaluate whether executions achieve their intended goals.Outcome-based validation accommodates multiple valid action paths rather than requiring a reference sequence.
  • Results: 14.41% was the best GPT-4-based agent’s end-to-end task success rate, versus 78.24% for humans.The results indicate substantial difficulty for current agents on complex realistic web tasks.

2 WE BAR E N A: WEBSITES AS AN ENVIRONMENT FOR AUTONOMOUS AGENTS

WebArena is a realistic, reproducible environment for autonomous web agents, combining operational websites, utility tools, external resources, and browser-like interaction. Its design supports long-horizon tasks across multiple sites while enabling deterministic functional evaluation.

  • Environment construction: WebArena uses standalone, self-hosted websites and reset scripts to avoid live-site changes and support reproducible comparisons.It combines open-source site implementations with sampled real-world data and deterministic initialization.
  • Task execution: WebArena models deterministic state transitions and evaluates task success by checking whether resulting states and action outcomes satisfy the natural-language intent.For example, placing an order is verified through the resulting order state and answer accuracy can also be checked.
  • Environment construction: Four operational website domains cover e-commerce, social forums, collaborative development, and business content management.The categories were selected after analyzing approximately 200 examples from the authors’ browser histories.
  • Environment construction: Utility tools include a map, calculator, and scratchpad, while external knowledge resources support information seeking and human-like task execution.The map supports navigation and point-of-interest searches.
  • Observation space: Observations include the URL, opened tabs, and focused-page content rendered as HTML DOM, screenshots, or accessibility trees.Multi-tab browsing is intended to replicate human web use more authentically.
  • Action space: The compound action space supports element, tab, and URL-navigation operations, with elements selectable by coordinates or unique IDs.ID-based selection turns element interaction into an n-way classification problem and supports different agent modalities.

3 BENCHMARK SUITE OF WEB-BASED TASKS

WebArena’s benchmark contains high-level, diverse web tasks designed to require realistic multi-step interaction, and evaluates them by functional correctness rather than matching one action sequence. It combines answer-based scoring with programmatic checks of execution states.

  • Benchmark design: 812 benchmark examples ground high-level natural-language instructions in WebArena interactions, with a metric for each task’s functional correctness.The benchmark emphasizes whether execution achieves the intended outcome.
  • Intent collection: Annotators were instructed to create abstract, creative, multi-action intents with optional constraints and replaceable variables.Template-based instantiation broadens task coverage without requiring identical implementations.
  • Intent collection: 241 templates generate 812 instantiated intents, averaging 3.3 examples per template across information seeking, site navigation, and content or configuration operations.Instantiations can share high-level semantics while having distinct execution traces.
  • Evaluation: Information-seeking tasks compare predicted answers with annotated references using exact_match, must_include, or fuzzy_match scoring.The functions accommodate standardized answers, required concepts, and semantically equivalent responses.
  • Evaluation: Navigation and content/configuration tasks use programmatic reward functions to inspect intermediate execution states and verify intended outcomes.Evaluation uses locators and keyword checks over relevant pages or stored data.
  • Evaluation: WebArena also includes unachievable tasks arising from inadequate evidence, permissions, or missing website functionality.These tasks test whether agents can recognize requests that cannot be completed in the environment.

4 BASELINE WEB AGENTS

The baseline study evaluates language-model agents using direct action prediction and chain-of-thought prompting under a common browser setup. Human performance provides a substantially higher reference point than the reported baseline result.

  • Human baseline: 78.24% Success Rateall is reported for humans after an average time of 110s.The component rates are 74.68% for info tasks and 81.32% for other tasks.
  • Agent prompting: Three LLMs are evaluated with direct next-action prediction and chain-of-thought reasoning, each using two in-context examples.The setup also describes the browser, allowed actions, rules, and an Unachievable hint.
  • Agent prompting: Agents use an accessibility-tree observation space with element IDs that identify interactable page elements.The configuration is designed to align with the guidelines given to human annotators for fair comparison.

5 RESULTS

WebArena results show that realistic, long-horizon web tasks remain difficult for current language-model agents, with performance varying across prompting and task templates.

  • Benchmark performance: 11.70% end-to-end task success was achieved by GPT-4 with CoT, versus 78.24% human performance.GPT-3.5 with CoT achieved 8.75%, while TEXT-BISON-001 achieved 5.05%.
  • Prompting strategies: 2.34% improvement came from explicit reasoning over GPT-4 without CoT.The comparison concerns end-to-end task success under the prompting strategies in Table 2.
  • Prompting strategies: 14.41% overall GPT-4 task success followed removal of the UA hint, which improved achievable-task performance.Without the hint, GPT-4 still recognized 44.44% of unachievable tasks, despite an overall decline in identifying them.
  • Error analysis: 54.9% of feasible tasks were incorrectly identified as impossible by GPT-4 during error analysis.The UA hint helped identify unachievable tasks but hindered performance on achievable tasks.
  • Consistency across templates: GPT-4 reached 100% success on only four of 61 templates, while GPT-3.5 reached full completion on none.Tasks from the same template can still differ in complexity, such as repetitive repository-forking operations.
  • Benchmark design: WebArena compares fully interactable realistic tasks with existing benchmarks and evaluates functional correctness rather than only matching reference action sequences.The benchmark emphasizes diverse tasks encountered in daily routines and outcome-based execution metrics.

6 RELATED WORK

Prior work studies natural-language control and interactive web agents, but existing benchmarks continue to struggle to balance functionality, authenticity, and environmental dynamics.

  • Natural-language control: Existing natural-language control benchmarks often compromise functionality, authenticity, or support for environmental dynamics.The comparison is summarized in Table 4.
  • Interactive decision-making: Interactive web-agent research includes search-based question answering, JavaScript synthesis, and screenshot-based action prediction.These approaches differ in whether agents rely on search results, generated code, or visual web-page inputs rather than text-based DOM trees.

7 CONCLUSION

WebArena is a realistic, standalone, reproducible web environment with functional applications, organic data, and a benchmark for outcome-based evaluation. GPT-4 achieved only 14.41% end-to-end success versus 78.24% for humans, underscoring the difficulty of robust autonomous web task execution.

  • Contribution: WebArena combines functional web applications and organic data with 812 natural-language web tasks evaluated through programmatic outcome validation.The environment is designed for development and testing of autonomous agents.
  • Findings: 14.41% end-to-end task success for GPT-4 lagged behind 78.24% human performance.The authors identify this gap as evidence that robust and effective autonomous agents remain an open research challenge within WebArena.

A.1 WEBSITE IMPLEMENTATION

WebArena recreates common websites using open-source frameworks and imported real-world data, while providing offline knowledge resources with search functionality.

  • Website construction: The e-commerce site contains approximately 90k products with prices, options, descriptions, images, and reviews.The implementation aims to reproduce functionality from commonly used sites in a reproducible way.
  • Knowledge resources: Kiwix hosts an offline English Wikipedia, while GitLab and Adobe Commerce Merchant manuals come from official websites.These knowledge resources are configured as individual searchable websites; Wikipedia has a May 2023 knowledge cutoff.

A.2 ENVIRONMENT DELIVERY AND RESET

WebArena is delivered as self-contained Docker images, enabling reproducible deployment and resetting after website data changes. Evaluation cases are often read-only, so resets add a small but non-negligible time cost.

  • Each website is packaged in a self-contained Docker image containing its code, database, and software dependencies.The images include pre-populated website data and do not rely on external volume mounts.
  • Website state can be reset by deleting the active container and restarting the original image.This restores the website to its initial state after tasks modify data.
  • All user intents require interactions with multiple web pages, according to the intent-distribution description.Cross-site intents additionally require interacting with multiple websites.
  • Resetting containers has a non-negligible but small impact on evaluation time because many intents are read-only and inference is also costly.

A.3 USER ROLES SIMULATION

WebArena simulates varied user experiences by assigning distinct roles, permissions, and interaction histories across its platforms. Profiles include realistic activity patterns and a pre-cached-cookie login mechanism rather than universally identical roles.

  • WebArena generates unique user profiles to emulate differences in roles, permissions, and interaction histories.The motivating example contrasts shop owners with employees who have narrower permissions.
  • Profiles represent realistic platform-specific activity, including orders, popular projects, merge requests, issues, posts, comments, and private projects.
  • All users are automatically logged into their accounts using a pre-cached cookie, unlike the universally identical user roles typically assumed in prior work.

A.4 INTENT DISTRIBUTION

The appendix distributes intents across websites and documents the evaluation setup and representative GPT-4 failure modes. These failures include relying on superficially relevant observations, missing granular state, and repeating actions.

  • A.4 INTENT DISTRIBUTION: The benchmark’s intent distribution spans different websites, with every intent requiring interaction across multiple web pages.Cross-site intents additionally require interaction with multiple websites.
  • A.5 HUMAN PERFORMANCE: Human-performance comparisons may vary with annotator demographics because some tasks require domain-specific knowledge.Examples include understanding Git merge requests or creating products in complex content-management systems.
  • Agent Evaluation Setup: The evaluation uses language-model agents with a maximum of 30 state transitions and early termination after repeated actions or three consecutive invalid actions.The experiments include GPT-3.5-TURBO-16K-0613, GPT-4-0613, and TEXT-BISON-001.
  • Agent Evaluation Setup: A reasoning agent performs reasoning before issuing an action, whereas a direct agent emits the next action from the observation.
  • A.10 ADDITIONAL ERROR ANALYSIS: GPT-4 may choose readily available but irrelevant information instead of generating the report needed for historically accurate results.The example concerns identifying the top-1 best-selling product in 2022 from an e-commerce CMS.
  • A.10 ADDITIONAL ERROR ANALYSIS: GPT-4 sometimes overlooks already-entered inputs and previous actions, causing repeated commands until the maximum step limit.The example repeats entering “DMV area” despite the observation showing that term already filled in.
Loading 2307.13854v4…