Source-linked AI summary
WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, Alexandre Lacoste
TL;DR
The paper asks whether web agents can perform the broad, everyday workflows of knowledge workers in enterprise software, rather than only simpler web tasks. It introduces WorkArena and BrowserGym for benchmarked browser interaction, and finds that enterprise automation remains challenging while BrowserGym-supported agents achieve strong results on other benchmarks.
Problem
Web-agent research lacks broad evidence on performing typical knowledge-worker tasks in complex enterprise software interfaces.
Method
The paper introduces WorkArena, a 33-task ServiceNow benchmark, and BrowserGym, an environment combining rich actions with multimodal observations for web-agent evaluation.
Results
WorkArena remains an unsolved benchmark requiring long-context understanding and advanced reasoning, while the GPT-4 agent reaches 25.4% on WebArena versus 14.4% in the original paper.
Takeaways & Limitations
The benchmark and environment provide open-source resources for evaluating web-agent capabilities and potential real-world impact across enterprise workflows.
Takeaways & Limitations
The benchmark is currently limited to existing WorkArena tasks, with future expansion planned toward compositional workflows requiring retrieval, memorization, visual perception, and advanced reasoning.
Abstract
from arXiv · showhide
We study the use of large language model-based agents for interacting with software via web browsers. Unlike prior work, we focus on measuring the agents' ability to perform tasks that span the typical daily work of knowledge workers utilizing enterprise software systems. To this end, we propose WorkArena, a remote-hosted benchmark of 33 tasks based on the widely-used ServiceNow platform. We also introduce BrowserGym, an environment for the design and evaluation of such agents, offering a rich set of actions as well as multimodal observations. Our empirical evaluation reveals that while current agents show promise on WorkArena, there remains a considerable gap towards achieving full task automation. Notably, our analysis uncovers a significant performance disparity between open and closed-source LLMs, highlighting a critical area for future exploration and development in the field.
1. Introduction
Web agents target burdensome, repetitive, and accessibility-limited UI work, but existing evaluations do not yet cover the breadth of enterprise knowledge-work tasks. The paper introduces WorkArena and BrowserGym to address this gap.
- Complex or repetitive UI tasks remain burdensome, while increasingly complex interfaces can discriminate against visually impaired users.
- UI assistants can provide transparent, inspectable automation because users can give or take back control during interaction.The supported automation range spans partial assistance, such as finding menus or filling forms, to complete task execution.
- Existing web-agent tasks range from simple commands on toy pages to complex requests on real-world websites, motivating broader enterprise evaluation.
- BrowserGym provides a web-agent development and evaluation environment with multimodal observations and a rich action set.The observations include HTML, an accessibility tree, and rendered pixels.
2. Related Works
Prior web-agent benchmarks span toy environments, simulated shopping, realistic websites, interaction datasets, and information retrieval. WorkArena complements them by targeting enterprise software and BrowserGym by unifying multimodal observations, code-based actions, and chat interaction.
- Early benchmarks such as MiniWoB use synthetic web environments with low-level interactions, while WebShop simulates e-commerce search and browsing.MiniWoB contains 125 toy tasks, including button clicking and basic text editing.
- WebArena evaluates 190 tasks on realistic websites spanning e-commerce, social forums, software development, and content management.The cited passage reports 14% success for a GPT-4 agent and 78% for human agents.
- Other resources provide human-curated web interactions, expert demonstrations, or information-retrieval tasks across real-world websites.The passage identifies Min2Web, WebLINX, and WebVoyager as examples of these directions.
- WorkArena complements existing benchmarks by focusing specifically on real-world enterprise software and end-to-end workflows performed by knowledge workers.
- BrowserGym aggregates multimodal observations and code-based actions and is the first environment in the cited comparison to support chat-based agent-user interactions.
3. WorkArena – An Enterprise Benchmark
WorkArena is an enterprise benchmark of natural-language-directed tasks spanning common ServiceNow workflows, with validation and oracle functions supporting evaluation. Its realistic interfaces create challenges through dynamic behavior, non-standard HTML, and very large DOMs.
- WorkArena Tasks: WorkArena contains 33 tasks and 19,912 instances covering filtering, forms, knowledge-base search, service catalogs, dashboards, and menu navigation.These operations represent common daily work for IT, administrative, and white-collar staff.
- WorkArena Tasks: Tasks use explicit natural-language goals generated from human-designed templates with predefined values.The goals provide the information required to solve each task.
- WorkArena Tasks: Validation functions provide real-time feedback on errors, while hand-crafted oracle functions automatically complete tasks through Playwright browser automation.
- WorkArena Tasks: Lists comprise 12 tasks for filtering or sorting, including hidden menus, multiple conditions, and up to three sort columns.The list tasks yield 6,900 instances.
- WorkArena Tasks: Form tasks create database entries across 1 to 26 fields, sometimes requiring tabs and dynamic auto-completion handling.The five form tasks yield 5,000 instances and are validated by querying created database entries.
- WorkArena Tasks: Knowledge-base tasks require keyword search and article browsing to answer questions, with validation accepting predefined equivalent answers.This task category yields 1,000 instances.
- WorkArena Tasks: Service catalogs, dashboards, and menus test item ordering, chart-data extraction, reasoning over charts, navigation, and user impersonation.The cited passages describe validation through database checks, answer matching, destination checks, or impersonation verification.
- Challenges: the World Wild Web of Work: ServiceNow interfaces include dynamic rules, overloaded interactions, nested iFrames, shadow DOMs, proprietary APIs, and non-standard HTML.These behaviors are presented as common characteristics of real-world enterprise software, although not necessarily web-development best practices.
4. BrowserGym
BrowserGym is a flexible browser environment for developing and evaluating multimodal web agents, with customizable observations and actions, support for complex navigation, and extensibility across benchmarks.
- BrowserGym provides a generic environment for designing benchmarks and evaluating multimodal web agents.
- Chat-based user interaction: BrowserGym supports chat-based interaction, including information retrieval, sequential instructions, and changing user requests.In WorkArena, the task goal begins as an initial user message and the agent may reply throughout the task.
- Rich observation space: At each time step, agents can receive chat history, open-page URLs, action errors, DOM and accessibility-tree snapshots, and a viewport screenshot.The environment follows a partially observable setup, so agents must decide how to manage history.
- Rich action space: Its customizable action space ranges from bid- and coordinate-based primitives to arbitrary Python code through the Playwright API.This lets researchers vary the interaction interface and agent flexibility.
- Multi-page navigation: The environment supports multiple pages, tabs, popups, iFrames, and shadow DOMs for handling heterogeneous real-world websites.This capability addresses navigation and encapsulation patterns that are difficult for existing web environments.
- Flexibility and extensibility: BrowserGym leaves observation, history, and action-space choices to agents while supporting concise task implementations and extensions to MiniWoB, WebArena, and WorkArena.New tasks use setup(), teardown(), validate(), and optionally cheat(), while the platform supports adding benchmarks and agents.
5. Experiments
The experiments evaluate zero-shot web agents across benchmarks and agent configurations, showing strong model- and task-dependent differences. WorkArena remains difficult, while reasoning and carefully selected interface features materially affect performance.
- Experimental protocol: The study evaluates a chain-of-thought web agent across MiniWoB, WebArena, and WorkArena while varying the underlying LLM and BrowserGym features.The experiments compare state-of-the-art general-purpose LLMs and ablate observation, action, history, prompting, and parsing features.
- Results: GPT-4o achieves 43% success on WorkArena, compared with 6.1% for GPT-3.5 and 17.9% for Llama3.GPT-4o also dominates GPT-3.5 and Llama3 across all evaluated benchmarks, with larger disparities on WorkArena and WebArena than on MiniWoB.
- Results: WorkArena is especially challenging because realistic enterprise interfaces create long contexts and non-trivial interactions despite high-level tasks being simple.List-based tasks using a non-standard HTML widget reach 0% success for every evaluated LLM, although humans can complete them relatively easily.
- Results: The GPT-4o agent reaches 82.9% on the WebGum subset and 23.5% on WebArena, exceeding prior reported zero-shot web-agent results.The WebArena result exceeds the 14.4% success rate reported for GPT-4 in the original WebArena paper.
- Ablation study: Chain-of-thought prompting is crucial: removing it reduces Llama3 performance by 10 points on both MiniWoB and WorkArena.This detrimental effect is reported consistently across all three evaluated agents.
- Ablation study: Adding features can overwhelm weaker LLMs, while think history can preserve erroneous early decisions and reduce self-correction.Longer prompts increase truncation for GPT-3.5 and Llama3; think history is most useful when tasks require information to persist across pages.
- Ablation study: Screenshot augmentation produces only minor gains for GPT-4o-V on MiniWoB and WebArena, leaving multimodal web-task performance limited.The authors suggest that current vision-language models may lack sufficient screen-related training data.
- Ablation study: Coordinate features improve GPT-4o performance on MiniWoB but do not improve WorkArena, where the bid-only action set performs best.The contrast suggests that precise 2D interaction is needed for specific tasks but not for most WorkArena interactions.
6. Conclusion
The paper introduces WorkArena and BrowserGym to evaluate web agents on realistic enterprise tasks, finding that enterprise interfaces remain challenging and that BrowserGym supports unified, multimodal evaluation.
- WorkArena evaluates web agents on tasks inspired by knowledge workers’ day-to-day workflows in ServiceNow.
- BrowserGym provides a general-purpose environment combining HTML, accessibility trees, screenshots, set-of-mark observations, code, high-level actions, and interactive chat.
- BrowserGym supports unified evaluation across MiniWoB, WebArena, and WorkArena.
- WorkArena remains an unsolved benchmark requiring advanced reasoning over long HTML or accessibility-tree contexts.
- Future work will expand BrowserGym and WorkArena with additional benchmarks and compositional workflows requiring retrieval, memorization, visual perception, and advanced reasoning.
Impact Statement
The paper discusses potential productivity and accessibility benefits of web agents alongside risks involving labor displacement, cybersecurity, privacy, and environmental costs.
- Positive impacts: Web agents could automate repetitive work, freeing employees for complex problem-solving and creative tasks.
- Positive impacts: Web agents could improve digital accessibility and open employment opportunities for people previously excluded by disabilities such as visual impairments.
- Negative impacts: WorkArena may help forecast which job roles are more exposed to automation and inform preventive measures such as reskilling.
- Negative impacts: Human-like web agents may enable more elaborate cyberattacks, motivating security measures such as constrained language models.
- Negative impacts: Workplace deployment raises privacy concerns because web agents may need to transmit sensitive information, while extensive computation carries energy costs.
A.1. Tasks
The appendix describes WorkArena’s task inventory and explains that task instances are parameterized, with list and form tasks capped because of combinatorial growth.
- WorkArena tasks are grouped by category in a complete task list.
- An instance represents one parameterized instantiation of a task, such as values entered into a specific field.
- List and form tasks are capped at 1,000 randomly selected instances because their combinatorial parameter spaces are exceedingly large.
A.2. Task User Interface Examples
The task examples illustrate explicit natural-language goals requiring agents to navigate enterprise interfaces, manipulate controls, retrieve information, and submit results for validation.
- FilterIncidentList requires opening the filter menu, adding conditions, and applying them with the Run button.
- KnowledgeBaseSearch requires searching, browsing resulting articles, extracting the requested information, and returning it through the chat box.
- OrderDeveloperLaptopMac requires navigating the service catalog, selecting configuration and quantity, and submitting the order.
- MultiChartValueRetrieval requires locating the relevant plot, finding a label, retrieving its count or percentage, and reporting it in the requested format.
- AllMenu requires searching or scrolling through the All menu and selecting the correct module among potentially similar items.
A.3. Knowledge Base Tasks – Additional Details
WorkArena’s knowledge-base tasks are built from generated articles, paraphrased questions, and validated answer variants. The construction uses GPT models while preserving exact facts and allowing controlled formatting variation.
- Task definition: The task asks agents to search the company knowledge base for specific information and answer a given question.The section describes knowledge-base generation, question and answer production, and validation as the task’s supporting pipeline.
- Generating the knowledge base: The knowledge base contains 100 GPT-4-generated HTML articles derived from item–value facts.Each article includes the exact string “the {fact} is {item}”; Figure 10 illustrates one generated article.
- Generating questions: Each fact receives ten GPT-4-generated question alternatives whose answers are constrained to the fact’s value.GPT-3.5 is then prompted with the article and each question to verify that every question is answered correctly.
- Answer validation: Agent answers are accepted against a set of GPT-4-generated alternative formats rather than requiring one exact string.The alternatives are inspected for coherence, with examples including abbreviated and expanded address formats.
B.1. Action Space
BrowserGym exposes a complete action space for browser interaction and adapts MiniWoB tasks so their goals are available in a separate chat window.
- Action space: BrowserGym provides a complete action space for agent interaction.The supplied passage identifies Table 10 as the complete action space, but does not enumerate its actions.
- MiniWoB integration: BrowserGym ports MiniWoB tasks while moving each task goal from the webpage HTML into a separate chat window.A minimal JavaScript snippet extracts the goal during setup and places it in the chat.
- MiniWoB integration: The MiniWoB adaptation removes the benchmark’s hard time limit per episode.The supplied passage describes this as an additional change beyond relocating the task goal.
BrowserGym
The supplied BrowserGym materials include a rendered MiniWoB task example and an instruction fragment ending with pressing submit.
- Task interaction: The task instruction includes a final step to press submit.The supplied text fragment begins with “two,” so only the submission step is explicit.
- MiniWoB rendering: Figure 11 presents an example of a MiniWoB task as rendered in BrowserGym.The passage identifies the figure but does not describe its visual encoding or task outcome.
- BrowserGym: The supplied passages connect BrowserGym with MiniWoB task rendering and browser-task submission interactions.They provide an example figure and a short instruction fragment rather than a fuller task specification.