WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?

• UI assistants are envisioned to ease complex, repetitive tasks accessibly, while offering inspectable, user-controlled automation from partial help to full execution.Source: §1 · p. 1
• Enterprise software remains an underexplored setting for web agents, despite repetitive work and interfaces that prioritize functionality over user experience.The work investigates agents' potential to improve accessibility, user experience, and worker productivity in enterprise settings.Source: §1 · p. 1
• WorkArena benchmarks 33 common ServiceNow web tasks across 19,912 unique task instances.Source: Figure 1 · p. 2, §1 · p. 2
• BrowserGym provides web-agent development and evaluation with multimodal observations, broader actions, and chat-based interactions.Source: §1 · p. 2
• 25.4% versus 14.4%: the authors report that BrowserGym features helped their GPT-4 agent top the WebArena leaderboard, compared with the original paper's score.Source: §1 · p. 2

Web-agent benchmarks

• Earlier benchmarks used synthetic low-level actions: MiniWoB offers 125 toy tasks, while WebShop (Yao et al., 2022) simulates e-commerce search and browsing.Tasks range from clicking a button to using a basic text editor.Source: §2 · p. 2
• WebArena (Zhou et al., 2023) contains 190 realistic-site tasks and reports 14% success for a GPT-4 state-of-the-art agent versus 78% for human agents.Its sites emulate domains including e-commerce, social forums, collaborative software development, and content management.Source: §2 · p. 2
• Min2Web (Deng et al., 2023) and WebLINX (L`u et al., 2024) curate cross-site interactions, while He et al. (2024) use 300 consumer-site retrieval tasks to evaluate WebVoyager.Min2Web includes 2,000 interactions from 137 websites; WebLINX has 2,337 demonstrations from 155 websites and averages 43 interactions per task.Source: §2 · p. 2

Related mobile benchmarks

• Mobile UI agent benchmarks include PixelHelp (Li et al., 2020), Android in the Wild, and Macro Mining (Huang et al., 2024).Source: §2 · p. 2

WorkArena and BrowserGym

• WorkArena targets real-world enterprise software and end-to-end knowledge-worker workflows, including challenges such as large DOMs, non-standard HTML, and complex UI elements.It is designed to complement existing benchmarks.Source: §2 · p. 2
• BrowserGym combines prior observation and action features and is the first environment described here to support chat-based agent-user interactions.It supports multimodal observations and code-based actions, and is flexible across interaction spaces used in prior research.Source: §2 · p. 3

LLM-based agents

• Prior LLM web agents used HTML inputs and high-level actions, while other work explored multimodal text-and-screenshot observations and Python-generated website actions.Examples include WebGPT (Nakano et al., 2021), HTML-based agents (Deng et al., 2023; Liu et al., 2023a;b; Yao et al., 2023), multimodal agents (He et al., 2024), and code-based methods (Gur et al., 2023a;b).Source: §2 · p. 3

3. WorkArena – An Enterprise Benchmark

• WorkArena covers 33 tasks and 19,912 instances of common ServiceNow work, and is open-source, extensible, and independent of proprietary code.Tasks run on real cloud-based ServiceNow Personal Developer Instances, available free through the developer program.Source: §3 · p. 3, §3.3 · p. 5

3.1. WorkArena Tasks

• Each task pairs an explicit natural-language goal with validation feedback and a Playwright oracle that tests feasibility, provides ground truth, and supports updates.Goals are generated from human-designed templates with predefined values; validators flag errors, while oracle functions automatically complete tasks.Source: §3.1 · p. 3
• List tasks cover filtering and sorting across 6,900 instances, while five form tasks test complex entry creation across 5,000 instances.Lists require constructing filters with 1 to 5 conditions or sorting by up to 3 columns; forms require filling 1 to 26 fields, sometimes hidden or dynamically completed.Source: §3.1 · pp. 3–4
• Knowledge-base search yields 1,000 instances, while catalog-ordering tasks yield 3,550 instances and require selecting items with specified amounts and configurations.Knowledge answers are checked against acceptable alternatives; catalog orders are validated against the requested items, quantities, and specifications.Source: §3.1 · p. 4
• Dashboard retrieval covers 1,862 instances, and menu navigation or user impersonation covers 1,600 instances, with validation checking chart answers or the resulting location and login.Dashboard tasks may involve extracting numbers from multiple charts and simple reasoning; menu tasks navigate to an application or log in as a specified user.Source: §3.1 · p. 4

3.2. Challenges: the World Wild Web of Work

• ServiceNow challenges UI assistants with dynamic, non-standard interfaces and HTML structures, including nested iFrames, shadow DOMs, proprietary APIs, and DOMs spanning 40k–500k tokens.Fields may become required or hidden based on other values, and finding an element can require long-context understanding of very large pages.Source: §3.2 · p. 5

4. BrowserGym

• BrowserGym provides a generic environment for designing benchmarks and evaluating multimodal web agents.It is implemented as an OpenAI Gym environment following a POMDP paradigm; observations are based only on the current page view, so agents must supply their own memory.Source: §4 · p. 5

4.1. Capabilities

• Agents can interact through chat, including sequential user instructions, and use unique element identifiers and visibility, clickability, and bounding-box attributes.The chat interface supports both answer-seeking information retrieval and tasks whose instructions change over time.Source: §4.1 · p. 5
• Observations combine chat and page context with DOM and accessibility-tree structures, error traces, and a viewport screenshot.Structured observations are enriched with element attributes and can be rendered as text for language models or combined with screenshots for vision-language models.Source: §4.1 · p. 5
• The customizable action space ranges from high-level bid- or coordinate-based primitives to arbitrary Python, and navigation supports multiple pages, iFrames, and shadow DOMs.These features accommodate heterogeneous real-world websites and interaction needs.Source: §4.1 · pp. 5–6

4.2. An Ideal Experimental Framework

• BrowserGym leaves agents free to choose which observations, histories, and action spaces to use, enabling comparable experiments across text-, vision-, and memory-augmented agents.Researchers can evaluate varied agent designs on the same tasks.Source: §4.2 · p. 6
• New tasks require setup, teardown, and validation functions, with an optional oracle function that can solve tasks using a hard-coded Playwright solution.Validation returns a reward, optional chat message, and completion flag; an oracle can help assess task feasibility.Source: §4.2 · p. 6
• BrowserGym supports MiniWoB, WebArena, and WorkArena, and is designed to make adding further benchmarks straightforward.The authors encourage the community to contribute benchmarks and agents.Source: §4.2 · p. 6

5.1. Agent Design

• The agents use chain-of-thought prompting and configurable BrowserGym features for observations, actions, interaction history, and zero-shot examples.The feature flags vary observation content and coordinates, action types and descriptions, and whether previous actions, errors, or thoughts are retained.Source: §5.1 · pp. 6–7

5.2. Experimental Protocol

• The evaluation used up to 15 steps per episode, tuned each LLM’s configuration on MiniWoB and WorkArena, then re-evaluated it on all benchmarks with a different seed.Experiments used 10 seeds per MiniWoB and WorkArena task and one per WebArena task; success rates and standard errors came from stratified bootstrap samples.Source: §5.2 · p. 7

5.3. Results

• On WorkArena, GPT-4o achieved 43% success, versus 6.1% for GPT-3.5 and 17.9% for Llama3.The authors report GPT-4o dominance across all three benchmarks, with the largest performance disparity on WorkArena.Source: §5.3 · p. 7, Table 2 · p. 7
• WorkArena challenged all agents: none achieved 100% on any task, and every LLM scored 0% on list-based tasks involving a non-standard HTML widget.The authors attribute the difficulty to real-world software interfaces, long contexts, and non-trivial interactions, despite tasks being simple at a high level.Source: §5.3 · pp. 7–8
• GPT-4o reached 23.5% on WebArena, above the prior zero-shot GPT-4 result of 14.4%, and its 82.9% WebGum-subset result surpassed prior zero-shot web agents on MiniWoB.The authors also report that GPT-4o outperformed the other evaluated agents on MiniWoB and surpassed prior studies on the full MiniWoB benchmark.Source: Table 2 · p. 7, §5.3 · p. 8

5.4. Ablation Study

• Removing chain-of-thought substantially harmed all three agents; Llama3 lost 10 points on both MiniWoB and WorkArena.The ablation text describes this conclusion as clear across GPT-4o, GPT-3.5, and Llama3.Source: §5.4 · p. 8
• Adding features could degrade weaker models, while think history sometimes reduced GPT agents’ self-correction by reinforcing early, even erroneous, decisions.The authors hypothesize that long prompts and non-crucial features can distract agents; more features also increased prompt truncation for GPT-3.5 and Llama3. Think history was most useful for tasks requiring information to be carried between pages, unlike most evaluated tasks.Source: §5.4 · pp. 8–9
• Screenshot input produced only minor MiniWoB and WebArena gains, while coordinate features helped GPT-4o on MiniWoB but not WorkArena, where bid-only worked best.The authors note that 2D actions are needed for specific tasks such as drawing on a web whiteboard, but may be unnecessary for most web interactions.Source: §5.4 · p. 9

6. Conclusion

• WorkArena benchmarks web agents on ServiceNow tasks inspired by knowledge workers’ daily workflows and exposes the challenges of large observations and complex, non-standard interfaces.Source: §6 · p. 9
• BrowserGym unifies web-agent evaluation across MiniWoB, WebArena, and WorkArena, combining established observation and action features with interactive chat; both contributions are open source.The authors present the benchmark and environment as catalysts for developing and evaluating agents’ capabilities and potential real-world impact.Source: §6 · p. 9
• WorkArena remains an unsolved, challenging benchmark requiring advanced reasoning over long HTML or AXTree contexts, capabilities that seem to emerge only in very large models.The evaluation examined GPT-3.5, GPT-4o, and Llama3 across MiniWoB, WebArena, and WorkArena.Source: §6 · p. 9
• Future work plans to add standard benchmarks to BrowserGym and expand WorkArena with compositional workflows requiring retrieval, memorization, visual perception, and advanced reasoning.The planned benchmarks include WebShop and WebVoyager.Source: §6 · p. 9
• The authors frame browser-based task automation as a testbed for evaluating multimodal LLM capabilities because web observations are multimodal and web-task complexity can vary widely.They hope this work will stimulate further community progress.Source: §6 · p. 9

Impact Statement

• The work frames browser-based UI assistants as potentially disruptive to software interaction and digital work, and reflects on their societal benefits and risks.The authors call for the research community to consider these impacts carefully.Source: Impact Statement · p. 10, Negative impacts · p. 10

Positive impacts

• Automating repetitive tasks could boost worker productivity, streamline workflows, and free employees for complex problem-solving and creative work.The authors suggest this could enhance work quality and innovation.Source: Positive impacts · p. 10
• UI assistants could improve digital accessibility for people with disabilities, including visual impairments, and expand employment opportunities and the labor pool.The authors also suggest broader access could help mitigate labor shortages.Source: Positive impacts · p. 10

Negative impacts

• Labor displacement could disrupt jobs, though the authors expect roles to evolve rather than disappear; WorkArena may help forecast affected roles and guide preventive reskilling.They cite the translation industry’s disruption by automatic translation while it continues to thrive (van der Meer, 2021).Source: Negative impacts · p. 10
• Human-like agents could enable more elaborate cyberattacks, motivating security measures such as constrained language models, including GPT-4 (OpenAI, 2023).Source: Negative impacts · p. 10
• Workplace deployment of LLM-based agents raises privacy concerns because it requires transmitting sensitive information, warranting further security research.Source: Negative impacts · p. 10
• Extensive LLM inference consumes significant energy, posing environmental challenges that must be addressed before widespread implementation.Source: Negative impacts · p. 10

A.1. Tasks

• Table 6 groups WorkArena tasks by category, caps combinatorial list and form task instances at 1,000 randomly selected cases, and reports results averaged over 10 sampled instances with standard deviations.Oracle-action counts indicate task complexity, but human-coded oracle functions may not be optimal.Source: Table 6 · p. 13

A.2. Task User Interface Examples

• FilterIncidentList requires opening the filter menu, adding conditions, and running the completed filter.The example shows the natural-language goal and a three-step workflow.Source: Figure 5 · p. 14
• KnowledgeBaseSearch requires searching, reading the resulting articles for an answer, and returning it through the chat box for validation.The goal specifies both the question and expected answer format.Source: Figure 6 · p. 15
• OrderDeveloperLaptopMac requires navigating the service catalog, choosing a configuration and quantity, and submitting the order.The example ends with the “Order Now” action.Source: Figure 7 · p. 16
• MultiChartValueRetrieval requires locating a plot on the dashboard, retrieving a labeled value in the requested count or percentage format, and reporting it in chat.The chat response is used for validation.Source: Figure 8 · p. 16
• AllMenu requires opening the “All” menu, searching or scrolling for a module, and selecting the right item while avoiding similarly named “Open” modules.The example highlights the need for caution when choosing a menu item.Source: Figure 9 · p. 17

A.3. Knowledge Base Tasks – Additional Details

• The task asks agents to search a company knowledge base for information needed to answer a given question.The appendix describes how the knowledge base, task questions and answers, and validation process are produced.Source: §A.3 · p. 18
• The WorkArena knowledge base contains 100 GPT-4-generated HTML articles, each built from a fact and including the exact string “the {fact} is {item}”.Figure 10 illustrates a generated article with its fact highlighted.Source: §A.3 · p. 18, Figure 10 · p. 18
• For each fact, GPT-4 generates 10 alternative wordings of an initial question, with instructions that answers must exactly match the fact’s value.The initial prompt is “What is {item}?”Source: §A.3 · p. 18
• GPT-3.5 checks each generated question against its article; if it answers incorrectly, GPT-4 is asked to improve the question and the check is repeated.Source: §A.3 · p. 18
• The authors use GPT-3.5 for question checks to avoid GPT-4 favoring its own ambiguous questions and succeeding anyway.Source: §A.3 · p. 18
• Answer validation accepts slight formatting or wording variations by comparing agent answers with a set of alternatives to the expected value.Source: §A.3 · p. 18
• GPT-4 generates 10 alternative answer formats from each expected value, using examples in the prompt; the authors inspect the outputs for coherence.Source: §A.3 · p. 18

B.1. Action Space

• Table 10 presents BrowserGym’s complete action space.Source: Table 10 · p. 20

B.2. MiniWoB

• BrowserGym moves each MiniWoB task goal from page HTML into an agent-accessible chat window and removes the hard per-episode time limit.The port required minimal effort beyond these changes.Source: §B.2 · p. 20
• Figure 11 shows an example MiniWoB task rendered in BrowserGym.Source: Figure 11 · p. 21