WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks

Motivation

• WorkArena++ addresses monotonous, low-value enterprise tasks as interest grows in agents that can interact directly with software UIs and websites.The introduction situates this motivation alongside LLM advances in reasoning and planning.Source: §1 · pp. 1–2

Benchmark

• WorkArena++ expands WorkArena from 33 to 682 tasks with realistic office-worker trajectories requiring complex skills, addressing the limits of predominantly atomic tasks.The paper describes it as the first web-agent benchmark requiring such complex skills.Source: §1 · p. 2

Technical contributions

• The benchmark adds visual diversity, task isolation, compositional task-building tools, and Oracle-based observation-action traces for fine-tuning.Its visual diversity includes 10 fictitious companies with customized interface styles.Source: §1 · p. 2

Empirical study

• Current agents generally struggle on WorkArena++, while humans perform reasonably well, supporting its value as an evaluation benchmark.The study includes state-of-the-art closed- and open-source LLM- and VLM-based agents and human workers.Source: §1 · p. 2

2.1 BrowserGym – A Gym Environment for Web Agents

• BrowserGym standardizes web-agent interaction through chat-based goals, multimodal observations, and a flexible action space.Observations can include HTML, accessibility trees, screenshots, set-of-marks, and element coordinates.Source: §2.1 · p. 3

2.2 WorkArena

• WorkArena tasks use ServiceNow UI components but do not require complex problem-solving, often giving agents explicit values to enter into a form.The paper characterizes these tasks as simplistic for humans.Source: §2.2 · p. 3
• WorkArena completion is certifiable through a human-coded oracle and a validator that checks task success; WorkArena++ extends these mechanisms to more complex tasks.Validation can inspect database state and the current page.Source: §2.2 · p. 3, Figure 3 · p. 3
• WorkArena++ adds L2 and L3 to WorkArena’s L1 tasks, with increased difficulty in L3 because its instructions are less explicit and more realistic.L2 and L3 cover the same workflows, presented as explicit or implicit goals through ServiceNow tickets.Source: §2.2 · p. 4

3 WorkArena++: Taking WorkArena to the Next Levels

• WorkArena++ composes simpler atomic tasks into realistic workflows, mostly reusing L1 tasks while adding new atoms such as editing existing records.Source: §3 · p. 4, §3.1 · p. 4

3.1 Difficulty Levels L2 and L3

• The same 341 workflows yield 682 tasks: L2 spells out steps in chat, while L3 uses tickets and knowledge-base instructions, requiring cross-source consolidation.Source: §3.1 · pp. 4–5

3.2 Skills and Abilities Evaluated in WorkArena++

• Five categories assess constrained planning, dashboard or list retrieval, data-driven logical or mathematical decisions, multi-page memorization, and recognizing or justifying infeasible tasks.Source: §3.2 · pp. 5–6

3.3 Salient Features of WorkArena++

• Randomly sampled company brands vary interface colors and logos, adding visual diversity intended to better assess generalization across enterprise settings.Source: §3.3 · p. 6
• An installer standardizes system parameters, themes, knowledge bases, and layouts, while per-task user accounts isolate changes and are deleted afterward.Source: §3.3 · p. 6
• Composed oracle and validator functions provide human-coded ground truths for complex tasks, and the framework collects observation-action traces for fine-tuning.Source: §3.3 · p. 6

4.1 Evaluation Curriculum: Standardizing WorkArena++ as a Benchmark

• The evaluation curriculum uses numerical seeds to sample any number of task instances uniformly across skills, reserving seeds 0–9 for evaluation.The procedure is intended to make evaluation reproducible and standardize skill coverage.Source: §4.1 · p. 7

4.2 Agent Design

• Agents select one BrowserGym action at a time using browser observations, including goals, accessibility trees, prior action history, and screenshots for VLMs.The study evaluates GPT-3.5, GPT-4o, Llama3-70b, and Mixtral-8x22b; outputs can be retried up to four times after parsing errors.Source: §4.2 · p. 7

4.3 Agent Results

• Agents generally achieved little success; GPT-4o outperformed GPT-3.5, GPT-4o-v solved some chart-reading retrieval tasks that GPT-4o did not, and no agent succeeded on L3 tasks.GPT-4o and GPT-4o-v succeeded on some L2 memorization tasks.Source: §4.3 · pp. 7–8

4.4 Human Evaluation

• Humans achieved 93.9% success versus GPT-4o’s 2.1% on the 98-instance human curriculum completed by 15 participants.Participants had little or no ServiceNow familiarity and received a 15-minute video and 15 minutes of self-guided platform exploration.Source: §4.4 · p. 8

4.5 Error Analysis

• Agents may reach relevant information sources but misretrieve values or target wrong elements; hidden tabs and folded sections also challenge exploration.Source: §4.5 · pp. 8–9
• Other errors include invented actions, reasoning–action mismatches, incomplete L3 subtask understanding, false progress beliefs, and repeated unchanged actions.Source: §4.5 · p. 9

• Earlier web-agent benchmarks used synthetic environments, while later work introduced realistic websites and scalable operating-system environments.Source: §5 · p. 9
• Prior web-interaction datasets include human-annotated or expert demonstrations, and WorkArena introduced 33 enterprise workflow tasks.Source: §5 · p. 9
• WorkArena++ adds complex, multi-step enterprise tasks requiring skills such as problem-solving and data-driven decision-making, with task isolation and standardized evaluation.The paper describes it as unique in evaluating LLM- and VLM-based web agents on these capabilities.Source: §5 · p. 9

• WorkArena++ comprises 682 realistic knowledge-work tasks evaluating planning, decision-making, retrieval, reasoning, and infeasible-task identification.The benchmark also supports standardized evaluation, visual diversity, and generation of web-interaction traces for fine-tuning.Source: §6 · p. 9
• Empirical evaluations find that state-of-the-art LLMs and VLMs struggle on WorkArena++, while humans achieve extremely high performance.Source: §6 · p. 9
• Future work aims to add ServiceNow tasks, including safety and cybersecurity assessments and a hidden test set, while using generated traces as fine-tuning data.The stated long-term goal is to narrow the gap between autonomous agents and humans on WorkArena++ and other benchmarks.Source: §6 · p. 10

7 Limitations and Potential Societal Impacts

• WorkArena++ does not exhaustively cover knowledge-work tasks or personas, excludes tasks requiring external software, and does not evaluate safety or robustness against malicious behaviors.The authors note that comprehensive task coverage would require thousands of additional tasks.Source: §7 · p. 10
• Potential benefits include productivity and accessibility gains, while potential harms include job displacement, privacy and security concerns, erosion of human skills, and energy use.Source: §7 · p. 10

A.1 Participants

• The 15 human-evaluation volunteers included ServiceNow research-team members and students from local institutions, with varied familiarity with ServiceNow.Source: §A.1 · p. 13

A.2 Protocol

• The in-person protocol included a 15-minute recorded training, 15 minutes of self-exploration, and up to seven individual tasks per participant.Participants were instructed not to discuss tasks; time to resolution and success rate were recorded.Source: §A.2 · p. 14

A.3 Task curriculum

• The human study used 98 task instances sampled uniformly across skills and both L2/L3 variants.Source: §A.3 · p. 14

A.4 Evaluation platform

• Humans used a UI matching the agents’ BrowserGym interface, with a console for validation and giving up; they received no progress feedback beyond incomplete or solved status.Source: §A.4 · p. 14

A.5 Limitations

• The human study did not directly account for participants becoming more efficient as they learned the interface; only three evaluators repeated a task.Repeated tasks were presented at different difficulty levels with different seeds.Source: §A.5 · p. 14
• The evaluator cohort may not represent typical ServiceNow users: about 75% had worked at ServiceNow, and all held undergraduate degrees.The paper notes that employment did not necessarily mean prior product interaction.Source: §A.5 · p. 16

B Agent Design

• The agent receives the goal, page HTML and/or AXTree, focused element, and any previous-action error, with clickable and visible flags.GPT-4o vision also receives a screenshot annotated with element identifiers.Source: B Agent Design · p. 18
• The agent uses chat, infeasibility-reporting, and element-interaction actions; its prompt includes a generic zero-shot example and the episode’s prior actions and thoughts.Reinjected history is intended to help with memory-intensive tasks.Source: B Agent Design · p. 18

C.1 Planning and Problem Solving (66 × 2 tasks)

• Planning tasks require constraint-aware global plans, including balancing workloads, assigning problems by expertise, scheduling requests, and deduplicating problems.Scheduling can involve time windows, overlaps, impact, criticality, and risk.Source: §C.1 · p. 21, §C.1.1 · p. 21, §C.1.2 · p. 21

C.2 Information Retrieval (39 × 2 tasks)

• Retrieval tasks ask agents to extract information from charts, forms, or filtered lists and then use it to answer questions or perform follow-up actions.Examples include chart-based stock ordering and extracting information from a form.Source: §C.2 · p. 23, §C.2.1 · p. 23, §C.2.2 · p. 23

C.3 Data-driven decision making and reasoning (156 × 2 tasks)

• Decision-making tasks cover duplicate-expense deletion, budget-constrained investment selection, and calculations over charts used to make subsequent decisions.Chart calculations include mean, median, and mode.Source: §C.3 · p. 23, §C.3.1 · p. 24, §C.3.2 · p. 24

C.4 Sophisticated Memorization (29 × 2 tasks)

• Memorization tasks require retaining details across navigation and multi-step workflows, including onboarding users with profiles, devices, and assignments or offboarding them.Navigation tasks also use remembered details to sort, filter, fill forms, or order items.Source: §C.4 · p. 25, §C.4.1 · p. 25, §C.4.2 · p. 25

C.5 Contextual understanding through infeasible tasks (52 × 2 tasks)

• Agents must identify impossible requests, such as a nonexistent form field or invalid item configuration, and notify the user through chat.The task examples include both L2 goals and L3 descriptions.Source: §C.5 · p. 25

C.6 Task Protocols

• The L3 protocols are available in the agent’s knowledge base, approximating knowledge workers’ use of company documents.The appendix presents protocols for the benchmark workflows.Source: §C.6 · p. 26

D Evaluation Curriculum Details

• Tasks are grouped into skill categories and buckets of similar required abilities, then sampled with weights to broaden skill coverage and limit redundancy.A meta-seed produces reproducible task instances across categories.Source: D Evaluation Curriculum Details · p. 37
• The curriculum yields 235 tasks for each of the L2 and L3 difficulty levels; seeds 0–9 are reserved for evaluation, while remaining seeds may be used for agent tuning.Source: D Evaluation Curriculum Details · p. 37

E.1 Application Themes

• Ten fictitious-company themes were created to add visual diversity.The appendix shows the themes individually.Source: §E.1 · p. 39

E.2 Extracting traces and designing more tasks

• Oracle functions solve tasks step by step, while BrowserGym captures page observations and the Playwright actions to create solution traces.Captured observations include screenshots, the AXTree, and DOM information.Source: §E.2 · p. 45, Figure 44 · p. 45
• Existing tasks can be chained into workflows, with validation performed sequentially or through a global database query.A workflow can be extended by adding another task, such as creating a task assigned to a newly onboarded user.Source: §E.2 · pp. 45–46

Information Retrieval

• Agents sometimes misread available page information, including repeatedly sorting a column already marked as ascending and failing to retrieve relevant values accurately.Confusion over element identifiers can also trigger an error or click a random page element.Source: Information Retrieval · pp. 47–48

Exploration

• Folded menus and tabbed forms can hide needed elements, and agents may fail to find them or overwrite a completed field instead of switching tabs.In one form example, the agent overwrote existing content rather than opening the tab containing the remaining field.Source: Exploration · pp. 48–49

Hallucinations - Made Up Actions

• Agents sometimes invent unavailable actions or buttons, including unsupported functions and shortcuts that replace element identifiers with target names.These fabricated interactions can cause errors rather than completing the intended task.Source: Hallucinations - Made Up Actions · p. 49, Hallucinations - Imaginary Buttons · pp. 50–51

Hallucination - Asking for help

• When confused, agents may falsely conclude a task is complete and ask the user for confirmation instead of taking the remaining actions.In the example, the agent treats a state update as task completion and sends a confirmation request.Source: Hallucination - Asking for help · p. 51

Goal Understanding - Thought/Action Consistency

• An agent can state the correct goal yet order an iPad case instead of the requested iPad Mini, causing the run to fail.WorkArena considers the order the end of the task.Source: Goal Understanding - Thought/Action… · p. 52

Goal Understanding - Low-Level Understanding in L3

• In L3 tasks, agents may mistake ticket information for completed work and close the ticket rather than carry out the task.The example agent interprets a hardware-asset task as already completed and changes its state to Closed Complete.Source: Goal Understanding - Low-Level… · pp. 52–53

Action Consequences Assessment - Hallucinated Consequences

• After entering information in the wrong form field, agents may believe the task is complete and proceed without correcting the mistake.One example has the agent overwrite the Serial Number with Cost center information, then continue as if the field were correct.Source: Action Consequences Assessment -… · p. 53

Action Consequences Assessment - Repeated Actions

• Agents can repeat an ineffective action until an episode ends, such as repeatedly clicking a search box that does not change the page.Missing information can also prompt aimless repeated scrolling.Source: Action Consequences Assessment -… · pp. 53–54

G.1 Limitation: Restricted to ServiceNow

• WorkArena++ tasks are confined to ServiceNow, while the authors identify multi-environment tasks as a future direction for a broader benchmark.They note that other environments could be reached by, for example, replacing the built-in knowledge base used in L3 tasks with an external one.Source: §G.1 · p. 55
• The authors expect insights to generalize because the benchmark uses common enterprise UI components and most analyzed failures are not ServiceNow-specific.They also characterize the evaluated reasoning and planning skills as largely independent of the specific interface.Source: §G.1 · p. 55

G.2 Expanding the Benchmark – Ensuring Long-Term Relevance

• WorkArena++'s modular design supports adding tasks by composing atomic tasks, with longer workflows, collaboration, and multi-software interactions identified as expansion directions.New atomic tasks involve setup, teardown, oracle, and validation functions.Source: §G.2 · pp. 55–56

G.3 Context-size sensitivity

• Llama3 performance on WorkArena-L1 was very similar across 130k, 64k, 32k, and 13k context lengths, but the authors caution against extending this result to more complex tasks.They note that more complex tasks will likely require greater memory and prompt detail.Source: §G.3 · p. 56