WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Setting

• LLMs’ inability to take actions limits their capabilities, while tool-using autonomous agents can perform actions such as replying to emails and booking meetings.Source: §1 · p. 2
• WorkBench targets workplace activities that existing benchmarks do not represent, including tasks such as sending emails and reviewing a calendar before acting.Source: §1 · p. 2
• Outcome-centric evaluation compares an agent’s resulting environment state with each task’s unique, unambiguous ground-truth outcome.This enables robust and automatic evaluation of action-taking agents.Source: §1 · p. 2
• 43% of WorkBench tasks are completed correctly by the state-of-the-art ReAct agent with GPT-4.The benchmark evaluates five agents and finds workplace tasks challenging.Source: §1 · p. 2
• 690 unique tasks are created by combining human-curated templates with programmatic task generation.Source: §1 · p. 2
• WorkBench contributes an open-source dataset, an outcome-centric benchmarking methodology, and benchmark results on current agents.Source: §1 · p. 2

Tool Use Methods

• ReAct-inspired methods decompose user intent into a task, plan, and tool-execution sequence, with GPT-4 showing effectiveness across multiple tool-use domains.Source: §2 · p. 2
• Prompt demonstrations, tool documentation, tool embeddings, and explicit API-call planning are proposed methods for improving tool selection and use.Source: §2 · p. 2

Tool Use Limitations

• Studies report that smaller language models often struggle with multi-step tool use and tool selection, whereas GPT-4 can make correct API calls without in-context examples.Source: §2 · p. 3

Tool Use Evaluation

• General capability datasets such as Big-Bench, MMLU, and HellaSwag are limited for tool-use evaluation because their questions can be answered without tools.Source: §2 · p. 3

• Existing tool-use benchmarks cover diverse tasks, but many focus on information retrieval, single actions, simulated environments, or answers that are not unique and unambiguous.Source: §2 · p. 3

Web Browsing Agents

• WorkBench extends related evaluation by focusing on workplace-relevant tasks and measuring unintended side effects outside the target outcome.Source: §2 · p. 3

3 Proposed Benchmark Dataset: WorkBench

• WorkBench evaluates workplace agents by comparing the updated state of five sandbox databases with each task’s unique, unambiguous ground-truth outcome.The benchmark avoids real APIs that require authentication or change over time.Source: §3 · p. 3, §3.4 · p. 6

3.1 Sandbox Environment

• The sandbox contains simulated Calendar, Email, website analytics, CRM, and project-management databases, with representative records for workplace interactions.The databases include 300 calendar events, 500 emails, 500 website visits, 200 customers, and 300 project-management tasks.Source: §3.1 · p. 4, Table 2 · p. 5

3.2 Task and outcome pairs

• The dataset combines single-domain and multi-domain templates into 690 varied tasks, including challenging requests that can require up to 12 actions across domains.Programmatic generation creates 10 tasks per template, with three linguistic phrasings for each template.Source: §3.2 · p. 5, Table 3 · p. 6, Figure 4 · p. 6

3.3 Task Execution

• Agents execute tasks with 26 documented tools across five domains, using observations from tool calls to determine whether further actions are needed.Tool documentation specifies descriptions, parameters, return values, examples, and limits such as a maximum of five returned events.Source: Figure 2 · p. 4, §3.3 · p. 6, Table 4 · p. 7

3.4 Outcome-Centric Evaluation

• Outcome-centric evaluation accepts any action sequence that produces the ground-truth database state, so agents can recover from erroneous calls without being penalized for extra steps.This enables evaluation of action-taking tasks beyond prior benchmarks limited to information retrieval.Source: §3.4 · pp. 6–7, Figure 5 · p. 7

4.2 Comparing Large Language Models

• GPT-4 greatly outperforms GPT-3.5, Claude-2, Llama2-70B, and Mixtral-8x7B on WorkBench.The weaker models commonly suffer from insufficient context windows or failure to follow the ReAct framework.Source: §4.2 · p. 8, Table 5 · p. 8
• 43% accuracy is achieved by GPT-4 with all 26 tools, while restricting it to required toolkits raises accuracy to 49%.The benchmark remains challenging for all evaluated models, and redundant tools appear to reduce performance.Source: §4.2 · p. 8, Table 5 · p. 8

4.3 Performance across domains

• GPT-4 accuracy ranges from 23% on CRM tasks to 65% on Calendar tasks, with 40% accuracy on tasks spanning multiple domains.Cross-domain performance is similar to the 43% average on single-domain tasks.Source: §4.3 · p. 8, Table 6 · p. 8

4.4 Sources of Error

• Agents can produce side effects by modifying the environment incorrectly, such as cancelling the wrong meeting instead of the ground-truth event.The example predicts deletion of event 00000196 instead of event 00000035.Source: §4.4 · p. 9
• Errors without side effects occur when agents fail to complete tasks without modifying the environment, such as searching the wrong date for meetings.In the example, no action is taken after searching 28 November instead of Tuesday 21 November.Source: §4.4 · p. 9
• The most frequent error is failing to produce the required ACTION keyword and JSON tool call, so no action is performed.This failure motivates resampling up to five times with a nonzero temperature.Source: §4.4 · p. 9
• Resampling increases GPT-4 accuracy from 43% to 49% when the agent fails to generate the required ACTION keyword.Other errors cannot be fixed by resampling, including hallucinated email addresses and failures to use search tools.Source: §4.4 · p. 9
• Agents also fail when searches return limited results, because they do not repeat searches and deletions to cover all matching future events.Incorrect date searches create additional failures on meeting-cancellation tasks.Source: §4.4 · p. 9

5 Discussion and Future Work

• The sandbox may underrepresent real-world complexity because actual inboxes contain many long, noisy, and erroneous emails, potentially overestimating agent capabilities.The authors propose adding more challenging sandbox data.Source: §5 · p. 10
• WorkBench uses single-turn chat despite multi-action tasks, while multi-turn interaction may better represent realistic workplace tasks.A human baseline could clarify task significance but would also measure GUI quality.Source: §5 · p. 10
• Agent performance dropped as the number of available tools increased, but this relationship was not fully explored with 26 tools.Adding tools from domains such as HR could support broader analysis of accuracy versus tooling breadth.Source: §5 · p. 10
• WorkBench does not assess pure retrieval tasks, although retrieval tools are intermediate steps in many benchmarked tasks.Future work could add dedicated retrieval evaluation.Source: §5 · p. 10
• 43% accuracy was achieved by the best agent, with errors including failed plans, incorrect tool arguments, and misunderstanding tool limits.These errors can have negative consequences, such as sending emails to the wrong people.Source: §5 · p. 10
• WorkBench offers a large volume of high-quality, unique tasks and can be extended to new domains through its scalable task-creation approach.The authors specifically mention adding accounting tasks and progressively more complex settings.Source: §5 · p. 10

A.1 Simulated database creation

• The paper presents five sample rows from each of the five sandbox databases.Source: §A.1 · p. 14

A.1.1 Analytics

• Analytics records each visit’s date, visitor ID, page views, session duration, traffic source, and engagement status.Source: §A.1.1 · p. 14

A.2 Example tasks

• WorkBench includes sampled analytics, calendar, CRM, email, project-management, and multi-domain tasks requiring conditional, coordinated workplace actions.Examples include plotting analytics, scheduling or canceling meetings, reassigning CRM records, handling email, updating tasks, and combining checks with follow-up actions.Source: §A.2 · p. 16, §A.2.1 · p. 16, §A.2.2 · p. 16

A.3 Template dependence

• Three linguistic templates per task introduce variation, while GPT-4 accuracy ranges from 10% to 90% for most templates.Figure 7 reports that only 15 of 69 templates have either 0% or 100% accuracy, indicating diverse tasks within templates.Source: §A.3 · p. 17, Figure 7 · p. 18

A.4 Impact of “no action” tasks

• 122 of 690 WorkBench tasks require no actions, such as checking whether a meeting condition is met before deciding not to schedule.The agent can use calendar.search events to determine that the condition is unmet, and Table 7 compares accuracy across subsets grouped by required actions.Source: §A.4 · p. 17, Table 7 · p. 18
• Agents perform better on tasks requiring no actions, although the comparison is affected because all tools cannot fit in the GPT3.5 and Llama2-70B context windows.Table 7 evaluates accuracy by the number of actions required and notes that all tools are provided in the prompt.Source: Table 7 · p. 18

A.5 Prompt and tool descriptions

• The full prompt provides the date and current time, requires meetings to start no earlier than 9am and end no later than 6pm, and lists available tools.Listed tools include searching emails across subject, body, or sender fields and deleting an email by ID.Source: §A.5 · pp. 18–20
• The accompanying analytics tools count engaged users, visitor traffic, and total visits over date ranges, retrieve visitor records, and calculate average session duration.Analytics queries use YYYY-MM-DD date bounds; visitor records include fields such as visit date, page views, session duration, traffic source, and engagement status.Source: §A.5 · pp. 25, 26, 27, 29
• Analytics data can also be plotted for selected values using bar, line, scatter, or histogram plots, with the tool returning the generated plot-file path.Supported plotted values include total visits, session duration, user engagement, and traffic-source categories.Source: §A.5 · p. 28
• Project management supports task lookup by ID and field, searching by task attributes, and creation, deletion, or updates using structured parameters.Search filters include task name, assignee email, list, due date, and board; task updates can modify task name, assignee, list, due date, or board.Source: §A.5 · pp. 29–32
• Task retrieval returns selected fields such as task name, assignee email, list, due date, or board, while task creation returns an 8-digit task ID.Deletion and update operations return messages indicating whether the requested operation succeeded.Source: §A.5 · pp. 29, 31
• The customer relationship manager supports searching customers with identity, status, assignment, product-interest, contact-date, and follow-up-date filters, returning at most 5 records.Search results include customer information for the supplied parameters.Source: §A.5 · pp. 33, 34, 35, 37
• CRM records can be updated, added, or deleted by customer ID, with supported fields, constrained status and product-interest values, and success or identifier outputs.Adding a customer accepts contact, assignment, status, product, notes, and follow-up information; updating changes one specified field.Source: §A.5 · pp. 34–36
• The agent must emit one JSON action at a time, repeat Thought/Action/Observation cycles as needed, and use Final Answer when it knows the response.Valid actions include email, calendar, analytics, project-management, CRM, and directory tools, with action_input specifying tool inputs or the final response.Source: §A.5 · pp. 37–38