OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automation

Motivation

• OFFICEBENCH targets realistic office workflows that integrate information across applications rather than stopping at document extraction.The paper motivates workflows such as extracting invoice data, integrating it into financial software, flagging discrepancies, and generating reminders.Source: §1 · pp. 1–2

Benchmark Contribution

• The benchmark requires agents to plan feasible operation chains, switch applications on time, and ground actions within a large combined action space.Its simulated environment includes Word, Excel, calendar, and email clients, among other office applications.Source: §1 · p. 2

Evaluation

• Customized exact, fuzzy, and execution-based evaluations assess agent outputs across diverse file formats and task requirements.These methods are tailored to individual test examples so correctness can be checked flexibly or through executable conditions.Source: §1 · p. 2

Results

• 47.00%: GPT-4 Omni achieves the highest reported pass rate, yet remains below the accuracy standards required by real-world office workflows.The benchmark evaluates GPT-3.5, GPT-4, Gemini, Llama 3, and Qwen 2 as language agents.Source: §1 · p. 2

Findings

• The benchmark identifies operation redundancy, hallucinated actions, and limitations in switching between applications and complex planning as major issues.These findings motivate continued development of more robust language-agent frameworks for office automation.Source: §1 · p. 2

Language Agent Benchmarks

• OFFICEBENCH is designed for real-world workflows requiring multiple software applications, unlike benchmarks focused on arithmetic, tool use, or single-interface interaction.Its tasks require agents to switch applications as needed while completing office workflows.Source: §2 · p. 2

Extensibility

• OFFICEBENCH provides an extensible, cost-effective evaluation platform that supports adding applications and Bash-compatible tasks with less manual effort than complex simulators.The comparison cites OSWorld’s approximately 1,800 human annotation hours.Source: §2 · p. 3

Document AI Benchmarks

• OFFICEBENCH goes beyond document extraction and question answering by evaluating agents that call different applications for general office automation.Document benchmarks typically assess understanding and extraction from invoices, receipts, forms, or tables.Source: §2 · p. 3

Benchmark Design

• The benchmark offers customized evaluation functions for individual tasks, supports a larger action space, and provides extensible annotation and environment creation.These capabilities are highlighted as distinguishing features for cross-application scenarios.Source: §2 · p. 3

3 OFFICEBENCH: Modeling Office Works Across Multiple Applications

• OFFICEBENCH simulates office work across applications including Word, Excel, PDF, Shell, email, and calendar, with agents constructing operation chains over a large valid action space.The environment is designed to test planning and reasoning in realistic multi-application scenarios.Source: §3 · p. 3, §3.1 · p. 4

3.1 Multi-Application Environment

• The environment contains 9 applications, each with specialized operations such as read_excel_file and set_cell_content, coordinated through a System application.System provides switch_app so agents can move between execution environments, such as Calendar and Email.Source: §3.1 · p. 4

3.2 Autonomous Workflow

• The workflow is a transition system whose state is the selected application, with restricted application-specific actions plus switch_app and submit.At each step, the agent selects among valid operations for the current application or changes applications or submits the task.Source: §3.2 · pp. 4–5
• Agents receive the full execution history and textual operation outputs as observations for predicting the next action.Structured results such as spreadsheet values are formatted with cell indices and contents for direct use in the agent’s history.Source: §3.1 · p. 4, §3.2 · p. 5
• Execution terminates normally through submit_task, or prematurely after five repeated operations or 50 total steps.These conditions classify stagnating or resource-exceeding runs as unsuccessful or prevent excessive resource use.Source: §3.2 · p. 5

3.3 Implementation Details

• The Docker implementation stores documents, emails, and calendar events in structured files, then saves the resulting file system for customized correctness evaluation.Python libraries automate operations over the simulated files and applications.Source: §3.3 · p. 5

4 Benchmark Annotation and Evaluation

• 300 tasks are organized into Single App, Two Apps, and Three Apps categories, with 93, 95, and 112 tasks respectively.The categories specify the number of involved applications, and difficulty increases as more applications are required.Source: §4 · p. 5, §4.1 · p. 5

4.1 Task Annotation

• Three-application tasks extend two-application workflows and require agents to manage dynamic switching across applications such as Excel, Calendar, and Email.For example, agents may read participants, create calendar events, and email training details.Source: §4.1 · pp. 5–6

4.2 Data Synthesis

• The benchmark synthesizes documents, emails, and calendar events with ChatGPT and random generators to avoid sensitive privacy issues while simulating realistic scenarios.HTML intermediates are converted into special formats such as images and PDFs before data are placed in the Docker environment.Source: §4.2 · p. 6

4.3 Evaluation Framework

• Exact matching compares final outputs with annotated ground truth, while fuzzy matching checks required properties when strict equality is inappropriate.Calendar evaluation can verify timestamps and required subject keywords while disregarding irrelevant details.Source: §4.3 · p. 6
• Execution-based evaluation runs code to verify valid outcomes when expected results are not unique, such as conflict-free meetings for mutually available participants.The evaluator checks both calendars and ensures that no overlapping commitments or time conflicts exist.Source: §4.3 · p. 6

5 Experiments

• OFFICEBENCH evaluates proprietary and open-weight agents on tasks grouped by whether they require one, two, or three applications.The experiments compare Gemini-1.0, Gemini-1.5, GPT-3.5, GPT-4, Llama 3, and Qwen 2, alongside human performance measurements.Source: §5 · p. 6, §5.1 · p. 7, Table 2 · p. 7
• Agents use end-to-end prompting to plan and execute workflows autonomously, with customized task-specific evaluation and pass rate as the final metric.The setup avoids manually selected demonstrations and evaluates whether each task outcome passes.Source: §5 · p. 7

5.1 Main Results

• 47.00% overall pass rate makes GPT-4 Omni the leading proprietary model, while Llama 3 leads open-weight models at 27.33%.Llama 3 also surpasses proprietary Gemini-1.5, indicating that open-weight models are not necessarily worse than proprietary models.Source: §5.1 · p. 7, Table 2 · p. 7
• GPT-4 Omni achieves only 21.43% on three-application tasks, showing a sharp decline as workflows require more applications.The paper links this difficulty to application-specific data formats and planning across applications.Source: §5.1 · p. 7, Table 2 · p. 7

5.2 Ablation Study for Application Switching

• OFFICEBENCH simulates complex office workflows in which agents switch between applications and construct operation chains step by step.The framework exposes application-specific actions through the switch_app operation.Source: §5.2 · pp. 7–8
• Application switching outperforms listing all operations, aided by concise instructions and an action space constrained to the current application.The ablation compares GPT-4 Omni and Llama 3 using the switch_app operation against vanilla prompts listing every operation.Source: Table 3 · p. 7, §5.2 · p. 8

5.3 Error Analysis

• Error analysis identifies repeated redundant operations, hallucinated actions, and complex-planning failures as typical agent failure modes.Figure 3 highlights these failure categories, while redundant read_file calls can cause operational stagnation.Source: §5.3 · p. 8, Figure 3 · p. 8
• Agents may hallucinate nonexistent actions even within a narrowed action space, producing non-executable commands and significant API or local inference costs.For multi-application workflows, agents may also attempt direct PDF edits instead of converting to Word, editing, and converting back.Source: §5.3 · p. 8

• OFFICEBENCH is one of the first office automation benchmarks for language agents, simulating a realistic execution environment.Source: §6 · p. 8
• The benchmark evaluates current powerful LLM agents on office tasks spanning different applications.Source: §6 · p. 8
• Application switching is effective for managing operations from multiple applications, while cross-application workflows remain a limitation for LLMs.Source: §6 · p. 8

• OFFICEBENCH currently focuses on planning complex workflows involving multiple applications.Source: Limitation · p. 9
• The authors anticipate that adding more applications will expand the benchmark to additional application scenarios.Source: Limitation · p. 9
• They are exploring instruction tuning tailored to office automation tasks to improve performance on OFFICEBENCH.Source: Limitation · p. 9

Ethical Statement

• The benchmark uses only synthesized file-system data, with fictitious individual and company names generated by ChatGPT.Source: Ethical Statement · p. 9
• The authors do not anticipate any major ethical concerns from this data construction.Source: Ethical Statement · p. 9

A Comparison with Recent Benchmarks

• OFFICEBENCH emphasizes cross-application scenarios with customized evaluation functions, a larger action space, and more extensible task and environment creation.Source: A Comparison with Recent Benchmarks · p. 12

B Applications and Operations

• The benchmark lists applications and their corresponding operations while simulating a realistic execution environment for office automation tasks.Source: B Applications and Operations · p. 12

C Observation Formats

• Representative operations are documented through their observation formats in Figure 4.Source: C Observation Formats · p. 12

D Evaluation Methods

• The appendix provides additional examples of OFFICEBENCH evaluation methods in Table 6.Source: D Evaluation Methods · p. 12

E OFFICEBENCH Prompts

• The prompt examples cover switching applications during a workflow and planning the next operation from the trajectory.Source: E OFFICEBENCH Prompts · p. 12
• An ablation-study prompt lists all operations available to the agent.Source: E OFFICEBENCH Prompts · p. 12

F Error Analysis for Human Annotators

• Human annotators sometimes omit one requested operation, such as sending an email while creating the requested poster.Source: F Error Analysis for Human Annotators · p. 12
• Human annotators can also miscount an answer when interpreting a scholarship condition across quarters.Source: F Error Analysis for Human Annotators · p. 12
• Table 4 compares OFFICEBENCH with recent benchmarks across cross-application scenarios, customized task evaluation, supported actions, and extensibility.Source: Table 4 · p. 13
• Table 5 catalogs the applications and corresponding operations implemented in OFFICEBENCH.Source: Table 5 · p. 13
• Table 6 describes exact matching, fuzzy matching, and execution-based evaluation for validating agent results.Source: Table 6 · p. 14