PaperlayerView exact revision on arXiv

Source-linked AI summary

OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding

Jingbo Zhou, Yusai Zhao, Qi Bao, Jingjia Cao, Zhenghai Chen, Chang Gao, Kaiqi Guo, Muxin Guo, Mingxuan Li, Xinjiang Lu, Yanru Ma, Yixiong Xiao, Zenghui Zhang, Le Zhang, Hua Wu

arXiv:2607.27155v1cs.AIcs.CLcs.HC

TL;DR

Existing benchmarks offer limited support for assessing whether LLM agents complete long-horizon office-suite workflows at reasonable cost. OmegaUse-OfficeVal adds task-level economic grounding and code-based verification, finding that agents are cheaper and faster than humans but still substantially behind them in deliverable quality.

  • Problem

    Existing benchmarks provide limited support for evaluating whether LLM agents complete long-horizon office-suite workflows at reasonable cost.

  • Method

    OmegaUse-OfficeVal benchmarks 100 long-horizon office-suite tasks with human labor time, task price proxies, and code-based verification from fine-grained rubrics.

  • Results

    17.91 versus 27.79 for humans: the best evaluated LLM substantially trails the human baseline in deliverable quality, despite being cheaper and faster.

  • Takeaways & Limitations

    OmegaUse-OfficeVal provides a reproducible testbed for tracking LLM-agent progress on practically valuable office-suite deliverables.

Abstract

from arXiv · show

Large language model (LLM) agents are increasingly expected to assist users in completing tasks. However, existing benchmarks provide limited support for evaluating whether agents can carry out office-suite workflows at a reasonable cost. We introduce OmegaUse-OfficeVal, a benchmark for evaluating LLM agents on long-horizon office-suite tasks with task-level economic grounding. The benchmark comprises 100 tasks derived from office-suite requests proposed by practitioners and adapted through a privacy-preserving process. On average, these tasks require 2.32 hours of human labor to complete. An important feature of the benchmark is that each task is paired with two economic signals: human labor time and task price proxy. These signals enable direct comparisons between human costs and LLM inference costs, as well as value-weighted evaluation. To support stable evaluation, we develop code-based verifiers from fine-grained rubrics. We evaluate several frontier LLMs together with a human baseline. Although all evaluated LLMs are substantially cheaper and faster than human workers, they have not yet approached human-level deliverable quality. The code and dataset are fully open-sourced, and more information is available on our project website: https://omegause-officeval.github.io.

1 Introduction

OmegaUse-OfficeVal benchmarks LLM agents on 100 long-horizon office-suite workflows with task-level economic grounding and code-based verification. Evaluations show that LLM agents are faster and cheaper than humans but remain behind human deliverable quality.

  • Benchmark: OmegaUse-OfficeVal comprises 100 long-horizon office-suite workflows derived from practitioners’ authentic requests and adapted through a privacy-preserving process.Each task includes a high-level instruction, input files, and required deliverables.
  • Economic grounding: Every task pairs human labor time with a task price proxy, enabling comparisons between human costs and LLM inference costs.Human labor time is measured without LLM assistance, while task price proxy uses practitioner price signals or aggregated expert estimates.
  • Results: 17.91 was the best LLM score, compared with 27.79 for the human baseline, showing that current agents make meaningful but limited progress.Value-weighted evaluation also shows that the highest average-scoring model is not necessarily the one capturing the most economically important work.
  • Results: Evaluated LLMs are faster and cheaper than human workers but still lag behind humans in deliverable quality.The benchmark evaluates GLM-5.2, Kimi K2.6, DeepSeek-V4-Pro, MiniMax M3, and Qwen3.7-Plus alongside a human baseline.
  • Evaluation: The benchmark uses code-based verification with usability checks and fine-grained rubrics to support stable evaluation.The verification protocol is designed around concrete deliverables and task-level criteria.

2 Related Work

OmegaUse-OfficeVal is positioned among productivity-agent, office-automation, and computer-use benchmarks. It is distinguished by long-horizon office-suite workflows, task-level economic grounding, and a fully open benchmark release.

  • Benchmark positioning: OmegaUse-OfficeVal surveys three related areas: productivity-agent benchmarks, office-automation benchmarks, and computer-use benchmarks.The paper defers a more comprehensive discussion to Appendix A.
  • Benchmark positioning: Its distinguishing features are long-horizon office-suite workflows, task-level economic grounding, and a fully open benchmark release.
  • Productivity-agent benchmarks: Productivity-agent benchmarks assess realistic professional tasks and whether agents produce deliverables comparable in quality to human professionals.Examples include GDPVal, the Remote Labor Index, and Agents’ Last Exam, alongside domain-specific benchmarks.
  • Office-automation benchmarks: Office-automation benchmarks cover multi-application and longer office workflows, as well as spreadsheets, presentations, and standardized office proficiency.OfficeBench evaluates multi-application workflows, while OdysseyBench extends this direction toward longer office workflows.
  • Computer-use benchmarks: Computer-use benchmarks evaluate GUI and computer-use agents in interactive execution environments spanning web, operating-system, and desktop settings.These benchmarks increasingly include office-suite tasks and general-purpose GUI agents for autonomous cross-platform execution.
  • Computer-use benchmarks: 108 tasks comprise OSWorld 2.0, whose median task takes a skilled human roughly 1.6 hours of active operation.OSWorld 2.0 evaluates whether an agent reaches the expected environment state, whereas OmegaUse-OfficeVal targets a fundamentally different evaluation target.

3 OmegaUse-OfficeVal Benchmark

OmegaUse-OfficeVal is a 100-task benchmark of office-suite workflows in which agents produce final artifacts from instructions and input files. Its task-level economic annotations, multimodal inputs, and code-based verification support evaluation of long-horizon task performance and cost.

  • Benchmark Structure: 100 tasks require agents to transform high-level user instructions and input files into final artifacts evaluated for quality and correctness.Each task centers evaluation on the produced final artifact.
  • Task Components: Each task includes an instruction, input files, human labor time, a task price proxy, and a code verifier.The price proxy uses explicit prices when available and expert estimation otherwise; the verifier assigns scores using executable evaluation code derived from instructions and rubrics.
  • Data Characteristics: Multimodal input files may include text, images, and videos as standalone files or embedded in word-processing documents, presentations, and spreadsheets.Task price proxies are recorded in Chinese yuan and reported in U.S. dollars using 1 CNY = 0.14609 USD.
  • Economic Grounding: $6.86 mean and $5.11 median task price proxies quantify the benchmark’s economic grounding.The task price proxies represent moderately priced office work plausibly assigned to a junior assistant or outsourced as a small office job.
  • Economic Grounding: 2.32 hours average, 2.03 hours median, and 8.35 hours maximum human labor time characterize the benchmark’s long-horizon tasks.These values indicate that tasks commonly require multiple hours of human work.

4 Task Construction Pipeline

OmegaUse-OfficeVal is built through a multi-stage pipeline that collects realistic office tasks, applies privacy-preserving expert screening, annotates economic signals, and creates deterministic code-based verification. The resulting benchmark evaluates final artifacts against fine-grained usability and task-completion rubrics.

  • Task collection: The pipeline begins with practitioner-proposed office workflows and representative materials, targeting realistic tasks assigned to junior workers, assistants, or interns.Tasks require concrete deliverables and nontrivial execution.
  • Task collection: 1,715 proposed tasks were initially collected, and expert screening reduced them to 595 tasks meeting real-world relevance, clarity, and deliverable criteria.Three senior experts next assess whether tasks are sufficiently nontrivial, long-horizon, and human-feasible.
  • Privacy-preserving adaptation: Privacy-preserving adaptation removes identifying information while preserving user intent, constraints, task-critical details, natural phrasing, and objectively evaluable requirements.Expert reviewers resolve ambiguities, clarify missing information, and remove subjective requirements.
  • Final acceptance: Three senior experts conduct final acceptance review, including privacy, executability, and fidelity to real office scenarios; inclusion requires unanimous agreement.The review covers language, structure, and artifact format.
  • Economic annotation: Each task receives human labor time and a task price proxy, enabling analysis of LLM performance relative to human effort and perceived task value.Human labor time records unaided completion time, while the task price proxy estimates market price.
  • Code-based verification: Fine-grained rubrics are converted into executable verifiers that assess final output files rather than fixed workflows, supporting reproducible artifact-level evaluation.Rubrics cover usability and task completion, while expert inspection, LLM reflection, and human-code discrepancy resolution calibrate the verifiers.

5 Experiments

Experiments compare several frontier LLM agents with a best-human reference using overall and economically weighted quality metrics, alongside runtime and cost. Humans achieve higher and more reliable deliverables, while LLMs are faster and cheaper but show substantial failures, with performance declining as task labor time increases.

  • Experimental setup: The evaluation compares GLM-5.2, Kimi K2.6, DeepSeek-V4-Pro, MiniMax M3, and Qwen3.7-Plus against the highest-scoring human submission per task.Human deliverable quality varies, so the best submission from at least two annotators is used as the reference artifact.
  • Evaluation metrics: Three metrics—score, time-weighted score, and price-weighted score—measure average quality and performance weighted by human labor time or task price proxy.The weighted metrics assess how well models perform on economically important tasks, not only average completion quality.
  • Overall performance: 27.79 is the human score, substantially exceeding all evaluated LLMs, although the benchmark remains difficult because scoring penalizes omissions, unintended changes, and avoidable damage.The long-horizon, complex tasks prevent even the human baseline from approaching a perfect score.
  • Overall performance: 17.91 is GLM-5.2’s highest average LLM score, whereas Qwen3.7-Plus leads both time-weighted and price-weighted scores.This ranking shift shows that average completion quality and economically weighted performance identify different leaders.
  • Efficiency trade-offs: DeepSeek-V4-Pro has the lowest runtime, Qwen3.7-Plus the lowest cost, and GLM-5.2 the best average score but higher cost and runtime than several alternatives.Overall, LLMs are substantially cheaper and faster than humans, yet still lag behind junior human workers in deliverable quality.
  • Score distributions and task difficulty: 21% of human tasks score above 50 and 29% score zero, while GLM-5.2 has 14% above 50, Qwen3.7-Plus has a 38% zero-score rate, and DeepSeek-V4-Pro and MiniMax M3 have 50% and 51%.Humans are more reliable overall; Qwen3.7-Plus makes relatively more partial progress, while longer human-labor tasks tend to receive lower scores from both humans and LLMs.

6 Conclusion · A More Related Work

OmegaUse-OfficeVal introduces a 100-task benchmark for evaluating LLM agents on office-suite deliverables with economic grounding and code-based verification. The related-work comparison positions it against productivity-agent, office-automation, and computer-use benchmarks along task horizon, economic grounding, and openness.

  • 6 Conclusion: OmegaUse-OfficeVal comprises 100 office-suite tasks paired with human labor time and a task price proxy.These economic signals provide task-level grounding for evaluation.
  • 6 Conclusion: Each task is verified through code-based verifiers built from fine-grained rubrics.The benchmark focuses verification on final office artifacts.
  • 6 Conclusion: The benchmark evaluates whether agents produce final office artifacts that are correct, usable, and practically valuable.Its final-artifact focus supports reproducible evaluation.
  • 6 Conclusion: The study introduces OmegaUse-OfficeVal and experimentally evaluates LLM agents on the benchmark.The conclusion summarizes both the benchmark contribution and its experimental evaluation.
  • A More Related Work: The benchmark comparison covers productivity-agent benchmarks, office-automation benchmarks, and computer-use benchmarks.These are the three comparison lines introduced in Section 2.
  • A More Related Work: The related-work analysis examines how OmegaUse-OfficeVal differs from prior work in task horizon, economic grounding, and openness.The appendix provides a comprehensive comparison along these dimensions.

A.1 Productivity-Agent Benchmarks · A.2 Office-Automation and Office-Suite Benchmarks

Prior benchmarks cover general-purpose, domain-specific, and office-automation workflows, but OmegaUse-OfficeVal distinguishes itself by combining a more complete public release with long-horizon office-suite tasks and task-level economic grounding.

  • A.1 Productivity-Agent Benchmarks: Prior productivity-agent benchmarks span general-purpose professional workflows and domain-specific workplace or enterprise environments.Examples include GDPVal, RLI, ALE, SWE-Lancer, TheAgentCompany, APEX, WorkArena/WorkArena++, WorkBench, and APEX-Agents.
  • A.1 Productivity-Agent Benchmarks: OmegaUse-OfficeVal focuses on office-suite tasks rather than general-purpose jobs and offers greater openness plus task-level economic grounding.This comparison is stated directly against GDPVal, RLI, and ALE.
  • A.1 Productivity-Agent Benchmarks: GDPVal and ALE release only subsets of their task pools, RLI releases 10 public tasks, and ALE gates access to reference outputs.OmegaUse-OfficeVal is described as providing a more complete public release than these broader productivity benchmarks.
  • A.1 Productivity-Agent Benchmarks: SWE-Lancer provides the strongest monetary grounding through real Upwork payouts, whereas other workplace benchmarks rely more on workflow realism or estimated duration.The cited comparisons distinguish real payouts from expert-estimated durations and realism-based grounding.
  • A.2 Office-Automation and Office-Suite Benchmarks: OfficeBench evaluates multi-application workflows across Word, Excel, PDF, email, and calendar in a simulated office environment.OdysseyBench extends this direction toward longer workflows involving extended interaction histories and multi-step reasoning.
  • A.2 Office-Automation and Office-Suite Benchmarks: Existing office and document benchmarks characterize difficulty through application count, action steps, dialogue length, rubric density, or runtime rather than measured human labor time.OfficeBench emphasizes bounded operation chains, while OdysseyBench uses extended dialogue histories and cross-application context.
  • A.2 Office-Automation and Office-Suite Benchmarks: OfficeBench, OdysseyBench, SpreadsheetBench, PPTC, PPT-Eval, Mind the Gap/OfficeEval, DocOps, and SheetCopilot evaluate office-automation capabilities without the task-level economic grounding emphasized by OmegaUse-OfficeVal.The listed capabilities include multi-application workflows, spreadsheet manipulation, PowerPoint editing, document operations, and fine-grained Office proficiency.

A.3 Computer-Use and GUI-Agent Benchmarks

Computer-use and GUI-agent benchmarks have progressed from web tasks to desktop and operating-system environments, while OmegaUse-OfficeVal evaluates long-horizon office-suite workflows without prescribing a specific execution environment or trajectory. OSWorld 2.0 also studies long-horizon workflows, but remains focused on computer-use evaluation.

  • OmegaUse-OfficeVal’s distinction: OmegaUse-OfficeVal differs by targeting long-horizon office-suite workflows rather than short-horizon interactions and by avoiding dependence on a particular interactive execution environment.It is not tied to a specific interactive environment, unlike benchmarks centered on GUI operation and environment-state transitions.
  • Benchmark landscape: Early benchmarks covered web-based tasks, including MiniWoB/MiniWoB++, WebShop, WebArena, VisualWebArena, Mind2Web, and WebVoyager.The benchmark landscape later expanded to operating-system and desktop environments.
  • Benchmark landscape: Desktop and operating-system benchmarks include OSWorld, Windows Agent Arena, MacOSWorld, OSUniverse, and WindowsWorld.These benchmarks span open-ended computer tasks, Windows 11 and macOS virtual machines, and process- or diagnostic-oriented evaluation.
  • OmegaUse-OfficeVal’s distinction: Agents in OmegaUse-OfficeVal may use GUI interactions, scripts, APIs, or hybrid strategies because the benchmark does not privilege a particular execution trajectory.This contrasts with computer-use benchmarks that primarily evaluate interactive task execution in controlled web, desktop, or operating-system environments.
  • Long-horizon comparison: OSWorld 2.0 contains 108 long-horizon tasks with a median skilled-human operation time of roughly 1.6 hours and hundreds of agent steps.Its completion grading uses many weighted checkpoints, but its evaluation target remains computer use rather than OmegaUse-OfficeVal’s office-suite workflow focus.

B Task Taxonomy by Operation Intent and Domain

The benchmark organizes tasks along two dimensions—operation intent and domain—with representative examples for each category.

  • Operation Intent: The task taxonomy is structured by operation intent, as presented in Table 4.
  • Domain: Tasks are also categorized by domain, as presented in Table 5.
  • Each taxonomy category includes representative examples.

C Incentive Mechanism and Computation Method for Human Labor Time … D.3 Human-Code Discrepancy Resolution

The benchmark combines quality-gated human-time measurement with an iterative code-based verification protocol. Human incentives and timing procedures are paired with expert-refined rubrics, executable verifiers, and discrepancy resolution to align automated scores with expert judgment.

  • C Incentive Mechanism and Computation Method for Human Labor Time: Annotator scores are aggregated daily, with quality-qualified top performers receiving bonuses of 1.5× or 1.25× the base daily salary.The top 25% receive 1.5 times the base salary, while those ranked between the top 25% and 50% receive 1.25 times.
  • C Incentive Mechanism and Computation Method for Human Labor Time: Each task is assigned to two annotators, and completion time includes revisions required after failed expert quality assessment.A revised deliverable that later passes is valid for time measurement, but its initial failure yields a task score of 0.
  • C Incentive Mechanism and Computation Method for Human Labor Time: Relative completion time and revision status determine annotator scores, while substantially divergent timings trigger a third measurement.A third annotator is recruited when the slower valid completion time exceeds 1.3 times the faster time.
  • D Code-Based Verification Protocol Details: The verification protocol iterates from LLM-assisted rubric generation through expert revision, code translation, and human-code discrepancy resolution.Iteration continues until code-based scores are sufficiently aligned with expert judgment.
  • D.1 Rubric Generation with Expert Revision: Each task rubric combines usability and task-completion dimensions, with usability failures assigning zero and preventing further completion evaluation.Usability checks include file format, openability, corruption, and continued editability.
  • D.1 Rubric Generation with Expert Revision: Task-completion rubrics reward explicit requirements and penalize unintended changes, using weighted positive items of +1, +3, or +5 points.Experts revise rubrics to remove vague items, add critical requirements, balance weights, and prioritize genuine completion quality.
  • D.2 Verification Code Generation and Reflection.: Coding agents generate initial verification code from expert-defined rubrics, after which experts revise the code for task- and format-specific artifact checking.The verifier automatically checks the final artifact and reports the corresponding rubric score.
  • D.3 Human-Code Discrepancy Resolution: Experts compare item-level manual and verifier scores, correcting implementation errors, ambiguous rubrics, interpretation inconsistencies, and coverage of reasonable task solutions.Repeated discrepancy analysis, rubric correction, and verifier revision improves reproducibility and alignment with expert judgment.

E More Experiment Settings … E.4 Prompt for Task Evaluation

The evaluation uses a shared programmatic agent scaffold and standardized sandboxed execution settings to test office-suite task completion without GUI control. Agents receive structured prompts, must preserve and expose deliverables in required formats and locations, and operate under fixed runtime and verification procedures.

  • E.1 Agent Scaffold.: Agents use a shared scaffold supporting programmatic tool use, shell execution, and file operations, but no GUI-level computer-use capabilities.The setup evaluates interaction with office artifacts through scripts and file-level APIs rather than visual office-application control.
  • E.2 Execution Environment.: Evaluations run in CPU-only Docker containers with Ubuntu 24.04.1 LTS, Python 3.10, remote model APIs, LibreOffice 24.2, up to 108 CPU-core equivalents, and 300 GiB memory.Token cost is calculated using the corresponding LLM’s token pricing.
  • E.3 Inference Configuration.: Each task supplies an instruction and input files, requires final deliverable artifacts in the specified output format, and uses the same system prompt and task-prompt template across models.Artifacts are evaluated with code-based verifiers, and each task has a wall-clock timeout of 14,400 seconds.
  • E.3 Inference Configuration.: Ten task environments run in parallel, with at most two concurrent inference threads per model.This configuration is part of the fixed inference procedure for each task.
  • E.4 Prompt for Task Evaluation.: The standardized task prompt specifies target file types, the user instruction, the required output location, and constraints on LibreOffice use within the sandbox.It preserves workflow open-endedness while keeping final deliverables directly accessible for evaluation.
  • E.4 Prompt for Task Evaluation.: The prompt instructs agents to modify the specified files according to the task requirements.The template uses placeholders for file types and task instructions.
  • E.4 Prompt for Task Evaluation.: Final deliverables must retain their original file formats and remain directly in the current directory rather than subdirectories.These constraints make the outputs consistently accessible for evaluation.
  • E.4 Prompt for Task Evaluation.: LibreOffice must be invoked through managed top-level Bash commands, while process-wide cleanup commands such as ‘pkill’ and ‘killall’ are prohibited.The prompt also specifies that each task has an isolated LibreOffice profile.

F Evaluation Metric Definitions

The evaluation reports an overall completion-quality score plus time- and price-weighted scores. The weighted metrics emphasize tasks by human labor time or estimated market value, complementing average-quality comparisons with captured labor or value.

  • Overall score: The overall score measures completion quality across all benchmark tasks using each task’s defined task score.For each task t, Score(t) is defined in Section 4.3.
  • Time-weighted score: The time-weighted score weights task completion by recorded human labor time, emphasizing tasks requiring more human effort.It can be interpreted as the total human labor time covered by an agent’s partial task completion.
  • Price-weighted score: The price-weighted score weights task completion by each task’s price proxy, emphasizing tasks with higher estimated market value.The price proxy is denoted by P_t.
  • Metric purpose: Together, the metrics compare models by average quality and by the human labor time or market value captured across the benchmark.The weighted measures provide different notions of economic value.

G Additional Breakdown Results of Performance

Performance varies substantially across domains, output file types, and operation intents. Human annotators generally perform best, while the strongest LLM differs by domain and task characteristic.

  • Human annotators achieve the best performance in most domains, indicating that current LLMs remain challenged across diverse office-suite scenarios.
  • The strongest LLM varies by domain: GLM-5.2 leads Engineering & Technology, Qwen3.7-Plus leads Business Operations, and DeepSeek-V4-Pro leads Academic Papers.
  • Models generally perform better on Word and Excel tasks than on PPT and mixed-file tasks.Excel tasks are relatively more tractable for several models, with performance approaching human annotators in some cases.
  • Annotation and extraction tasks show the highest scores for several models, with GLM-5.2 particularly strong on annotation and Qwen3.7-Plus on extraction.
  • Beautification and other loosely specified tasks tend to be more difficult because they require layout judgment, visual consistency, and preservation of existing content.
Loading 2607.27155v1…