Source-linked AI summary
Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies
Zirui Tang, Xuanhe Zhou, Yumou Liu, Linchun Li, Yukai Wu, Weizheng Wang, Hongzhang Huang, Wei Zhou, Jun Zhou, Jiachen Song, Shaoli Yu, Jinqi Wang, Zihang Zhou, Hongyi Zhou, Yuting Lv, Jinyang Li, Jiashuo Liu, Ruoyu Chen, Chunwei Liu, GuoLiang Li, Jihua Kang, Fan Wu
TL;DR
Existing benchmarks inadequately test AI agents on authentic workplace tasks involving complex dependencies across heterogeneous files. Workspace-Bench fills this gap with realistic, dependency-driven workspaces and tasks, showing current agents remain unreliable, with an average Rubrics Pass Rate of 43.3%.
Problem
Existing benchmarks inadequately simulate authentic workplace workflows and complex relationships among heterogeneous files, limiting evaluation of workspace learning.
Method
Workspace-Bench evaluates agents on realistic workspaces spanning five user profiles, 20,476 interconnected files, and 388 dependency-driven tasks with grounded rubrics.
Results
43.3% average Rubrics Pass Rate shows current agents remain unreliable on workspace tasks requiring cross-file retrieval, reasoning, and adaptive execution.
Takeaways & Limitations
Workspace-Bench establishes a demanding testbed for shifting agent evaluation from isolated skills toward workspace-aware reasoning.
Takeaways & Limitations
Current agents cannot yet close the Data Association Gap between workspace-learning requirements and isolated-file processing.
Abstract
from arXiv · showhide
Workspace learning requires AI agents to identify, reason over, exploit, and update explicit and implicit dependencies among heterogeneous files in a worker's workspace, enabling them to complete both routine and advanced tasks effectively. Despite its importance, existing relevant benchmarks largely evaluate agents on pre-specified or synthesized files with limited real-world dependencies, leaving workspace-level evaluation underexplored. To this end, we introduce Workspace-Bench, a benchmark for evaluating AI agents on Workspace Learning involving Large-Scale File Dependencies. We construct realistic workspaces with 5 worker profiles, 74 file types, 20,476 files (up to 20GB) and curate 388 tasks, each with its own file dependency graph, evaluated across 7,399 total rubrics that require cross-file retrieval, contextual reasoning, and adaptive decision-making. We further provide Workspace-Bench-Lite, a 100-task subset that preserves the benchmark distribution while reducing evaluation costs by about 70%. We evaluate 4 popular agent harnesses and 7 foundation models. Experimental results show that current agents remain far from reliable workspace learning, where the best reaches only about 60%, substantially below the human result of 80.7%, and the average performance across agents is only 43.3%.
1 Introduction
Workspace-Bench addresses the gap between AI agents’ apparent capabilities and their unreliable performance on real-world workplace tasks by evaluating Workspace Learning over large-scale, heterogeneous file dependencies. Its results show that current agents remain substantially below reliable workspace learning, with the best combination reaching nearly 60% accuracy.
- Benchmark Design: Each task is paired with 19.1 rubrics on average that assess final-output correctness and critical intermediate decisions.The evaluation therefore covers more than whether the final answer is correct.
- Results: Nearly 60% accuracy is achieved by the best-performing DeepAgent + GLM-5.1 combination across 27 combinations of 4 agent harnesses and 7 backbone LLMs.The reported result indicates that even the strongest evaluated combination remains far from fully reliable workspace learning.
- Results: 58.1 interaction turns and 0.61 million tokens per task are consumed by DeepAgent + MiniMax-M2.7 while its average pass rate reaches only 45%.The result is described as a severe cost explosion that still fails to produce competitive success rates.
- Benchmark Design: Workspace-Bench evaluates Workspace Learning through five realistic user workspaces, heterogeneous files, and 388 file-dependency-driven tasks.The benchmark is designed to assess workspace tasks involving large-scale file dependencies rather than isolated skills.
- Evaluation Framework: 7,000+ rubrics support fine-grained assessment of correctness, intermediate reasoning, and operational efficiency through sandboxed execution, workspace recovery, and dependency-graph-based assessment.The framework combines sandboxed parallel execution, automatic workspace recovery, and dependency-graph-based assessment.
- Results: 57.6% on Easy tasks versus 40.5% on Hard tasks reveals consistent degradation across workspace-task difficulty levels.Heterogeneous file understanding and lineage tracing are identified as the primary capability bottlenecks across configurations.
2 Related Work
Prior work advances GUI interaction, retrieval, memory, and environment-based evaluation, but often abstracts away the relational structure of real workspaces. Workspace-Bench targets this gap by evaluating reasoning over diverse files and their semantic, aggregative, and lineage relationships.
- GUI and Desktop Agents: GUI agents excel at localized, single-application operations but struggle with implicit relationships across scattered data sources in complex file systems.
- Memory and RAG for Agents: RAG and persistent memory expand accessible information, yet typically represent retrieved context as a flat collection of text chunks.
- Benchmark Paradigms: Existing benchmarks span prompt-driven, environment-driven, task-file-driven, and workspace-relevant settings, but commonly emphasize reasoning, tool execution, document handling, or action grounding.Environment-driven benchmarks include operating-system, GUI, browser, and data-centric tasks, while task-file-driven benchmarks provide source files and reference documents.
- Workspace-Relevant Benchmarks: Workspace-Bench targets relational workspace structure through diverse user personas, over 70 file modalities, and tasks requiring navigation of semantic, aggregative, and lineage relationships.
3 Collection and Curation of Workspace-Bench
Workspace-Bench is curated as a realistic, dependency-rich benchmark that models heterogeneous professional workspaces rather than isolated files. Its tasks are sourced from authentic workflows, manually annotated, and evaluated with fine-grained, data-grounded rubrics.
- Workspace construction: Workspace-Bench constructs realistic workspaces with thousands of interconnected artifacts, implicit conventions, role-specific organization, and noisy file structures.The benchmark targets relational workspaces unlike clean collections of independent files.
- Workspace construction: The benchmark explicitly annotates dependencies such as references, semantic relations, modality transformations, and version lineage for cross-file reasoning.Agents must retrieve contextually related files and reason over their dependencies rather than treating files as isolated evidence.
- Workspace construction: Five professional roles—Operations Manager, Logistics Manager, Product Manager, Backend Developer, and Researcher—define diverse workspace structures and corresponding tasks.Role-specific persona profiles capture responsibilities, workflows, file usage patterns, and domain terminology.
- Task curation: 388 tasks are written as under-specified natural-language requests requiring agents to recover file dependencies across routine and complex multi-step workflows.The tasks are based on 154 representative scenarios collected from authentic workplace workflows and filtered by domain experts.
- Task curation: 25 human annotators create tasks and fine-grained binary rubrics, while auxiliary-agent verification and expert cross-validation convert vague criteria into data-grounded assertions.Rubrics assess output quality, procedural correctness, and task completion through foundational, procedural, and result-oriented criteria.
4 Benchmark Analysis
Workspace-Bench is a large-scale benchmark spanning realistic professional workspaces, heterogeneous files, cross-file dependencies, and fine-grained evaluation. Its 100-task Lite subset preserves the full benchmark’s distribution while reducing evaluation costs by approximately 70%.
- Benchmark scale and composition: 388 tasks span five professional workspaces, with Operations Manager contributing 122 tasks and Logistics Manager 115.The remaining workspaces are Researcher (67), Backend Developer (43), and Product Manager (41).
- Workspace Learning abilities: 238 tasks require Task-Supporting Files Utilization, 211 require Result-Providing Files Utilization, and 170 require Content Relations Understanding.Semantic Heterogeneous File Understanding appears in 140 tasks, while Lineage Tracing appears in 136 tasks.
- Dependency and evaluation complexity: 19.1 rubrics per task evaluate 7,399 rubrics across result-oriented, foundation, and process-oriented criteria.These categories account for 54.8%, 25.0%, and 20.2% of rubrics, respectively; tasks require an average of 5.1 dependency edges across 4.7 files.
- Workspace-Bench-Lite: 100 Workspace-Bench-Lite tasks preserve the original distribution across workspaces, difficulty levels, and Workspace Learning dimensions while reducing evaluation costs by approximately 70%.The subset is designed for lightweight and rapid evaluation.
5 Experiments
Workspace-Bench-Lite results show current agents remain substantially below human performance, with accuracy declining as task complexity rises. Performance depends on task dimensions, harness–model combinations, and inference efficiency, while human collaboration still outperforms full autonomy.
- Overall Results: 45.1% mean pass rate across 15 configurations, versus 80.7% for human experts, with configuration pass rates ranging from approximately 27% to 60%.DeepAgent + GLM-5.1 achieves the highest performance among tested combinations.
- Finding 1: Task Complexity: 35.7% pass rate on Hard tasks follows 46.0% on Medium and 51.4% on Easy, confirming substantial degradation with higher task complexity.Hard tasks require dependency discovery, long-horizon planning, state tracking, and error recovery, while degradation reflects both base-LLM reasoning limits and harness orchestration constraints.
- Finding 2: Capability Bottlenecks: Three of six workspace dimensions are primary bottlenecks, with Heterogeneous File Understanding and Lineage Tracing ranking lowest across most agents.Agents perform better in Workspace Exploration and Result-Providing Files Utilization, while DeepAgent + GLM-5.1 reaches nearly 50% in exploration but approximately 30% in heterogeneous understanding.
- Finding 3: Harness–Model Disparities: GLM-5.1 generally forms the strongest cross-persona performance envelope, while harness orchestration reshapes capability distributions and business-oriented personas reduce average performance.Hermes is optimized for code development and research-oriented tasks, whereas business personas require strategic planning, resource allocation, and ambiguous-semantic comprehension.
- Finding 4: Inference Efficiency: Over 55% average accuracy with fewer than 30 interaction turns demonstrates that high interaction counts and token costs do not guarantee better performance.Hermes + GLM-5.1 achieves this efficiency, while DeepAgent + GLM-5.1 reaches nearly 60% accuracy with many turns; Gemini configurations use 40–60 turns yet remain at 30%–45%.
- Finding 5: Human Collaboration: Human experts significantly outperform fully autonomous agents across all task tiers, indicating a substantial gap in complex office-workflow capabilities.The results support continued human-in-the-loop intervention as agents develop higher-level workspace-task capabilities.
6 Five Stages of Workspace Learning
Workspace Learning is framed as a five-stage progression from data-insensitive assistance to workspace-native self-evolution, centered on connecting tasks with relevant data and file relationships. Current agents degrade from 57.6% on Easy rubrics to 40.5% on Hard rubrics and face a fundamental Data Association Gap at higher stages.
- Motivation: Workspace Learning requires agents to connect tasks with relevant data and understand lineage and logical relationships across numerous workspace files.The paper presents this capability as the key to completing real-world workplace tasks beyond existing harness capabilities such as advanced reasoning.
- Five stages: The first two stages progress from passive procedural advice to user-specified file execution, where agents process isolated files without broader dependency awareness.L0 leaves humans as primary contributors, while L1 depends on explicit file paths and operational sequences and struggles with fragmented workspace intent.
- Five stages: L2 requires agents to identify explicit and implicit file-to-file dependencies, but Relationship Omission can cause errors such as selecting outdated file versions.This stage marks the transition from isolated-file processing to understanding how disparate files function collectively.
- Five stages: L3 enables proactive task-to-file discovery across the workspace, while L4 adds continuous adaptation through feedback from task execution, workspace changes, and historical context.The paper calls L3’s end-to-end independent task processing the Capability Singularity and describes L4 as workspace-native self-evolution.
- Evaluation and bottleneck: 57.6% to 40.5%: rubric pass rates fall from Easy to Hard as agents approach L3, while the Data Association Gap remains unclosed at L3/L4.From L2 onward, the harness contributes more consistently than the foundation model; the gap reflects a mismatch between required Workspace Learning and isolated-file processing.
7 Conclusion
Workspace-Bench is introduced as a large-scale benchmark for evaluating workspace learning in autonomous AI agents, focusing on cross-file dependency reasoning in realistic digital workspaces. It targets the gap between existing agent benchmarks and real-world workplace demands by addressing heterogeneous file ecosystems and complex inter-file dependencies.
- Workspace-Bench evaluates Workspace Learning in autonomous AI agents using realistic digital workspaces.The benchmark is designed around cross-file dependency reasoning.
- The benchmark focuses on reasoning over cross-file dependencies within realistic digital workspaces.
- Over 70 formats constitute the heterogeneous file ecosystems that Workspace-Bench addresses.
A Appendix · A.1 Appendix A. Detailed Statistics of Workspace-Bench Evaluation.
The appendix details computational cost, interaction efficiency, and task-level robustness across agent configurations, harnesses, and backbone LLMs. It shows substantial variation in resource use and declining accuracy under stricter completion criteria and harder tasks.
- A.1 Appendix A. Detailed Statistics of Workspace-Bench Evaluation.: OpenClaw and DeepAgent configurations consume the most tokens, with OpenClaw + Kimi-2.5 exceeding 1M tokens per task.Hermes configurations are generally more token-efficient, while the overall benchmark average is 563.1K tokens per task.
- A.1 Appendix A. Detailed Statistics of Workspace-Bench Evaluation.: DeepAgent configurations require the most interventions, peaking at nearly 60 turns for DeepAgent + GLM-5.1.OpenClaw and Hermes setups often resolve tasks in under the average 40.7 turns per task, and high costs reflect lengthy trial-and-error loops.
- A.1 Appendix A. Detailed Statistics of Workspace-Bench Evaluation.: Performance degrades as completion criteria tighten from Pass@50% to Pass@100% across the evaluated harnesses.Figure 16 compares task-level accuracy across DeepAgent, Hermes, and OpenClaw at Pass@50%, 70%, 90%, and 100%.
- A.1 Appendix A. Detailed Statistics of Workspace-Bench Evaluation.: GLM-5.1 shows the smallest drop-off from Pass@50% to Pass@100%, indicating the most resilient performance across harnesses.OpenClaw and Hermes generally facilitate higher accuracy at the strictest Pass threshold.
- A.1 Appendix A. Detailed Statistics of Workspace-Bench Evaluation.: GLM-5.1 pairs best with OpenClaw and Hermes, achieving the highest scores even on Hard tasks.Table 4 reports Total, Easy, Medium, Hard, and Pass@30 to Pass@100 results for all tested combinations.
- A.1 Appendix A. Detailed Statistics of Workspace-Bench Evaluation.: Seed-2.0-Code and Gemini-3.1-Pro experience a steep decline in accuracy as task complexity increases.The appendix emphasizes advanced reasoning capabilities as necessary for Workspace Learning tasks.
A.2 Appendix B. Prompts.
The appendix defines strict workspace-scoped execution and output requirements, then specifies evidence-based procedures for judging task results and extracting dependency graphs. Agents must rely on inspected files and execution traces while avoiding unsupported assumptions or direct ground-truth copying.
- Execution constraints: Agents must complete tasks exclusively using the current workspace and access it only through relative paths.The authorized working directory is strictly restricted, and conflicting path instructions must be disregarded.
- Execution constraints: Generated artifacts must be duplicated, and the final response must be only a Python list[str] of generated file paths.Conversational text, explanations, and markdown wrappers are prohibited.
- Agent-as-a-Judge evaluation: Judges must base rubric verdicts on inspected files and document exact paths and observed evidence, marking unsupported rubrics as passed=false.They may proactively inspect paths and must not treat inputs/ or original task directories as answer keys.
- Dependency graph extraction: Dependency graphs must use execution traces as the only valid evidence for establishing file-level edges.Inputs, candidate outputs, and the ground-truth reference may assist alignment, but cannot independently validate edges.
- Dependency graph extraction: Edges require concrete endpoint evidence and explicit source-reading to destination-writing chains; co-occurrence, semantics, and unbound multi-input relationships must be excluded.The ground-truth graph is for standardizing names and reference only, not for direct copying.