Source-linked AI summary
DocOps: A Verifiable Benchmark for Autonomous Agents in Complex Document Operations
Jiazhen Jiang, Boxi Cao, Lingyong Yan, Yaojie Lu, Hongyu Lin, Shuaiqiang Wang, Dawei Yin, Xianpei Han, Le Sun
TL;DR
Reliable manipulation of structured documents remains insufficiently measured by existing evaluations. DocOps introduces a verifiable benchmark and finds that current agents struggle substantially with complex, long-range workflows and preserving global document consistency.
Problem
Quantitative understanding of how agents interact with real-world documents remains constrained by evaluations that treat documents as read-only knowledge repositories.
Method
DocOps provides a rigorously verifiable framework with a hierarchical taxonomy for evaluating end-to-end document manipulation across escalating workflow complexities.
Results
0.671 overall success was achieved by the strongest evaluated configuration, with performance rapidly dropping on L3/L4 tasks and three recurring failure modes identified.
Takeaways & Limitations
DocOps provides a reproducible diagnostic foundation for developing agents that maintain global consistency and avoid destructive modifications across complex digital artifacts.
Takeaways & Limitations
DocOps covers deterministic offline editing tasks but excludes workflows requiring live external services, collaborative editing, or interactive user clarification.
Abstract
from arXiv · showhide
As autonomous agents rapidly evolve, their ability to reliably manipulate ubiquitous digital documents has become critical for enabling general-purpose AI assistants and automating complex workspace workflows. In this paper, we introduce DocOps, a deterministically verifiable evaluation framework underpinned by a hierarchical taxonomy that deconstructs document operations inspired by real-world practices into atomic dimensions and escalating workflow complexities. Based on DocOps, we systematically evaluate representative closed- and open-source models across various agentic harnesses, revealing that even the most advanced frontier configurations still exhibit profound limitations when handling highly coupled, long-range tasks. Furthermore, a fine-grained analysis of existing agents' manipulation behaviors uncovers 3 key failure modes: long-term state tracking collapse, shallow semantic verification, and destructive editing of structural metadata. Ultimately, our work exposes the capability boundaries of agents in maintaining global document consistency, shedding light on the future design of robust, non-destructive agents for complex digital ecosystems.
1 Introduction
DocOps introduces a rigorously verifiable benchmark that decomposes document manipulation into atomic capabilities and escalating workflow depths, then evaluates agents through deterministic artifact-level validation. Experiments reveal substantial limitations in complex workflows, three pervasive failure modes, and strong dependence on agent harness design.
- Benchmark design: DocOps evaluates complex end-to-end document manipulation with a taxonomy spanning content, format, structure, and four escalating workflow levels.L1 and L2 isolate atomic edits, while L3 and L4 cover long-horizon, cross-document workflows inspired by real-world interactions.
- Benchmark design: A deterministic validator directly checks each requested final document state, reducing lenient evaluation and exposing native failures missed by surface-level or task-level signals.The design targets valid, nondestructively edited office artifacts through direct artifact-level verification.
- Empirical findings: 0.671 overall success rate is achieved by GPT-5.5 with Codex and skills, while performance rapidly drops on L3 and L4 tasks.This result demonstrates that even the most capable frontier configuration struggles with workflow-level complexity.
- Empirical findings: Three pervasive failure modes are identified: long-term state tracking failure, shallow semantic verification, and destructive editing of document data and structural metadata.These modes account for a majority of failed runs and include losing global state, accepting surface-plausible outputs, and breaking formulas, tables, or object structures.
- Empirical findings: Open-ended programmable coding environments with file-system feedback vastly outperform static RPC-style tool calling, while procedural skills mainly benefit mid-tier open-source models.Skills provide marginal utility for frontier models capable of zero-shot orchestration.
2 Related Work
Related work on autonomous agents advances both model capabilities and agent harnesses, while document benchmarks largely emphasize understanding or format-specific manipulation. DocOps distinguishes itself by treating documents as stateful computational objects and verifying valid state transitions that preserve relevant surrounding state.
- Autonomous agents: Autonomous-agent research follows two trajectories: improving model capabilities and constructing harnesses that scaffold execution loops, context management, and tool registries.Model-focused techniques include tool use, long-context window management, and memory mechanisms.
- Autonomous agents: Model-capability work develops techniques for tool use, long-context window management, and sophisticated memory mechanisms.These techniques are presented as one major direction in autonomous-agent research.
- Document-related benchmarks: Existing document benchmarks generally target read-only document understanding or format-specific manipulation rather than deterministic artifact-level verification.Examples include DocVQA, DocBench, OmniDocBench, SheetCopilot, and SpreadsheetBench.
- DocOps novelty: DocOps evaluates each document operation as a state transition that must reach the requested state while preserving native-format validity and relevant out-of-scope state across heterogeneous documents.This comparison frames documents as first-class, stateful computational objects.
3 DocOps
DocOps evaluates preservation-aware editing of native-format office artifacts through a two-axis taxonomy, controlled task construction, and deterministic final-artifact verification. Its benchmark spans atomic edits through cross-document workflows while checking both scoped changes and preservation of document-native state.
- Taxonomy: DocOps classifies tasks by editable operation layers and workflow difficulty, diagnosing local execution, planning, global consistency, and cross-document state tracking.The difficulty levels progress from L1 atomic edits through L4 multi-document artifact production or alignment.
- Task construction: The controlled pipeline screens practical sources, synthesizes native-format artifacts, and applies human review for clarity, scope, taxonomy alignment, integrity, and path consistency.Source artifacts include spreadsheet records and formulas, Word sections, slide objects, and PDF pages.
- Task construction: 210 tasks comprise 50 L1 atomic-edit, 40 L2 compositional, 60 L3 single-document workflow, and 60 L4 cross-document workflow tasks.Each task is packaged with source artifact(s), a natural-language instruction, optional document skills, and a deterministic verifier.
- Verification: Deterministic in-container verifiers score final artifacts offline through targeted predicates rather than whole-artifact exact matching, including preservation checks for out-of-scope elements.Preservation predicates test whether elements such as protected styles and unmodified worksheets remain intact.
- Verification: 95.31% verifier fidelity was observed in a human audit of 128 sampled decisions, with three false passes and three false fails.The verifier agreed with the human artifact-level pass/fail judgments in 122 cases.
4 Experimental
DocOps evaluates closed- and open-source models across multiple execution harnesses and skill settings on a deterministically verified 210-task suite. Results show that frontier agents remain limited on long-horizon document workflows, with performance shaped by harness–model interactions and failures concentrated in state tracking, semantic verification, and structural preservation.
- Evaluation setup: The evaluation covers closed- and open-source models across four harnesses spanning constrained document APIs, stateful shell access, interactive coding, and open script execution.The harnesses are DocTools, Terminus-2, Claude Code, and Codex.
- Evaluation setup: Claude Code and Codex are tested with and without official document skills while keeping the underlying harness unchanged.The skills cover xlsx, docx, pptx, and pdf operations.
- Evaluation setup: Each configuration is evaluated on the full 210-task suite, and success requires both a submitted artifact and deterministic verifier success.Runtime errors, missing outputs, verifier failures, and timeouts count as failures.
- Overall performance: 0.671 is the highest reported pass rate, achieved by GPT-5.5 with Codex and skills, leaving nearly one-third of tasks failed.GPT-5.5 decreases from 0.725 on L1 to 0.237 on L4 when averaged over available harness settings, indicating severe workflow-level degradation.
- Failure analysis: Semantic verification gaps dominate failures, alongside substantial long-term state-tracking failures and destructive editing of document structure.Observed problems include stale or misplaced components, violations of formulas or structural constraints, and failure to preserve native document state.
- Harness and model effects: Performance is shaped by the interaction between model capability and execution framework, while current open-source configurations remain below the strongest frontier configurations.Skill injection does not consistently improve performance and can constrain effective problem solving.
Conclusion
DocOps establishes a controlled, preservation-aware benchmark for evaluating autonomous agents on native-format artifacts. Results show that performance declines on longer-horizon, dependency-rich tasks, with recurring failures in global state maintenance, semantic verification, and native-structure preservation.
- Conclusion: DocOps establishes a controlled, preservation-aware benchmark for evaluating autonomous agents on native-format artifacts.It provides a reproducible diagnostic framework for complex digital document operations.
- Conclusion: Performance declines substantially as tasks require longer-horizon state tracking, cross-step dependency management, and preservation of document-native structure.These demands expose limitations across evaluated model–harness configurations.
- Conclusion: Recurring failures involve maintaining global state, verifying task-specific semantics, and preserving native structure.The benchmark provides an empirical foundation for developing agents that maintain global consistency and avoid destructive modifications.
Limitations · Appendix
DocOps evaluates deterministic, offline document-editing tasks but excludes workflows involving live services, collaboration, or interactive clarification. Its structurally demanding tasks make benchmark scaling labor-intensive, and the current benchmark contains 210 tasks.
- Limitations: DocOps focuses on deterministic document-editing tasks.
- Limitations: The benchmark is restricted to offline workflows.
- Limitations: DocOps does not cover workflows requiring live external services.
- Limitations: The benchmark excludes collaborative editing and interactive user clarification.
- Limitations: Scaling DocOps is substantially more labor-intensive than collecting read-only document examples.Complex tasks require structurally valid artifacts, clear editing scopes, and manual review of instructions and generated files.
- Appendix: DocOps currently contains 210 tasks.
A Task Statistics and Full Task List
The expanded DocOps benchmark contains 210 tasks, characterized by source-artifact formats and difficulty levels alongside operation-label distributions. Appendix Table 8 lists every task with its identifiers, difficulty, labels, input formats, and required output format.
- Figure 8 benchmark views: Figure 8 characterizes the 210-task benchmark through source-artifact counts by format and difficulty level, plus operation-label distributions under Content, Format, and Structure.The two panels provide complementary views of the expanded benchmark.
- Figure 8 benchmark views: Operation-label values represent label occurrences rather than mutually exclusive task counts because composite and workflow-level tasks may involve multiple primitives.This distinction applies to the full benchmark’s Content, Format, and Structure taxonomy.
- Complete task listing: Table 8 lists all 210 tasks with task identifiers, difficulty levels, operation labels, normalized input formats, and required output formats.For L4 cross-document tasks, input formats include all unique source formats in the task environment.
B Task Formalization Prompt · C Harness Cost Accounting
The construction pipeline formalizes informal document-editing seeds into structured task metadata that serves as a contract for synthesis and review. Harness costs are reported using standardized runtime and token-accounting procedures over valid run records.
- B Task Formalization Prompt: Stage 2 converts an informal document-editing seed into a structured metadata specification rather than directly generating the final document artifact.The metadata serves as the task-level contract for later artifact synthesis and review.
- B Task Formalization Prompt: The task-level metadata fixes the document format, difficulty level, operation labels, paths, user-facing instruction, editing scope, expected output type, and task information.
- C Harness Cost Accounting: Harness runtime is measured from task start to termination using Harbor timing records, independently of harness-specific token accounting.It includes agent interaction, tool execution, file operations, verifier invocation, and applicable timeout duration.
- C Harness Cost Accounting: Token usage counts prompt/input and completion/output tokens across the full interaction trajectory, but accounting can differ because runtimes expose different usage metadata.
- C Harness Cost Accounting: DocTools uses agent-reported usage fields when available, while missing reliable metadata is estimated from visible LLM-call trajectories and calibrated.
- C Harness Cost Accounting: Codex estimates marked with † use visible trajectories calibrated against reliably accounted runs, whereas Claude Code reads recorded usage fields without additional calibration.
- C Harness Cost Accounting: Harness averages include only model settings with valid run records under the reported configuration; unavailable or invalid executions are excluded to avoid distorting costs.
D Full operation-level capability map.
Figure 10 provides a complete operation-level heatmap of L1 atomic-task pass rates across evaluated models and harness settings, confirming recurring bottleneck operations in the broader model set.
- D Full operation-level capability map.: Figure 10 reports the complete operation-level heatmap for all evaluated models.It complements the representative heatmap presented in the main text.
- D Full operation-level capability map.: Each heatmap cell gives the average pass rate for one model and one atomic operation across that model’s available harness settings.The measurements are reported for L1 atomic tasks.
- D Full operation-level capability map.: The broader model set exhibits the same recurring bottleneck operations, particularly those requiring document-native state control.The passage identifies hierarchy editing and theme-related operations as examples, but the provided text truncates the latter example.
E Full Skill-Injection Results
Table 5 evaluates how skill injection changes pass rates for Claude Code and Codex across difficulty levels and operation families. Positive, negative, and zero deltas respectively denote gains, regressions, and unchanged performance.
- Overall results: Table 5 reports full skill-injection deltas for Claude Code and Codex across difficulty levels L1–L4.The table reports overall deltas by difficulty level for each harness.
- Operation-family results: The analysis breaks deltas down by Content (C), Format (F), and Structure (S) operation families.These families provide operation-level views of skill-injection effects.
- Delta interpretation: Positive deltas indicate improved pass rates, negative deltas indicate regressions, and zero deltas indicate identical with-skill and no-skill pass rates.Delta signs define whether skill injection helps, harms, or leaves performance unchanged on a subset.
- Delta interpretation: Boldface marks the largest positive and largest negative delta within each harness block and column.This highlights the strongest gains and regressions associated with skill injection.
F Verifier Fidelity Evaluation · G Representative Verifier Snippets
DocOps evaluates verifier fidelity through manual artifact audits and controlled mutation tests, achieving high agreement and mutation detection while exposing limited edge cases. Representative verifiers enforce semantic and structural postconditions across Excel, Word, PowerPoint, and PDF artifacts rather than relying on visual similarity alone.
- F Verifier Fidelity Evaluation: Manual artifact-level auditing sampled 128 verifier decisions across document formats and difficulty levels, with an experienced PhD auditor assigning pass or fail judgments.The audit compared instructions, source artifacts, and submitted outputs against requested edits and correctness criteria.
- F Verifier Fidelity Evaluation: 95.31% agreement was observed, with three false passes and three false fails.The false decisions reflected peripheral document properties or implementation-specific representations rather than central task failures.
- F Verifier Fidelity Evaluation: The verifier detected 174 of 180 controlled mutations (96.67%), including every missing-output and parse-corruption case.Mutations changed requested content, native structure, preservation requirements, output existence, or file integrity while retaining the extension.
- F Verifier Fidelity Evaluation: An undetected Excel mutation was accepted because it changed an unrelated worksheet-view detail while preserving task-related content and workbook validity.The case study treated this as acceptable under the localized rewrite task scope.
- G Representative Verifier Snippets: Each task verifier deterministically checks whether the submitted artifact satisfies the task’s semantic and structural contract rather than enforcing one edit script.Figures 11–14 provide representative snippets from four document formats.
- G Representative Verifier Snippets: Excel verification requires executable formulas with valid semantics grounded in required source ranges, rejecting static values and unsupported formulas.The verifier accepts multiple valid formula families and recomputes expected totals from workbook state.
- G Representative Verifier Snippets: Word verification inspects styled headings and native table cells through the .docx object model, preventing visually simulated headings or tables from passing.It checks heading hierarchy, section boundaries, and native table structure.
- G Representative Verifier Snippets: PowerPoint and PDF verifiers preserve native structure by requiring real table grids with row content and jointly checking PDF page order with bookmark hierarchy.These checks prevent text-box overlays or ignored navigation state from substituting for native document structures.
H Model Serving Details · I Harness Configuration Details
The evaluation standardizes model serving and harness execution across API and local deployments, while preserving deterministic task isolation and verification. Harness configurations also control tool-call parsing, interaction limits, and skill availability through reproducible settings.
- H Model Serving Details: All evaluated models were publicly accessible, with GPT-5.5 and GPT-5.4 accessed via OpenAI, Claude Sonnet 4.6 via Anthropic, and DeepSeek-V4-Pro via DeepSeek APIs.Qwen, Gemma, and GLM models were served locally through vLLM on a single A800 node with 8 NVIDIA A800-SXM4-80GB GPUs.
- H Model Serving Details: Claude Code used vLLM’s Anthropic-compatible endpoint, whereas Codex, Terminus-2, and DocTools used its OpenAI-compatible endpoint.Qwen3.6 and Gemma4 Codex runs additionally used explicit chat templates, while serving fixed model loading, context length, parallelism, and tool-call parsing.
- I Harness Configuration Details: Every harness followed the same Harbor lifecycle: container setup, workspace mounting, isolated agent launch, artifact collection, and deterministic verification.This separation ensures agents edit only the received workspace while the verifier evaluates the submitted artifact independently.
- I Harness Configuration Details: Terminus-2 provided a stateful shell with full file-system feedback under a 200-turn limit, JSON action parser, and temperature 0.2.Adaptive thinking was disabled for vLLM-backed runs so open-weight models used the same external tool loop without provider-specific reasoning budgets.
- I Harness Configuration Details: Skill-enabled and no-skill runs were deterministically materialized from the same benchmark project and source documents.The no-skill script removed skill declarations, advertisements, Dockerfile copy steps, and task-local skill folders.
- I Harness Configuration Details: The harness configuration covered DocOps tasks spanning document formats, operation levels, and combinations of extraction, editing, generation, computation, reasoning, formatting, and structural operations.The supplied task listings include L3 and L4 workflows across Excel, PPT, PDF, Word, and cross-format inputs and outputs.