PaperlayerView exact revision on arXiv

Source-linked AI summary

The Last Mile Is the File: OfficeEditBench for Preservation-Aware Office Editing

Zhiwen Wu, Chengxu Wu

arXiv:2609.34469v1cs.SEcs.CL

TL;DR

OfficeEditBench addresses the challenge of making required Office changes without altering protected state. It defines task contracts and analyzes archived outcomes across 170 tasks; its diagnostics connect maintenance obligations to frozen verifier results, whose validity remains uncalibrated by independent human assessment.

  • Problem

    Office edits must propagate required consequences across files while preserving prior work, because changing too little or too much can leave the deliverable outside its authorized scope.

    Source: §1 · p. 1

  • Method

    OfficeEditBench defines required updates, protected state, native structures, and applicable interactions in task contracts, then analyzes 510 archived task–system outcomes.

    Source: §1 · pp. 1–2, §3.1 · p. 4

  • Results

    Selected WorkBuddy diagnoses show that locally correct edits can still leave deliverables incomplete through formula-behavior failures or mismatches in required document text.

    Source: §5.4 · p. 7

  • Takeaways & Limitations

    The observations motivate treating edit scope as a first-class constraint and reporting acceptance vectors alongside scalar scores, as design implications not experimentally validated improvements.

    Source: §6 · p. 8

    The archived records lack complete per-attempt budgets and selection data, so results do not estimate standardized first-attempt or repeated-trial reliability.

    Source: §6 · p. 8

Abstract

from arXiv · show

A small Office edit creates two obligations: propagate every required update and leave protected state untouched. Updating too little leaves dependencies inconsistent; updating too much changes content the user did not authorize. We introduce OfficeEditBench, a 170-task benchmark for change-scoped maintenance of spreadsheets, presentations, and documents. Task contracts specify required updates, protected state, native structures, and applicable interaction requirements. Across 510 archived task-system outcomes from WorkBuddy, Doubao, and Codex, we distinguish file delivery, target completion, and verifier-defined acceptance. Hard package-valid delivery ranges from 92% to 100%, yet no selected output satisfies the complete contract. Case analysis highlights why local correctness is insufficient: an updated value can lose its generating formula, a revised rule can fail to reach related conclusions, and a new deadline can omit a retained prerequisite. These mechanisms connect artifact-level checks to the continued maintainability of Office files. We analyze maintenance failures while distinguishing frozen automatic verdicts from human acceptability. OfficeEditBench provides a testbed for completing required changes while preserving the logic and scope of existing work.

1 INTRODUCTION

OfficeEditBench frames Office editing as change-scoped maintenance: required consequences must be completed while protected content and native editability remain intact. It evaluates these obligations across native Office tasks and analyzes archived outcomes against multiple acceptance criteria.

1 INTRODUCTION

  • A California electricity correction must propagate from the workbook to a native presentation chart and related conclusions, without altering raw records or unrelated values.The example contrasts under-editing with over-editing and is not an observed agent-run failure.Source: §1 · p. 1, Figure 1 · p. 2
  • The maintenance objective is to complete every required update within its authorized scope while preserving protected content and native editability.Authorized changes may span multiple files; authorization alone does not make every possible edit mandatory.Source: §1 · p. 1
  • OfficeEditBench builds on execution-based maintenance and Office-manipulation benchmarks, focusing on specified cross-representation consequences, protected observations, and delivery or clarification requirements.The paper presents preservation as an existing evaluation principle, not a new one.Source: §1 · p. 1
  • The study asks whether delivery implies complete maintenance, what failures remain after target completion, and which obligations explain the gap.It compares criteria on the same selected outputs and complements them with case analysis of formulas, shared definitions, and retained conditions.Source: §1 · p. 2
  • OfficeEditBench contributes a 170-task native Office suite, an evaluation design separating required updates from protected state, and analysis of 510 archived task–system outcomes.Its package preserves frozen verdicts, reproducible aggregates, provenance, and focused case evidence.Source: §1 · pp. 1–2

2 RELATED WORK

Related work evaluates maintenance, Office manipulation, preservation, and interactive execution, but these strands use different evaluation units and validation practices. OfficeEditBench focuses on task-specific authorized consequences and protected observations, with strict acceptance and partial diagnostics.

2 RELATED WORK

  • OfficeEditBench applies regression-testing logic to native Office objects, where a displayed value may be correct even if its formula, chart linkage, or protected context is not preserved.The paper relates this to SWE-bench’s fail-to-pass and pass-to-pass tests.Source: §2 · p. 2
  • Office editing and preservation checking already appear in prior benchmarks; this work focuses on task-specific authorized consequences and protected observations under conjunctive acceptance.Prior examples include SpreadsheetBench, OfficeBench, PPTArena, PPT-Eval, DeckEdit-Bench, and OmegaUse-OfficeVal.Source: §2 · pp. 2–3
  • Interactive evaluation benchmarks differ in task scope and reliability evidence, while human assessment and expert validation provide distinct evidence about measurement validity.OfficeEditBench’s fixed-reply clarification tasks do not estimate repeated-trial reliability, and its validation gap remains explicit.Source: §2 · p. 3

3 BENCHMARK DESIGN AND CONSTRUCTION

The benchmark defines each Office task through its starting artifacts, request, visible evidence, authorization, verifier, and required deliverables. Its task suite combines varied tiers and formats with explicit target, preservation, and provenance boundaries.

3.1 TASK CONTRACT AND DESIGN PRINCIPLES

  • A task contract identifies starting artifacts, request, visible evidence, authorization, and a versioned verifier, alongside edited artifacts and any required interaction record.The benchmark encodes dependencies in instructions, evidence, and predicates rather than inferring a complete dependency graph from arbitrary files.Source: §3.1 · p. 3
  • Preservation compares protected observations with the actual materialized starting state, while required target predicates must pass.The task-specific comparison is neither universal semantic equivalence nor byte identity; protected state can include values, formulas, text, notes, and native structures.Source: §3.1 · p. 4
  • The deliverable is an editable artifact, and file validity, target correctness, and preservation are distinct checks; specified clarification interactions are checked without verifying every trajectory action.A parseable package alone does not establish a completed edit or native-application usability.Source: §3.1 · p. 4

3.2 COVERAGE AND CONSTRUCTION

  • The active suite spans Easy v2, Medium, and Hard; Easy v2 contains 25 English and 25 Chinese tasks, while Medium includes pilot and cohort tasks.Medium and Hard instructions are predominantly Chinese; tiers express construction intent rather than calibrated difficulty, and some Medium tasks include linked pairs.Source: §3.2 · p. 4, Table 1 · p. 4
  • Hard varies four shared source scenarios across fourteen workflow profiles, including partial approvals, late amendments, release decisions, and duplicate labels.The 50 task IDs are variations over shared workspaces, not 50 independent organizations; eight tasks require first-turn clarification.Source: §3.2 · p. 4
  • Task construction combines source fixtures, materialized starting states, requests, authority evidence, target and protected-state predicates, and specified file returns.Hard uses public-data-derived source context with synthetic workflow overlays; this describes implemented components, not uniform QA coverage or independent annotation across tiers.Source: §3.2 · p. 4
  • Requests are benchmark-authored around public-data-derived or synthetic workspaces, not sampled as a distribution of naturally occurring user requests.Source boundaries and construction records are documented in Appendix B.Source: §3.2 · p. 4

4 EVALUATION PROTOCOL AND CONSTRUCTION CHECKS

Evaluation separates strict verifier-defined acceptance from partial diagnostic scores and checks construction with reference, unchanged, and adversarial inputs. Those implementation checks do not independently establish evaluator validity.

4.1 STRICT ACCEPTANCE AND DIAGNOSTIC SCORES

  • Strict acceptance requires every required predicate to pass, including any specified interaction, and the reported strict pass rate is verifier-defined acceptance rather than human acceptability or pass@1.Partial scores describe satisfied requirements but do not replace the conjunction.Source: §4.1 · p. 5
  • Easy v2 weights readability, structure, target, and non-target preservation; a failed PPTX structural guard also marks downstream checks failed because object correspondence is unavailable.Those downstream content checks are unknown, not independently observed content errors.Source: §4.1 · p. 5
  • Medium and Hard report semantic and preservation categories with fixed weights, while Hard clarification tasks add an interaction component; tier composites are not a common calibrated scale.Hard uses C = 0.8S + 0.2P, or C = 0.7S + 0.2P + 0.1I on clarification tasks.Source: §4.1 · p. 5

4.2 CONSTRUCTION CHECKS AND THEIR SCOPE

  • Archived QA tests reference outputs, unchanged inputs, and adversarial variants, including variants with correct targets plus unrelated mutations.Easy v2 rejects all 50 such mutation variants; Hard rejects 12/12 adversarial variants spanning location, authorization, preservation, interaction, and missing files.Source: §4.2 · p. 5
  • Construction checks are not independent estimates of evaluator validity, and exact text, style identifiers, or formula representations may reject legitimate alternatives.No completed blind adjudication or comprehensive native-application rendering audit is available for agent outputs.Source: §4.2 · p. 5

5 EVALUATION RESULTS

Across the evaluated outputs, package delivery and target completion did not guarantee strict acceptance, with failures spanning semantic requirements and preservation of existing state. Case analyses show how locally correct edits can still leave files unmaintainable.

5.1 EVALUATION SCOPE AND EVIDENCE

  • The archive does not establish budget matching or uniform gold/verifier isolation, so the selected-output results are diagnostic rather than a controlled blind comparison.Recorded retries, corrections, and submission-selection policies do not constitute a uniform first-attempt protocol.Source: §5.1 · p. 5

5.2 RQ1: DOES DELIVERY IMPLY COMPLETE MAINTENANCE?

  • On Hard, WorkBuddy, Doubao, and Codex delivered 46/50, 50/50, and 49/50 package-valid triples, respectively, yet each recorded 0/50 strict passes.The results compare selected outputs under tier-specific criteria; WorkBuddy Hard is exploratory and Doubao Hard uses the corrected snapshot.Source: §5.1 · p. 5, §5.2 · p. 6
  • Neither Doubao nor Codex had a Hard task with S = 100; their all-task semantic means were 35.11 and 43.78, respectively.WorkBuddy likewise had no S = 100 result among its 46 scored deliveries; strict zeros can collapse distinct partial outcomes.Source: §5.2 · p. 6
  • Among package-valid triples, non-target document-text checks failed in 34/46 WorkBuddy, 45/50 Doubao, and 49/49 Codex cases; spreadsheet checks failed in 33/46, 44/50, and 35/49.These are non-exclusive task-level verdict counts, not counts or severity estimates of damaged objects.Source: §5.2 · p. 6

5.3 RQ2: WHAT REMAINS AFTER TARGET COMPLETION?

  • On Easy, Codex passed all 50 target checks but only 42 strict contracts; all eight failures were XLSX files whose Lists worksheet changed from hidden to visible.On Medium, 8/45 WorkBuddy and 5/49 Codex target-perfect outputs failed strict success; these denominators exclude non-target-perfect outputs.Source: §5.3 · p. 7

5.4 RQ3: WHICH MAINTENANCE OBLIGATIONS EXPLAIN THE GAP?

  • OBM-034 retained a correct displayed result after an AVERAGE formula was replaced with a constant, while OBM-058 missed a revised rule and OBM-062 omitted a retained prerequisite.OBM-032 passed every stored check, demonstrating bounded cross-file synchronization under its task contract.Source: §5.4 · pp. 7–8
  • WorkBuddy Medium’s 33 rejected tasks included eight with S = 100 and P < 100, eleven with P = 100 and S < 100, and fourteen with both below 100.The overlap rules out a single-axis account, though the categories include structural as well as content checks.Source: §5.4 · p. 8
  • Exploratory WorkBuddy Hard cases included approved workbook/chart values that passed while review-note, source-use, governance-record, or non-target checks failed.These are check-level mismatches, not findings that every rejected note was substantively wrong; an operator-reported internal edit does not establish correctness of an undelivered artifact.Source: §5.4 · p. 8

6 DISCUSSION AND LIMITATIONS

The observed disagreements motivate scope-aware agents and reporting an acceptance vector, but these are design implications rather than validated improvements. The results also have measurement, execution, population, exposure, and release limits.

Implications for agent and evaluator design

  • The observations support treating edit scope as a first-class constraint and reporting an acceptance vector alongside a scalar score.These are design implications, not experimentally validated improvements; the benchmark makes the hypotheses testable on native artifacts.Source: §6 · p. 8

Measurement validity

  • Independent human/native-application calibration has not been conducted, and precision- and wording-sensitive checks rejected two outputs not treated as omitted updates.Frozen scores remain unchanged; subsequent checker revisions should be versioned and applied uniformly.Source: §6 · p. 8

Execution and population limits

  • Incomplete attempt budgets and submission-selection records prevent estimating standardized first-attempt performance or repeated-trial reliability.Hard reuses four source scenarios, and tiers and languages are not controlled experimental factors; source-family holdouts and fully logged executions are needed to measure generalization and reliability.Source: §6 · p. 8

Exposure and release scope

  • Public gold enables reproduction but does not establish secrecy or gold/verifier isolation for every historical run.The static suite lacks ongoing contamination mitigation, and the package does not reproduce every artifact judgment or UI trajectory.Source: §6 · p. 8

7 CONCLUSION

Reliable Office maintenance requires completing required changes while preserving protected state and native editability; OfficeEditBench makes that contract explicit and connects acceptance to maintainable artifact structures.

7 CONCLUSION

  • OfficeEditBench connects strict acceptance to live formulas, consistent definitions, retained conditions, and complete delivery while distinguishing frozen verifier outcomes from human acceptability.The benchmark provides a testbed for scope-aware agents, but its automatic verdicts require independent human calibration.Source: §7 · p. 9

REPRODUCIBILITY STATEMENT

The source package provides reproducibility materials and hash-linked case evidence, but not complete historical execution replay.

REPRODUCIBILITY STATEMENT

  • The source package includes de-identified snapshots, source digests, analysis scripts, a selection ledger, and hash-linked case evidence; historical replay requires version-pinned fixtures, verifiers, outputs, and execution manifests.A separate ten-case bundle includes unchanged inputs and inspected outputs, not all 170 executable tasks.Source: REPRODUCIBILITY STATEMENT · p. 9

ETHICS AND AI-USE STATEMENTS

The paper describes ethical and AI-use safeguards and documents what the evidence ledger supports, while distinguishing check-level analysis from complete execution replay.

ETHICS AND AI-USE STATEMENTS

  • Approvals and personnel are synthetic, private conversation links and account identifiers are omitted, and the benchmark provides no deployment-safety assurance.AI assisted drafting, source review, code, artifact extraction, and Figures 2 and 7; AI review is not independent human annotation.Source: ETHICS AND AI-USE STATEMENTS · p. 9

A EVIDENCE LEDGER AND SNAPSHOT SELECTION

  • Table 4 specifies the evidence available for each main-results cell, while its “first delivery” label is a recorded selection rule rather than a verified universal first-attempt claim.The records support check-level analysis, not complete execution replay.Source: A EVIDENCE LEDGER AND SNAPSHOT SELECTION · p. 11, Table 4 · p. 11
  • WorkBuddy attempt histories and selection rules are retained rather than reconstructed as a common first-attempt rate.Easy tasks E2-003 and E2-004 reached formal delivery on the third and second attempts, respectively; Medium excludes initial DNS failures on OBM-055/056 from its final-delivery record.Source: A EVIDENCE LEDGER AND SNAPSHOT SELECTION · p. 11
  • The Doubao Medium prefix changed from 30/63 with mean 77.54 to 35/70 with mean 78.36 across different coverage freezes, not repeated trials.Hard historical means vary, but every stored stage has zero strict passes; reruns, interaction corrections, and evaluation changes prevent attributing the trajectory to one capability improvement.Source: A EVIDENCE LEDGER AND SNAPSHOT SELECTION · p. 11
  • Composites are equal-task averages of stored scores, and aggregate S/P means cannot reconstruct them because task-type weights and denominators vary.For Codex Hard, conditioning on 49 package-valid triples gives S = 44.68 and P = 81.47, versus all-task means of 43.78 and 79.84.Source: A EVIDENCE LEDGER AND SNAPSHOT SELECTION · p. 11
  • WorkBuddy Medium has 22 tasks with component-based preservation failures: 18 fail the check named preservation, while the other four fail under different check identifiers.The 33 rejected tasks comprise eight target-perfect preservation failures, eleven preservation-perfect target failures, and fourteen with both components below 100.Source: A EVIDENCE LEDGER AND SNAPSHOT SELECTION · p. 11

A.1 WORKBUDDY OPERATOR-SUPPLIED CASE ANALYSIS

The case analyses combine archived WorkBuddy checks and direct Codex artifact inspection to distinguish recorded outcomes from explanations of how edits were made. They show that passing requested-value checks can coexist with failures in protected state, formulas, or related content.

A.1 WORKBUDDY OPERATOR-SUPPLIED CASE ANALYSIS

  • The WorkBuddy analysis relies on archived checks and operator explanations, while Codex cases use direct artifact inspection rather than historical execution replay.The Codex extraction is purposive and programmatic, not blind human annotation, native rendering, or an error-rate estimate.Source: §A.1 · p. 12, §A.2 · p. 12
  • OBM-034 passes recalculation and native chart-data checks but fails formula checks; the operator attributes this to replacing a formula with a constant, which logs do not trace.OBM-058 and OBM-062 also record on-time classifications and an omitted invoice condition where delayed classification and the condition were required.Source: §A.1 · p. 12
  • WorkBuddy records show target and non-target failures across Easy cases, but some reported mechanisms are not independently established by the stored checks.For E2-013, the operator reports correct requested values, but those values are absent from check details; the reported renumbering cause in DOCX cases is also unverified.Source: §A.1 · p. 12

A.2 ARTIFACT-LEVEL CASE INSPECTION

  • Codex’s eight Easy XLSX outputs pass all target checks but fail strict contracts because the Lists worksheet changes from hidden to visible.Other checked structural properties remain unchanged; the inspection does not establish that the workbooks are unusable.Source: §A.2 · p. 12
  • In OBM-037, the chart and embedded workbook contain the approved January value, but a rounded reference triggers a frozen numeric-check failure.OBM-035’s required literal exclusion phrase is absent despite the relevant numbers and cancellation information; these observations raise equivalence and authorization questions, not complete human-acceptance findings.Source: §A.2 · p. 12

B DATASET COMPOSITION AND INTENDED USE

The benchmark combines shared-source Office tasks and workflow profiles with strict contracts and decomposed verifier outcomes. Its archived results expose gaps between delivery, target completion, and strict acceptance, while the authors distinguish those automatic judgments from independent validation, which remains unconducted.

B DATASET COMPOSITION AND INTENDED USE

  • Language, file format, and request type co-vary in the active suite, so language-slice scores do not isolate a language capability effect.The stated intended use is local diagnostics and verifier research, not contamination-resistant ranking or unattended consequential deployment.Source: B DATASET COMPOSITION AND INTENDED USE · p. 13
  • Hard comprises 50 tasks across four shared source scenarios and fourteen workflow profiles, so task IDs are variations over shared workspaces rather than independent organizations.The set includes eight tasks requiring first-turn clarification; profiles do not eliminate template dependence.Source: B DATASET COMPOSITION AND INTENDED USE · p. 13

C COHORT AND PRESERVATION DIAGNOSTICS

  • WorkBuddy and Codex have stored strict failures despite some perfect component scores; comparable Doubao component counts are unavailable for Medium.Figure 5 partitions Medium outcomes into strict passes, perfect semantic scores that still fail strict success, and remaining outcomes; these are verifier results, not human judgments.Source: C COHORT AND PRESERVATION DIAGNOSTICS · p. 15, Figure 5 · p. 15

C.1 SHARED-SOURCE COMPOSITION AND SENSITIVITY

  • Leave-one-source-scenario-out Hard composite means vary by system, but the analysis is descriptive and its zero strict-pass result is algebraically expected, not independent robustness evidence.The reported retained-mean ranges are 43.58–49.05 for WorkBuddy, 42.40–44.06 for Doubao, and 50.54–52.12 for Codex.Source: §C.1 · pp. 15–16

D MEASUREMENT VALIDITY AND CONFIRMATORY PROTOCOL

  • The measurement section separates construction QA and archived-record analysis from independent human/native-application validation, which the study has not conducted.Construction probes are not agent executions or a human baseline; the proposed protocol reviews accepted and rejected outputs, with sampling and explicit denominators.Source: D MEASUREMENT VALIDITY AND… · pp. 16–18, Figure 6 · p. 17

E COMPLETE EASY V2 OUTCOMES

  • The Easy v2 outcome table identifies eight Codex structural failures, while the five reported WorkBuddy style-ID cases are not reclassified as successes without independent equivalence judgments.All other Easy Codex checks pass.Source: E COMPLETE EASY V2 OUTCOMES · p. 20

F COMPLETE MEDIUM OUTCOMES

  • The Medium outcome table covers all 70 tasks for each system, preserving selected-snapshot delivery failures and retry policies rather than reconstructing first-attempt scores.The results are not uniform first-attempt comparisons.Source: F COMPLETE MEDIUM OUTCOMES · p. 21, Table 14 · p. 21

C COHORT AND PRESERVATION DIAGNOSTICS

  • All Hard strict verdicts are N, including package-valid deliveries; package validity alone does not establish verifier-defined acceptance.Figure 4 contrasts package-valid triples with zero strict passes; these are frozen verifier outcomes, not human acceptability judgments.Source: Figure 4 · p. 14, G COMPLETE HARD OUTCOMES · p. 22

H RELATIONSHIP TO EXISTING EVALUATION SETTINGS

OfficeEditBench is positioned within an existing, overlapping area of preservation-aware Office evaluation rather than as an entirely new evaluation principle. Its contribution is the specific task contracts and diagnostic suite, with independent validity evidence from adjacent work remaining complementary.

H RELATIONSHIP TO EXISTING EVALUATION SETTINGS

Related papers

Loading 2609.34469v1…