SpreadsheetBench 2: Evaluating Agents on End-to-End Business Spreadsheet Workflows

1 Introduction

• Existing benchmarks focus on isolated operations, small workbooks, and synthetic or forum-derived tasks rather than complete professional business workflows.Real spreadsheet work also involves cross-sheet reasoning, long-range dependencies, and multi-step coordination.Source: §1 · pp. 1–2
• SpreadsheetBench 2 evaluates generation, debugging, and visualization as stages of workflow-level tasks grounded in professional business scenarios.Generation covers financial modeling and template completion; visualization tasks produce analysis-ready charts.Source: §1 · p. 2, Figure 1 · p. 2
• The benchmark contains 321 tasks built from authentic business data and expert-annotated instructions, with complex multi-sheet structures and cross-sheet dependencies.Construction required over 1,500 hours of expert effort.Source: §1 · p. 2
• Current LLMs and spreadsheet products achieve below 35% overall accuracy, with substantially lower debugging performance, and failures primarily involve insufficient inspection and incorrect target-cell selection.The study also reports that its SWE-agent-based scaffold outperforms three coding-agent scaffolds on 50 samples in task-level accuracy despite comparable modification scores.Source: §1 · p. 2

2 SPREADSHEETBENCH 2

• The benchmark is designed to reflect realistic business workflows through real-world task formulation, authentic business data, and evaluation of both cell-level and task-level outcomes.The section describes task categories, dataset construction, and evaluation metrics.Source: §2 · p. 3

2.1 Task Categories

• Generation builds or completes workbooks, debugging identifies and repairs unspecified errors, and visualization turns spreadsheet data into analysis-ready charts and summaries.Generation includes simpler template tasks averaging 1.2 sheets and financial modeling tasks averaging 15.3 sheets; debugging requires locating errors without specified types or target cells, while visualization may draw on irregular data across sheets.Source: §2.1 · p. 3

2.2 Benchmark Construction

• Experts construct gold-standard workbooks from authentic financial sources, transform them into instructed partial tasks, and validate each task through independent expert review and discrepancy resolution.Tasks are designed to have a unique deterministic solution and require multi-step reasoning rather than isolated edits.Source: §2.2 · p. 4

2.3 Benchmark Statistics

• The 321-task benchmark averages 11.8 worksheets and 593.5 modified cells per task; Financial Modeling averages 1,164.5 modified cells, while Debugging averages 656.6.Compared with prior benchmarks, it averages 11.8 sheets per file versus roughly 1–2, and instructions average 429.0 words; it also adds systematic debugging, native chart generation, and workflow-level evaluation.Source: §2.3 · pp. 4–5, Table 1 · p. 5, Table 2 · p. 5

2.4 Evaluation Metrics

• Cell-based tasks are evaluated against a gold spreadsheet using target-cell Modification and full-spreadsheet Accuracy, while visualization uses expert-designed rubric pass rates from a VLM judge.Visualization rubrics assess data correctness and format compliance.Source: §2.4 · pp. 5–6, Table 3 · p. 6

3.1 Experimental Settings

• Eight language models were evaluated through a shared SWE-agent-based scaffold, alongside spreadsheet products as complementary baselines.The scaffold uses bash, view_xlsx, and submit, with a maximum of 50 interaction turns.Source: §3.1 · p. 6

3.2 Main Results

• 34.89% overall accuracy for Claude Opus 4.6, but only 12.00% Accuracy on Debugging; visualization is easiest and debugging hardest.Claude Opus 4.6 reaches 89.69% Modification but 34.00% Accuracy on Financial Modeling, illustrating the gap between cell edits and end-to-end correctness.Source: §3.2 · p. 7
• On a 30-example subset, none of four spreadsheet products surpasses foundation models; Claude for Excel leads the products at 15.4% accuracy.Human operators tested the products on 10 examples each from Financial Modeling, Debugging, and Visualization.Source: §3.2 · p. 7, Figure 3 · p. 7

3.3 Analysis and Discussions

• Errors was the hardest debugging subcategory, barely exceeding 20% for the best model, while Insufficient Inspection and Wrong Target Selection dominated unresolved-task failures.Errors involve diagnosing formulas that produce explicit spreadsheet error values, often linked to complex cross-sheet dependencies or invalid references.Source: §3.3 · pp. 7–8, Figure 4 · p. 8, Figure 5 · p. 8
• Claude Opus 4.6 outperformed MiniMax M2.5 across task categories with fewer average interaction steps and a substantially higher effective-step ratio.An effective step is executed without errors and directly contributes to task progress.Source: §3.3 · p. 8, Figure 6 · p. 9
• With GLM-5 fixed on 50 samples, the proposed scaffold achieved 46.32% modification score and 15.45% accuracy, versus 43.47% / 14.20% for Claude Code.Cline and Kilo Code each achieved only 8.66% accuracy, despite modification scores staying within a narrow four-point band.Source: §3.3 · p. 8, Table 4 · p. 8
• Claude Opus 4.6 inspected more in Template, Visualization, and Financial Modeling, but both models devoted over 60% of Debugging trajectories to inspection.The authors attribute this reversal to debugging instructions that lack explicit step-by-step guidance, requiring extensive exploration and information localization.Source: §3.3 · pp. 8–9, Figure 6 · p. 9
• In a banking-model case, textbook-correct formulas yielded only 4.1% modification accuracy because their cell references did not match the workbook’s row layout.The task required filling six groups of empty cells across multiple sheets; the example illustrates the need to ground domain knowledge in workbook dependencies.Source: §3.3 · p. 9

Spreadsheet and Table Benchmarks

• SpreadsheetBench 2 covers end-to-end generation, debugging, and visualization tasks, averaging 11.8 sheets and 593.5 modified cells per instance.These tasks target multi-step, cross-sheet workflows characteristic of real business spreadsheet use.Source: §4 · p. 9
• Prior table benchmarks span question answering, fact verification, and numerical reasoning, while spreadsheet benchmarks cover representations and natural-language manipulation.Spreadsheet manipulation work includes formula generation, cell editing, and task completion, but predominantly focuses on atomic operations.Source: §4 · p. 9

LLM-based Spreadsheet Agents

• Existing spreadsheet agents use varied strategies, but their evaluations leave realistic business workflows requiring sustained cross-sheet reasoning and coordinated tool use unassessed.SpreadsheetBench 2 provides a testbed for comparing agents on multi-step reasoning and structurally coherent spreadsheet artifacts.Source: §4 · p. 9

5 Conclusion

• 34.89% accuracy is the best model's result on the 321-task benchmark, with debugging accuracy at 12%.The benchmark covers workflow-level generation, debugging, and visualization on multi-sheet workbooks with rich cross-sheet dependencies.Source: §5 · p. 10
• The benchmark evaluates generation, debugging, and visualization using 321 expert-annotated tasks over multi-sheet workbooks with rich cross-sheet dependencies.Source: §5 · p. 10
• Stronger models benefit from more reliable per-step execution rather than larger interaction budgets, while structured reasoning over interdependent workbooks remains a key bottleneck.Source: §5 · p. 10

A.1 Limitations

• The benchmark focuses on financial and business workflows; its findings may not generalize to scientific computing or engineering, and evaluation uncertainty and scoring noise remain concerns.API inference costs precluded error bars or confidence intervals; VLM judging may add noise, and exact-match scoring may miss semantically equivalent cell-task solutions.Source: §A.1 · p. 14

A.2 Ethical Considerations

• Authors reviewed files and instructions for inappropriate content and independently checked for personally identifiable information; source financial data came from public filings and reports.Source: §A.2 · p. 14

A.3 Broader Impacts

• The authors note potential benefits from reduced repetitive work and error detection, but warn that inadequate oversight could leave spreadsheet errors undetected.The benchmark's 34.89% best-model accuracy is presented as a reason to discourage premature deployment and encourage reliability research.Source: §A.3 · p. 14

A.4 License

• The dataset uses a CC BY-SA 4.0 license, while the evaluation code uses the MIT License.Source: §A.4 · p. 14

B.1 Data Sources

• Financial data sources include publicly available regulatory filings and datasets, with coverage across multiple geographies and industry sectors.For sources with redistribution restrictions, the authors release derived task metadata and evaluation artifacts where permitted and document access and licensing constraints.Source: §B.1 · pp. 14–15

B.2 Annotation Effort

• Annotation exceeded 1,500 hours across four task categories, excluding additional quality-assurance rounds and solvability checks.Reported effort included around 400 hours for Financial Modeling, 500 for Debugging, 380 for Visualization, and 200 for Template tasks.Source: §B.2 · p. 15

B.3 Visualization Task Rubric Examples

• Visualization rubrics assess format compliance and data correctness, with each criterion independently verified by a VLM judge.The example rubric concerns Task 129, a football field valuation chart.Source: §B.3 · p. 15, Table 5 · p. 15

B.4 Debugging Task Types

• Table 6 presents ten error categories identified through iterative annotation of real-world financial-modeling and accounting errors.Source: §B.4 · p. 16, Table 6 · p. 16
• The taxonomy spans low-level syntactic issues, including formula errors and incorrect sign assignments, and structural problems such as cross-sheet reference failures and unit mismatches.Source: §B.4 · p. 16
• Each debugging instance contains one or more injected errors, and agents must localize and correct all of them to restore the spreadsheet's intended state.Source: §B.4 · p. 16

C.1 Environment.

• Each task runs in a network-disabled Docker container with Python 3.11, common data-science libraries, LibreOffice, and its Python UNO bridge.The setup supports both programmatic and application-level spreadsheet manipulation.Source: §C.1 · p. 16

C.2 Tools.

• The agent uses bash for programmatic access, view_xlsx to inspect workbook sheets and formulas, and submit to finalize its verified spreadsheet.view_xlsx is read-only and can list sheets or display selected sheet content, including formulas and evaluated values.Source: §C.2 · pp. 16–17

C.3 Prompt.

• A system prompt sets the interaction protocol, while separate instance-prompt variants provide task instructions for generation and debugging versus visualization.The visualization variant adds a visualization-focused objective and restricts the library set; prompts instruct agents to modify only required cells and preserve formatting.Source: §C.3 · p. 17, Figure 7 · p. 17, Figure 8 · p. 17
• Figure 10 tracks error occurrence, per-error fixes, and end-to-end passes; MiniMax can fix individual errors but rarely produces a correct final output on error-affected tasks.The figure covers Template, Financial Model, and Debugging tasks.Source: Figure 10 · p. 18

D.1 Error-Recovery Analysis

• 72% on Template, 47% on Financial Model, and 34% on Debugging errors fixed by MiniMax M2.5 correspond to final pass rates of only 7%, 9%, and 3%, respectively.By contrast, Opus 4.6's Financial Model rate drops from 50% after fixing an error to 45% passing end-to-end.Source: §D.1 · p. 19
• MiniMax encounters errors in 62–88% of tasks, compared with 16–67% for Opus 4.6, so it both recovers worse and needs to recover more often.Figure 10 distinguishes error occurrence, fixing at least one error, and passing end-to-end among tasks with errors.Source: §D.1 · pp. 18–19

D.3 Failure Taxonomy

• Six mutually exclusive primary-cause categories organize failed trajectories, with Insufficient Inspection and Wrong Target Selection identified as dominant modes.The taxonomy separates comprehension, execution, planning-efficiency, and other failures.Source: §D.3 · p. 19, Table 7 · p. 20

D.4 Failure Cases

• The six representative cases come from Claude Opus 4.6 failed trajectories, with two examples each from Debugging, Financial Modeling, and Visualization.They illustrate reasoning breakdowns leading to partial or complete task failure.Source: §D.4 · p. 19

D.2 Fine-Grained Evaluation of Visualization Tasks

• Claude Opus 4.6 leads Overall at 76.2, followed by GPT-5.2 at 70.4 and Gemini 3.1 Pro at 69.0; Figure 11 reports rubric-based scores across data correctness and format compliance.Every model scores higher on Format Compliance than Data Correctness; Claude Opus 4.6 scores 82.0 versus 72.7, while MiniMax M2.5 scores 64.7 versus 31.4.Source: §D.2 · p. 19, Figure 11 · p. 20

D.4.1 Debugging Cases

• In debugging, one agent found three double-counting bugs but fixed only two, while another drifted from requested color fixes into harmful, out-of-scope edits.The latter changed C90 from −0.3 to 0.3 rather than recoloring the font.Source: §D.4.1 · p. 20

D.4.2 Financial Modeling Cases

• The Debt/Equity formula copied an adjacent ratio's numerator pattern without checking definitions, although Debt/Equity used long-term borrowings rather than both long- and short-term borrowings.The agent reasoned through more complex subtasks but treated this formula as a trivial copy operation.Source: Figure 14 · p. 22, §D.4.2 · p. 22
• A banking model's formulas were conceptually reasonable but mismatched its actual layout, yielding ∼4% cell accuracy and cascading errors after previously empty cells were filled.The agent relied on abstract financial knowledge instead of verifying workbook row and column structure.Source: §D.4.2 · p. 22, Figure 15 · p. 23

D.4.3 Visualization Cases

• For clustered-stacked charts unsupported by openpyxl, an unverified dual-axis workaround produced overlapping bars, and the agent claimed verification despite negative or empty checks.The case describes the agent dismissing its own anticipated overlap risk and contradicting its tool output.Source: §D.4.3 · p. 23, Figure 16 · p. 24
• Unable to create a sunburst with available libraries, the agent rejected a nested doughnut approximation and injected raw XML, but evaluation found no chart objects.The nested doughnut approximation was supported by openpyxl and likely would have earned partial credit.Source: §D.4.3 · p. 23, Figure 17 · p. 24

• Figure 18 provides an example of a debugging task.Source: Figure 18 · p. 25
• Figure 19 provides an example of a financial modeling task.Source: Figure 19 · p. 26
• Figure 20 provides an example of a template task.Source: Figure 20 · p. 27
• Figure 21 provides an example of a visualization task.Source: Figure 21 · p. 28