PaperlayerView exact revision on arXiv

Source-linked AI summary

Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise Workflows

Haoyu Dong, Pengkun Zhang, Yan Gao, Xuanyu Dong, Yilin Cheng, Mingzhe Lu, Zikun Zhu, Adina Yakefu, Shuxin Zheng

arXiv:2512.13168v5cs.AIcs.CEcs.IRcs.MA

TL;DR

Assessing complex finance and accounting workflows requires judging structural correctness and semantic fidelity beyond exact matching. FINCH brings together annotated enterprise workflows and human and automated evaluation; even its strongest evaluated agents pass fewer than half, with GPT 5.1 Pro reaching 38.4% under human evaluation after 16.8 minutes per workflow.

  • Problem

    Evaluating complex finance and accounting workflows requires assessing structural correctness and semantic fidelity beyond exact matching.

    Source: §1 · p. 2

  • Method

    FINCH combines expert-annotated enterprise workflows derived from emails, spreadsheet histories, and financial artifacts with expert human evaluation and a multimodal LLM-as-judge pipeline.

    Source: §1 · pp. 2–3, §5 · p. 14

  • Results

    Even the strongest frontier agents pass fewer than 50% of FINCH workflows; GPT 5.1 Pro averages 16.8 minutes per workflow and achieves 38.4% under human evaluation.

    Source: §1 · p. 3

  • Takeaways & Limitations

    FINCH reveals a substantial gap between current AI capabilities and the demands of real enterprise practice.

    Source: §5 · p. 14

    Automated evaluation can miss subtle spreadsheet errors or over-penalize benign conversions, while tooling and data-pipeline limits can mark valid outputs as failures.

    Source: §3.2.1 · p. 12

Abstract

from arXiv · show

We introduce FinWorkBench (a.k.a. Finch) for evaluating AI agents on real-world, enterprise-grade finance and accounting workflows that interleave data entry, structuring, formatting, web search, cross-file retrieval, calculation, modeling, validation, translation, visualization, and reporting. Finch is sourced from authentic enterprise workspaces from Enron (15,000 files and 500,000 emails) and other financial institutions, covering the period 2000--2025 and preserving the in-the-wild messiness of multimodal artifacts such as tables and charts across diverse domains including budgeting, trading, asset management, and operational management. We propose a workflow construction process that combines LLM-assisted mining of workflows from authentic enterprise environments with expert annotation: (1) LLM-assisted, expert-verified derivation of workflows from real-world email threads and spreadsheet version histories, and (2) meticulous annotation requiring over 700 hours of expert effort. This yields 172 composite workflows with 384 tasks, involving 1,710 spreadsheets with 27 million cells, along with PDFs and other artifacts, capturing the intrinsically messy, long-horizon, knowledge-intensive, and collaborative nature of real-world enterprise work. We conduct both human and automated evaluations of frontier AI systems, including GPT-5.1, Claude Sonnet 4.5, Claude Opus 4.5, Gemini 3 Pro, Grok 4, and Qwen 3 Max. Under human evaluation, GPT-5.1 Pro spends an average of 16.8 minutes per workflow yet passes only 38.4% of workflows. Comprehensive case studies further surface the challenges that real-world enterprise workflows pose for AI agents.

1 Introduction

FINCH targets the messy, long-horizon, knowledge-intensive workflows of finance and accounting by building a benchmark from authentic enterprise artifacts and evaluating frontier agents. Even the strongest systems pass fewer than half of its workflows, with case analyses linking failures to interacting sources of real-world complexity.

1 Introduction

  • FINCH addresses whether frontier agents can handle finance and accounting work that is messy, long-horizon, and knowledge-intensive.Work spans interconnected, evolving spreadsheets and other artifacts, with cross-sheet references, complex layouts, inconsistent formatting, cryptic terms, and multimodal content.Source: §1 · p. 2
  • Its construction pipeline derives workflows from enterprise email threads and spreadsheet version histories, using LLM assistance with expert verification and annotation.Annotators infer analysts’ goals from multi-sheet workbooks and subtle version changes, rather than curating isolated table questions.Source: §1 · p. 2
  • FINCH contains 172 annotated workflows built on 1,710 spreadsheets, plus PDFs and other artifacts, covering interdependent tasks from data entry through reporting.A 20-workflow case study found a median of 16 tool calls per workflow, with a range of 6–107.Source: §1 · p. 2
  • Evaluation combines expert human judgments of end-to-end completion with a scalable multimodal LLM judge that compares inputs, outputs, and reference artifacts.The judge uses structured diffs, compact snapshots, and multimodal renderings; Figure 2 presents overall workflow success rates for product-side agents and API-based models.Source: §1 · pp. 2–3, Figure 2 · p. 2
  • Even the strongest frontier agents pass fewer than 50% of FINCH workflows; GPT 5.1 Pro takes 16.8 minutes per workflow and passes 38.4% under human evaluation.Case analyses associate degradation with the combination of messy inputs, long-horizon dependencies, domain knowledge, and heterogeneous formats.Source: §1 · p. 3
  • Figure 3 illustrates a financial analyst’s predictive-modeling workflow, from web search and data import through retrieval, modeling, visualization, and report generation.The illustrated sequence includes cross-sheet and cross-file retrieval, calculation, and financial modeling.Source: Figure 3 · p. 3

2 FINCH: A Real-world Finance & Accounting Workflow Benchmark

FINCH builds workflows from enterprise emails, spreadsheet histories, and deliverable artifacts, then evaluates agents on composite finance and accounting work. Its dataset spans varied task and business types, irregular spreadsheets, and multimodal files; evaluation combines human review with an LLM judge.

2.1 Dataset Construction

  • FINCH derives workflows from enterprise email threads, spreadsheet version histories, and high-quality spreadsheets and reports, with experts turning these sources into instructions and input/reference artifacts.Email-derived cases may be strongly grounded in attached input and final files or partially grounded, requiring experts to identify or create missing artifacts; version-history candidates are checked for coherent workflows and refined against actual workbook versions.Source: §2.1 · p. 3, §2.1.1 · p. 4, §2.1.2 · pp. 4–5

2.1.3 Workflow Sourced from Final Deliverable Spreadsheets and Reports

  • For deliverable-based workflows, domain experts turn artifacts such as valuation models, reports, and bilingual documents into modeling, visualization, translation, and consistency-checking tasks.The construction also adapts 10 WideSearch cases and 3 DABStep examples into multi-step workflows.Source: §2.1.3 · p. 5

2.1.4 Quality Control

  • Quality control uses expert review, iterative revisions, and secondary AI checks verified by humans; annotation and review required over 700 hours.About 40% of workflows underwent at least one revision, and more than 20 went through three or more rounds.Source: §2.1.4 · p. 5

2.2 Dataset Characteristics

  • The benchmark covers nine task categories, led by calculation (119 workflows) and structuring/formatting (86), and multiple finance and business areas, led by reporting (48).Other business areas include trading and risk management, predictive modeling, operations, planning and budgeting, valuation, accounts payable/receivable, procurement and sales, and asset management.Source: §2.2 · p. 6

2.2.1 Task Compositionality

  • 135 workflows (78.5%) involve multiple interleaved tasks, often progressing from data structuring or import to retrieval and then calculation, modeling, or reporting.Tasks share spreadsheets and files rather than forming independent subtasks.Source: §2.2.1 · p. 6, Figure 5 · p. 6

2.2.2 Messiness

  • 92.4% of workflows involve multiple input and reference sheets, with an average of 8 sheets and a long tail reaching 91; the median workflow covers 15K cells.Workflows touch up to 14 files, and the largest covers 3.7 million cells; complex layouts mix text, values, formulas, charts, and irregular tables.Source: §2.2.2 · p. 7, Figure 6 · p. 7

2.2.3 Multimodality

  • Around 10.5% of workflows link spreadsheets to non-spreadsheet artifacts, while 7.6% require reasoning over PDFs or images; 20.3% include charts.The benchmark also includes workflows with pivot tables and other aggregation structures.Source: §2.2.3 · p. 7

2.3.1 Human Evaluation

  • Human evaluation assigns workflow-level pass/fail based on instruction fulfillment and critical errors, while the LLM judge handles modify, generate, and QA tasks using artifact diffs, snapshots, screenshots, and task-specific rubrics.The judge assesses completeness, correctness, over-edit avoidance, and readability without requiring exact reference equality when multiple solutions are acceptable; human judgments serve as the gold standard.Source: §2.3.1 · p. 7, §2.3.2 · p. 8

3 Experiments

The experiments compare product-side and API-based agents on FINCH, finding that even leading systems struggle with multi-step workflows. Error analyses identify interacting workflow and artifact complexities behind failures.

3.1.1 Product-side Agents

  • Product-side evaluations use GPT 5.1 Pro and Claude Sonnet/Opus 4.5 with browsing enabled and historical chats disabled, evaluating each workflow independently.The study uses final-round model versions because updates are frequent and manual evaluation is time-consuming.Source: §3.1.1 · p. 9

3.1.2 API-based Models

  • The API evaluation adapts SpreadsheetBench for larger, messier, long-horizon workflows and encodes cells with addresses, values, types, and formulas to preserve spreadsheet structure and semantics.Vision-capable models receive images through provider APIs; PDF handling varies by model, and Qwen 3 Max receives text descriptions that lose layout and visual context. Large inputs are truncated from the end after reserving 32K output tokens, with a notice of data loss.Source: §3.1.2 · pp. 9–10

3.2 Experimental Results

  • Both product-side agents pass fewer than 50% of workflows; beyond two tasks, GPT 5.1 Pro falls from 44.3% to 23.5% and Claude Sonnet 4.5 from 30.3% to 11.8%.The pass-rate comparison treats a multi-task workflow as successful only when all its tasks are completed successfully.Source: §3.2 · p. 10, Figure 8 · p. 11
  • Data Entry / Import and Structuring / Formatting are among the hardest task types, while Translation also performs poorly when preserving financial-table structure and layout is difficult.Single-task GPT 5.1 Pro workflows average 13.1 minutes, versus 17.4 minutes for two-task, 18.7 for three-task, and 17.4 for four-or-more-task workflows; on non-spreadsheet workflows, GPT 5.1 Pro passes 35.0% and Claude Sonnet 4.5 passes 25.0%.Source: §3.2 · pp. 10–11, Table 2 · p. 10, Table 3 · p. 11
  • Under automated evaluation, GPT 5.1 Pro passes 41.9% of workflows versus 32.0% for the GPT 5.1 API-based agent, whose single-call design lacks iterative interaction and self-correction.The API design narrows the gap with more efficient spreadsheet encodings and task-appropriate tool outputs within its single-call budget.Source: §3.2 · pp. 11–12

3.2.1 Consistency Between Human and Automated Evaluation

  • The LLM judge agrees with human labels on 82.1% of GPT 5.1 Pro workflows and 90.2% of Claude Sonnet 4.5 workflows, but automated scores remain approximate.The judge can miss subtle errors or interpret instructions too literally, while spreadsheet tooling and data-pipeline limitations can also misclassify valid outputs.Source: §3.2.1 · p. 12, Table 5 · p. 12

3.3 Error Analysis

  • Claude Sonnet 4.5 failures are attributed mainly to formula reasoning (35%), data retrieval (25%), and code generation (25%), amid fragmented spreadsheets, irregular layouts, latent formula logic, and multimodal inputs.The analysis argues that these complexity factors co-occur in FINCH workflows, a composition associated with sharp performance drops.Source: §3.3 · pp. 12–13

4 Related Work

Related work spans enterprise AI tools, agentic task completion, spreadsheet methods, and benchmarks for financial and document reasoning. FINCH focuses on messy, long-horizon workflows in real-world finance and accounting.

4 Related Work

  • Enterprise productivity tools now embed AI for document drafting, spreadsheet analysis, and workflow automation, while agentic systems extend from question answering toward autonomous task completion.Examples include ChatGPT Agent, Microsoft Copilot, Google Gemini, and Claude Excel.Source: §4 · p. 13
  • Spreadsheet research addresses structural encoding and compression [17], formatting, and formula generation through numerical reasoning, synthesis, recommendation, and interactive prediction.Source: §4 · p. 13
  • Prior benchmarks cover financial, spreadsheet, multimodal document, table, chart, and spreadsheet reasoning, whereas FINCH targets messy artifacts and long-horizon workflows in real-world finance and accounting.Source: §4 · p. 14

5 Conclusion

FINCH is presented as a benchmark for evaluating agents on real-world finance and accounting workflows involving large, messy, multimodal artifacts and long-horizon spreadsheet reasoning. Experiments find that even the strongest systems pass fewer than half of workflows.

5 Conclusion

  • FINCH combines workflows derived from enterprise email threads and spreadsheet version histories with high-quality financial artifacts, expert annotation, and a calibrated LLM-as-judge framework.Source: §5 · p. 14
  • Even the strongest frontier systems pass fewer than 50% of workflows after an average of 16.8 minutes per workflow, exposing a gap between current AI capabilities and real enterprise practice.Source: §5 · p. 14
  • The authors propose FINCH as a foundation for developing agents to tackle messy, long-horizon professional work.Source: §5 · p. 14

B Experiment Details

The API experiments treat spreadsheet work as code generation, executing model-written Python in a sandbox to produce output artifacts. The setup uses a one-shot protocol and structured prompting.

B.1 API-based Models

  • Models generate executable Python scripts as the action space for spreadsheet manipulation and generation workflows, and scripts run in an isolated sandbox to produce artifacts.The setup uses libraries including openpyxl, pandas, matplotlib, and scikit-learn, with each execution in a Docker container.Source: §B.1 · p. 21
  • Each model produces one solution per workflow without iterative refinement or retry after code-execution failure, isolating raw code-generation performance from interactive debugging.Source: §B.1 · p. 21
  • Zero-shot prompts specify the model role, spreadsheet encoding, task and file paths, library practices, and required code output; reasoning-capable models also provide reasoning traces for qualitative analysis.Temperature is set to 0.7 across models.Source: §B.1 · pp. 21–22

C Detailed Analysis

The analysis contrasts web-agent strategies and spreadsheet tool use, then introduces a case study measuring tool-call overhead across representative workflows.

C Detailed Analysis

  • ChatGPT 5.1 Pro takes more steps and has higher latency than Claude Sonnet 4.5, while Claude’s charts are often more accurate and polished.The comparison notes that ChatGPT’s intermediate checks create more validation opportunities, but its hidden code limits error attribution to observed behavior and reasoning.Source: C Detailed Analysis · p. 22
  • A single-call API method with efficient encoding and tool use achieves a pass rate closer to product-side agents than the web agents’ spreadsheet encodings.The passage does not identify the method’s specific implementation.Source: C Detailed Analysis · p. 22
  • Single-call API agents lack interleaved execution feedback and reflection, so mistaken spreadsheet assumptions cannot be corrected and errors rise, especially in schema understanding and table manipulation.The passage recommends exploring API agents with multiple rounds of calls.Source: C Detailed Analysis · p. 22

D Tool-Call Overhead Case Study

  • A 20-workflow case study examines how many tool calls an LLM agent needs across tasks of differing types and complexity.The selected examples come from the full set of 172 workflows.Source: D Tool-Call Overhead Case Study · p. 22

D.1 Methodology

The overhead study logs an agent’s tool-use trajectory on selected tasks and checks outputs against reference answers, adjusting failed-run counts to approximate successful completion.

D.1 Methodology

  • Claude Coworker Opus 4.6 solved each selected task in an isolated session, logging every tool call; a second session checked outputs against reference files.The agent configuration was current as of March 2026.Source: §D.1 · p. 22
  • For incorrect outputs, the study distinguishes faulty execution within a complete trajectory from missing critical steps, which occurred twice in 20 examples.Missing steps were added to the original trajectory, so final counts approximate calls needed for correct completion rather than raw failed-run calls.Source: §D.1 · p. 22

D.2 Results and Analysis

The case study finds that web-grounded tasks require substantial retrieval and verification overhead, while non-web call counts vary for reasons beyond task-type labels. Its findings are bounded by a small, nonrandom sample and one agent configuration.

D.2 Results and Analysis

  • Web search and fetch use 60 of 107 calls (56%) in Task 19 and 34 of 71 calls (48%) in Task 88, with external retrieval and validation dominating spreadsheet manipulation.Table 6 summarizes tool-call counts for 20 selected tasks.Source: §D.2 · pp. 22–23, Table 6 · p. 23
  • Without the two web-search tasks, tool calls range from 6 to 25, averaging 13.2 with a median of 14 and an interquartile range of 8.The first and third quartiles are 9 and 17.Source: §D.2 · p. 23
  • After excluding web-search outliers, task-type count shows no clear positive relationship with call count; Task 47 needs nearly twice Task 46’s calls due to added business-logic reasoning.The cited example involves sign conventions, Purchase versus Pooling Method tax-base logic, and business-consistency checks.Source: §D.2 · p. 23
  • The 20 tasks are representative rather than randomly sampled, counts depend on Claude Coworker Opus 4.6, and reconstructed missing steps introduce judgment.The authors characterize the analysis as a case study, not a full-scale statistical evaluation.Source: §D.2 · p. 23

D.3 Representative Tool-Call Logs

  • Task 26 illustrates a standard successful tool-use trajectory, while Task 139 shows critical steps added when the original trajectory omitted them.The added steps are marked with + Bash.Source: §D.3 · p. 23

D.3.1 Task 26: Chart Generation across Scenario Sheets

Task 26 builds two chart types on each of three scenario worksheets, using workbook inspection to identify the relevant data ranges before generation and verification.

D.3.1 Task 26: Chart Generation across Scenario Sheets

  • The task requires six charts: a first-year Hedge Coverage stacked column chart and an all-years Portfolio Cost line chart on each Scenario1–3 worksheet.The Hedge Coverage chart uses months and contract MW as stacked series with Short as a reference line; Portfolio Cost plots Portfolio Weighted Avg and Monthly Price in $/MWh.Source: §D.3.1 · pp. 23–24
  • The workflow confirms the source file, reads task metadata and Excel-operation guidance, then uses openpyxl to inspect workbook sheets, initial rows, column mappings, and Scenario-sheet column semantics and data types.Source: §D.3.1 · p. 24
  • After identifying dates in column Z, rows 4–63, the agent copies the workbook, generates charts with openpyxl, and verifies their count, titles, types, and series.The generated charts are created by running add_charts.py.Source: §D.3.1 · p. 24

D.3.2 Task 139: XNPV Sensitivity Analysis with Corrected Trajectory

The corrected trajectory for Task 139 builds and validates missing intermediate data for an XNPV sensitivity analysis. The ethics statement describes the benchmark’s public data sources, curation safeguards, licensing, and language scope.

D.3.2 Task 139: XNPV Sensitivity Analysis with Corrected Trajectory

  • The corrected log adds steps to construct monthly cash flows, multi-year analysis data, and native Excel XNPV formulas that were absent from the original trajectory.The added work includes annual capacity payments under four capacity assumptions, conversion to monthly sequences, and recalculation and verification of outputs.Source: §D.3.2 · pp. 24–25
  • The trajectory inspects workbook structure, values, formulas, and input variables, including date conventions, discounting logic, and the plant-capacity input.It compares adjustment sheets and validates the sensitivity-table parameters before building the missing data.Source: §D.3.2 · pp. 24–25
  • The completed workbook fills XNPV sensitivity formulas and multi-year Cedar Brakes data, then verifies cash flows, formula results, and summary outputs against the 263 MW baseline.The log also records writing Q1–Q4 capacity values and deltas in the Summary sheet.Source: §D.3.2 · p. 25

E Ethics Statement

  • FINCH workflows draw on public Enron and EUSES corpora and other institutional and public artifacts, with upstream licenses respected and redistribution limited to their terms.The listed sources include the World Bank, Canadian and British government websites, WideSearch, and DABStep.Source: E Ethics Statement · p. 26
  • Curation filters and normalizes source materials, removes unnecessary obviously sensitive fields, avoids tasks dependent on sensitive personal attributes, and releases FINCH under CC BY 3.0 US.The authors state that curation introduces no new personally identifiable information.Source: E Ethics Statement · p. 26
  • FINCH is primarily English, while a small fraction of workflows include Chinese or French content.The authors connect this language distribution to the underlying corpora and some institutional sources.Source: E Ethics Statement · p. 26

F Examples

The examples illustrate spreadsheet-centric enterprise workflows involving cross-sheet checks, document-grounded extraction, financial modeling, transformation, and reporting. Their figures are illustrative excerpts, not complete workflow inputs or specifications.

F.1 Example 1

  • Headcount verification requires reconciling 39 department summaries against detailed rosters, correcting counts and membership while handling slight schema variation.The task includes removing obsolete departments and adding omitted ones.Source: Figure 9 · p. 27

F.2 Example 2

  • PDF table-and-chart extraction requires interpreting layout and hierarchy, mapping values to spreadsheet cells, and reconstructing formulas from aggregates.Source: Figure 10 · p. 28

F.3 Example 3; F.5 Example 5

  • Cross-sheet reference validation is presented as relatively easy for frontier agents, while other checks require retrieval and calculation across sheets.The latter examples include the sum of A&B and the equity roll-forward test.Source: Figure 11 · p. 28, Figure 13 · p. 29

F.4 Example 4

  • The XNPV example requires retrieving contract adjustments from supporting documents, constructing monthly cash flows under alternative assumptions, and discounting them to a valuation date.The stated inputs include capacity rates, plant capacity, and a discount rate; adjustments may be included or excluded by scenario.Source: Figure 12 · p. 29

F.6 Example 6; F.9 Example 9

  • Two document transformations require preserving structure: translating a French report with tables and charts, and converting spreadsheet values to USD while retaining formula references and layout.Source: Figure 14 · p. 30, Figure 17 · p. 31

F.7 Example 7

  • Table restructuring requires cross-sheet retrieval and aggregation, while distinguishing value-driven from volume-driven operators and validating the reorganized data.Source: Figure 15 · p. 30

F.8 Example 8

  • Formula inspection can overturn apparent header semantics: one column encodes a 55-day payment timing schedule, and misreading it can propagate through later computations.The example formula draws on values from multiple sheets.Source: Figure 16 · p. 31

F.10 Example 10; F.11 Example 11

  • The examples also cover reporting from tabular data and an assumption update that calculates blended prices and portfolio costs and charts contract coverage by type.Reporting may use a secondary y-axis to plot series with different units.Source: Figure 18 · p. 32, Figure 19 · p. 32
Loading 2512.13168v5…