PaperlayerView exact revision on arXiv

Source-linked AI summary

FinMaster: A Holistic Benchmark for Mastering Full-Pipeline Financial Workflows with LLMs

Junzhe Jiang, Chang Yang, Aixin Cui, Sihan Jin, Ruiyu Wang, Bo Li, Xiao Huang, Dongning Sun, Xinrun Wang

arXiv:2505.13533v1cs.AIcs.LGq-fin.GN

TL;DR

Existing finance benchmarks fall short of real-world financial complexity. FinMaster combines synthetic data, a 183-task suite, and unified evaluation; experiments show a sharp accuracy drop from fundamental to complex tasks.

  • Problem

    Existing LLM finance benchmarks fall short on the complexity, dynamics, and domain-specific nuances of real-world finance, amid financial tasks' fragmented data and low error tolerance.

    Source: §1 · pp. 1–2, §5 · p. 9

  • Method

    FinMaster combines synthetic financial-data simulation, 183 financial tasks, and unified evaluation to assess LLMs across financial literacy, accounting, auditing, and consulting.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    96% accuracy on fundamental financial literacy tasks falls to 40% on complex tasks requiring rigorous multi-source data integration and domain-specific reasoning.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Computational error propagation significantly impairs accuracy on complex financial tasks.

    Source: §5 · p. 10

    FinMaster's current financial tasks are text-based, while real-world financial analysis can involve multimodal data such as charts and scanned documents.

    Source: §A.4 · p. 13

Abstract

from arXiv · show

Financial tasks are pivotal to global economic stability; however, their execution faces challenges including labor intensive processes, low error tolerance, data fragmentation, and tool limitations. Although large language models (LLMs) have succeeded in various natural language processing tasks and have shown potential in automating workflows through reasoning and contextual understanding, current benchmarks for evaluating LLMs in finance lack sufficient domain-specific data, have simplistic task design, and incomplete evaluation frameworks. To address these gaps, this article presents FinMaster, a comprehensive financial benchmark designed to systematically assess the capabilities of LLM in financial literacy, accounting, auditing, and consulting. Specifically, FinMaster comprises three main modules: i) FinSim, which builds simulators that generate synthetic, privacy-compliant financial data for companies to replicate market dynamics; ii) FinSuite, which provides tasks in core financial domains, spanning 183 tasks of various types and difficulty levels; and iii) FinEval, which develops a unified interface for evaluation. Extensive experiments over state-of-the-art LLMs reveal critical capability gaps in financial reasoning, with accuracy dropping from over 90% on basic tasks to merely 40% on complex scenarios requiring multi-step reasoning. This degradation exhibits the propagation of computational errors, where single-metric calculations initially demonstrating 58% accuracy decreased to 37% in multimetric scenarios. To the best of our knowledge, FinMaster is the first benchmark that covers full-pipeline financial workflows with challenging tasks. We hope that FinMaster can bridge the gap between research and industry practitioners, driving the adoption of LLMs in real-world financial practices to enhance efficiency and accuracy.

1 Introduction

Existing financial LLM benchmarks have limited coverage of dynamic, domain-specific reasoning and practical workflows. FinMaster combines synthetic-data simulation, financial task evaluation, and a unified evaluation interface; experiments reveal a sharp accuracy drop on complex, multi-step tasks.

1 Introduction

  • Financial work is labor-intensive and error-sensitive, while fragmented data and limited tools complicate accurate, efficient execution.The introduction notes that even minor financial-statement errors can trigger compliance risks or market volatility.Source: §1 · pp. 1–2
  • Earlier benchmarks vary in scope, but several rely on static data or overlook complex reasoning and real-world financial workflows.FinQA focuses on structured numerical reasoning, while SECQUE evaluates multi-step tasks but uses a static dataset.Source: §1 · p. 2
  • FinMaster combines synthetic financial data generation, a task suite for financial domains, and a unified interface for systematic LLM evaluation.FinSim generates company data to replicate market dynamics and address privacy-related data scarcity; FinSuite covers accounting, auditing, and consulting tasks.Source: §1 · p. 2
  • Accuracy falls from over 90% on basic tasks to 40% on complex scenarios requiring multi-step reasoning.The experiments include widely used models such as GPT-4o-mini, Claude-3.7-Sonnet, DeepSeek-V3, and OpenAI o3-mini.Source: §1 · p. 2
  • General-purpose LLMs can produce hallucinated conclusions or statistically invalid outputs when professional judgment is needed.The paper presents FinMaster as a benchmark for multi-step financial operations and a testbed for advanced LLMs.Source: §1 · p. 3

2 Preliminaries

The preliminaries define LLMs and outline the financial records and workflows that FinMaster evaluates, from transaction recording through audit and consulting analysis.

Preliminaries

  • LLMs model the joint probability of token sequences through conditional probabilities and can generalize across NLP tasks without task-specific fine-tuning.The text describes them as autoregressive models trained on textual data through unsupervised learning.Source: §2 · p. 3
  • Transactions are recorded to produce financial statements: income statements report period performance, balance sheets show position at a point in time, and cash flow statements track cash movements.These statements are interconnected and prepared under standards such as IFRS and GAAP.Source: §2 · p. 3
  • Accounting records and reports transactions, auditing independently verifies compliance and information reliability, and consulting uses financial analysis to inform business decisions.Consulting diagnostics include profitability, efficiency, and solvency frameworks such as DuPont Analysis and Altman Z-scores.Source: §2 · p. 3

3 FinMaster

FinMaster combines simulated company data, a 183-task suite across four financial domains, and a unified interface for evaluating LLMs. Its tasks cover financial literacy, accounting, auditing, and consulting.

3.1 FinSim: Financial Data Simulator

  • FinSim models five company types and generates transaction records and financial statements from simulated daily financial activities.The types represent distinct business profiles, including capital-intensive manufacturers, high-margin producers, asset-light companies, and high-turnover sellers.Source: §3.1 · pp. 3–4
  • FinSim simulates asset, operational, and financial records, including purchases, sales, depreciation, expenses, cash transfers, and interest receivables.Source: §3.1 · p. 4

3.2 FinSuite: Financial Task Suite

  • FinSuite covers 183 tasks: 64 in financial literacy, 49 in accounting, 35 in auditing, and 35 in consulting.The suite also describes interdependencies among accounting, auditing, and consulting tasks.Source: §3.2 · p. 4
  • FinSuite characterizes task complexity with α for computational base cardinality, β for cross-source integration, and γ for output dimensionality.The framework is presented as supporting fine-grained error attribution and applying across accounting, auditing, and consulting.Source: §3.2 · pp. 4–5
  • FinSuite evaluates literacy through definition matching, accounting through transaction-to-statement generation, auditing through error detection, and consulting through financial-indicator analysis.Its consulting tasks use 18 indicators across profitability, operational efficiency, liquidity, solvency, and cash flow quality.Source: §3.2 · pp. 5–6

3.3 FinEval: LLM Evaluation

  • FinEval provides a unified interface for LLM completion, using adaptive prompts, standardized execution, and parsing of JSON-formatted answers.The prompt template includes a task description, examples, the problem to solve, and an instruction specifying JSON output.Source: §3.3 · p. 6

4 Results

FinMaster results show strong basic financial literacy but sharp drops on more complex financial tasks, alongside performance differences by task type, token use, company setting, and input length. Across these analyses, models struggle with multi-step reasoning and some reasoning failures reflect specific knowledge, data, precision, or consistency errors.

4.1 Analysis of Financial Literacy

  • Top models reach nearly 100% on financial-literacy questions, while GPT-4.1-nano and GPT-4o-mini score 40–60%, especially on multiple-output tasks.The average accuracy across models is 96%.Source: §4.1 · p. 6

4.2 Analysis of Performance

  • FinMaster’s selected models average 96% accuracy on basic financial-literacy tasks, while accounting-task accuracy falls below 20% for multistep calculations and to 3% for statement generation.Source: §4.1 · p. 6, §4.2 · p. 7
  • Auditing accuracy is higher on multiple simultaneous errors than on isolated errors; consulting performance varies with task complexity and model.o3-mini’s accuracy drops 35% on interconnected cross-statement calculations; DeepSeek-V3 and Claude-3.7-Sonnet range from 40–85%, while GPT-4.1 is usually at 55–65%.Source: Figure 6 · p. 7

Takeaways

  • LLMs struggle with multi-step reasoning and source integration; performance also varies with company operations and input length, while output variance across cycles suggests inherent uncertainty.The takeaways say that audit capabilities rely on explicit patterns and are weaker at isolated-anomaly detection.Source: Takeaways · pp. 7–9

4.3 Analysis of Tokens

  • Token use generally rises with task complexity, but more tokens do not necessarily mean better performance.In auditing, o3-mini reaches 85% accuracy with lower token use than Claude-3.7-Sonnet at 70% and DeepSeek-V3 at 69%.Source: §4.3 · pp. 7–8

4.4 Ablation

  • Across five company types, higher-accuracy models are more consistent, while weaker models show greater variability and are more affected by organizational differences.The results indicate that company-specific operations increase performance variability, especially for weaker models.Source: §4.4 · p. 8

4.5 Analysis of Reasoning Failure Cases

  • The identified o3-mini failure cases include a financial knowledge gap, missing data, floating-point error, and inconsistent reasoning; the benchmark uses text-only inputs.The authors also identify context-window and specialized financial-knowledge constraints for general-purpose LLMs.Source: §4.5 · p. 9

5 Conclusion

FinMaster combines synthetic financial data, a broad task suite, and a unified evaluation framework to assess LLMs across financial workflows. Its experiments show high accuracy on fundamental financial-literacy tasks but a sharp drop on complex tasks requiring multi-source integration and domain-specific reasoning.

5 Conclusion

  • FinMaster comprises FinSim for synthetic financial data, FinSuite with 183 tasks across four financial domains, and FinEval for unified model assessment.The benchmark is designed to evaluate LLM performance and capabilities across financial tasks.Source: §5 · p. 9
  • Accuracy drops from 96% on fundamental financial-literacy tasks to 40% on complex tasks requiring multi-source integration and domain-specific reasoning.The conclusion attributes the decline to computational error propagation.Source: §5 · p. 10
  • The authors describe FinMaster as the first benchmark designed to rigorously evaluate LLM reasoning in real-world financial workflows.They present it as a testbed intended to guide future research on robustness, precision, and practical utility in finance.Source: §5 · p. 10

A.1 Why Using Simulated Data is Enough?

The appendix argues that simulated data can support financial LLM research through privacy protection, scalable and controllable evaluation, and standardized task structures. It focuses on accounting, auditing, and consulting, while noting that model coverage is limited by budget.

A.1 Why Using Simulated Data is Enough?

  • FinSim’s synthetic records avoid exposing confidential financial information and can generate diverse company and market-condition datasets at scale.The authors describe controllable ground truth, task complexity, and injected audit errors as evaluation benefits.Source: §A.1 · p. 12

A.2 Why Focusing on Accounting and Auditing and Consulting?

  • The authors focus on accounting, auditing, and consulting because these workflows span data processing, verification, and strategic interpretation, while existing benchmarks underrepresent multi-step financial reasoning.They contrast FinMaster’s workflow tasks with FinQA’s focus on single-step numerical question answering from static reports [6].Source: §A.2 · p. 12

A.3 Model Selection

  • Budget constraints limit evaluation to a curated set of online advanced models, and the authors plan to add more recent models in a future benchmark update.The selected models include GPT-4o-mini, Claude-3.7-Sonnet, DeepSeek-V3, and o3-mini.Source: §A.3 · p. 12

A.4 Discussion about Limitations and Future Works

The discussion identifies text-only analysis, context limits, and general-purpose models’ financial knowledge as boundaries, and proposes multimodal data, retrieval, and domain-specific training as future directions.

A.4 Discussion about Limitations and Future Works

  • FinMaster’s current financial tasks are text-based; generating multimodal data and considering multimodal LLMs are proposed to extend its coverage.The paper says a multimodal version could better simulate real-world scenarios.Source: §A.4 · p. 13
  • Limited context windows and long-context retention are challenges for LLMs handling the large datasets financial reasoning often requires; dynamic retrieval is proposed for future work.The paper identifies RAG as a promising direction to reduce hallucination and improve accuracy.Source: §A.4 · p. 13
  • General-purpose LLMs lack deep financial knowledge, while FinSim’s training data could support fine-tuning or finance-specific foundation models.The paper links the knowledge gap to misinterpretation of financial concepts.Source: §A.4 · p. 13

A.5 Negative Impacts

  • The paper reports no anticipated negative impacts.Source: §A.5 · p. 13

B Related Work

  • The related-work review contrasts benchmarks’ task coverage and data settings with systems and models for financial agents and applications.It discusses limitations of FinQA, FinBen, FinTSB, FinanceBench, PIXIU, BizBench, and SECQUE, as well as financial LLMs and agents including FinGPT, FinAgent, FinCon, Baichuan4-Finance, and DianJin-R1.Source: B Related Work · p. 14

C Preliminaries of Finance Management Workflows

  • The workflow preliminaries cover accounting, auditing, and consulting, while the accompanying statements describe balance sheets, income statements, and cash flows.Auditing includes financial statements, operations, and compliance; consulting uses financial diagnostics and peer benchmarking to inform organizational decisions.Source: C Preliminaries of Finance Management… · p. 15, Table 3 · p. 16, Table 4 · p. 17

D.1 Types of Companies

The section specifies five company archetypes and outlines the FinSuite task catalogue and its financial-statement reference and classification materials.

D.1 Types of Companies

  • Five company types range from capital-intensive manufacturers and transaction-driven traders to high-margin consumer goods, asset-light services, and high-turnover businesses.Their profiles distinguish capital requirements, margins, sales frequency, inventory, and purchasing patterns.Source: §D.1 · p. 19, Table 6 · p. 19

E FinSuite

  • FinSuite documents each task’s name, difficulty, description, and input and output specifications.Source: E FinSuite · p. 20

E.1 Financial Statement Items Definition

E.2 Element Categorization

E.3 Critical Financial Indicators Display

  • A dedicated table presents critical financial indicators with their descriptions and formulas.Source: Table 12 · p. 23

E.4 Financial Statement Tasks Information

The task-information tables span financial-statement detection and generation, auditing errors, and consulting capabilities.

E.4 Financial Statement Tasks Information

F Extended Experiment Result

The extended-results materials cover model comparisons, company types, operation time, and prompt and completion tokens; the accompanying appendix lists models and a shared prompt template.

F Extended Experiment Result

  • The extended-results figures and tables identify comparisons by main task, company type, and operation time, alongside prompt and completion token results.The supplied captions state the scope of these visuals but provide no comparison values.Source: Figure 11 · p. 36, Figure 10 · p. 36, Table 25 · p. 37

G.1 Models

  • The model appendix includes a table of API-based LLMs considered through FinEval.Source: Table 34 · p. 46

G.2 Prompt Template

  • The prompt template combines task description, in-context examples, a problem to solve, and an instruction to return a JSON solution.Source: §G.2 · p. 46

H Accuracy of Each Company Type for Specific Task

The figures present accuracy for five company types across financial literacy, accounting, audit, and consulting.

I Analysis of Reasoning Failures

The section presents a comparative analysis of critical errors across LLMs and describes how task-running costs are calculated and reported.

I Analysis of Reasoning Failures

  • Table 35 is titled as a comparative analysis of critical errors across large language models.Source: Table 35 · p. 51

J Costs

  • LLM task-running costs are calculated from input and completion token counts and their corresponding prices.The costs for running all tasks once are shown in Table 36.Source: J Costs · p. 52, Table 36 · p. 52
  • Table 36 reports costs for LLMs.Source: Table 36 · p. 52
Loading 2505.13533v1…