PaperlayerView exact revision on arXiv

Source-linked AI summary

Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements

Xinke Tong, Xuanming Zhang, Tianyi Tang, An Yang, Jiatu Hu, Guojie Lin, Zhenzhen Shi, Lingfeng Zeng, Boyu Yang, Bing Zhao, Hu Wei, Lin Qu, Dayiheng Liu

arXiv:2607.28661v1cs.CL

TL;DR

Existing financial benchmarks undermeasure the numerical-table reliability and structural reasoning required by real-world, long-horizon financial analysis. FinIndices evaluates this capability on uncropped financial statements and finds knowledge and structural bottlenecks, while supervised fine-tuning partially mitigates the deficits.

  • Problem

    Existing financial benchmarks emphasize factual questions, visual perception, or end-to-end tool use rather than data-intensive numerical-table workflows in financial practice.

  • Method

    FinIndices benchmarks calculation fidelity over uncropped financial statements through single-value computation and unified-caliber multi-indicator tabulation across accounting, temporal, and caliber reasoning.

  • Results

    70.07% to 38.22%: Gemini-3.1-Pro’s Table-Index performance collapses without explicit formula hints, while models also degrade when generating multi-period matrices.

  • Takeaways & Limitations

    The findings indicate that current LLMs rely on fragile pattern matching under complex accounting and structural demands, while data-centric alignment can mitigate these deficits.

  • Takeaways & Limitations

    The dataset is restricted primarily to core corporate financial statements and excludes diverse sources such as market, macroeconomic, and operational data.

Abstract

from arXiv · show

Do Large Language Models (LLMs) possess genuine structural reasoning, or merely rely on surface-level pattern matching? The financial domain, demanding numerical precision and multi-step logic over long contexts, is an ideal testbed. Existing benchmarks fail to capture real-world industrial complexity, predominantly relying on multiple-choice questions or single-hop QA over cropped tables while ignoring intricate cross-statement dynamics and temporal de-cumulation. To bridge this gap, we introduce FinIndices, a large-scale benchmark evaluating data-processing fidelity over uncropped financial statements (up to 32K tokens). Utilizing an automated synthesis pipeline with adversarial traps, FinIndices encompasses Single-Index computation and Table-Index tabulation to test complex domain, temporal, and caliber reasoning. Our evaluation reveals two severe LLM vulnerabilities. First, a "Knowledge Bottleneck": despite memorizing formulas during pre-training, models demonstrate fragile pattern matching. Removing explicit formula hints causes performance to collapse (e.g., Gemini-3.1-Pro drops from 70.70% to 38.22% on table tasks), exposing fatal flaws in temporal de-cumulation and stock-flow caliber mismatch. Second, a "Structural Bottleneck": the intense cognitive load of generating multi-metric, multi-period tables actively drains reasoning capacity. Under structural pressure, LLMs that flawlessly execute isolated derivations regress to shallow heuristics, such as fetching incorrect adjacent columns or substituting deep accounting adjustments with lazy literal arithmetic. Finally, Supervised Fine-Tuning (SFT) yields substantial zero-hint gains (+8.54% Single, +3.82% Table), validating that structured logic can be partially restored via data-centric alignment.

1 Introduction

Real-world financial analysis requires reliable multi-source data processing, caliber reconciliation, temporal alignment, composite calculation, and audit-ready tabular reporting. Existing benchmarks underrepresent this reliability challenge, motivating FinIndices as a benchmark for calculation fidelity in realistic financial workflows.

  • Motivation: Financial agents must reconcile heterogeneous sources across statistical calibers, produce composite indicators, and deliver clean, audit-ready tables rather than isolated answers.The workflow spans market feeds, financial statements, operating disclosures, macroeconomic databases, and distinctions such as stock versus flow.
  • Problem: Cascading extraction, alignment, and calculation errors make financial-agent outputs untrustworthy despite potentially sound high-level reasoning.The problem is rooted largely in domain-specific accounting and finance knowledge failures during intermediate processing.
  • Gaps in existing benchmarks: Existing tabular benchmarks generally emphasize simple numerical answers, while knowledge-focused benchmarks use multiple-choice questions that diverge from practitioners’ data-intensive workflows.These formats are useful for factual knowledge but do not adequately measure intermediate data-processing reliability in table-centric financial work.
  • Our benchmark: FinIndices is introduced to evaluate calculation fidelity and data-processing reliability in realistic financial workflows using continuous financial statements and standards-grounded indicators.Its resources span multiple reporting frequencies and statistical calibers, and the benchmark is organized into two task formats.

2 Method

FININDICES is built through a three-stage synthesis and validation pipeline over authentic, long-horizon financial statements. Its evaluation design combines temporal and adversarial reasoning demands with single-value and structured multi-value outputs.

  • Dataset Synthesis: FININDICES covers 829 publicly listed companies, 384 financial indices, and 28 reporting periods.The dataset is designed for broad cross-sectional and temporal coverage.
  • Dataset Synthesis: The pipeline extracts financial statements, aligns temporal contexts with adversarial traps, and generates instruction–QA pairs using canonical formulas and programmatic execution.The three stages are Financial Statement Extraction, Context Construction & Index Alignment, and Instruction & QA Pair Generation, followed by expert validation.
  • Context Construction & Index Alignment: Adversarial contexts inject irrelevant adjacent periods and omit necessary reports, requiring models to identify distractors or output “Insufficient information.”These traps test comprehension beyond blind computation and discourage hallucinated numeric answers.
  • Expert Validation: Expert validation refined 684 initial test instances into a final 640-instance test set after checking answerability, numerical accuracy, and rounding discrepancies.Three experts produced 484 unanimous judgments and 171 two-person majorities before pruning some unanswerable samples.
  • Benchmark Design: FININDICES uses uncropped statements averaging over 16,000 tokens, with contexts reaching 33,126 tokens and requiring alignment across up to 9 historical periods.This contrasts with existing cropped-table datasets averaging 500–1,000 tokens and introduces long-horizon temporal alignment.
  • Task Paradigms: Table-Index tasks require structured comparison tables averaging 7.75 target values and reaching 20 data points per sample, alongside consistent calibers, scaling, and formatting.Single-Index tasks instead isolate calculation fidelity by requiring one numerical answer from distractor-rich inputs.

3 Experiment

Experiments on FININDICES show severe structural and knowledge bottlenecks: models lose accuracy on table synthesis and depend heavily on explicit formula hints. Domain-specific pretraining performs poorly, whereas SFT improves zero-hint performance and transfers to broader financial reasoning.

  • Structural Bottleneck: 79.61% to 70.07%: Gemini-3.1-Pro’s hinted accuracy drops from Single-Index to Table-Index tasks, exposing a structural bottleneck in 2D matrix synthesis.Qwen3.7-Max reaches 72.82% on hinted Table tasks, but multi-period caliber alignment and tabular formatting remain difficult.
  • Knowledge Bottleneck: 70.07% to 38.22%: removing hints causes Gemini-3.1-Pro’s Table-Index accuracy to collapse, while Claude-Opus-4.8 reaches only 38.85%.The results indicate reliance on in-context pattern matching rather than independently deriving complex indicators using accounting knowledge such as GAAP rules.
  • Performance of Domain-Specific LLMs: 11.41% w/ Hint and 5.33% w/o Hint: DianJin-R1-32B fails on Table tasks despite achieving 41.29% on hinted Single tasks.Fin-R1 and Llama-Fin-8B score near 0%, showing that unstructured financial pretraining is insufficient for structured tabular logic.
  • Effectiveness and Generalization of SFT: +8.54% Single, +3.82% Table: SFT with 6,301 Gemini-distilled reasoning trajectories improves Qwen3.5-35B-A3B in the Without Hint setting.SFT also improves FINMATH by +2.19% and transfers positively to Chinese financial QA benchmarks without catastrophic forgetting.

4 Conclusion

FININDICES shows that current LLMs face knowledge and structural bottlenecks in long-context financial reasoning. Explicit formulas and lower structural pressure are important because models otherwise regress to shallow heuristics, while domain-specific SFT partially mitigates these deficits.

  • Conclusion: FININDICES evaluates whether LLMs exhibit genuine structural reasoning or rely on surface-level pattern matching using uncropped, long-context financial statements.The benchmark is presented as a rigorous testbed for complex accounting reasoning.
  • Knowledge Bottleneck: Without explicit in-context formulas, model performance collapses on complex accounting tasks, revealing a fundamental Knowledge Bottleneck.The conclusion identifies this as one of two vulnerabilities in current foundation models.
  • Structural Bottleneck: Generating multi-metric, multi-period tables imposes high cognitive load that drains reasoning capacity under structural pressure.This is identified as the Structural Bottleneck affecting current models.
  • Structural Bottleneck: Under structural pressure, LLMs abandon rigorous logic and regress to shallow heuristics, causing severe temporal misalignments and aggregation shortcuts.These failures reflect the interaction between structural load and reasoning quality.
  • Mitigation: Supervised Fine-Tuning confirms that data-centric alignment can mitigate these deficits without catastrophic forgetting.The conclusion characterizes domain-specific SFT as an effective mitigation strategy.

5 Limitations · A Case Study

FinIndices rigorously evaluates ultra-long-context tabular reasoning but is limited by narrow data-source and task coverage. Future iterations could integrate multimodal and multi-source financial data to support broader real-world investment and operational scenarios.

  • 5 Limitations: FinIndices provides a rigorous testbed for ultra-long-context tabular reasoning while presenting several opportunities for future research.The limitations concern dataset scope, data modalities, and application coverage.
  • 5 Limitations: Engineering complexity restricts the dataset primarily to Balance Sheets, Income Statements, and Cash Flow Statements.The restriction results from extracting, aligning, and verifying ultra-long textual and tabular contexts.
  • 5 Limitations: The benchmark lacks integration with diverse financial data sources, including high-frequency stock market data such as trading volumes and real-time pricing.This limitation follows from the dataset’s focus on core corporate financial statements.
  • 5 Limitations: Restricting data modalities limits the benchmark’s coverage of application scenarios.The passage identifies data-modality restriction as an inherent constraint on evaluated use cases.
  • 5 Limitations: The benchmark heavily focuses on fundamental accounting analysis and metric derivation.Its current scope does not evaluate broader financial tasks.
  • 5 Limitations: It does not evaluate qualitative sentiment analysis of earnings calls or cross-asset correlation modeling.These examples illustrate broader financial tasks outside the benchmark’s current coverage.
  • 5 Limitations: Future iterations could integrate multimodal and multi-source financial data to evaluate autonomous agents across a wider spectrum of real-world scenarios.The stated target includes broader investment and operational scenarios.

A.1 Single-Index, Domain Understanding · A.2 Single-Index, Temporal Reasoning + Domain Understanding

The Single-Index cases expose two distinct demands in financial reasoning: domain-specific accounting classification and temporal reconstruction across reporting periods. Models must infer unstated liability taxonomies, apply rolling-window formulas, reject distractors, and normalize results.

  • A.1 Single-Index, Domain Understanding: Total Invested Capital requires subtracting non-interest-bearing liabilities from total capital, making expert accounting knowledge essential beyond verbatim table extraction.The injected hint supplies the high-level formula, but the balance sheet does not explicitly identify which liabilities bear interest.
  • A.1 Single-Index, Domain Understanding: The balance sheet presents dozens of raw liability items without explicit interest-bearing labels, forcing classification under Chinese accounting conventions.This classification challenge differs from ordinary reading comprehension because the relevant categories must be inferred rather than copied.
  • A.1 Single-Index, Domain Understanding: Notes Payable is typically non-interest-bearing, whereas Lease Liabilities and Current Maturities of Non-Current Liabilities are interest-bearing.These examples illustrate the domain taxonomy required to compute the indicator correctly.
  • A.2 Single-Index, Temporal Reasoning + Domain Understanding: TTM metrics require synthesizing a rolling one-year window because interim reports disclose year-to-date rather than rolling figures.The model cannot obtain the answer through direct extraction alone.
  • A.2 Single-Index, Temporal Reasoning + Domain Understanding: TTM = Current YTD+Previous Annual−Previous YTD provides the required temporal reconstruction formula.Applying this identity converts year-to-date disclosures into a trailing-twelve-month measure.
  • A.2 Single-Index, Temporal Reasoning + Domain Understanding: Six reporting periods require deliberate rejection of Q1 and Q3 distractors while selecting the relevant reporting windows.The listed periods are 2025H1, 2025Q1, 2024Y, 2024Q3, 2024H1, and 2024Q1.
  • A.2 Single-Index, Temporal Reasoning + Domain Understanding: The temporal task also requires identifying consolidated income statements instead of parent-company distractors and extracting distinct operating-expense lines.Financial expenses must be handled correctly when reported as negative values.
  • A.2 Single-Index, Temporal Reasoning + Domain Understanding: After applying TTM to each expense line, the model must sum the results and convert the raw figure into ten-thousands.This combines temporal reasoning, line-item extraction, arithmetic aggregation, and unit normalization.

A.3 Single-Index, Adversarial Temporal and Caliber Alignment … A.6 Table-Index, Temporal and Caliber Alignment

FinIndices tests whether LLMs can reason across temporal boundaries, stock-flow calibers, domain-specific accounting items, and rigid multi-period table structures. Its cases expose the need for de-cumulation, calibrated denominators, selective liability classification, and error-free matrix generation.

  • A.3 Single-Index, Adversarial Temporal and Caliber Alignment: LLMs often confuse temporal semantics by extracting “Current Year” and “Prior Year” values and directly applying a YoY formula.This reveals a bias toward nearest-column pattern matching rather than understanding temporal boundaries.
  • A.3 Single-Index, Adversarial Temporal and Caliber Alignment: Cash Flow Q4 requires subtracting Q3 YTD from Annual YTD, so omitting the Q3 report should yield “Insufficient information.”YTD figures cannot substitute for standalone quarterly figures.
  • A.4 Table-Index, Complex Caliber Alignment: Turnover ratios and cash conversion cycles require aligning flow variables such as Revenue with point-in-time stock variables such as Accounts Receivable.The task therefore tests complex caliber alignment within Table-Index generation.
  • A.4 Table-Index, Complex Caliber Alignment: Caliber alignment requires an average balance denominator from current and prior-year ending balances plus annualization multipliers of 90, 180, 270, and 360.The multipliers correspond respectively to Q1, H1, Q3, and Annual periods.
  • A.5 Table-Index, Domain Understanding: Total Invested Capital cannot be obtained by keyword search; the hinted formula is Total Equity + Total Liabilities − Non-interest-bearing Liabilities.The calculation spans multiple years and requires parsing raw balance sheets.
  • A.5 Table-Index, Domain Understanding: Domain understanding requires deducting Accounts Payable, Contract Liabilities, Taxes Payable, and other operating payables while excluding interest-bearing liabilities.The exclusions include Lease Liabilities and Current Maturities of Non-Current Liabilities.
  • A.6 Table-Index, Temporal and Caliber Alignment: The final case combines Table-Index structure, temporal de-cumulation, and caliber alignment in a single data-processing task.The LLM must subtract prior cumulative figures, isolate standalone Revenue, Profit, and Cost, calculate Gross Profit as Revenue minus Cost, and preserve the JSON matrix.

B Detailed Results across Capability Axes · C Weakness Study

The detailed breakdown evaluates leading general LLMs across four capability axes under hinted and hint-free formulas. Results expose severe weaknesses in temporal-domain reasoning, stock-flow caliber alignment, and formula-free Table-Index performance.

  • B Detailed Results across Capability Axes: The evaluation decomposes Test-set accuracy across four axes: Domain Understanding, Complex Caliber Alignment, Temporal Reasoning + Complex Caliber Alignment, and Temporal Reasoning + Domain Understanding.The breakdown targets fine-grained domain, caliber, and temporal capabilities.
  • B Detailed Results across Capability Axes: Tables 11 and 12 compare performance with explicitly provided formulas versus withheld formulas, using explicit reasoning modes for all general models.Table 11 reports With Hint results, while Table 12 reports Without Hint results.
  • B Detailed Results across Capability Axes: GPT-5.4 scores 58.33% with hints and 38.78% without hints on Temporal + Domain in Single-Index, among the model’s weakest axes.This challenge requires TTM calculations across 4–5 non-sequential historical reports while applying accounting taxonomy adjustments.
  • B Detailed Results across Capability Axes: 63.95% is Gemini-3.1’s Complex Caliber Alignment accuracy with formulas in Table-Index, while GPT-5.4 reaches 50.00%.The task combines point-in-time stock metrics with period-aggregated flow metrics to calculate turnover rates across consecutive periods.
  • C Weakness Study: GPT-5.4’s Table-Index Domain Understanding accuracy collapses from 72.73% to 18.18% without formulas.The absence of explicit formulas disproportionately harms Domain Understanding and Caliber Alignment.
  • C Weakness Study: Gemini’s Table-Index Caliber Alignment accuracy drops from 63.95% to 23.26% when formulas are withheld.The comparison indicates strong dependence on external formula hints for financial reasoning.

C.1 Anomaly on the Leaderboard: Why Do Large-Scale or Closed-Source Models with Strong General Capabilities Fail? · C.2 Small Parameter Models (e.g., 35B variants): Magnitude Scaling and Format Brittleness

Large or closed-source models fail on complex financial tasks because surface matching, formula disobedience, and structural reasoning pressures trigger domain-specific errors. Smaller models additionally suffer from magnitude-scaling mistakes and formatting hallucinations that collapse table-task performance.

  • C.1 Anomaly on the Leaderboard: Why Do Large-Scale or Closed-Source Models with Strong General Capabilities Fail?: Large or closed-source models show anomalously poor performance on complex financial tasks despite formidable general capabilities.The analysis traced generation trajectories across isolated extraction and structured-table tasks.
  • C.1 Anomaly on the Leaderboard: Why Do Large-Scale or Closed-Source Models with Strong General Capabilities Fail?: Their failures predominantly reflect profound knowledge bottlenecks and rigid formula disobedience under structural constraints.Rather than activating financial knowledge, models often rely on semantic surface matching.
  • C.1 Anomaly on the Leaderboard: Why Do Large-Scale or Closed-Source Models with Strong General Capabilities Fail?: Models fall into semantic-anchor traps, cognitive overload during temporal de-cumulation, stock-flow mismatches, and fine-grained accounting-taxonomy errors.These failure modes were conceptualized through five representative cases in Table 13.
  • C.2 Small Parameter Models (e.g., 35B variants): Magnitude Scaling and Format Brittleness: Roughly 35B small models exhibit Magnitude Scaling Brittleness, often combined with Helpfulness Bias, causing performance collapse on structured table tasks.Table 14 shows correct extraction logic being undermined by faulty unit conversion and formatting hallucination.
  • C.2 Small Parameter Models (e.g., 35B variants): Magnitude Scaling and Format Brittleness: Helpfulness Bias causes models to violate tuple_list constraints by hallucinating descriptive keys such as "Working_Capital_10k".The unsolicited readability optimization breaks the programmatic JSON schema and triggers automatic parsing failures.

D Analysis of the “Structure Bottleneck”

The Structure Bottleneck is a format-induced cognitive collapse: LLMs reason effectively on isolated financial metrics but degrade when producing multi-metric, multi-period structured outputs. Controlled ablations attribute this degradation primarily to Temporal Misalignment and Aggregation Shortcut.

  • Isolated single_indice queries elicited expert-like execution of complex multi-period temporal logic and deep accounting derivations.
  • Structured table_indice tasks induced reasoning degradation when models generated multi-metric, multi-period outputs such as nested JSONs or tabular matrices.
  • A controlled indicator-level ablation showed that the format-induced collapse primarily manifests as Temporal Misalignment and Aggregation Shortcut.The study contrasted isolated metrics with identical indicator types embedded in batch-generation tasks.

D.1 Case 1: Temporal Misalignment in Time-Series Logic

The Structure Bottleneck causes LLMs to lose temporal anchoring in complex time-series calculations. Under multi-metric table-generation pressure, models replace cross-report retrieval with adjacent-column shortcuts, corrupting YoY denominators and outputs.

  • Temporal anchoring: YoY growth requires precise cross-period mapping, including the Prior Year Same Period rather than the Beginning of Year balance.This temporal distinction is correctly identified when Net Assets YoY is evaluated in isolation.
  • Temporal anchoring: Multi-metric HTML table generation overwhelms temporal reasoning, prompting models to retrieve an adjacent Beginning of Period column instead of the correct cross-report denominator.Balance Sheets typically omit an explicit Prior Year Same Period column, making cross-report retrieval necessary.
  • Temporal anchoring: −4.22% ground truth is transformed into the hallucinated artifact −3.25% when structural coercion corrupts temporal logic.The error arises from the model’s lazy adjacent-column shortcut under comprehensive table-generation constraints.

D.2 Case 2: Aggregation Shortcut in Accounting Logic · E Source Materials · E.1 Formula Source Reference

The aggregation shortcut shows that bundling five accounting indicators into one table can make Gemini abandon valid deep derivations for shallow aggregate parsing, while the benchmark’s formulas are grounded in authoritative accounting, CFA, and CPA sources. FinIndices therefore tests domain classification, temporal de-cumulation, stock-flow alignment, and period-specific scaling across demanding financial-index cases.

  • D.2 Case 2: Aggregation Shortcut in Accounting Logic: Bundling five indicators into one table triggers an aggregation shortcut in which the model abandons deep accounting reasoning for shallow top-level aggregate parsing.This failure is attributed to the demands of sustaining a long, multi-metric list.
  • D.2 Case 2: Aggregation Shortcut in Accounting Logic: 104.0%: Gemini correctly executes complex cross-period deductions and nuanced accounting derivations when the Operating Net Income / Total Profit formula is queried individually.The result is reported for ST Tongde.
  • D.2 Case 2: Aggregation Shortcut in Accounting Logic: 129.82%: In the bundled table, Gemini substitutes a literal (Revenue −Cost)/Profit shortcut for the accounting-sensitive metric and ignores CAS impairment line items.The same model maintains rigor on simpler ratios but collapses on the most accounting-sensitive metric under the tuple-list constraint.
  • D.2 Case 2: Aggregation Shortcut in Accounting Logic: Rigid JSON matrices, HTML tables, and tuple lists drain the reasoning resources needed for deep financial parsing and multi-step temporal mathematics.The paper characterizes this as a zero-sum cognitive mechanism within the LLM.
  • E.1 Formula Source Reference: FinIndices maps over 300 indices to authoritative statutory accounting standards, textbooks, and frameworks central to CFA and CPA curricula.These sources provide the benchmark’s ground truth for multi-dimensional financial evaluation, with distributions and representative formulas in Table 17.
  • E.1 Formula Source Reference: The benchmark tests invested-capital classification, TTM computation across non-sequential periods, and refusal of impossible quarterly de-cumulation from annual data alone.These cases require accounting-taxonomy decisions, temporal navigation, unit conversion, and recognizing when Q3 data is missing.
  • E.1 Formula Source Reference: Operating-turnover cases require averaging beginning and ending balance-sheet stocks with income-statement flows and applying 90, 270, or 360 scaling multipliers by reporting period.Other table cases require item-by-item liability classification across four periods or unrolling YTD statements into standalone quarters before cross-metric and YoY calculations.
  • E.1 Formula Source Reference: The benchmark materials include accuracy tables, failure-mode taxonomies, scaling and format-hallucination analyses, and ablations of temporal misalignment and aggregation shortcuts.These analyses contrast isolated competence with failures under unified table-generation constraints, including wrong adjacent-column substitutions and abandoned operating adjustments.
Loading 2607.28661v1…