PaperlayerView exact revision on arXiv

Source-linked AI summary

FinanceComplexQA: Benchmarking Agentic Reasoning on Industrial-grade Financial Documents

Xianfu Cheng, Shiwei Zhang, Jiyu Zhao, Jian Yang, Xinyuan Wang, Ming Zhou, Weixiao Zhou, Xiangyuan Guan, Xiang Li, Zhenhe Wu, Ziyi Ni, Zhoujun Li, Bingjing Xu

arXiv:2607.19238v1cs.CE

TL;DR

Financial QA systems need evaluation that reflects numerical precision, domain knowledge, long-document reasoning, and stable scoring. The paper develops Finance-LaTeX SKILL and FinanceComplexQA, then evaluates retrieval and agentic systems on expert-level, bilingual financial document tasks. The benchmark exposes persistent weaknesses in numerical reasoning, evidence use, layout handling, and analytical synthesis.

  • Problem

    Existing financial QA benchmarks rarely represent complex document-grounded reasoning across layouts, domain knowledge, and stable open-ended evaluation.

  • Method

    The paper combines Finance-LaTeX SKILL with a bilingual benchmark of expert-level questions over structured financial documents and dual-context, cross-layout evidence.

  • Results

    Current RAG and agentic systems struggle with long-chain numerical reasoning, cross-layout evidence fusion, industry-level synthesis, numerical drift, evidence omission, and layout confusion.

  • Takeaways & Limitations

    Reliable financial agents require layout-preserving retrieval, explicit evidence planning, calculation verification, and faithful analytical synthesis.

  • Takeaways & Limitations

    The benchmark excludes live trading, real-time forecasting, personalized investment advice, and broader multilingual transfer, while Agent-as-a-Judge may inherit evaluator bias.

Abstract

from arXiv · show

Agentic Reasoning has become a transformative force in financial analysis due to its ability to integrate large-scale information and generate reliable and accurate content. However, when handling complex real-world problems, different agents still show significant performance variation. In this work, we design Finance-LaTeX SKILL, a skill for synthesizing financial documents with complex layouts based on expert knowledge. Using an agent workflow built on this skill, we generate 2,000 professional financial documents along with 6,000 high-quality question-answer pairs. To evaluate the overall capability of agents, we introduce FinanceComplexQA, a comprehensive open-ended generation benchmark for financial documents that closely resembles real-world scenarios. It contains 2,026 deep research tasks targeting 1009 financial documents. FinanceComplexQA has 8 key features: bilingual support; coverage of six mainstream scenarios and seven tasks; expert-level document reasoning questions; deep research of complex layouts; relatively stable and permanent reference answers; and precise evaluation through an Agent-as-a-Judge with multiple evaluation metrics. Using FinanceComplexQA, we conduct a comprehensive evaluation of leading RAG systems and agentic reasoning tools for financial document QA. Through identifying and analyzing failure cases, we provide an in-depth study of their capabilities in numerical computation, multi-hop reasoning, content summarization, and industry analysis.

1 Introduction

FinanceComplexQA addresses the difficulty of evaluating reliable financial reasoning over long, heterogeneous documents. It introduces a benchmark and construction framework targeting expert-level, open-ended analysis across numerical, multi-hop, and cross-layout tasks.

  • Financial analysis demands numerical precision, domain expertise, multi-step reasoning, and strict factual reliability because errors can affect investment or risk decisions.
  • Existing financial QA benchmarks rarely capture cross-document, table, chart, and domain-knowledge synthesis, while open-ended evaluation remains unstable.
  • FinanceComplexQA contains 2,026 expert-level questions from 1,009 real-world rich-text documents across 8 financial subdomains and 9 research tasks.
  • Finance-LaTeX SKILL combines expert knowledge, automated document generation, terminology constraints, and financial-logic verification to synthesize scalable QA data.
  • The benchmark evaluates RAG, MCP, and agentic systems and finds persistent weaknesses in long-chain numerical reasoning, cross-layout fusion, and industry-level synthesis.
  • The framework contributes a dual-context, cross-layout methodology and systematic error analysis covering reasoning, planning, evidence use, and factual consistency.

2 Finance-LaTeX SKILL

Finance-LaTeX SKILL generates realistic, layout-rich financial documents and verified QA pairs through expert-guided evidence planning, structured LaTeX generation, and multi-stage quality control. Its synthetic data support development while remaining separate from held-out benchmark evaluation.

  • Finance-LaTeX combines expert financial knowledge, evidence planning, layout-aware LaTeX writing, and multi-stage verification for realistic analyst workloads.
  • The skill addresses short-excerpt and simplified-table limitations by treating documents as structured evidence environments with consistency and solvability checks.
  • The workflow samples domains, audiences, document types, terminology constraints, and stable reasoning targets before document generation.
  • Evidence plans assign facts, quantities, assumptions, and reasoning functions to paragraphs, tables, captions, and notes before writing.
  • Generated documents preserve row labels, column labels, periods, units, entity scopes, and cross-layout dependencies so questions require contextual binding.
  • LLM, web/MCP, harness, expert, and independent checks remove documents or QA pairs with broken layouts, ambiguous evidence, or irreproducible calculations.
  • 2,000 professional documents and 6,000 QA pairs support development, ablations, controlled testing, and failure analysis rather than held-out performance reporting.

3 FinanceComplexQA Pipeline

FinanceComplexQA is a bilingual benchmark built from professional financial documents and designed to test the full reasoning loop from evidence retrieval through faithful analytical synthesis. Its layout-aware evidence hierarchy and dual-context tasks support fine-grained diagnosis of retrieval, reasoning, and calculation failures.

  • The benchmark combines real financial corpora, expert evidence extraction, Finance-LaTeX construction, multi-channel verification, human review, and judge-oriented evaluation.
  • FinanceComplexQA contains 2,026 deep-research tasks over 1,009 financial documents, split evenly between 1,013 Chinese and 1,013 English questions.
  • Its documents span corporate reports, investment strategy, market analysis, FinTech, banking, customer service, compliance, insurance, innovation, and official bulletins.
  • Layout-aware parsing preserves paragraphs, tables, forms, captions, section boundaries, headers, units, periods, and nearby explanatory text.
  • Page-local blocks support layout-sensitive retrieval, while topic blocks support cross-page reasoning and annotation of necessary, optional, or misleading evidence.
  • The metadata distinguishes evidence omission from reasoning or calculation errors, enabling targeted diagnosis of system failures.
  • Dual-context questions require explicit document evidence and implicit financial knowledge, such as accounting relations, margin interpretation, or sector cyclicality.
  • Cross-layout questions connect narrative, tables, charts, and notes, while analytical references preserve conclusions, calculations, evidence, and caveats.

4 Experiments

The experiments compare retrieval, layout-aware, and agentic systems across financial scenes and tasks using common documents, questions, and evaluation dimensions. Results show complementary strengths, substantial quality–cost trade-offs, and persistent weaknesses in cross-layout, numerical, planning, and multi-hop reasoning.

  • 4.1 Systems: Seven LLMs are evaluated across lightweight retrieval, layout-aware retrieval, and agentic systems using shared documents, questions, formats, and recorded quality, cost, and latency measures.The evaluated systems include LightRAG, PageIndex, and Codex- or Claude Code-style agents paired with closed- or open-source models.
  • 4.1 Systems: Agent-as-a-Judge scores accuracy, numeric correctness, evidence coverage, faithfulness, completeness, and deployment cost.Numeric correctness checks formulas, units, periods, signs, and entity scopes; evidence coverage checks required paragraphs, tables, charts, and captions.
  • 4.3 Main Findings: PageIndex consistently improves overall scene scores over LightRAG, indicating that page structure and table context matter for long-document and cross-layout evidence.The comparison uses different core models, so the improvement cannot be attributed to indexing alone.
  • 4.3 Main Findings: Claude Code with Sonnet 5 reaches the best overall Chinese and English scene scores, 76.01 and 69.39, while Codex leads aggregate closed-source and open-source settings in Table 7.Codex with GPT-5.5 scores 60.63 Chinese and 62.97 English; Codex with DeepSeek-V4-Flash scores 60.44 Chinese and 61.55 English.
  • 4.3 Main Findings: No system dominates every task: PageIndex is competitive on retrieval, multi-hop, numerical comparison, judgment, and planning, while agentic systems lead several reasoning, scene, faithfulness, and coverage measures.The fragmentation indicates that financial QA requires capabilities that do not always improve together.
  • 4.3 Main Findings: Agentic quality gains require more resources: LightRAG uses 12.8k tokens and 8.6 seconds per question, PageIndex 15.6k and 11.9 seconds, versus agents’ 31.4k–45.8k tokens and 22.8–50.6 seconds.The experiments therefore assess quality together with practical deployment cost.
  • 4.5 Task-Level Observations: Task metrics separate correctness from overlap, groundedness, and completeness: PageIndex obtains 65.19 ACC but 27.93 ROU on Chinese multi-hop reasoning.Planning similarly separates faithfulness from coverage; Codex with GPT-5.5 scores 83.54 FS and 32.77 Cov on Chinese planning.
  • 4.5 Task-Level Observations: Retrieved evidence does not prevent numerical errors, including incompatible periods, percentage-versus-percentage-point confusion, wrong aggregation scope, and lost negative signs.Reliable systems must bind numbers to row labels, periods, units, and entity scopes before verifying calculations.

5 Further Analysis

The analysis identifies evidence-alignment failures as a major source of financial QA errors, even after retrieval succeeds. It also shows that layout preservation, explicit evidence planning, verification, and cost-aware routing are important for reliable deployment.

  • Failure Modes: Many failures occur after relevant evidence is retrieved, including wrong rows, periods, trends, or table interpretations.These errors can reverse an investment implication despite fluent answers.
  • Failure Modes: Numeric drift lowers ACC, evidence omission lowers Cov, and layout confusion harms both scene-level and task-level performance.Over-synthesis and weak planning also reduce faithfulness and completeness.
  • Human Evaluation: Human-evaluation difficulty varies beyond document length across languages, domains, and subsets.One English IS group averaged 34,045 seconds with a 92.0 Corr&Comp score, while another English GFB group scored 0.0.
  • Benchmark Difficulty: Cross-layout questions require combining paragraphs, tables, charts, and notes without overstating conclusions.Systems may retrieve relevant elements yet miss the evidence that changes interpretation.
  • System Trade-offs: PageIndex can be competitive when layout-preserving retrieval suffices, whereas agentic systems improve some open-ended settings but incur higher cost and miss coverage requirements.Shorter documents can remain difficult when domain structure and evidence relations are challenging.
  • Future Directions: Future systems should use explicit evidence plans, layout-preserving retrieval, metric-aware self-checking, and cost-aware routing.Full agent loops should be reserved for decomposition, calculation, or cross-layout synthesis.

6 Related Work

Financial QA benchmarks have progressed from short-context arithmetic toward longer, grounded document reasoning, but existing evaluations remain limited in scope and realism. Related systems add retrieval, structure preservation, planning, and tool use, yet professional bilingual financial documents remain underrepresented.

  • Financial QA Benchmarks: FinQA and ConvFinQA established multi-step arithmetic over financial report excerpts, while TAT-QA combined tabular and textual evidence.These datasets remain tightly scoped in context.
  • Financial QA Benchmarks: FinanceBench introduced open-ended SEC-filing questions, while DocFinQA extended reasoning to full filings but remained single-document.Enterprise benchmarks broadened grounded reasoning over long office corpora.
  • Retrieval-Augmented Generation: RAG research improves retrieval through re-ranking, query rewriting, modular pipelines, iterative retrieval, and graph- or page-level structure.These approaches target more structured and multi-hop evidence access.
  • Research Gap: Most evaluations still use general QA or scientific corpora rather than professional bilingual financial documents with tables, charts, and domain-specific inference.This leaves a gap in realistic financial document evaluation.
  • Agentic Reasoning: Agentic RAG combines planning, tool use, reflection, and collaboration into retrieval loops.Frameworks such as AutoGen, Voyager, Claude Code, and Codex demonstrate tool-using agent workflows.

7 Conclusion

The paper presents Finance-LaTeX for generating layout-rich financial documents and verified QA pairs, and FinanceComplexQA for evaluating agentic reasoning over industrial-grade documents. The benchmark emphasizes bilingual deep research, dual-context reasoning, cross-layout evidence aggregation, and stable evaluation, while experiments show that structure preservation remains crucial and numerical, evidence, and layout failures persist.

  • Conclusion: Finance-LaTeX generates layout-rich financial documents and verified QA pairs for benchmark construction.FinanceComplexQA is a bilingual benchmark for agentic reasoning over industrial-grade financial documents.
  • Conclusion: FinanceComplexQA contains 2,026 deep research tasks over 1009 documents and uses dual-context reasoning, cross-layout evidence aggregation, stable reference answers, and Agent-as-a-Judge evaluation.These design choices target complex document-grounded financial reasoning.
  • Conclusion: Experiments show that preserving document structure is crucial, while current agents still struggle with numerical drift, evidence omission, and layout confusion.The conclusion connects benchmark results with observed system failure modes.

Limitations

The benchmark evaluates document-grounded financial reasoning within a defined scope and relies partly on an automated judge. Its current coverage excludes live and personalized financial decision settings and broader multilingual transfer, while evaluator bias remains possible.

  • Scope: FinanceComplexQA does not cover live trading, real-time market forecasting, or personalized investment advice.Its scope is document-grounded financial reasoning.
  • Scope: The bilingual benchmark currently emphasizes Chinese and English and does not test broader multilingual transfer.This limits conclusions about multilingual generalization.
  • Evaluation: Agent-as-a-Judge improves scalability but may inherit evaluator bias.Future releases should add stronger human calibration and adversarial judge checks.

A Prompt Templates

Part A prompt templates share a financial-advisor role, the finance-latex skill, indirect-answer constraints, and layoutCoT plus Reference_documents outputs. The variants generate different document genres from benchmark-specific inputs, with A1 adding sub-question expansion and richer layouts.

  • A Prompt Templates: All six Part A prompts share a financial-advisor role, the finance-latex skill, indirect-answer constraints, and layoutCoT plus Reference_documents outputs.Prompts are presented in English, use programmatically substituted placeholders, and typically batch up to ten QA pairs per call.
  • A Prompt Templates: Part A prompts require answers to demand investigation or calculation rather than direct or single-step retrieval from generated documents.The shared constraints also require consistency with Relevant_passage and prohibit changes that affect answer correctness.
  • A1 | Real-Industry Document Generation: A1 generates Chinese real-industry financial documents from Question, Answer, Relevant_passage, and Additional_information inputs.Additional_information contains decomposed SubQuestions and Related_questions, and the output includes a LaTeX source file, layoutCoT, and Reference_documents.
  • A1 | Real-Industry Document Generation: A1 expands passage themes using search and knowledge-base content, while requiring complex tables for numerical tasks and rich-text forms for named-entity retrieval.The generated content must remain consistent with the factual information in Relevant_passage.
  • A Prompt Templates: A1 emits one layoutCoT navigation chain per sub-question, whereas A2–A5 emit one chain per main question.The chains describe how to obtain information from document paragraphs, tables, or forms.
  • A Prompt Templates: A2 uses a fixed corporate-report filename, while A1 embeds the financial document type and document title in its institution–year path.Both prompts store the generated file path in Reference_documents.
  • A.2 Corporate Financial Report Generation: A2 generates Chinese BizFinBenchV2 counterfactual corporate financial reports from Question, Answer, and Relevant_passage triples.It expands reference data and randomly distributes items across rich-text paragraphs, complex tables, and forms without repeating conversion of the same item.
  • A.3 Investment Strategy Report Generation (Chinese Source): A3a generates investment strategy reports from Chinese anomaly-tracing benchmark items using Question, Answer, and Relevant_passage reference content.The prompt specifies LaTeX investment strategy reports together with layoutCoT and Reference_documents outputs.

A.5 General Financial Document Generation

The English financial-document prompt generates one of five FinanceBench document genres from reference triples and uses structured rewriting, layout complexity, and navigation hints. The government-bulletin variant instead organizes one OfficeQA triple with knowledge-base support.

  • A.4 General Financial Document Generation: The general English prompt generates FinanceBench documents in five permitted genres from Question, Answer, and Relevant_passage inputs.The genres are Investment Strategy Report, Corporate Financial Report, FinTech Research Report, Market Trend Analysis Report, and Market Regulation and Compliance Audit Report.
  • A.4 General Financial Document Generation: Each general-document invocation outputs a LaTeX document, layoutCoT, and Reference_documents, with answers requiring in-depth investigation or calculation.The document must be written entirely in English and may not use Chinese.
  • A.4 General Financial Document Generation: The general prompt rewrites and polishes reference content into a coherent document while integrating rich-text paragraphs, complex tables, forms, and optional transitional text.Added transitional segments should improve coherence without repeating the reference content’s main idea.
  • A.4 General Financial Document Generation: The general prompt requires checks ensuring that answers cannot be obtained directly or instantly from the generated document.This preserves the benchmark’s intended investigation-oriented question resolution.
  • A5 | Government Fiscal Bulletin Generation (English): The government-bulletin variant produces an English LaTeX bulletin for OfficeQA announcement items using one Question, Answer, Relevant_passage triple per invocation.Its outputs are a LaTeX bulletin, layoutCoT, and Reference_documents.
  • A5 | Government Fiscal Bulletin Generation (English): Government-bulletin generation organizes Relevant_passage with the knowledge base and must preserve answer correctness without adding explanations or formulas for financial terminology.The prompt also requires publicly accessible content and excludes confidentiality or classified statements.
  • A5 | Government Fiscal Bulletin Generation (English): The government-bulletin workflow creates a path-style layoutCoT describing how to obtain information from paragraphs, tables, or forms.The resulting file uses a UID-based government fiscal bulletin path stored in Reference_documents.

Overview of Part A Prompt Variants

Part A shares a common prompt skeleton but varies in benchmark, document type, input richness, content-generation strategy, and layoutCoT granularity. Generation intensity increases from A5 through A4 and A2/A3 to A1.

  • Overview of Part A Prompt Variants: Part A variants differ mainly in benchmark source, document type, input fields, content-generation strategy, layoutCoT granularity, and file-naming conventions.All variants retain the shared prompt skeleton and indirect-answer requirement.
  • Overview of Part A Prompt Variants: A1 uses Chinese real-industry QA with a variable document type and Q+A+Passage+Additional_info, while A2–A5 use simpler Q+A+Passage inputs.A5 uses one Q+A+Passage triple per invocation.
  • Overview of Part A Prompt Variants: A1 leaves the genre variable, A2 and A3 fix report genres, A4 permits five English genres, and A5 always produces a government fiscal bulletin.A3a and A3b both use investment strategy reports despite different source-language benchmarks.
  • Overview of Part A Prompt Variants: A1 treats Relevant_passage as a writing theme and expands it, whereas A2/A3 reorganize reference items across paragraphs, tables, and forms.A1 specifically mandates complex tables for numerical sub-questions and rich-text forms for entity-retrieval sub-questions.
  • Overview of Part A Prompt Variants: A4 polishes reference content into coherent English with possible transitional paragraphs, while A5 organizes passage text into bulletin format using only the knowledge base.A5 does not use search-based expansion.
  • Overview of Part A Prompt Variants: A1 emits layoutCoT per sub-question, whereas A2–A5 emit layoutCoT per main question.This changes the granularity of the document-navigation guidance attached to each benchmark item.
  • Overview of Part A Prompt Variants: A1 and A4 embed document type in filenames, A2/A3 use fixed genre suffixes, and A5 uses a UID instead of an institution–year identifier.The naming rules distinguish the generated document families in their stored paths.
  • Overview of Part A Prompt Variants: The appendix summarizes generation intensity as A5 → A4 ≈ A2/A3 → A1.The ordering reflects differences in expansion, input richness, and document-construction demands.

A.4 Investment Strategy Report Generation (English Benchmark)

The English investment-strategy prompt converts BizFinBenchV2 summary items into LaTeX reports whose answers require investigation or calculation. It expands reference content and varies its presentation across tables, paragraphs, and forms.

  • A3b | Investment Strategy Report Generation (English Benchmark): The A3b prompt generates English investment strategy reports in LaTeX for BizFinBenchV2 summary items.Each invocation accepts up to ten Question, Answer, Relevant_passage sets and outputs the report, layoutCoT, and Reference_documents.
  • A3b | Investment Strategy Report Generation (English Benchmark): Each question’s answer must be obtainable from the generated document through in-depth investigation or calculation rather than direct retrieval.The prompt separately requires checking that answers cannot be obtained directly or instantly.
  • A3b | Investment Strategy Report Generation (English Benchmark): The prompt expands Relevant_passage with search engines and a knowledge base while requiring consistency with the supplied factual information.It frames the passage as reference content for the investment strategy report.
  • A3b | Investment Strategy Report Generation (English Benchmark): Multiple reference items are randomly rewritten into complex tables or rich-text paragraphs and complex forms, without repeatedly converting the same item.The layout choices increase the complexity of resolving questions from the report.
  • A3b | Investment Strategy Report Generation (English Benchmark): The generated investment strategy report uses an institution–year, fixed-genre, and document-title filename stored under Reference_documents.The required filename suffix is Investment_Strategy_Report.
  • A3b | Investment Strategy Report Generation (English Benchmark): For each question, the prompt creates a path-style layoutCoT explaining how to obtain information from document paragraphs, tables, or forms.The chain is stored in the JSON dictionary at QAR_TRIPLES_PATH.
Loading 2607.19238v1…