Source-linked AI summary
Enhancing Financial Question Answering: A Novel Benchmark Dataset of Banks' financial statements
Arianna Miola, Bruno Spaccavento, Lorenzo Silotto, Marco Bianchetti, Luca Cagliero
TL;DR
Financial question answering over bank disclosures is difficult because the documents are complex, lengthy, and heterogeneous, while analysts need scalable extraction of key indicators. The paper introduces FinRAG-QA and evaluates a multi-stage RAG pipeline with retrieval and generation optimizations. The benchmark reports 38.8% retrieval-accuracy and 34.4% generation-accuracy improvements, while noting that results reflect models available in 2024–2025.
Problem
Extracting key financial indicators from large, technical, and heterogeneous bank disclosures requires substantial specialist effort and is difficult to scale.
Method
The paper introduces FinRAG-QA and evaluates a financial-document RAG architecture using contextual chunk enrichment, retrieval strategies, reranking, and generation strategies.
Results
38.8% retrieval accuracy and 34.4% generation accuracy improvements are reported from domain-specific optimizations.
Takeaways & Limitations
FinRAG-QA supports cross-institutional financial question answering evaluation across standardized indicators and international bank disclosures.
Takeaways & Limitations
The experiments are a snapshot based on GPT-4o and GPT-o1-high available in 2024–2025, so later models may produce different accuracy and latency trade-offs.
Abstract
from arXiv · showhide
The comparative analysis of banks' financial statements poses significant challenges for automated question answering systems due to their complexity, substantial length, technical language, and inhomogeneity of both textual and numerical content across different jurisdictions and institutions. We introduce FinRAG-QA, a novel benchmark dataset for financial question answering, which comprises 999 practitioner-curated questions on 10 standardised indicators, grounded in 209 annual and Pillar 3 reports from 24 major European and U.S. banks spanning 2019-2023. Unlike prior financial QA benchmarks, which centre on U.S. filings and single-institution analysis, FinRAG-QA targets cross-institutional retrieval over documents averaging 198k words, longer than any existing financial QA resource. On this benchmark we evaluate a multi-stage RAG pipeline and isolate the contribution of each component. Contextual chunk enrichment combined with a retrieval-optimised embedding model raises NDCG@10 from 0.322 to 0.710; conditional on the ground truth being retrieved, a reasoning-optimised generator raises answer accuracy from 44.6% to 79.0% (+34.4 percentage points), at roughly 20x the generation latency. We further show that cross-encoder reranking degrades retrieval when the first-stage ranking is already strong, and that a single top-ranked chunk outperforms larger contexts at generation time. Experiments were run in late 2024-early 2025 with the models available at that time.
1 Introduction
FinRAG-QA addresses the difficulty of extracting financial indicators from long, heterogeneous bank disclosures by introducing a benchmark and evaluating an optimized RAG architecture. The reported improvements span both retrieval and generation accuracy.
- The benchmark targets financial reports whose length, technical language, and heterogeneous numerical and textual formats make automated analysis difficult to scale.
- The proposed RAG architecture combines contextual chunk enrichment, hybrid retrieval, and advanced reranking for financial document analysis.
- The evaluation compares retrieval approaches, embedding models, and generation strategies to establish performance baselines for financial question answering.
- 38.8% retrieval accuracy and 34.4% generation accuracy improvements demonstrate gains from domain-specific optimizations.
2 Related Work
Existing financial QA datasets provide valuable retrieval, reasoning, and conversational benchmarks but often emphasize U.S. filings or single-institution settings. FinRAG-QA complements them with cross-institutional banking and regulatory coverage.
- FinDER provides 5,703 expert-annotated queries but is limited to U.S. public companies’ annual reports.
- FinanceBench contains approximately 10,000 questions from 40 U.S. public companies but was not designed for comparative analysis across institutions.
- DocFinQA increases average context length to approximately 123k words, while reporting degradation for retrieval-based and long-context models on the longest documents.
- Existing benchmarks largely center on U.S. regulation, overlook Basel III Pillar 3 disclosures, and favor single-institution analysis over cross-bank comparisons.
- FinRAG-QA enables cross-institutional analysis, banking compliance evaluation, and systematic metric extraction for professional workflows.
- The benchmark is positioned as a complement to existing resources rather than a replacement for them.
3 Dataset
The dataset standardizes financial questions around indicators, years, banks, document types, and source values. It contains 999 rows drawn from 209 reports covering 24 international banks and reflects the structural and numerical challenges of real financial analysis.
- The dataset contains 999 rows covering 10 key financial indicators.
- Each record identifies the indicator, year from 2019–2023, bank, and document type, distinguishing Financial Reports from Pillar 3 documents.
- Queries use the standardized form “What is the consolidated <Data> value in millions for <Bank> for the year <Year>?”
- Ground-truth values are decimal numbers in millions and are simple values rather than combinations derived from other values.
- The corpus comprises 209 documents: 120 annual financial reports and 89 Pillar 3 reports covering 24 major European and U.S. banks.
- Dataset challenges include long hierarchical documents, technical abbreviations, numerical reasoning and unit conversion, and information distributed across tables, text, or absent fields.
4 Experimental Setup
The paper evaluates a multi-stage RAG pipeline for financial question answering, covering stages from document ingestion through generation. Experiments used the most advanced generation models available between late 2024 and early 2025.
- The evaluation methodology assesses the proposed RAG pipeline from document ingestion to answer generation.
- Experiments were conducted between late 2024 and early 2025 using the most advanced generation models available at that time.
4.1 RAG Architecture
The RAG architecture combines document preprocessing, chunk indexing, retrieval, reranking, and generation, with contextual enrichment and specialized embedding models addressing lost chunk context and retrieval quality.
- Pipeline overview: The pipeline ingests and preprocesses documents, indexes chunks, retrieves and reranks candidates, then generates an answer from the top-ranked context.RAG grounds generation in retrieved external knowledge rather than relying only on pretrained model knowledge.
- Embedding models: The system evaluates OpenAI text-embedding-3-large as a baseline against VoyageAI voyage-3-large, which is optimized for retrieval.The indexed chunks are stored in a Qdrant vector store for searching.
- Contextual enrichment: Traditional chunking can discard context and make fragments difficult to interpret, weakening retrieval.Enrichment is designed to preserve enough document background for accurate retrieval and interpretation.
- Contextual enrichment: Contextual enrichment prepends each chunk with a concise synthetic summary describing its role in the document.The summaries are generated with GPT-4.1 and typically contain 50–100 tokens.
- Reranking: Reranking re-evaluates initially retrieved chunks with a computationally intensive cross-encoder before selecting the top-k results.Cohere rerank-3.5 is used for this second-stage relevance scoring.
- Generation: The generator receives the user query together with top-ranked, context-rich chunks; GPT-4o provides the baseline and GPT-o1-high tests reasoning-oriented generation.GPT-o1-high prioritizes accuracy over response latency for numerical extraction and interpretation.
4.2 Evaluation Metrics
Retrieval is evaluated by how highly relevant chunks rank, while generation accuracy is evaluated only when the ground truth appears in the retrieved context.
- Retrieval metrics: NDCG@k evaluates ranked retrieval lists by prioritizing relevant items near the top, with k tested at 1, 10, and 20.Its value ranges from 0 to 1, where 1 represents a perfect ranking and 0 indicates no relevance in the first k results.
- Retrieval metrics: NDCG@k compares DCG@k with the ideal ranking value IDCG@k, using relevance scores for chunks at each rank.The relevance score Gi is computed from occurrences of the numerical ground truth in each chunk.
- Retrieval metrics: Queries with non-numerical ground truths are excluded from retrieval evaluation.The evaluated relevance score is based on the number of times the numerical ground truth appears in a retrieved chunk.
- Generation metrics: Generation accuracy is measured only for queries whose ground truth appears in the retrieved context, isolating generator performance from retrieval errors.An answer is correct when it matches the ground truth, both values are null, or their absolute numerical difference is at most 0.01.
- Generation metrics: Accuracy is the proportion of evaluated queries whose generated values satisfy the correctness criterion.The indicator function contributes 1 when the criterion is met and 0 otherwise.
5 Results
The evaluation identifies strong retrieval and generation configurations for financial RAG, while exposing trade-offs involving reranking, context size, recall, and latency.
- Retrieval Performance: 0.710 NDCG@10 was achieved by combining contextual chunk enrichment with VoyageAI embeddings, improving over the OpenAI baseline’s 0.322.This configuration also had the fastest average retrieval time.
- Retrieval Performance: Reranking improved OpenAI contextualized embeddings by 5.1% on average but degraded VoyageAI retrieval by 19.1%.The result suggests reranking can be detrimental when first-stage retrieval quality is already high.
- Retrieval Performance: Increasing k from 10 to 20 reduced NDCG@10 from 0.710 to 0.705 but increased retrieved-ground-truth queries from 929 to 956 of 978.This reflects a trade-off between ranking quality and overall recall.
- Retrieval Performance: Document-heading enrichment provided negligible benefit over the baseline, indicating that semantic context mattered more than structural metadata alone.The comparison concerns enrichment strategies within retrieval.
- Generation Performance: 79.0% weighted average accuracy for GPT-o1-high exceeded GPT-4o’s 44.6% by 34.4 percentage points when the ground truth was retrieved.Generation evaluation measured accuracy only on queries with the ground truth present in retrieved context.
- Generation Performance: For generation, k=1 often produced the highest accuracy, while GPT-o1-high required 35.0s on average versus 1.8s for GPT-4o.The results indicate a trade-off between focused context, answer accuracy, and response latency.
6 Conclusions
The paper establishes a benchmark and optimized RAG approach for financial question answering, with gains from contextual enrichment, advanced embeddings, and reasoning-focused generation. It also identifies computational, reranking, reproducibility, and temporal boundaries for the current system.
- Key findings: 999 expert-validated questions across 10 financial indicators and 24 major banks provide a rigorous benchmark for financial QA.The benchmark is intended to support evaluation across European and U.S. regulatory disclosures.
- Key findings: NDCG@10 increased from 32.2% to 71.0%, a 38.8% retrieval-accuracy improvement, through contextual chunk enrichment and advanced embedding models.The reported improvement compares the baseline and optimized retrieval configurations.
- Key findings: The evaluation indicates that contextual enrichment and financial-domain embeddings provide substantial gains, while domain-specific optimizations outperform generic approaches.The paper also reports that semantic enrichment was more valuable than structural enhancements.
- Limitations: Contextual enrichment introduces substantial computational overhead that may be prohibitive for smaller organizations.The approach processes entire documents with large language models.
- Limitations: Reranking remains inconsistent, reasoning-model outputs may be difficult to reproduce, and results are a snapshot of 2024–2025 model capabilities.The paper identifies mixed reranking outcomes, stochastic reasoning models, and temporal validity as boundaries of the current approach.