Source-linked AI summary
Enhancing Financial Report Question-Answering: A Retrieval-Augmented Generation System with Reranking Analysis
Zhiyuan Cheng, Longying Lai, Yue Liu, Kai Cheng, Xiaoxi Qi
TL;DR
Financial question-answering over lengthy 10-K reports requires reliable retrieval, while the contribution of neural reranking in modern financial RAG systems remains underexplored. The paper builds a hybrid RAG pipeline and evaluates reranking on FinDER, finding higher correctness and fewer completely incorrect answers across five experimental groups.
Problem
The impact of neural reranking on financial RAG question-answering remains underexplored, despite limited systematic end-to-end evaluation with realistic queries.
Method
The paper evaluates an end-to-end financial RAG system combining full-text and semantic retrieval with optional neural reranking on 1,500 FinDER queries across five groups.
Results
15.5 percentage points: Score≥8 correctness rises from 33.5% to 49.0%, while Score=1 answers fall from 35.3% to 22.5% across five experimental groups.
Takeaways & Limitations
Neural reranking provides consistent correctness gains and reduces critical errors in this financial 10-K RAG evaluation.
Takeaways & Limitations
The evaluation uses an LLM-as-judge approach that may favor verbose answers or penalize valid alternative phrasings; human validation is needed.
Abstract
from arXiv · showhide
Financial analysts face significant challenges extracting information from lengthy 10-K reports, which often exceed 100 pages. This paper presents a Retrieval-Augmented Generation (RAG) system designed to answer questions about S&P 500 financial reports and evaluates the impact of neural reranking on system performance. Our pipeline employs hybrid search combining full-text and semantic retrieval, followed by an optional reranking stage using a cross-encoder model. We conduct systematic evaluation using the FinDER benchmark dataset, comprising 1,500 queries across five experimental groups. Results demonstrate that reranking significantly improves answer quality, achieving 49.0 percent correctness for scores of 8 or above compared to 33.5 percent without reranking, representing a 15.5 percentage point improvement. Additionally, the error rate for completely incorrect answers decreases from 35.3 percent to 22.5 percent. Our findings emphasize the critical role of reranking in financial RAG systems and demonstrate performance improvements over baseline methods through modern language models and refined retrieval strategies.
I. INTRODUCTION
Financial 10-K reports are comprehensive but difficult to search manually because of their length, complexity, and dispersed disclosures. The paper presents an end-to-end financial RAG system and evaluates neural reranking as a response to these challenges.
- 10-K reports typically span 100-300 pages and contain financial, operational, risk, and management information.
- Manual extraction is time-consuming, error-prone, and difficult to scale across companies or reporting periods.
- RAG combines retrieval with language-model generation to ground answers in relevant passages rather than relying only on parametric knowledge.
- Financial RAG remains challenging because irrelevant or marginally relevant retrievals can produce incorrect or incomplete answers.
- The paper addresses the underexplored contribution of neural reranking to financial document question-answering with modern language models.
- The system implements an end-to-end hybrid RAG pipeline and evaluates it using 1,500 FinDER queries across five independent groups.
II. RELATED WORK
Prior work establishes RAG, hybrid retrieval, rank fusion, and neural reranking as complementary approaches for difficult financial question-answering. The paper targets limited end-to-end evidence about reranking in realistic financial settings.
- RAG combines parametric language-model memory with external document collections to improve factual accuracy in knowledge-intensive tasks.
- A standard RAG architecture retrieves relevant documents and conditions a generator on those documents and the query.
- FinDER contains realistic financial query-evidence-answer triplets from 10-K filings, including ambiguous queries with domain-specific abbreviations and jargon.
- FinDER includes 84.5% qualitative and 15.5% quantitative questions, with relevant evidence often scattered across document sections.
- Hybrid search combines exact-match keyword retrieval with semantic retrieval for complementary retrieval strengths.
- RRF combines ranked lists without training data, while cross-encoders jointly encode query-document pairs to assess fine-grained relevance.
- Existing evidence is limited because studies often isolate retrieval or generation and lack systematic end-to-end evaluation with realistic financial queries.
III. SYSTEM ARCHITECTURE
The document-processing architecture converts S&P 500 10-K reports into searchable lexical and semantic representations. Offline processing preserves document structure, creates overlapping chunks, and indexes both text and embeddings.
- The pipeline processes 10-K reports from all 500 S&P 500 companies, converting HTML documents to PDF for consistent extraction.
- PyMuPDF extracts textual content while preserving document structure before chunking.
- Chunks contain 2,500 characters with 1,250-character overlap to preserve boundary-spanning information while retaining retrieval granularity.
- SQLite FTS5 stores extracted chunks for Boolean queries, phrase matching, keyword retrieval, and relevance ranking.
- The text-embedding-3-small model generates 1,024-dimensional embeddings in batches of 32.
- FAISS indexes dense vectors for approximate nearest-neighbor search with sub-linear time complexity.
B. Query Processing Pipeline
The query pipeline rewrites user questions, retrieves candidates through lexical and semantic search, and fuses the results with Reciprocal Rank Fusion. This design addresses ambiguity while combining exact terminology matching with semantic similarity.
- Query Rewriting: GPT-4.1 rewrites queries by clarifying vague terms, correcting errors, and extracting keywords.
- Query Rewriting: The rewriting module outputs a clarified semantic-search query and extracted keywords for keyword retrieval.
- Query Rewriting: Query preprocessing improves retrieval recall by addressing query formulation issues.
- Hybrid Retrieval: Full-text search retrieves the top-20 chunks using extracted keywords and supports exact terminology and named-entity matching.
- Hybrid Retrieval: Semantic search retrieves the top-30 similar chunks using the clarified query and filters matches with a distance threshold of 2.0.
- Result Fusion: Reciprocal Rank Fusion combines candidates from both retrieval methods using k = 60.
- Result Fusion: The unsupervised fusion strategy combines complementary retrieval strengths without requiring training data.
3) Neural Reranking:
The reranking stage refines hybrid-retrieval candidates with a cross-encoder and adaptive cutoff strategies before answer generation.
- Neural Reranking: The reranker jointly encodes each query and candidate chunk to compute fine-grained relevance scores.It processes up to 30 chunks from the RRF-fused candidate list.
- Cutoff Strategies: The pipeline applies cumulative probability and score-cliff cutoffs to select the most relevant chunks.The probability threshold retains chunks through 55% cumulative mass, while cliff detection truncates after a score drop exceeding 0.15.
- Cutoff Strategies: Reranking typically produces 5-10 high-quality chunks and filters marginally relevant context before generation.These cutoffs are intended to prevent low-quality candidates from diluting context quality.
- Answer Generation: The selected chunks are assembled into a context string and provided to GPT-4.1 for answer generation.The system prompt requires answers to rely only on the supplied context and avoid speculation when information is insufficient.
- Answer Generation: Evaluation uses deterministic sampling with temperature=0.0 and includes the retrieved context plus the original query in the user prompt.This configuration supports reproducible evaluation.
A. Dataset
The study evaluates reranking on FinDER financial queries using five controlled groups and compares otherwise identical RAG configurations with and without neural reranking.
- Dataset: FinDER contains 5,703 query-answer pairs, with 84.5% qualitative and 15.5% quantitative questions.The queries involve multi-hop reasoning, domain-specific jargon, and scattered evidence.
- Experimental Configurations: The full pipeline combines query rewriting, hybrid search, RRF fusion, neural reranking, and answer generation.The ablation instead selects the top 10 fused results before generation.
- Experimental Configurations: The only configuration difference is whether the neural reranking stage is present.This design attributes observed performance differences to reranking.
- Evaluation Methodology: Answer quality is scored from 1 to 10 by comparing each generated answer with the FinDER ground truth.Score 1 denotes a completely incorrect answer, score 8 a basically correct answer, and score 10 a fully correct answer.
- Evaluation Methodology: The evaluation reports average score, Score=1, Score≥8, and Score=10 rates to capture overall quality and success or failure thresholds.These metrics provide complementary views of system performance.
D. Implementation Details
The implementation combines standard search, embedding, generation, and reranking components, with results reported across five experimental groups.
- Implementation Details: The software stack uses SQLite FTS5, FAISS vector search, OpenAI embeddings and GPT-4.1, and Jina AI reranking.The listed components support full-text retrieval, semantic retrieval, generation, and neural reranking.
- Implementation Details: Table II in the appendix provides the complete hyperparameter configuration.The main text points to the appendix for these settings.
- Quantitative Results: A 1.07-point average-score improvement raises performance from 4.95 to 6.02 with reranking.This corresponds to a 21.6% relative increase.
- Quantitative Results: 15.5 percentage points separate Score≥8 correctness with reranking at 49.0% from the no-reranking result at 33.5%.The relative improvement is 46.3%.
- Quantitative Results: 12.8 percentage points separate Score=1 error rates of 22.5% with reranking and 35.3% without reranking.This represents 36.3% fewer critical errors.
- Quantitative Results: All five independent groups show similar improvement patterns, with average-score gains ranging from 1.00 to 1.21 points.The reported pattern indicates consistent and reproducible benefits across groups.
B. Analysis and Discussion
Neural reranking improves financial RAG performance by filtering and prioritizing relevant context, but important failure modes remain.
- Impact of Reranking: Neural reranking plays a critical role in the reported financial RAG performance improvement.The analysis attributes the improvement to better candidate selection before generation.
- Impact of Reranking: Cross-encoder scoring promotes semantically relevant chunks that first-stage RRF may miss through superficial matches.Joint query-document processing captures relevance interactions unavailable to biencoder retrievers.
- Impact of Reranking: Adaptive cutoff strategies reduce the risk that marginally relevant passages dilute the generator’s context.Without reranking, RRF’s top-10 results may include passages that mislead generation.
- Error Analysis: 22.5% of queries still receive completely incorrect answers despite reranking.The reported failure analysis identifies ambiguity, absent information, multi-hop reasoning, and retrieval failures as contributing patterns.
- Error Analysis: Queries requiring synthesis across multiple non-adjacent sections remain challenging for chunk-based retrieval.This limitation affects multi-hop questions even when reranking is applied.
- Error Analysis: Reranking cannot recover relevant chunks absent from the top-30 hybrid-search candidates.This boundary suggests that improvements to first-stage retrieval remain necessary.
3) Comparison with Prior Work:
The system improves on earlier FinDER baseline performance, with reranking producing higher correctness and fewer completely incorrect answers. Benefits remain consistent across five independent query groups, while evaluation is limited to 10-K reports, FinDER queries, and LLM-based judging.
- Comparison with Prior Work: 49.0% correctness (Score≥8) with reranking exceeds the approximately 33% earlier-model baseline and the 33.5% no-reranking result.The reported improvement is attributed to modern LLMs, neural reranking, and hybrid retrieval, with reranking contributing 15.5 percentage points.
- Comparison with Prior Work: Five independent experimental groups show consistent improvements, with reranking average scores varying by only 0.24 points standard deviation.This consistency is reported as evidence that reranking benefits are not artifacts of particular query selections.
- Comparison with Prior Work: LLM-as-judge scoring may favor verbose answers or penalize valid alternative phrasings, motivating future human evaluation.The automated evaluation is described as cost-effective and scalable but potentially biased.
- Comparison with Prior Work: The evaluation covers 1,500 FinDER queries, while performance may differ for other financial documents or non-financial domains.The scope boundary includes 10-K reports rather than 8-K filings, earnings-call transcripts, or non-financial documents.
- Comparison with Prior Work: Neural reranking adds latency and API costs, requiring production systems to weigh response-time requirements against budget constraints.The paper identifies computational cost as a deployment consideration.
- Comparison with Prior Work: Real-world analyst-query performance may differ because the evaluation is restricted to the FinDER benchmark and can encounter distribution shift.This limitation concerns dataset coverage rather than the reported benchmark comparison.
- Comparison with Prior Work: 12.8 percentage points fewer completely incorrect answers occur with reranking, declining from 35.3% to 22.5%.The evaluation also reports that 22.5% of reranked queries remain completely incorrect.
B. Contributions
The paper contributes an end-to-end financial RAG system, rigorous FinDER evaluation, and controlled evidence for neural reranking. It also identifies extensions for retrieval, evaluation, efficiency, modality, domain coverage, and explainability.
- Contributions: The system processes all 500 S&P 500 companies’ 10-K reports using production-ready components.The contribution is framed as a complete end-to-end RAG pipeline.
- Contributions: Rigorous evaluation uses 1,500 FinDER queries divided into five independent groups of 300 queries each.The grouping is intended to provide statistical controls and robustness.
- Contributions: A controlled ablation study provides empirical evidence for the importance of neural reranking in financial RAG systems.The study also supplies updated performance baselines using modern LLMs and techniques.
- Future Directions: Graph-based retrieval could better support multi-hop queries by modeling relationships between document sections.This is proposed as a direction for advanced retrieval research.
- Future Directions: Combining automated LLM evaluation with human assessment could provide broader quality analysis and validate automated scoring reliability.The proposal addresses evaluation coverage rather than retrieval performance.
- Future Directions: Distilled or quantized rerankers could reduce computational costs while maintaining quality.Hardware-aware model co-design is identified as a possible route to lower latency in production reranking.
- Future Directions: Multimodal models could incorporate tables, charts, and figures that text-only approaches do not currently address.The proposed expansion targets financial-report content beyond text.
- Future Directions: Extending evaluation to 8-K filings, earnings-call transcripts, and analyst reports would broaden applicability.The paper also proposes comparing rerankers such as ColBERT, BGE-reranker, and Cohere rerank.
APPENDIX
The appendix identifies Table II as containing the complete system configuration.
- APPENDIX: Table II contains the complete system configuration.The passage provides the table title but no parameter values.