Source-linked AI summary
Improving Retrieval for RAG based Question Answering Models on Financial Documents
Spurthi Setty, Harsh Thakkar, Alyssa Lee, Eden Chung, Natan Vidra
TL;DR
Standard RAG can miss relevant financial-document context, contributing to the retrieval challenge in domain-specific question answering. The paper examines retrieval improvements, and reports higher LLM evaluation scores for zero-shot query expansion and re-ranking than base RAG, while correct context yields the strongest reported score.
Problem
Standard RAG can miss relevant information because uniform chunking and similarity search may fail to identify the most relevant context in complex documents.
Source: §1.3 · p. 4Method
The paper examines retrieval improvements including sophisticated chunking, query expansion, metadata annotations, and re-ranking for financial-document RAG.
Source: §2.1 · p. 6, §2.3 · p. 7, §2.4 · p. 8Results
LLM evaluation scores were 0.24 for query expansion and 0.256 for re-ranking, versus 0.204 for base RAG; fake RAG scored 0.573.
Source: §4 · p. 11Takeaways & Limitations
The authors conclude that better retrieval improves document-based question answering and that their framework is applicable beyond financial documents, including healthcare and legal tech.
Source: §6 · p. 13Traditional similarity search may retrieve irrelevant or opposing information, and uniform chunking may miss answers distributed across sections.
Source: §1.3 · p. 4
Abstract
from arXiv · showhide
The effectiveness of Large Language Models (LLMs) in generating accurate responses relies heavily on the quality of input provided, particularly when employing Retrieval Augmented Generation (RAG) techniques. RAG enhances LLMs by sourcing the most relevant text chunk(s) to base queries upon. Despite the significant advancements in LLMs' response quality in recent years, users may still encounter inaccuracies or irrelevant answers; these issues often stem from suboptimal text chunk retrieval by RAG rather than the inherent capabilities of LLMs. To augment the efficacy of LLMs, it is crucial to refine the RAG process. This paper explores the existing constraints of RAG pipelines and introduces methodologies for enhancing text retrieval. It delves into strategies such as sophisticated chunking techniques, query expansion, the incorporation of metadata annotations, the application of re-ranking algorithms, and the fine-tuning of embedding algorithms. Implementing these approaches can substantially improve the retrieval quality, thereby elevating the overall performance and reliability of LLMs in processing and responding to queries.
1 Introduction
LLMs have limited and uneven domain knowledge, while retrieval-augmented generation supplies external document context without updating model weights. The section motivates improving retrieval because financial documents are too large and complex to include in full, and standard retrieval can miss relevant information.
1 Introduction
- Domain-specific performance is constrained by general, static pretraining data and depends on how often a topic appears in that data.Financial information is identified as a domain where standard LLMs perform poorly.Source: §1 · pp. 1–2
1.1 Fine-tuning
- Fine-tuning adds domain information to model parameters, but unsupervised fine-tuning may not robustly retain or retrieve it, especially when domain data is small.Supervised fine-tuning, by contrast, depends on access to high-quality labeled data.Source: §1.1 · p. 2
1.2 Retrieval Augmented Generation
- RAG injects external knowledge in context without changing model weights, but its context window limits how much information can accompany a query.The retrieval step selects document chunks to supply as context for financial question answering.Source: §1.2 · p. 3
1.3 Limitations of Current RAG Pipelines
- Uniform chunks can split information spanning sections, while similarity search may return semantically similar but irrelevant or opposing content.Across multiple financial reports, disregarding document identity and year can also return outdated or irrelevant chunks; multi-step questions may require reasoning beyond the original query.Source: §1.3 · pp. 4–5
2 Techniques to Improve Retrieval
The section surveys retrieval improvements for financial-document RAG, from document-aware chunking and query expansion to metadata, re-ranking, and embedding fine-tuning. It notes trade-offs in chunking and that this paper focuses on zero-shot methods requiring no additional training data.
2.1 Chunking Techniques
- Chunking choices should match the documents and queries: element-based chunking can preserve tables, while agentic chunking is computationally expensive and may exceed embedding-model token limits.Recursive and semantic chunking preserve context better than character-based or uniform methods but are more complex to implement.Source: §2.1 · p. 6
2.2 Query Expansion
- HyDE expands retrieval by having an LLM generate a hypothetical document and searching with it alongside the original query; it can outperform standard retrievers but may return incorrect results.The method depends on another LLM to generate additional context.Source: §2.2 · p. 7
2.3 Metadata Annotations and Indexing
- Metadata annotations address retrieval confusion across documents by adding information about chunks, such as the type of table they contain.The section suggests aligning annotations with element-based chunking and adding summaries or representative keywords.Source: §2.3 · pp. 7–8
2.4 Re-ranking Algorithms
- A cross-encoder scores retrieved chunks for question relevance and re-ranks them, prioritizing relevance over the initial similarity ranking.The highest-ranked chunks after re-ranking are then used as context.Source: §2.4 · p. 8
2.5 Fine-tuning Embedding Algorithms
- Domain-specific embedding fine-tuning requires queries, a text corpus, and relevant documents; this paper instead focuses on zero-shot retrieval methods.The authors leave the possible benefits of additional annotations or data for embedding fine-tuning to future work.Source: §2.5 · pp. 8–9
3 Evaluation
The evaluation distinguishes retrieval quality from answer quality and uses both structured measures with human-analyst ground truth and unstructured measures that do not require it. It treats structured evaluation as more robust, while unstructured evaluation offers ways to assess context relevance and answer faithfulness.
3 Evaluation
- The evaluation covers both retrieval of context and answers generated from that context, using analyst-provided ground-truth contexts and answers where available.Structured evaluation uses ground truth; unstructured evaluation is used when ground truth is unavailable.Source: §3 · p. 9
3.1 Retrieval Quality
- Page-level and paragraph-level retrieval accuracy compare retrieved chunks with the analyst-referenced document page or section.A match on the same page counts as high page-level retrieval accuracy.Source: §3.1 · p. 9
3.2 Answer Accuracy
- Unstructured retrieval uses context relevance, the proportion of context sentences an LLM judges relevant to answering the question, penalizing redundancy.For answer accuracy, the section also describes ground-truth comparison with BLEU, Rouge-L, cosine similarity, and LLM evaluation; answer faithfulness measures the share of answer statements supported by context.Source: §3.1 · p. 9, §3.2 · pp. 9–10
4 Results
On the 150 publicly available FinanceBench rows, Fake RAG achieved the highest reported average LLM evaluation score, while query expansion and reranking made smaller gains over base RAG. Whole-dataset comparisons for chunking and metadata tagging were unavailable.
4 Results
- The evaluation used the 150 FinanceBench rows publicly available from a benchmark of 10,231 financial questions.The benchmark covers publicly traded U.S. companies and financial documents released between 2015 and 2023.Source: §4 · p. 10
- Chunking methods lacked a comprehensive whole-dataset metric, preventing direct comparison with the other methods.The authors cite token-limit and compute constraints.Source: §4 · p. 11
- Metadata-tagging results were unavailable because the technique had not been integrated clearly into the existing pipeline.Source: §4 · p. 11
- Fake RAG scored 0.573 on average LLM evaluation, versus base RAG at 0.204; query expansion scored 0.24 and reranking 0.256.The authors report Fake RAG had the highest score and the greatest score variation.Source: §4 · p. 11
- With the same LLM and prompt across methods, the authors attribute an accuracy gap of over 35 percent to differences in retrieved context.Source: §4 · p. 11
5 Impact
The paper identifies three RAG pipeline concerns: retrieving context spread across a document, treating similarity as relevance, and chunking without regard to document structure.
5 Impact
- Relevant context can span multiple locations, while retrieval often returns only one or two chunks; even more retrieved chunks may miss the sections an analyst would select.Source: §5 · p. 13
- Current pipelines assume similarity is equivalent to relevance, although the paper identifies these as distinct.Source: §5 · p. 13
- Uniform chunking can disregard document structure and leave parts of the source incomplete.Source: §5 · p. 13
6 Conclusion and Next Steps
The paper concludes that better retrieval can improve document-based question answering, and frames its work around financial-domain use cases. It identifies multi-part context retrieval and embedding adaptation as areas for further work.
6 Conclusion and Next Steps
- Improved retrieval can improve document-based question-answering quality and provide better citations and answers.Source: §6 · p. 13
- The work focuses on financial question answering, primarily on 10-K documents, while describing the framework as applicable to healthcare and legal technology.Source: §6 · p. 13
- Retrieving and using context spread across multiple document sections remains a challenge; the authors suggest knowledge graphs as a possible aid.Source: §6 · p. 13
- The authors also propose exploring user-labeled data to fine-tune embeddings for domain-specific word and phrase nuances.Source: §6 · p. 13