PaperlayerView exact revision on arXiv

Source-linked AI summary

Financial Report Chunking for Effective Retrieval Augmented Generation

Antonio Jimeno Yepes, Yao You, Jan Milczek, Sebastian Laverde, Renyu Li

arXiv:2402.05131v3cs.CL

TL;DR

RAG chunking research has largely focused on paragraph-level approaches that do not account for document structure, leaving best practices for financial reports uncertain. The paper evaluates chunks built from document elements identified by document-understanding models. It reports state-of-the-art FinanceBench Q&A performance for element-based chunking and says the approach adapts to different documents without tuning.

  • Problem

    Research has largely centered on paragraph-level chunking, leaving the best practice for chunking documents for effective RAG document understanding unresolved.

    Source: §1 · pp. 1–2

  • Method

    The paper evaluates RAG chunking based on structural elements annotated by document-understanding models on FinanceBench.

    Source: §1 · p. 3, §3.5 · p. 7

  • Results

    Element-based chunking attains state-of-the-art FinanceBench Q&A performance, with reported accuracy of 50% vs 53.19%, and shows consistent results between retrieval and Q&A accuracy.

    Source: §5 · p. 11

  • Takeaways & Limitations

    The method adapts to different documents without tuning, while basic chunk-size results support that the best size varies by data.

    Source: §5 · p. 11

    Aggregating retrieval outputs from different chunking methods exceeded GPT-4’s token limit for Q&A, making that solution ineffective with the selected model.

    Source: §5 · p. 12

Abstract

from arXiv · show

Chunking information is a key step in Retrieval Augmented Generation (RAG). Current research primarily centers on paragraph-level chunking. This approach treats all texts as equal and neglects the information contained in the structure of documents. We propose an expanded approach to chunk documents by moving beyond mere paragraph-level chunking to chunk primary by structural element components of documents. Dissecting documents into these constituent elements creates a new way to chunk documents that yields the best chunk size without tuning. We introduce a novel framework that evaluates how chunking based on element types annotated by document understanding models contributes to the overall context and accuracy of the information retrieved. We also demonstrate how this approach impacts RAG assisted Question & Answer task performance. Our research includes a comprehensive analysis of various element types, their role in effective information retrieval, and the impact they have on the quality of RAG outputs. Findings support that element type based chunking largely improve RAG results on financial reporting. Through this research, we are also able to answer how to uncover highly accurate RAG.

1 Introduction

The paper studies how to chunk documents for RAG, where chunk size affects retrieval and answer generation. It focuses on SEC financial reports and examines structural-element chunking as an alternative to simple token-based strategies.

1 Introduction

  • RAG retrieves document segments relevant to a query and uses them with an LLM to generate an answer.The paper frames RAG as a way to constrain answers to retrieved material when recovering factual information.Source: §1 · p. 2
  • What constitutes effective chunking for RAG document understanding remains an open question.Chunk size is one of several dimensions to consider.Source: §1 · p. 2
  • Chunk size matters because it affects token limits, retrieval results, and the amount of retrieved context processed during answer generation.The paper notes that embedding models may constrain chunk length and that retrieved-chunk token counts may affect generation [25].Source: §1 · p. 2
  • The study examines chunking U.S. SEC financial reports, including 10-Ks, 10-Qs, and 8-Ks, whose layouts and tabular content vary.The authors argue that better document preprocessing and chunking configuration may improve their use.Source: §1 · p. 2
  • Prior work used simple strategies such as token-based chunking, while this study investigates structural-element chunking for financial reports.The authors call this the first systematic study of chunking for document understanding, particularly financial-report processing [14].Source: §1 · p. 3

2 Related work

The related work covers RAG, document-structure analysis, financial question-answering datasets, and chunking strategies. It identifies limited evaluation of chunking for RAG on financial reports as the gap this study addresses.

2 Related work

  • Prior RAG research examines configurations [12], dynamic retrieval, reasoning-chain storage and retrieval optimization, and RAG with fine-tuning.Source: §2 · p. 3
  • Common chunking strategies include fixed-size, recursive, contextual, and hybrid approaches, while element-type chunking uses document structure such as headings, paragraphs, and tables.The authors distinguish structure-aware element-type chunking from Markdown or LaTeX chunking.Source: §2 · p. 3
  • Prior financial-document work has addressed structure in UK and French reports, including El-Haj et al. and the FinTOC challenges.Related work also includes U.S. SEC report resources DocLayNet and FinTabNet.Source: §2 · pp. 3–4
  • Financial QA datasets such as FinQA [6], LLMWare, ConFIRM [8], and TAT-QA [46] support testing LLM methods on financial-data snippets, including tables.Source: §2 · p. 4
  • FinanceBench [14] proposes questions about financial-report content with evidence snippets, addressing a gap in work focused on layouts or report snippets.The passage describes broader research on understanding financial reports as limited.Source: §2 · p. 4
  • Standard RAG chunking approaches include token-length spans and sentence-based chunks, while the cited open-source tools do not explicitly consider table structure.The passage names Unstructured, Llamaindex, and Langchain as examples.Source: §2 · p. 4
  • The authors identify exhaustive evaluation of chunking for RAG, particularly on financial reporting, as lacking, and compare a broad range of strategies.They analyze methods for questions about different aspects of financial reports; prior chunking analysis was limited [14].Source: §2 · p. 4

3 Methods

The experiments hold the RAG setup constant while varying document chunking, then evaluate retrieval and generation on FinanceBench. The method compares token-sized chunks with chunks assembled from document elements and enriched with metadata.

3.1 RAG setting for the experiments

  • The RAG setting splits documents into chunks for vector indexing, retrieves the top-k chunks most similar to an encoded question, and uses them to generate an answer.The question is also turned into instructions for the LLM to answer from retrieved chunks; chunking is the experimental variable while other settings remain constant.Source: §3.1 · pp. 4–5

3.2 Indexing and retrieval

  • For indexing, each chunk is encoded and stored in a vector database; retrieval ranks chunks by similarity to the question vector, with the top 10 retrieved per question.The experiments use Weaviate and a sentence-transformer encoder; the number of chunks varies with chunking strategy.Source: §3.2 · pp. 5–6

3.3 Generation

  • GPT-4 generates answers from prompts containing the question and retrieved chunks, with instructions to answer from the source or return “No answer.”The prompt was designed in a similar RAG implementation with different document types.Source: §3.3 · p. 6

3.4 Chunking

  • Token baselines use chunks of 128, 256, or 512 tokens, with aggregation across chunking configurations also evaluated.Source: §3.4 · p. 6
  • Structural chunks use Chipper’s document-element predictions, focusing on titles, text, and tables, with new chunks beginning at titles and tables preserved whole.Elements shorter than 2,048 characters are iteratively merged with following elements until the desired length is reached, without breaking an element.Source: §3.4 · p. 7
  • Each element-based chunk can be indexed with GPT-4-generated keywords or a summary, or with a naive representation using its first two sentences and, for tables, the caption description.Source: §3.4 · p. 7

3.5 Dataset

  • The study evaluates chunking strategies on FinanceBench, a dataset of realistic open-book financial questions, including questions requiring computational reasoning.The dataset has 150 question-answer instances from 84 reports; the authors recovered 80 source documents, leaving 141 questions for evaluation.Source: §3.5 · p. 7
  • FinanceBench reports span 4 to 549 pages, with 147.34 pages on average (standard deviation 97.78) across 11,787 pages.Evidence locations range from the first page to page 304, averaging page 54.58 (standard deviation 43.66).Source: §3.5 · pp. 7–8

4 Results

The evaluation tests chunking strategies on FinanceBench using factual accuracy to assess retrieval and generation. It covers 80 documents and 141 questions.

Chunking Efficiency

Q&A Accuracy

  • GPT-4-based and manual evaluations show element-based strategies offer the best Q&A accuracy, consistent with retrieval results.Source: Q&A Accuracy · p. 10
  • 62,529 versus 112,155 chunks, element-based chunking achieves the highest retrieval scores with half the chunks of methods that ignore document structure.The paper says this can reduce indexing cost and improve query latency.Source: Q&A Accuracy · p. 10

Retrieval Accuracy

5 Discussion

The discussion reports FinanceBench Q&A accuracy of 50% vs 53.19% and consistent results between retrieval and Q&A for element-based chunking. It also describes automatic chunk sizing and notes limits involving aggregation and GPT-4 evaluation.

5 Discussion

  • FinanceBench Q&A accuracy is reported as 50% vs 53.19%, and element-based chunking yields consistent retrieval and Q&A accuracy.Source: §5 · p. 11
  • Basic 512-token chunks most resemble the Unstructured approach, but may lose coherent context and required information; their chunk contexts score lower on ROUGE and BLEU in Table 4.Source: §5 · p. 11
  • The best basic chunk size varies across data, while structural layout lets the method adapt chunk size automatically without tuning.Source: §5 · pp. 11–12
  • Aggregation appears effective for retrieval, but its Q&A input exceeded GPT-4’s token limit and produced a non-effective solution.Source: §5 · p. 12
  • Reordering prompt context and question produced no statistically different results, while changing prompt verbs seemed to lower answer quality.The authors conclude that prompt engineering is relevant to RAG.Source: §5 · p. 12
  • GPT-4 evaluation failed on some elaborate answers, including a 39.73% estimate accompanying an answer of 39.7%.Source: §5 · p. 12

6 Conclusions and Future Work

The conclusion reports improved state-of-the-art Q&A performance from element-based chunking. Future work would test other domains and examine additional element types, element relations, and RAG configuration.

6 Conclusions and Future Work

  • Element-based chunking improves state-of-the-art Q&A performance for the task by providing a better chunking strategy for processed documents.Source: §6 · p. 12
  • Future evaluation in domains such as biomedicine would test how findings apply beyond financial reporting, while further study could examine additional element types and their relations.Source: §6 · p. 12
  • The authors also propose studying how RAG configuration interacts with element-type-based chunking.Source: §6 · p. 13

Related papers

Loading 2402.05131v3…