PaperlayerView exact revision on arXiv

Source-linked AI summary

FinRAGBench-V: A Benchmark for Multimodal RAG with Visual Citation in the Financial Domain

Suifeng Zhao, Zhuoran Jin, Sujian Li, Jun Gao

arXiv:2505.17471v2cs.CL

TL;DR

Financial RAG benchmarks largely focus on text, while financial documents also contain charts and tables and require traceable evidence. FinRAGBench-V combines a bilingual retrieval corpus and varied QA dataset with RGenCite and automatic visual-citation evaluation; experiments expose different strengths and challenges across retrieval, reasoning, and citation.

  • Problem

    Existing financial RAG benchmarks predominantly focus on text, while multimodal documents and the need for traceable, verifiable answers motivate broader coverage.

    Source: §1 · p. 1

  • Method

    FinRAGBench-V combines a diverse financial retrieval corpus and QA dataset with RGenCite, which generates answers and page- and block-level visual citations, plus automatic citation evaluation.

    Source: §1 · pp. 1–2

  • Results

    Multimodal retrievers preserve chart and table information better than text-only retrievers, but MLLMs struggle with numerical reasoning and multi-page inference, and systems struggle more with block-level than page-level citation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    FinRAGBench-V exposes MLLM limitations and serves as a resource to guide future improvements in financial visual RAG systems.

    Source: §8 · p. 9

    The work did not train a dedicated multimodal RAG model for the financial domain.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) plays a vital role in the financial domain, powering applications such as real-time market analysis, trend forecasting, and interest rate computation. However, most existing RAG research in finance focuses predominantly on textual data, overlooking the rich visual content in financial documents, resulting in the loss of key analytical insights. To bridge this gap, we present FinRAGBench-V, a comprehensive visual RAG benchmark tailored for finance which effectively integrates multimodal data and provides visual citation to ensure traceability. It includes a bilingual retrieval corpus with 60,780 Chinese and 51,219 English pages, along with a high-quality, human-annotated question-answering (QA) dataset spanning heterogeneous data types and seven question categories. Moreover, we introduce RGenCite, an RAG baseline that seamlessly integrates visual citation with generation. Furthermore, we propose an automatic citation evaluation method to systematically assess the visual citation capabilities of Multimodal Large Language Models (MLLMs). Extensive experiments on RGenCite underscore the challenging nature of FinRAGBench-V, providing valuable insights for the development of multimodal RAG systems in finance.

1 Introduction

FinRAGBench-V addresses the limits of text-focused financial RAG with a multimodal benchmark and grounded visual citation. It also introduces RGenCite, an integrated retrieval-generation-citation baseline, and automatic citation evaluation; experiments reveal distinct retrieval, reasoning, and attribution challenges.

1 Introduction

  • FinRAGBench-V targets gaps in financial benchmarks that rely on text-only retrieval or isolated screenshots and omit traceable visual evidence.The introduction notes that text conversion can lose critical information and that isolated screenshots do not reflect retrieval across diverse sources and document types.Source: §1 · p. 1
  • The benchmark combines diverse real-world financial documents with a bilingual QA dataset spanning text, charts, tables, and varied financial reasoning tasks.The corpus includes 60,780 Chinese pages and 51,219 English pages; the QA set contains 855 Chinese and 539 English pairs.Source: §1 · pp. 1–2
  • RGenCite unifies retrieval, answer generation, and page- and block-level visual citations grounded in retrieved content.Its block-level citation approach adapts Ma et al. (2024b) to multimodal RAG.Source: §1 · p. 2
  • The proposed citation evaluation measures precision and recall at page and block levels using box-bounding and image-cropping strategies.Visual citation lacks dedicated evaluation methodologies, motivating these measures and strategies.Source: §1 · pp. 2–3
  • Multimodal retrievers preserve chart and table information better than text-only retrievers, while MLLMs struggle with numerical reasoning and multi-page inference.Systems perform well at page-level citation but struggle with block-level citation.Source: §1 · p. 2

2 Related Work

Prior financial multimodal benchmarks cover parts of the problem but often omit integration across data types, question diversity, or document-level retrieval. This work situates its benchmark against those gaps and adapts coordinate-based visual citation to financial RAG.

Benchmarking Multimodal RAG

  • Financial multimodal benchmarks vary in scope: EvoChart focuses on charts, Cho et al. (2024) and Suri et al. (2024) support limited question types, and MME-Finance relies on isolated screenshots.The cited comparison identifies missing text-and-table integration or document-level retrieval as limitations.Source: §2 · p. 3
  • Text-only financial RAG benchmarks may overlook information in charts and graphs, motivating multimodal RAG benchmark design.The passage cites FinQA (Chen et al., 2021) and OmniEval (Wang et al., 2024c) as text-only examples.Source: §2 · p. 3

Citation and Its Evaluation

  • The paper adapts Ma et al. (2024b)’s coordinate-based multimodal citation approach to financial RAG and proposes automatic visual-citation evaluation.The related-work discussion contrasts this with prior focus on textual citations.Source: §2 · p. 3

3 Task Definition

The task is organized into benchmark construction followed by retrieval and answer generation with citations. The second phase returns an answer together with citations to retrieved pages and supporting blocks.

3 Task Definition

  • The task has two phases: construct FinRAGBench-V, then implement the RGenCite baseline.The paper presents these phases in Figure 2.Source: §3 · p. 3
  • Benchmark construction forms a page corpus S and a QA dataset D, whose entries pair a question, answer, question type, and corresponding page set.Each dataset entry is defined as d_i = (q_i, a_i, t_i, P_i).Source: §3 · p. 3
  • Given a question, a retriever selects top-k pages from S, and a generator produces an answer with citations to pages and their supporting blocks.Each citation pairs a cited page with its corresponding block set.Source: §3 · p. 3

4 The Construction of FinRAGBench-V

FinRAGBench-V pairs a bilingual, multimodal financial-page corpus with a human-inspected QA dataset designed to cover varied data formats and reasoning tasks.

4.1 Retrieval Corpus Collection

  • The team selected 1,104 Chinese and 1,105 English documents, converting each document page into a single image and producing a retrieval corpus of 60,780 Chinese and 51,219 English pages.Source: §4.1 · p. 4

4.2.1 QA Pairs Synthesis

  • GPT-4o generates QA pairs from selected pages, covering single- and multi-page questions, text, charts and tables, and short and long answers.The dataset organizes questions into seven financial-domain categories.Source: §4.2.1 · pp. 4–5
  • The seven categories span text inference, chart extraction and calculations, and time-sensitive chart queries.Source: §4.2.1 · p. 5
  • The remaining categories cover table calculations, table comparison and sorting, and queries requiring multiple pages.Source: §4.2.1 · p. 5

4.2.2 Quality Inspection

  • Human inspection checks question clarity and categorization, answer accuracy, and page attribution for multi-page queries, reducing 11,328 generated pairs to 1,394.The retained set contains 855 Chinese and 539 English pairs.Source: §4.2.2 · p. 5

5 RGenCite: Retrieval, Generation, and Visual Citation

RGenCite combines retrieval and answer generation with visual citations, while FinRAGBench-V supplies the corpus and QA dataset on which the baseline operates.

5 RGenCite: Retrieval, Generation, and Visual Citation

  • RGenCite is a baseline built on FinRAGBench-V that integrates retrieval, generation, and visual citation in the generation stage.Source: §5 · p. 5

5.1 Retrieval

  • For a query, retrievers identify the top-k relevant pages from the corpus.Source: §5.1 · p. 5

5.2 Generation with Visual Citation

  • The generator receives the query and retrieved pages and produces an answer with page-level citations and block-level citations represented by coordinates.A citation pairs a reference page with the specific cited regions on that page.Source: §5.2 · pp. 5–6

6 Evaluation Metrics

The evaluation framework measures retrieval quality, answer accuracy, and visual citation quality, including citation coverage and relevance at page and block levels.

6.1 Retrieval Quality

  • Retrieval quality is measured with nDCG@5/10, Recall@5/10, and MRR@10, capturing ranking quality, coverage, and early relevance.Source: §6.1 · p. 6

6.2 Answer Accuracy

  • Answer evaluation combines ROUGE with GPT-4o assessment of whether responses match ground truths and remain consistent with visual context.Source: §6.2 · p. 6

6.3 Citation Quality

  • Citation quality uses recall and precision at page and block levels, evaluated through box-bounding or image-cropping.Source: §6.3 · p. 6
  • Recall tests whether cited images jointly support the answer, while precision measures whether citations are essential; precision is evaluated only when recall is 1.Source: §6.3 · p. 6
  • Image-cropping is the default citation evaluation approach because it aligns better with IoU scores and human judgments than box-bounding.An evaluator MLLM assesses the cited content under both approaches.Source: §6.3 · p. 7

7 Experiments and Results

Experiments compare multimodal and text retrievers and evaluate proprietary and open-source MLLMs on generation and citation. Results show strong retrieval gains for multimodal systems, alongside continuing challenges in complex reasoning and block-level citation.

7.1 Basic Settings

  • Generation experiments supply the top 10 pages from the best retriever and compare proprietary and open-source MLLMs across tasks.The retrieval comparison includes five multimodal and four text retrievers.Source: §7 · p. 7, §7.1 · pp. 7–8

7.2 Main Results

  • ColQwen2 reaches recall@10 of 90.13 (Chinese) and 85.86 (English), versus 42.71 and 36.36 for the best textual retriever, BGE-M3.Source: §7.2 · p. 8
  • Proprietary MLLMs outperform open-source counterparts, performance varies across languages, and models struggle more with numerical calculations and multi-page inference than text inference and direct extraction.The language strengths differ by model, with several proprietary models stronger on English and Qwen models balanced or stronger on Chinese.Source: §7.2 · p. 8, Figure 5 · p. 8
  • Most MLLMs perform well on page-level citation, but block-level citation remains difficult, especially for open-source models.Source: §7.2 · p. 8

7.3 Consistency between Automatic Citation Evaluation with Human Evaluations

  • Image-cropping achieves Pearson correlations of 65.06 with IoU and 68.01 with human ratings, while box-bounding underperforms due to redundant visual content.IoU can be affected by blank space or missing key information despite a high overlap score.Source: Table 6 · p. 8, §7.3 · p. 9

8 Conclusion

FinRAGBench-V is a multimodal RAG benchmark for finance, pairing diverse real-world financial documents with a QA dataset spanning financial tasks and visual citations. Experiments expose MLLM limitations and are intended to guide future visual RAG improvements.

  • FinRAGBench-V combines a retrieval corpus from diverse real-world financial documents with a financial QA dataset and visual citations.Source: §8 · p. 9
  • Experiments expose MLLM limitations and position the benchmark as a resource for guiding future visual RAG improvements.Source: §8 · p. 9

Limitations

The authors did not train a dedicated financial multimodal RAG model. The appendix provides prompts for QA-pair construction, answer generation with visual citations, and evaluation.

Limitations

  • The study did not train a dedicated model for multimodal RAG in finance, leaving development of models tailored to this domain for future work.Source: Limitations · p. 9

A Prompts for QA Pairs Construction, Generation, and Evaluations

B Examples of Six Real-World Data Sources of Retrieval Corpus

The appendix illustrates retrieval-corpus sources and QA examples across financial question types, then shows visual citation outputs, evaluation strategies, and potential RGenCite errors.

B Examples of Six Real-World Data Sources of Retrieval Corpus

C.1 Text Inference:

  • Text-inference questions cover summarization and extracting information or trends from text.Source: §C.1 · p. 12

C.2 Chart-Information Extraction

C.5 Table-Numerical Calculations

  • Table questions include numerical calculations, comparing and ranking entities, and identifying highest or lowest values.Source: §C.5 · p. 12, §C.6 · p. 12

C.7 Multi-page Queries

  • Multi-page queries require combining information across pages, including text, tables, or charts split between pages.Source: §C.7 · p. 12

D Example for Visual Citation and the Two Evaluation Methods

E.1 Error Case Study Based on Different Stages in RGenCite

  • The case study identifies three RGenCite error types: retrieved images without relevant information, graphical reasoning errors, and incorrect citations despite a correct answer.Source: §E.1 · pp. 12–13

E.2 Error Case Study Based on Typical Task Types in the Financial Domain

The case studies illustrate errors in chart interpretation, financial-table reading, and cross-page aggregation. The appendix also reports resource-use details and warns of possible corpus and QA-data risks.

E.2 Error Case Study Based on Typical Task Types in the Financial Domain

  • In a candlestick chart example, models fail to report opening and closing prices despite the report containing them; the correct values are 14.25 and 13.55.Source: §E.2 · p. 13
  • A table misread of AMBAC’s 2020 exposure value as $6,325 million rather than $6,352 million produced a calculated decrease of $1,840 million instead of $1,813 million.Source: §E.2 · p. 13
  • A multi-page GDP query failed because the relevant U.S. and Brazil data were split across pages, leading to an inaccurate comparison.Source: §E.2 · p. 13

G Resource Usage

  • The study reports $3,021.47 in proprietary API costs, local open-source experiments on 4×A100 80GB GPUs, and manual annotation by three experienced annotators.Source: G Resource Usage · p. 13

H Potential Risks

  • The corpus may include noisy, outdated, or irrelevant documents, while QA annotations may contain errors, ambiguities, or biases that could affect evaluation accuracy and result generalizability.Source: H Potential Risks · pp. 13–14

F Annotation guidelines.

  • Appendix F identifies Table 27 as the QA-pair verification guideline and Table 28 as the guideline for rating-based human evaluation of visual citations.Source: Table 27 · p. 35, Table 28 · p. 35

Related papers

Loading 2505.17471v2…