PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answering

Kuicai Dong, Yujing Chang, Shijie Huang, Yasheng Wang, Ruiming Tang, Yong Liu

arXiv:2505.16470v2cs.IRcs.CLcs.CV

TL;DR

Long multimodal documents challenge DocVQA with evidence discovery and cross-modal reasoning, while DocRAG benchmarks lack robust evaluation of multimodal selection and integration. MMDocRAG supplies annotated evidence chains and multimodal-generation evaluation, and experiments show persistent model challenges alongside advantages for proprietary models and fine-tuned systems.

  • Problem

    Long multimodal documents complicate evidence identification and cross-modal reasoning, while existing DocRAG benchmarks lack evaluation of multimodal evidence selection and integration.

    Source: §1 · pp. 1–2

  • Method

    MMDocRAG provides 4,055 expert-annotated QA pairs with multi-page, cross-modal evidence chains, quote-selection evaluation, and interleaved multimodal answers.

    Source: §1 · p. 2

  • Results

    Experiments find that current models struggle with multimodal evidence selection and interleaved answers; proprietary models outperform open-source alternatives, and fine-tuning and detailed visual descriptions improve performance.

    Source: §1 · p. 3, §6 · p. 11

  • Takeaways & Limitations

    MMDocRAG offers a benchmark for evaluating multimodal retrieval, quote selection, and answer generation across models and retrieval methods.

    Source: §1 · pp. 2–3

    The benchmark evaluates selecting and integrating candidate multimodal content, not generating that content from scratch.

    Source: §3 · p. 5

Abstract

from arXiv · show

Document Visual Question Answering (DocVQA) faces dual challenges in processing lengthy multimodal documents (text, images, tables) and performing cross-modal reasoning. Current document retrieval-augmented generation (DocRAG) methods remain limited by their text-centric approaches, frequently missing critical visual information. The field also lacks robust benchmarks for assessing multimodal evidence selection and integration. We introduce MMDocRAG, a comprehensive benchmark featuring 4,055 expert-annotated QA pairs with multi-page, cross-modal evidence chains. Our framework introduces innovative metrics for evaluating multimodal quote selection and enables answers that interleave text with relevant visual elements. Through large-scale experiments with 60 VLM/LLM models and 14 retrieval systems, we identify persistent challenges in multimodal evidence retrieval, selection, and integration.Key findings reveal advanced proprietary LVMs show superior performance than open-sourced alternatives. Also, they show moderate advantages using multimodal inputs over text-only inputs, while open-source alternatives show significant performance degradation. Notably, fine-tuned LLMs achieve substantial improvements when using detailed image descriptions. MMDocRAG establishes a rigorous testing ground and provides actionable insights for developing more robust multimodal DocVQA systems. Our benchmark and code are available at https://mmdocrag.github.io/MMDocRAG/.

1 Introduction

Long, multimodal documents make evidence discovery and cross-modal reasoning difficult, while existing DocRAG methods and benchmarks are text-centric. MMDocRAG addresses the evaluation gap with multimodal evidence chains, quote-selection evaluation, and interleaved text-image answers.

1 Introduction

  • Current DocRAG systems often favor plain text over visual evidence, while existing benchmarks primarily assess retrieved-quote recall or textual answer quality.The introduction identifies evaluation of multimodal evidence selection and coherent text-image integration as missing benchmark capabilities.Source: §1 · pp. 1–2
  • MMDocRAG provides 4,055 expert-annotated QA pairs with multi-page, cross-modal evidence chains, noisy quote selection, and answers that interleave text with visual content.Its evaluation includes selecting and integrating relevant multimodal quotes.Source: §1 · p. 2
  • 60-model experiments find persistent challenges in retrieval, evidence selection, and integration; proprietary VLMs outperform open-source models, and fine-tuning improves performance.Advanced proprietary VLMs generally gain modestly from multimodal rather than text-only input, whereas open-source or smaller proprietary models perform better with pure-text inputs; VLM-generated image descriptions outperform OCR text for LLMs.Source: §1 · p. 3

2 MMDocRAG Benchmark

MMDocRAG builds a multimodal DocVQA benchmark from long, diverse documents, with annotated questions, evidence, and interleaved answers. Its construction combines document parsing, answer generation, hard-negative quotes, and annotation quality checks.

2.1 Construction

  • Document parsing turns layout-detected text and visual elements into quotes stored as text, original images, OCR text, and VLM-generated descriptions.The corpus comprises 313 MMDocIR documents across 10 domains, averaging 65.1 pages; 2,373 high-quality pages were selected for annotation.Source: §2.1 · pp. 3–4
  • The benchmark combines 821 revised multimodal answers from existing QA pairs with 3,349 new single- and multi-page QA pairs paired with multimodal answers.New questions cover eight types and may use one or multiple document pages as evidence.Source: §2.1 · p. 4
  • Hard text and image negatives are mixed with gold quotes in candidate sets of 15 or 20, testing whether models distinguish relevant evidence from similar distractors.Each quote is annotated with its layout and page identifier for traceability.Source: §2.1 · p. 4

2.2 Dataset Analysis

  • 4,055 questions cover 222 documents averaging 67 pages and about 33k words, including 2,107 cross-page, 1,590 multi-image, and 2,503 cross-modal questions.The questions are split 2,055/2,000 for development and evaluation; each has an average of 2.7 gold quotes among 15/20 candidates.Source: §2.2 · p. 5

2.3 Quality Assurance

  • Quality assurance retains 90.2% of answers and 93.5% of gold quotes after validation, while cross-validation yields quote-selection F1 scores of 89.7 and 91.4 for the two groups.Expert review follows semi-automated validation, with up to three regeneration attempts for answers that fail checks.Source: §2.3 · p. 5

3 Task Definition

The task separates retrieving relevant document quotes from generating a multimodal answer using a fixed candidate context. Evaluation focuses on selecting and integrating existing multimodal content, not generating it from scratch.

3 Task Definition

  • Retrieval ranks text and image quotes by query similarity, while answer generation receives fixed contexts of 15 or 20 quotes to isolate selection and generation performance.The contexts contain 10 text and 5 image quotes, or 12 text and 8 image quotes; irrelevant quotes should be excluded from the answer.Source: §3 · p. 5
  • The task evaluates selecting and integrating multimodal content from candidate quotes rather than generating multimodal content from scratch.This scope distinguishes the benchmark task from end-to-end visual-content creation.Source: §3 · p. 5

4 Experiments

The experiments evaluate quote selection, multimodal answer generation, and retrieval across varied models and input settings. Results show persistent challenges in selecting and integrating multimodal evidence, with performance affected by model type, quote representation, evidence position, and retrieval quality.

4.1 Evaluation Metric

  • Evaluation combines text and image quote-selection precision, recall, and F1 with lexical similarity and five LLM-judged answer-quality dimensions.Retrieval is measured with recall@k; answer judging covers fluency, citation quality, text-image coherence, reasoning logic, and factuality.Source: §4.1 · p. 6

4.2 Baseline Models

  • The study evaluates six text and four visual retrievers, hybrid retrieval configurations, 60 models, and nine models fine-tuned on MMDocRAG’s dev set.Hybrid retrieval combines text and visual retriever quotes; the fine-tuned set includes LLMs and VLMs.Source: §4.2 · p. 6

4.3 Main Results

  • With 20 quotes, GPT-4.1 leads quote-selection F1 at 70.2 and answer quality at 4.14; proprietary models generally outperform smaller and open-source models.Thinking models do not show advanced performance despite costing 3 times more output tokens, while fine-tuning can significantly improve selection and generation.Source: §4.3 · p. 6, Table 3 · p. 7

4.4 Multimodal vs Pure-text Quotes: Comparison and Analysis

  • Gemini, Claude, and GPT models show better quote selection and comparable answer quality with multimodal quotes, while Qwen models perform better with pure-text inputs.Qwen-7B and Qwen-32B LLMs have 175.9% and 62.7% higher quote-selection F1, respectively, than their equivalent VLMs.Source: §4.4 · p. 8, Table 4 · p. 8

4.5 Multimodal Quotes as text: VLM-text vs OCR-text

  • VLM-generated image descriptions outperform OCR text for image quote selection and multimodal answer generation, with additional overhead and latency.VLM-text is 0.5 times longer for tables and 2.8 times longer for figures than OCR-text.Source: Table 5 · p. 8, §4.5 · p. 9

4.6 Quotes Selection Analysis

  • Gold-quote selection accuracy declines at later sequence positions, with first-position image quotes especially likely to be selected and the final quotes least likely.The analysis separates 12 text-quote positions from eight image-quote positions.Source: §4.6 · p. 9, Figure 8 · p. 9

4.7 Quotes Retrieval Results

  • Visual retrievers outperform text retrievers on image retrieval but lag on text retrieval, while hybrid retrieval leverages both strengths.The experiments identify long-document retrieval as an ongoing challenge.Source: §4.7 · p. 9, Table 6 · p. 9

4.8 End-to-end Multimodal RAG Analysis

  • Under end-to-end retrieval, GPT-4.1’s quote-selection F1 falls from 70.2 to 54.4 and answer quality from 4.14 to 3.53 when recall drops from 100% to 71.0%.Query expansion improves BGE recall by 7.9 percentage points; multi-retriever recall reaches 84.9%, yet leading models still experience about 10% performance degradation under realistic retrieval conditions.Source: Table 7 · p. 9, §4.8 · p. 10

5 Related Work

Prior DocVQA benchmarks extend from single-page tasks to long-context and retrieval-augmented settings, while multimodal RAG work addresses image-text evidence. The paper identifies interleaved multimodal generation as a gap in existing DocVQA and DocRAG benchmarks.

5 Related Work

  • Interleaved text-image generation methods include diffusion-plus-LLM pipelines and autoregressive approaches, but image generation from scratch is prone to hallucinations and noise.The cited interleaved benchmarks reflect these issues.Source: §5 · p. 10
  • Prior multimodal RAG systems support multimodal answers, but their benchmarks are webpage-limited and lack supporting-evidence annotations.MuRAR addresses source attribution, while M2RAG uses a multi-stage image-insertion framework.Source: §5 · p. 10
  • DocVQA benchmarks progress from single-page tasks to long-context documents and retrieval-augmented tasks using retrieved multimodal evidence.Long-context benchmarks input entire documents averaging 50–70 pages; DocRAG approaches retrieve evidence before answer generation.Source: §5 · p. 10
  • Existing DocVQA and DocRAG benchmarks do not focus on multimodal interleaved generation.This is the gap the paper positions its benchmark to address.Source: §5 · p. 11

6 Conclusion

MMDocRAG provides expert-annotated multimodal QA evidence chains and evaluation metrics for quote selection and interleaved answers. The experiments report continuing model challenges, proprietary-model advantages, and gains associated with fine-tuning and high-quality visual descriptions.

6 Conclusion

  • MMDocRAG contains over 4,000 expert-annotated QA pairs with multimodal evidence chains and metrics for quote selection and interleaved answer generation.The paper reports persistent difficulty with multimodal evidence selection and interleaved image-text answers, especially in noisy and diverse documents.Source: §6 · p. 11

Appendix Overview

  • The appendix covers evaluation metrics, supplementary experiments, implementation details, annotated examples, prompts, qualitative studies, and artifact licenses.Supplementary experiments include comparisons by quote representation and analyses by document, question, and evidence type.Source: Appendix Overview · p. 20
  • Additional appendix material includes qualitative analysis of answer generation and documentation of licenses for MMDocRAG and its construction artifacts.Source: Appendix Overview · p. 20

A Evaluation Metric of Multimodal Answer Generation

A.1 Related Work of multimodal generation

  • MMDocRAG extends generation evaluation to RAG by adding F1-based image and text quote-selection metrics plus citation quality, reasoning logic, and factuality criteria.Source: §A.1 · p. 20

A.2 Evaluation Metrics: details and implementations

The section defines evidence-selection, answer-similarity, and judge-based evaluation, then reports extended results across input settings, model modes, document domains, and question types.

A.2 Evaluation Metrics: details and implementations

  • Evaluation combines text and image quote precision, recall, and F1 with BLEU, ROUGE-L, and LLM-judge scores for fluency, citation quality, text-image coherence, reasoning logic, and factuality.BLEU and ROUGE-L capture surface-level similarity but are limited for deep semantic or cross-modal relationships, so task-specific metrics and human-aligned evaluation supplement them.Source: §A.2 · p. 21

B.1 Main results by using 15 quotes for Multimodal Generation

  • With 15 quotes, proprietary VLMs lose more performance than open-source and smaller proprietary models when inputs switch from multimodal to pure text.The 15- and 20-quote comparisons show similar patterns overall.Source: §B.1 · p. 22

B.2 Comprehensive Results and Comparison Between Thinking and Non-Thinking Modes

  • Disabling thinking typically cuts output tokens by 50%–80%, but consistently degrades reasoning-trained Qwen3 models; DeepSeek-R1 also underperforms its non-thinking DeepSeek-V3 counterpart.Qwen3 performance varies by size, while disabling thinking improves QVQ-Max quote selection with almost the same output-token count.Source: §B.2 · p. 22

B.3 Full results by using OCR and LLM text

  • The OCR-text versus VLM-text comparison reports full quote-selection and interleaved-answer results for the two image-text representation approaches.These results extend the OCR-text and VLM-text performance analysis.Source: §B.3 · p. 22

B.4 Fine-grained Results by Document Domains

  • Models perform best on Workshop and Others documents and worst on Brochures, while advanced VLMs score more highly and evenly, especially on Brochure and Academic documents.The passage attributes the domain pattern to simpler Workshop images and complex, non-textual Brochure content; answer quality correlates positively with quote-selection F1.Source: §B.4 · pp. 22–23

B.5 Fine-grained Results by Question Types

  • Descriptive and Comparative questions yield the highest performance, while Interpretative and Inferential questions yield the lowest; advanced VLMs are more balanced on complex questions.Quote-selection F1 and answer quality correlate positively across question types, most strongly for Analytical and Comparative questions.Source: §B.5 · p. 23

B.6 Fine-grained Results by Evidence Types

Evidence-type results compare model performance across image, page, and modality configurations. The appendix also describes model deployment and input handling.

B.6 Fine-grained Results by Evidence Types

  • All models perform better with single-image and single-page evidence than with multiple images or pages; cross-modal preferences vary by model, and GPT-4.1 leads across configurations.Gemini2.5-pro gains in cross-modal settings, while Llama4 shows the most limited adaptability as evidence complexity varies.Source: §B.6 · pp. 23–24

C Implementation Details

  • The appendix implementation section covers VLM/LLM inference, LLM fine-tuning, and retriever implementation, and points to the project code and datasets.The cited passage identifies the implementation topics and repository.Source: C Implementation Details · p. 31

C.1 Implementation Details of Large Models Inference

  • The evaluation includes 60 models—33 VLMs and 27 LLMs—and excludes two Llama vision models that cannot accept multiple images in an input sequence.The set comprises 38 open-source and 22 proprietary models.Source: §C.1 · p. 32
  • VLM multimodal quotes are supplied as interleaved text and images for quote selection and answer generation.The implementation uses the described inference setting and prompt template.Source: §C.1 · p. 32
  • In pure-text settings, images are represented using OCR-derived text or VLM-generated text for both LLMs and VLMs.A common prompt template consolidates the quotes and question.Source: §C.1 · p. 32

C.2 Implementation Details of LLM Finetuning

The appendices document MMDocRAG’s fine-tuning and retrieval implementations, annotation examples, answer-generation and quality-assurance procedures, and evaluation criteria. They also outline qualitative analyses of model errors, input formats, and fine-tuning.

C.2 Implementation Details of LLM Finetuning

  • Fine-tuning uses 4,110 training instances from 2,055 development questions, with one epoch of supervised training; inference converts multimodal quotes to text.The models include five Qwen2.5 LLMs, two Qwen2.5-VL VLMs, and two InternVL-3 VLMs.Source: §C.2 · p. 32

D Annotation Examples

  • Six annotation examples illustrate cross-page reasoning over noisy and gold quotes, including visual-only, multi-table, alignment, and comparative tasks.The examples include numerical and categorical reasoning, as well as extraction of financial values from visually similar tables.Source: D Annotation Examples · p. 34, Figure 12 · p. 34, Figure 13 · p. 35

E.1 Dataset Creation

  • Initial multimodal answers are generated from questions, page screenshots, cropped images, and text snippets, then explicitly cite gold quotes.The procedure uses the templates shown in Figures 18 and 19.Source: §E.1 · p. 40

E.2 Dataset Quality Assurance

  • Automated quality assurance uses VLMs to assess visual-content selection and insertion, and LLMs to check integrated text accuracy and coherence.Source: §E.2 · p. 40

E.3 Inference using Pure-text/Multimodal Quotes

  • Answer generation represents quotes either as interleaved text-image sequences for VLMs or as pure-text sequences for both VLMs and LLMs.Source: §E.3 · p. 40

E.4 LLM Evaluation

  • LLM-as-Judge scores generated answers on fluency, citation quality, text-image coherence, reasoning logic, and factuality.Source: §E.4 · p. 40

F Qualitative Study

  • The qualitative study examines four common errors, compares multimodal with pure-text inputs for VLMs, and assesses fine-tuning effectiveness.Source: F Qualitative Study · p. 46

F.1 Error Analysis: Qualitative Study on 4 Common Errors

A 200-case manual analysis of GPT-4o answers identifies citation errors, visual-reasoning problems, and text-image coherence mismatches, despite generally fluent language.

Citing quality

  • 34.5% of cases showed excessive citations, 30.0% inadequate citations, and 16.5% citation-position errors.Excessive citation included irrelevant or repeated images or text; inadequate citation omitted evidence needed for a complete answer.Source: §F.1 · p. 46

Reasoning and factual consistency

  • Visual misunderstanding and confusion between similar images sometimes led to omitted information, incorrect evidence selection, and inaccurate or incomplete answers.Source: §F.1 · p. 46

Text-image coherence

  • Citation placement often mismatched the gold standard, affecting coherence without significantly changing answer correctness.Image placement was generally satisfactory and reflected flexible interleaving rather than a rigid order.Source: §F.1 · p. 46

Fluency and summary

  • Most answers were linguistically coherent and had few grammatical errors, but visual understanding, multimodal integration, and citation coherence still needed improvement.Source: §F.1 · p. 46

F.2 Multimodal vs Pure-text Quotes: Qualitative Analysis

The qualitative comparison finds that original images can help advanced VLMs interpret visual details and ground citations, while text-only inputs can miss visual information. The relative benefit varies by model size.

Multimodal reasoning

  • GPT-4o’s original-image inputs improved visual interpretation and integration compared with relying solely on text descriptions.Source: §F.2 · p. 48

Citation quality

  • Pure-text inputs increased missed or incorrect image citations, whereas original images supported higher citation precision and stronger evidence grounding.Source: §F.2 · p. 48

Answer quality

  • Text-only inputs sometimes produced hallucinations or factual inaccuracies, although GPT-4o retained comparable logical coherence and some factuality.Source: §F.2 · p. 48

Summary

  • Original images substantially improved advanced VLMs’ citation accuracy, factual grounding, and multimodal reasoning, while smaller VLMs struggled with multiple images and generally handled pure-text inputs more reliably.Source: §F.2 · p. 48

F.3 Finetuning Effectiveness: Qualitative Analysis

A qualitative review reports that fine-tuning strengthened evidence selection and answer quality. The appendix also specifies distinct research-use terms for reused data and new annotations.

F.3 Finetuning Effectiveness: Qualitative Analysis

  • In a 100-case review of Qwen2.5-14B-Instruct, fine-tuning improved selection of relevant visual and textual evidence.The base model often selected incorrect images or irrelevant passages; the fine-tuned model more consistently aligned selected evidence with gold answers.Source: §F.3 · p. 50
  • Fine-tuned answers showed higher factual accuracy, stronger reasoning consistency, more coherent image integration, and greater concision.The passage attributes accuracy and reasoning improvements primarily to improved evidence selection.Source: §F.3 · p. 50

G License Agreement

  • Reused MMDocIR data is distributed under Apache License 2.0, while the new annotations are available for academic and noncommercial research use.Other use of the new annotations requires explicit written permission from the authors.Source: G License Agreement · p. 51

Related papers

Loading 2505.16470v2…