PaperlayerView exact revision on arXiv

Source-linked AI summary

VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation

Manan Suri, Puneet Mathur, Franck Dernoncourt, Kanika Goswami, Ryan A. Rossi, Dinesh Manocha

arXiv:2412.10704v2cs.CL

TL;DR

Multi-document QA over visually rich documents requires locating relevant information across sources and interpreting text alongside tables and charts. VisDoMRAG runs parallel textual and visual RAG pipelines with consistency-constrained fusion, and outperforms long-context and unimodal RAG baselines by 12-20%.

  • Problem

    Multi-document QA over visually rich documents requires locating relevant evidence across sources and interpreting textual and visual content, while existing benchmark coverage for these settings is limited.

    Source: §1 · pp. 1–2

  • Method

    VisDoMRAG runs parallel textual and visual RAG pipelines with evidence curation and chain-of-thought reasoning, then aligns their reasoning through consistency-constrained modality fusion.

    Source: §1 · p. 2, §5 · p. 5

  • Results

    12-20%: VisDoMRAG improves end-to-end QA performance over long-context, textual RAG, and visual RAG baselines on VisDoMBench.

    Source: §1 · pp. 2–3, §8 · p. 9

  • Takeaways & Limitations

    Evidence curation provides answer verifiability through built-in context attribution, especially when critical information is distributed across modalities.

    Source: §1 · p. 2

    VisDoMRAG retains text-extraction and document-parsing overhead and requires three LLM calls per query.

    Source: §10 · p. 9

Abstract

from arXiv · show

Understanding information from a collection of multiple documents, particularly those with visually rich elements, is important for document-grounded question answering. This paper introduces VisDoMBench, the first comprehensive benchmark designed to evaluate QA systems in multi-document settings with rich multimodal content, including tables, charts, and presentation slides. We propose VisDoMRAG, a novel multimodal Retrieval Augmented Generation (RAG) approach that simultaneously utilizes visual and textual RAG, combining robust visual retrieval capabilities with sophisticated linguistic reasoning. VisDoMRAG employs a multi-step reasoning process encompassing evidence curation and chain-of-thought reasoning for concurrent textual and visual RAG pipelines. A key novelty of VisDoMRAG is its consistency-constrained modality fusion mechanism, which aligns the reasoning processes across modalities at inference time to produce a coherent final answer. This leads to enhanced accuracy in scenarios where critical information is distributed across modalities and improved answer verifiability through implicit context attribution. Through extensive experiments involving open-source and proprietary large language models, we benchmark state-of-the-art document QA methods on VisDoMBench. Extensive results show that VisDoMRAG outperforms unimodal and long-context LLM baselines for end-to-end multimodal document QA by 12-20%.

1 Introduction

Multi-document QA over visually rich documents requires locating relevant information across sources and interpreting text alongside tables, charts, and other visuals. The paper introduces VisDoMBench and proposes VisDoMRAG to address this setting.

1 Introduction

  • Multi-document QA requires identifying relevant context across a large, varied collection, a challenge common in finance, science, and policy analysis.Queries may require finding the source document containing the answer.Source: §1 · p. 1
  • Existing multi-document benchmarks are predominantly text-focused, while tables, charts, and slides can contain evidence needed to answer multimodal questions.Such questions may require interpreting table values and graph trends alongside surrounding text.Source: §1 · p. 2
  • VisDoMBench is introduced as a multi-document multimodal QA benchmark covering visually rich content such as tables, charts, and slides.It is designed for comprehensive evaluation of multimodal QA systems.Source: §1 · p. 2
  • VisDoMRAG runs textual and visual RAG in parallel, using evidence curation and chain-of-thought reasoning before consistency-constrained modality fusion.The fusion aligns reasoning chains across modalities and ensembles their answers.Source: §1 · p. 2
  • 12-20%: VisDoMRAG outperforms long-context processing, textual RAG, and visual RAG on VisDoMBench across open- and closed-source LLM settings.The reported comparison is for end-to-end QA performance.Source: §1 · pp. 2–3

2 Related Work

Prior work includes RAG methods that retrieve text and visual information, but benchmark coverage remains divided between text-centric multi-document tasks and visually rich single-document tasks.

Retrieval Augmented Generation

  • MuRAG retrieves text and image data, while RAMM retrieves biomedical images and captions through distinct networks (Chen et al., 2022; Yuan et al., 2023).The cited passage also describes RAG applications across unimodal NLP and visual question answering.Source: §2 · p. 3

Long Context Document Benchmarks

  • L-Eval, Marathon, and LooGLE focus on text-based multi-domain content without multi-document inputs; LongBench and Loong add multi-document settings but remain text-centric (An et al., 2023; Zhang et al., 2023; Li et al., 2023; Bai et al., 2023; Wang et al., 2024c).These comparisons concern long-context document QA benchmarks.Source: §2 · p. 3

Comparison with existing datasets

  • MPDocVQA, UDA, and MMLONGBENCH-DOC include tables, charts, or slides but are single-document benchmarks; VisDoMBench supports multi-document QA across text, tables, charts, and slides (Hui et al., 2024; Ma et al., 2024).The paper characterizes VisDoMBench as a more comprehensive multi-domain evaluation framework.Source: §2 · p. 3

3 Problem Formulation

The task is to answer a query over a collection of multi-page documents by retrieving relevant evidence from one or more documents and generating an answer from that context.

3 Problem Formulation

  • Given a query and a collection of documents, the system aims to generate text that accurately answers the query.Each document may contain multiple pages.Source: §3 · p. 3
  • A query may require information distributed across different pages in one or more associated documents.Source: §3 · p. 3
  • Answer generation relies on retrieving relevant evidence at the level of pages, paragraphs, or text chunks before using the retrieved context.Source: §3 · p. 3

4 VisDoMBench

VisDoMBench combines existing document-QA sources into a multi-document benchmark with visually rich content, curated evidence, and questions tailored to identify their answering documents.

4 VisDoMBench

  • VisDoMBench repurposes five document-QA datasets; Table 2 summarizes its splits by statistics, QA type, and content type.Each data point consists of a question, document set, and ground-truth answer.Source: §4 · p. 3, Table 2 · p. 4

4.1 VisDoMBench

  • Dataset sourcing prioritizes visually rich content, publicly accessible documents, and grounded evidence, drawing on table, chart, and slide QA sources.The source sets include PaperTab, FeTaTab, SciGraphQA, SPIQA, and SlideVQA; the benchmark removes overlapping or duplicate QA pairs.Source: §4.1 · pp. 3–4
  • Table 1 compares long-context document-QA benchmarks with VisDoMBench.The supplied caption identifies the comparison's scope but does not report its individual table values.Source: Table 1 · p. 4
  • Document augmentation adds randomly sampled distractors to target an expected 50-200 pages per query, varying the amount of multi-document evidence.Source: §4.1 · p. 4
  • For ambiguous source questions, GPT-4o generates more specific variants that are human-reviewed and checked against the document collection; ambiguous examples are discarded.Source: §4.1 · pp. 4–5

5 VisDoMRAG

VisDoMRAG runs textual and visual retrieval and reasoning in parallel, then fuses their outputs under a consistency constraint. Each pipeline curates evidence, reasons over it, and generates an answer.

5 VisDoMRAG

  • VisDoMRAG combines parallel vision and text RAG pipelines with modality fusion that constrains consistency between their reasoning chains.The fusion stage uses an LLM to identify discrepancies, reconcile them, and produce a coherent answer.Source: §5 · p. 5, §5.2 · p. 6

5.1 Evidence-driven Parallel Unimodal RAG

  • The textual pipeline extracts and chunks document text, preserves source and page metadata, indexes chunks, and retrieves relevant context for answer generation.OCR supports traceability by retaining each chunk’s source document and page number.Source: §5.1 · p. 5
  • The visual pipeline indexes PDF pages, retrieves relevant pages, and supplies them as visual context to multimodal LLMs.Both pipelines curate relevant evidence, build chain-of-thought reasoning, and generate answers in the requested format.Source: §5.1 · pp. 5–6

6 Experiments

The experiments evaluate retrievers on VisDoMBench before comparing end-to-end QA approaches across LLMs. Retrieval metrics assess evidence extraction and source-document identification; QA is evaluated with word-overlap F1.

6 Experiments

  • The experiments first compare retrieval and indexing models, then evaluate end-to-end QA using selected retrievers and multiple LLMs.The end-to-end comparison includes VisDoMRAG, textual and visual RAG, and long-context approaches.Source: §6 · p. 6, §6.2 · p. 6

6.1 Retrieval

  • Text retrieval uses BM25, MPNet, MiniLM, and BGE-1.5 on OCR-extracted text segmented into overlapping 3,000-character chunks.Visual retrieval uses ColPali and ColQwen2, late-interaction multi-vector models built on PaliGemma and Qwen2.Source: §6.1 · p. 6

6.2 End-to-End QA

  • Retrieval evaluation measures evidence extraction against ground truth and whether the retrieved context identifies the correct source document.End-to-end QA compares approaches using word-overlap F1, with a modified version for PaperTab; Figures 3 and Table 3 present retrieval and QA comparisons.Source: §6.1 · p. 6, Figure 3 · p. 7, Table 3 · p. 7

7 Results

VisDoMRAG outperforms the evaluated RAG and long-context baselines across datasets, while retrieval and modality comparisons reveal dataset-dependent patterns. Its performance also holds at high page counts, and ablations show lower scores without late fusion or structured prompting.

7.1 Retrieval Evaluation on VisDoMBench

  • ColQwen2 outperforms other retrieval baselines across datasets and exceeds BGE1.5 by 4.5% in source-document identification.Visual models substantially outperform text-only models on SlideVQA, while BM25 does better than other text-only models there.Source: §7.1 · p. 7

7.2 End-to-End Evaluation

  • VisDoMRAG gains 2.1-21.6% (PaperTab), 0.67-36.14% (FetaTab), 0.24-11.24% (SciGraphQA), 0.81-32.87% (SPIQA), and 0.40-52.16% (SlideVQA) over baselines.The comparison spans Visual RAG, Textual RAG, and Long Context across Qwen2VL (7B), Gemini Flash, and GPT-4.Source: §7.2 · p. 7
  • Long-context performance drops as page counts rise, while VisDoMRAG remains consistent at high counts; a PaperTab example shows it synthesizing unimodal reasoning and answers.Figure 4 compares the approaches across page-volume ranges, averaged across LLMs.Source: Figure 4 · p. 8, §7.2 · p. 8, Figure 5 · p. 8
  • Visual RAG consistently outperforms Textual RAG, with a smaller gap on text-rich SciGraphQA and SPIQA and a substantial gap on SlideVQA.Gemini often performs better in the textual than visual modality across datasets.Source: §7.2 · p. 8

7.3 Ablations

  • Early fusion averages 43.63 across datasets, while late fusion performs better; prompt ablation lowers scores in text-only and vision-only settings and reduces VisDoMRAG's average from 50.01 to 45.98.The largest prompt-ablation declines occur on SPIQA and SlideVQA, which require complex reasoning.Source: Table 5 · p. 9, §7.3 · p. 9

8 Conclusion and Future Work

The paper introduces VisDoMBench for evaluating multi-document QA on visually rich content and presents VisDoMRAG, which fuses visual and textual pipelines under a consistency constraint.

8 Conclusion and Future Work

  • VisDoMBench is designed to evaluate multi-document QA systems on visually rich elements, including tables, charts, and slides.The benchmark targets documents requiring both textual and visual comprehension.Source: §8 · p. 9
  • VisDoMRAG improves over long-context, textual, and visual RAG baselines by 12-20%; future work will extend the approach to end-to-end trained models, especially in low-resource settings.VisDoMRAG fuses visual and textual pipelines using consistency-constrained modality fusion.Source: §8 · p. 9

9 Ethics Statement

The study uses publicly available datasets, keeps human evaluators’ identities confidential, and uses no personally identifiable information.

9 Ethics Statement

  • The experiments use publicly available datasets, keep evaluators’ identities confidential, and do not use personally identifiable information.The stated intended application is document question answering.Source: §9 · p. 9

10 Limitations

The authors identify text-processing overhead, multiple LLM calls, hallucination risk, and limits on isolating visual performance as limitations.

10 Limitations

  • The approach retains text extraction and document-parsing overhead, makes three LLM calls per query, and remains subject to hallucination risks.The authors say it is more cost-effective than long-context models, despite requiring multiple calls.Source: §10 · p. 9
  • Because the framework incorporates document context, relevant text from other sections can contribute to answers, making visual-only performance difficult to isolate.The authors note this reflects real-world multimodal interdependence.Source: §10 · p. 10

A.1 Baselines

This appendix describes the retrieval models, datasets, evaluation setup, illustrative QA cases, prompts, query review, and experiment resources used in the study.

A.1.1 Retrieval Models

  • The retrieval baselines include BM25, MiniLM, MPNet, and BGE-1.5 for text, plus ColPali and ColQwen2 for page-image retrieval.ColPali indexes page images with projected token embeddings and does not require OCR or document parsing; ColQwen2 uses Qwen2 as its base VLM.Source: §A.1.1 · pp. 12–13

A.1.2 LLMs

  • Experiments use Qwen2-VL-7B-Instruct, ChatGPT-4o, and Gemini-1.5-Flash; long-context evaluation uses Qwen2-7B-Instruct with YaRN.The reported results are from a single experimental run.Source: §A.1.2 · p. 13

A.2 Datasets

  • FetaTab and PaperTab are built from complete source-document question–answer triplets, with UDA matching source documents and standardizing and filtering examples.FetaTab derives from FetaQA, while PaperTab derives from QASPER.Source: §A.2 · p. 13
  • The benchmark includes FetaTab; SPIQA covers scientific-paper figures, tables, and text; SciGraphQA focuses on academic graphs; and SlideVQA contains over 2,600 slide decks, 52,000 slide images, and 14,500 questions.SlideVQA requires single-hop, multi-hop, and numerical reasoning. Figures 6–10 show pages-per-query distributions in all data splits.Source: §A.2 · pp. 13–14, Figure 6 · p. 13

A.2.1 Distracting Documents

  • The distracting documents are randomly selected, contextually similar in-domain documents that are irrelevant to the query.The oracle-removal experiment tests whether the system can refuse when the answer document is unavailable.Source: §A.2.1 · pp. 14–15
  • When the oracle document was removed, refusal rates rose to 94–98% across datasets; with it included, PaperTab and FetaTab had 26% and 4% refusal rates.The authors interpret the increase as evidence that the correct document is important for answering queries amid distracting documents.Source: Table 6 · p. 15, §A.2.1 · p. 15

A.3.1 Query Augmentation

A.3.2 End-to-End QA Examples

  • In examples, VisDoMRAG combines complementary retrieval outputs on PaperTab and FetaTab, corroborates correct outputs on SciGraphQA, and favors visual multi-hop evidence on SlideVQA.On SPIQA, it fails when it favors the longer incorrect VisualRAG response.Source: §A.3.2 · p. 15

Related papers

Loading 2412.10704v2…