PaperlayerView exact revision on arXiv

Source-linked AI summary

VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documents

Ryota Tanaka, Taichi Iki, Taku Hasegawa, Kyosuke Nishida, Kuniko Saito, Jun Suzuki

arXiv:2504.09795v1cs.CLcs.AIcs.CVcs.IR

TL;DR

Text-centric RAG is insufficient for questions over visually-rich documents with mixed modalities and formats. VDocRAG retrieves and generates from unified document images, adapts LVLMs with retrieval-oriented pre-training, and introduces OpenDocVQA for open-domain evaluation. Across four datasets, it significantly outperformed conventional text-based RAG, while the authors note unresolved index-cost and joint-training limitations.

  • Problem

    Most RAG frameworks assume text-only context, while open-domain DocumentVQA requires retrieving and answering from visually-rich documents across diverse formats.

    Source: §1 · p. 1

  • Method

    VDocRAG represents documents as images, retrieves relevant document images, generates answers from them, and pre-trains LVLMs to compress image representations using OCR targets.

    Source: §1 · p. 1, §4.2 · p. 4

  • Results

    Across four datasets, VDocRAG significantly outperformed conventional text-based RAG and showed strong generalization performance.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    OpenDocVQA provides a unified open-domain resource for training and evaluating retrieval and question answering on visually-rich documents.

    Source: §1 · p. 2

    The study does not reduce the computational cost of creating search indexes for extensive image collections and does not jointly train QA and retrieval components.

    Source: §6 · p. 8

Abstract

from arXiv · show

We aim to develop a retrieval-augmented generation (RAG) framework that answers questions over a corpus of visually-rich documents presented in mixed modalities (e.g., charts, tables) and diverse formats (e.g., PDF, PPTX). In this paper, we introduce a new RAG framework, VDocRAG, which can directly understand varied documents and modalities in a unified image format to prevent missing information that occurs by parsing documents to obtain text. To improve the performance, we propose novel self-supervised pre-training tasks that adapt large vision-language models for retrieval by compressing visual information into dense token representations while aligning them with textual content in documents. Furthermore, we introduce OpenDocVQA, the first unified collection of open-domain document visual question answering datasets, encompassing diverse document types and formats. OpenDocVQA provides a comprehensive resource for training and evaluating retrieval and question answering models on visually-rich documents in an open-domain setting. Experiments show that VDocRAG substantially outperforms conventional text-based RAG and has strong generalization capability, highlighting the potential of an effective RAG paradigm for real-world documents.

1. Introduction

VDocRAG addresses the mismatch between text-only RAG and visually rich documents by retrieving and generating directly from document images. It combines visual document understanding, retrieval-oriented LVLM adaptation, and the OpenDocVQA benchmark.

  • Visually-rich documents combine textual and visual objects whose content is distributed across diverse formats and locations.Source: §1 · p. 1
  • Existing DocumentVQA datasets mostly use closed settings, while recent document-image retrieval methods do not fully integrate retrieved information into final answers.Source: §1 · p. 1
  • VDocRAG unifies varied documents and modalities as images to avoid parsing-related information loss, using VDocRetriever and VDocGenerator.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • The framework adapts pretrained LVLMs for document-image encoding and interaction with retrieved visual information.Source: §1 · p. 2
  • OpenDocVQA is presented as the first unified open-domain DocumentVQA collection spanning diverse document types and formats.Source: §1 · p. 2
  • Experiments show that VDocRAG substantially outperforms conventional text-based RAG and generalizes strongly.Source: §1 · p. 2

2. Related Work

Prior work applies RAG and LVLMs across modalities, but visual document retrieval and visual RAG remain limited by document coverage, question settings, and retrieval-specific training. OpenDocVQA targets these gaps with broader, retrieval-requiring evaluation.

Retrieval-augmented generation

  • Most RAG research retrieves from plain-text documents or non-text media, leaving knowledge extraction from visually-rich documents less addressed.Source: §2 · p. 2

Visual document retrieval and visual RAG

  • Existing visual document retrieval and visual RAG methods use LVLMs to encode documents as images but face dataset and training-strategy limitations.Source: §2 · p. 2
  • ViDoRe includes questions that may not require retrieval and covers limited document types, whereas OpenDocVQA uses human-verified retrieval-requiring questions across open document types.Source: §2 · p. 2
  • Previous LVLM-based approaches lack specific training to bridge the gap between generative pre-training and retrieval tasks.Source: §2 · p. 2

Document visual question answering

  • Closed-setting DocumentVQA studies generally avoid retrieval, while OpenDocVQA targets broader formats, multi-hop reasoning, and visual semantic search.Source: §2 · p. 2

3. OpenDocVQA Task and Dataset

OpenDocVQA formulates open-domain DocumentVQA as retrieving relevant document images before answer generation, and assembles diverse, retrieval-requiring datasets with single- and multi-hop challenges. Its construction includes filtering, screenshot reformulation, multi-hop creation, and hard negative mining.

3.1. Task Formulation

  • OpenDocVQA retrieves k relevant images from N document images for a question, with k ≪ N, then generates an answer from the retrieved images.Source: §3.1 · pp. 2–3
  • The benchmark evaluates both single-pool retrieval within each source dataset and all-pool retrieval across documents from diverse domains.Source: §3.1 · p. 3

3.2. Dataset Collection

  • Heuristic filtering removed 20.9% of the initial question pool, followed by manual verification of context-independent questions requiring retrieval.Source: §3.2 · p. 3
  • Open-WikiTable QA pairs were reformulated by replacing textual HTML tables with screenshot images from the corresponding Wikipedia pages.Source: §3.2 · p. 3
  • MHDocVQA was created by linking bridge entities across single-hop questions, combining them with Mixtral-8x22B, and filtering candidates with GPT-4o.Source: §3.2 · p. 3
  • Negative image candidates were mined from COYO-700M using OCR lexical overlap while excluding images containing the correct answer.Source: §3.2 · p. 3

3.3. Comparison with Related Datasets

  • Compared with related datasets, OpenDocVQA spans open document types, requires visual semantic search, and supports multi-hop extractive and abstractive answers.Source: §3.3 · p. 3

4. Proposed Model

VDocRAG uses LVLMs to represent varied documents as images, retrieve relevant document images, and generate answers from the retrieved results. Its retriever is trained with self-supervised representation-compression tasks that align visual document information with OCR text.

Architecture Overview

  • VDocRAG unifies varied document formats and modalities as images, using VDocRetriever and VDocGenerator for retrieval-augmented question answering.The retriever selects document images, while the generator uses the question and retrieved images to produce answers.Source: §4.1 · p. 4
  • Dynamic cropping splits high-resolution documents into 336×336 patches while preserving the original aspect ratio before visual feature projection.Each patch is encoded separately, and a two-layer MLP projector converts the outputs into visual document features.Source: §4.1 · p. 4
  • VDocRetriever independently encodes questions and document images, then retrieves the k documents with the highest question-document similarity scores.Question and document embeddings are obtained from the final <EOS> vector, and similarity is computed using maximum inner product search.Source: §4.1 · p. 4

Self-Supervised Pre-training Tasks

  • Two self-supervised tasks compress an entire document image representation into its terminal <EOS> token using OCR text as a pseudo-target.The combined objective is L = L_RCR + L_RCG.Source: §4.2 · pp. 4–5
  • RCR learns image retrieval from corresponding OCR text with contrastive learning and in-batch negatives.Positive OCR text-image pairs are contrasted against other images in the batch using InfoNCE; τ scales logits and B denotes batch size.Source: §4.2 · p. 4
  • RCG predicts OCR tokens while allowing subsequent OCR representations to attend only to <EOS> and preceding OCR tokens.This attention design pools image information into <EOS> for OCR-token generation.Source: §4.2 · pp. 4–5

5. Experiments

Experiments evaluate VDocRAG’s retrieval and question-answering performance across supervised and zero-shot settings, ablations, document lengths, and efficiency. VDocRAG consistently improves over text-based RAG, with visual encoding, pre-training, and OpenDocVQA contributing to the gains.

Retrieval Results

  • VDocRetriever achieved significantly higher retrieval performance than the text-based Phi3 retriever across all evaluated datasets under the same conditions.It also showed superior zero-shot generalization on unseen ChartQA and SlideVQA datasets.Source: §5.2 · p. 6

Retrieval-Augmented Generation Results

  • VDocRAG significantly outperformed closed-book and text-based RAG models on DocumentVQA, including when all models shared the same initialization.With ground-truth documents, VDocRAG also surpassed text-based RAG, while both methods improved substantially.Source: §5.3 · pp. 6–7

Analysis

  • Removing either or both RCG and RCR pre-training tasks decreased retrieval performance, indicating complementary contributions from both tasks.The authors associate these tasks with compressing image features and aligning them with textual contents.Source: §5.4 · p. 7
  • VDocRAG consistently outperformed text-based RAG across document lengths, with larger relative gains for images containing 0-10 words than 500+ words.The relative gains were +66.0 in retrieval and +21.1 in QA for 0-10 words, versus +28.4 in retrieval and +16.7 in QA for 500+ words.Source: §5.4 · p. 7
  • VDocRAG required 69% less inference time to retrieve documents than text-based RAG.Although document encoding and generation took longer, VDocRAG eliminated the OCR processing required by text-based RAG.Source: §5.4 · p. 7
  • Qualitative analysis found stronger handling of charts, multi-hop reasoning, graph understanding, and tables with cells spanning multiple rows and columns.The analysis also identified challenges on text-heavy documents, primarily associated with OCR capabilities.Source: §5.4 · p. 8

6. Conclusion

The paper concludes that VDocRAG directly understands diverse real-world documents through visual features and combines this framework with visual-information pre-training and OpenDocVQA. Across four datasets, VDocRAG significantly outperformed conventional text-based RAG, while the authors identify remaining efficiency and training limitations.

6. Conclusion

  • VDocRAG directly understands various real-world documents through visual features.The framework is enhanced by pre-training tasks for efficient image representation and the unified OpenDocVQA dataset.Source: §6 · p. 8
  • Holistic evaluations on four datasets showed that VDocRAG significantly outperformed conventional text-based RAG.The authors present this result as evidence supporting effective RAG over real-world documents.Source: §6 · p. 8

Limitations

  • The study did not address reducing the computational cost of creating search indexes for extensive image collections.The authors also identify caption-based retrieval for text-free images and joint QA-retrieval training as future directions.Source: §6 · p. 8

Supplementary Material

OpenDocVQA contains 43,474 questions from nine source datasets, with both single-hop and multi-hop questions and both extractive and abstractive answers. Its reported statistics also include question and answer length bounds and averages.

Dataset Statistics

  • OpenDocVQA contains 43,474 questions, including 33,244 single-hop questions and 10,230 multi-hop questions.Single-hop questions comprise 76.5% and multi-hop questions 23.5% of the collection.Source: Supplementary Material · p. 12
  • The collection includes 19,797 extractive answers and 23,677 abstractive answers.These represent 45.5% and 54.5% of the answers, respectively.Source: Supplementary Material · p. 12
  • OpenDocVQA draws from nine source datasets: seven existing DocumentVQA datasets, one existing TableQA dataset, and one newly created dataset.Source: Supplementary Material · p. 12
  • The maximum question and answer lengths are 58 and 130, while average lengths are 13.7 and 3.7.The reported averages are given for questions and answers, respectively.Source: Supplementary Material · p. 12

A. OpenDocVQA Details

OpenDocVQA unifies diverse open-domain DocumentVQA resources, including multiple document formats, modalities, question types, and answer types. Its construction includes multi-hop question generation, filtering, and dataset-specific retrieval formulations.

Dataset composition

  • OpenDocVQA contains nine open-domain DocumentVQA datasets, including the newly created MHDocVQA dataset for multi-hop questions over multiple documents.The collection includes datasets covering industry documents, infographics, webpage screenshots, charts, tables, PDFs, and slides.Source: A. OpenDocVQA Details · p. 12
  • 45.5% of questions are single-hop and 23.5% are multi-hop, while 45.5% of answers are extractive and 54.5% are abstractive.Source: A. OpenDocVQA Details · p. 12
  • The collection spans document images with handwritten or digital text, complex layouts, and graphical elements such as plots and charts.Source: A. OpenDocVQA Details · p. 12

Dataset analysis

  • Question-answer word clouds and question-prefix distributions indicate broad topical and lexical coverage in OpenDocVQA.Source: A. OpenDocVQA Details · p. 12

Multi-hop construction

Retrieval formulation

  • The constituent datasets frame retrieval around finding relevant industry documents, infographics, screenshots, charts, tables, PDF pages, or other document evidence.Source: A. OpenDocVQA Details · p. 14

B. Experimental Details

The experiments use instruction templates for visual document retrieval and brief-answer prompting for DocumentVQA. Training and evaluation involve both LLM-based and non-LLM-based retrievers, with listed checkpoints and hyperparameters.

Instructions

  • Instruction templates prepend a task description and query to questions for the visual document retrieval task.Source: B. Experimental Details · p. 14
  • Applying instructions during both training and evaluation improved LLM-based retrievers, whereas applying them to Contriever reduced performance.Source: B. Experimental Details · p. 14

Evaluation

  • The experiments report retrieval metrics including nDCG@5 and encoding time, alongside QA metrics including ANLS and generation time.Source: B. Experimental Details · p. 14

Instructions

Configuration

C. Additional Experimental Analysis

Additional analyses examine image resolution, the number of retrieved documents, and qualitative differences between VDocRAG and text-based RAG. Higher resolution improves visual understanding but increases inference time, while three retrieved documents perform best.

Image resolution

Retrieved-document count

Qualitative comparison

Related papers

Loading 2504.09795v1…