VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documents

Ryota Tanaka; Taichi Iki; Taku Hasegawa; Kyosuke Nishida; Kuniko Saito; Jun Suzuki

Source-linked AI summary

VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documents

Ryota Tanaka, Taichi Iki, Taku Hasegawa, Kyosuke Nishida, Kuniko Saito, Jun Suzuki

arXiv:2504.09795v1Apr 14, 2025cs.CLcs.AIcs.CVcs.IR

TL;DR

Text-centric RAG is insufficient for questions over visually-rich documents with mixed modalities and formats. VDocRAG retrieves and generates from unified document images, adapts LVLMs with retrieval-oriented pre-training, and introduces OpenDocVQA for open-domain evaluation. Across four datasets, it significantly outperformed conventional text-based RAG, while the authors note unresolved index-cost and joint-training limitations.

Problem
Most RAG frameworks assume text-only context, while open-domain DocumentVQA requires retrieving and answering from visually-rich documents across diverse formats.
Method
VDocRAG represents documents as images, retrieves relevant document images, generates answers from them, and pre-trains LVLMs to compress image representations using OCR targets.
Results
Across four datasets, VDocRAG significantly outperformed conventional text-based RAG and showed strong generalization performance.
Takeaways & Limitations
OpenDocVQA provides a unified open-domain resource for training and evaluating retrieval and question answering on visually-rich documents.
Takeaways & Limitations
The study does not reduce the computational cost of creating search indexes for extensive image collections and does not jointly train QA and retrieval components.

Abstract

from arXiv · show

We aim to develop a retrieval-augmented generation (RAG) framework that answers questions over a corpus of visually-rich documents presented in mixed modalities (e.g., charts, tables) and diverse formats (e.g., PDF, PPTX). In this paper, we introduce a new RAG framework, VDocRAG, which can directly understand varied documents and modalities in a unified image format to prevent missing information that occurs by parsing documents to obtain text. To improve the performance, we propose novel self-supervised pre-training tasks that adapt large vision-language models for retrieval by compressing visual information into dense token representations while aligning them with textual content in documents. Furthermore, we introduce OpenDocVQA, the first unified collection of open-domain document visual question answering datasets, encompassing diverse document types and formats. OpenDocVQA provides a comprehensive resource for training and evaluating retrieval and question answering models on visually-rich documents in an open-domain setting. Experiments show that VDocRAG substantially outperforms conventional text-based RAG and has strong generalization capability, highlighting the potential of an effective RAG paradigm for real-world documents.

1. Introduction

VDocRAG addresses the mismatch between text-only RAG and visually rich documents by retrieving and generating directly from document images. It combines visual document understanding, retrieval-oriented LVLM adaptation, and the OpenDocVQA benchmark.

Visually-rich documents combine textual and visual objects whose content is distributed across diverse formats and locations.
Existing DocumentVQA datasets mostly use closed settings, while recent document-image retrieval methods do not fully integrate retrieved information into final answers.
VDocRAG unifies varied documents and modalities as images to avoid parsing-related information loss, using VDocRetriever and VDocGenerator.
The framework adapts pretrained LVLMs for document-image encoding and interaction with retrieved visual information.
OpenDocVQA is presented as the first unified open-domain DocumentVQA collection spanning diverse document types and formats.
Experiments show that VDocRAG substantially outperforms conventional text-based RAG and generalizes strongly.

2. Related Work

Prior work applies RAG and LVLMs across modalities, but visual document retrieval and visual RAG remain limited by document coverage, question settings, and retrieval-specific training. OpenDocVQA targets these gaps with broader, retrieval-requiring evaluation.

Retrieval-augmented generation: Most RAG research retrieves from plain-text documents or non-text media, leaving knowledge extraction from visually-rich documents less addressed.
Visual document retrieval and visual RAG: Existing visual document retrieval and visual RAG methods use LVLMs to encode documents as images but face dataset and training-strategy limitations.
Visual document retrieval and visual RAG: ViDoRe includes questions that may not require retrieval and covers limited document types, whereas OpenDocVQA uses human-verified retrieval-requiring questions across open document types.
Visual document retrieval and visual RAG: Previous LVLM-based approaches lack specific training to bridge the gap between generative pre-training and retrieval tasks.
Document visual question answering: Closed-setting DocumentVQA studies generally avoid retrieval, while OpenDocVQA targets broader formats, multi-hop reasoning, and visual semantic search.

3. OpenDocVQA Task and Dataset

OpenDocVQA formulates open-domain DocumentVQA as retrieving relevant document images before answer generation, and assembles diverse, retrieval-requiring datasets with single- and multi-hop challenges. Its construction includes filtering, screenshot reformulation, multi-hop creation, and hard negative mining.

3.1. Task Formulation: OpenDocVQA retrieves k relevant images from N document images for a question, with k ≪ N, then generates an answer from the retrieved images.
3.1. Task Formulation: The benchmark evaluates both single-pool retrieval within each source dataset and all-pool retrieval across documents from diverse domains.
3.2. Dataset Collection: Heuristic filtering removed 20.9% of the initial question pool, followed by manual verification of context-independent questions requiring retrieval.
3.2. Dataset Collection: Open-WikiTable QA pairs were reformulated by replacing textual HTML tables with screenshot images from the corresponding Wikipedia pages.
3.2. Dataset Collection: MHDocVQA was created by linking bridge entities across single-hop questions, combining them with Mixtral-8x22B, and filtering candidates with GPT-4o.
3.2. Dataset Collection: Negative image candidates were mined from COYO-700M using OCR lexical overlap while excluding images containing the correct answer.
3.3. Comparison with Related Datasets: Compared with related datasets, OpenDocVQA spans open document types, requires visual semantic search, and supports multi-hop extractive and abstractive answers.

4. Proposed Model

VDocRAG uses LVLMs to represent varied documents as images, retrieve relevant document images, and generate answers from the retrieved results. Its retriever is trained with self-supervised representation-compression tasks that align visual document information with OCR text.

Architecture Overview: VDocRAG unifies varied document formats and modalities as images, using VDocRetriever and VDocGenerator for retrieval-augmented question answering.The retriever selects document images, while the generator uses the question and retrieved images to produce answers.
Architecture Overview: Dynamic cropping splits high-resolution documents into 336×336 patches while preserving the original aspect ratio before visual feature projection.Each patch is encoded separately, and a two-layer MLP projector converts the outputs into visual document features.
Architecture Overview: VDocRetriever independently encodes questions and document images, then retrieves the k documents with the highest question-document similarity scores.Question and document embeddings are obtained from the final <EOS> vector, and similarity is computed using maximum inner product search.
Self-Supervised Pre-training Tasks: Two self-supervised tasks compress an entire document image representation into its terminal <EOS> token using OCR text as a pseudo-target.The combined objective is L = L_RCR + L_RCG.
Self-Supervised Pre-training Tasks: RCR learns image retrieval from corresponding OCR text with contrastive learning and in-batch negatives.Positive OCR text-image pairs are contrasted against other images in the batch using InfoNCE; τ scales logits and B denotes batch size.
Self-Supervised Pre-training Tasks: RCG predicts OCR tokens while allowing subsequent OCR representations to attend only to <EOS> and preceding OCR tokens.This attention design pools image information into <EOS> for OCR-token generation.

5. Experiments

Experiments evaluate VDocRAG’s retrieval and question-answering performance across supervised and zero-shot settings, ablations, document lengths, and efficiency. VDocRAG consistently improves over text-based RAG, with visual encoding, pre-training, and OpenDocVQA contributing to the gains.

Retrieval Results: VDocRetriever achieved significantly higher retrieval performance than the text-based Phi3 retriever across all evaluated datasets under the same conditions.It also showed superior zero-shot generalization on unseen ChartQA and SlideVQA datasets.
Retrieval-Augmented Generation Results: VDocRAG significantly outperformed closed-book and text-based RAG models on DocumentVQA, including when all models shared the same initialization.With ground-truth documents, VDocRAG also surpassed text-based RAG, while both methods improved substantially.
Analysis: Removing either or both RCG and RCR pre-training tasks decreased retrieval performance, indicating complementary contributions from both tasks.The authors associate these tasks with compressing image features and aligning them with textual contents.
Analysis: VDocRAG consistently outperformed text-based RAG across document lengths, with larger relative gains for images containing 0-10 words than 500+ words.The relative gains were +66.0 in retrieval and +21.1 in QA for 0-10 words, versus +28.4 in retrieval and +16.7 in QA for 500+ words.
Analysis: VDocRAG required 69% less inference time to retrieve documents than text-based RAG.Although document encoding and generation took longer, VDocRAG eliminated the OCR processing required by text-based RAG.
Analysis: Qualitative analysis found stronger handling of charts, multi-hop reasoning, graph understanding, and tables with cells spanning multiple rows and columns.The analysis also identified challenges on text-heavy documents, primarily associated with OCR capabilities.

6. Conclusion

The paper concludes that VDocRAG directly understands diverse real-world documents through visual features and combines this framework with visual-information pre-training and OpenDocVQA. Across four datasets, VDocRAG significantly outperformed conventional text-based RAG, while the authors identify remaining efficiency and training limitations.

6. Conclusion: VDocRAG directly understands various real-world documents through visual features.The framework is enhanced by pre-training tasks for efficient image representation and the unified OpenDocVQA dataset.
6. Conclusion: Holistic evaluations on four datasets showed that VDocRAG significantly outperformed conventional text-based RAG.The authors present this result as evidence supporting effective RAG over real-world documents.
Limitations: The study did not address reducing the computational cost of creating search indexes for extensive image collections.The authors also identify caption-based retrieval for text-free images and joint QA-retrieval training as future directions.

Supplementary Material

OpenDocVQA contains 43,474 questions from nine source datasets, with both single-hop and multi-hop questions and both extractive and abstractive answers. Its reported statistics also include question and answer length bounds and averages.

Dataset Statistics: OpenDocVQA contains 43,474 questions, including 33,244 single-hop questions and 10,230 multi-hop questions.Single-hop questions comprise 76.5% and multi-hop questions 23.5% of the collection.
Dataset Statistics: The collection includes 19,797 extractive answers and 23,677 abstractive answers.These represent 45.5% and 54.5% of the answers, respectively.
Dataset Statistics: OpenDocVQA draws from nine source datasets: seven existing DocumentVQA datasets, one existing TableQA dataset, and one newly created dataset.
Dataset Statistics: The maximum question and answer lengths are 58 and 130, while average lengths are 13.7 and 3.7.The reported averages are given for questions and answers, respectively.

A. OpenDocVQA Details

OpenDocVQA unifies diverse open-domain DocumentVQA resources, including multiple document formats, modalities, question types, and answer types. Its construction includes multi-hop question generation, filtering, and dataset-specific retrieval formulations.

Dataset composition: OpenDocVQA contains nine open-domain DocumentVQA datasets, including the newly created MHDocVQA dataset for multi-hop questions over multiple documents.The collection includes datasets covering industry documents, infographics, webpage screenshots, charts, tables, PDFs, and slides.
Dataset composition: 45.5% of questions are single-hop and 23.5% are multi-hop, while 45.5% of answers are extractive and 54.5% are abstractive.
Dataset composition: The collection spans document images with handwritten or digital text, complex layouts, and graphical elements such as plots and charts.
Dataset analysis: Question-answer word clouds and question-prefix distributions indicate broad topical and lexical coverage in OpenDocVQA.
Multi-hop construction: Multi-hop questions are generated by prompting a language model to combine two single-hop questions while preserving the second answer, then filtered with a dedicated prompt.
Retrieval formulation: The constituent datasets frame retrieval around finding relevant industry documents, infographics, screenshots, charts, tables, PDF pages, or other document evidence.

B. Experimental Details

The experiments use instruction templates for visual document retrieval and brief-answer prompting for DocumentVQA. Training and evaluation involve both LLM-based and non-LLM-based retrievers, with listed checkpoints and hyperparameters.

Instructions: Instruction templates prepend a task description and query to questions for the visual document retrieval task.
Instructions: Applying instructions during both training and evaluation improved LLM-based retrievers, whereas applying them to Contriever reduced performance.
Evaluation: The experiments report retrieval metrics including nDCG@5 and encoding time, alongside QA metrics including ANLS and generation time.
Instructions: DocumentVQA inputs receive an additional instruction requesting brief answers.
Configuration: Model initialization checkpoints and pre-training or fine-tuning hyperparameters are documented in Tables E and F.

C. Additional Experimental Analysis

Additional analyses examine image resolution, the number of retrieved documents, and qualitative differences between VDocRAG and text-based RAG. Higher resolution improves visual understanding but increases inference time, while three retrieved documents perform best.

Image resolution: Higher image resolution improves document understanding and encoding but significantly increases retrieval and QA inference time.
Image resolution: QA performance is more sensitive to image resolution than retrieval performance, indicating greater visual-detail demands for QA.
Retrieved-document count: Three retrieved documents produce the best VDocRAG results under the analyzed setting.
Retrieved-document count: Adding more low-ranked or randomly sampled documents introduces noise and deteriorates generation because of retriever imperfections.
Qualitative comparison: Compared with text-based RAG, VDocRAG better handles layouts and visual content including tables, charts, figures, and diagrams.