PaperlayerView exact revision on arXiv

Source-linked AI summary

VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents

Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, Maosong Sun

arXiv:2410.10594v2cs.IRcs.AIcs.CLcs.CV

TL;DR

Text-based RAG requires parsing multimodal documents into text, leaving open how to use their visual information directly. VisRAG uses VLMs to retrieve document images and generate answers, and outperforms text-based RAG in retrieval and generation.

  • Problem

    Text-based RAG requires parsing complex multimodal documents into text, leaving open whether VLMs can use their visual information directly for retrieval and generation.

    Source: §1 · p. 2, §3.1 · p. 4

  • Method

    VisRAG uses VLMs to retrieve document pages directly as images and generate answers from the retrieved images, bypassing text parsing.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Across six evaluation datasets, VisRAG consistently outperforms text-based RAG in retrieval and generation; on InfographicsVQA, accuracy rises from 25% to 51%.

    Source: §1 · p. 2, §5.1 · p. 9, §6 · p. 10

  • Takeaways & Limitations

    The findings support VLM-based RAG as a promising approach for handling multimodal documents.

    Source: §6 · p. 10

Abstract

from arXiv · show

Retrieval-augmented generation (RAG) is an effective technique that enables large language models (LLMs) to utilize external knowledge sources for generation. However, current RAG systems are solely based on text, rendering it impossible to utilize vision information like layout and images that play crucial roles in real-world multi-modality documents. In this paper, we introduce VisRAG, which tackles this issue by establishing a vision-language model (VLM)-based RAG pipeline. In this pipeline, instead of first parsing the document to obtain text, the document is directly embedded using a VLM as an image and then retrieved to enhance the generation of a VLM. Compared to traditional text-based RAG, VisRAG maximizes the retention and utilization of the data information in the original documents, eliminating the information loss introduced during the parsing process. We collect both open-source and synthetic data to train the retriever in VisRAG and explore a variety of generation methods. Experiments demonstrate that VisRAG outperforms traditional RAG in both the retrieval and generation stages, achieving a 20--40% end-to-end performance gain over traditional text-based RAG pipeline. Further analysis reveals that VisRAG is efficient in utilizing training data and demonstrates strong generalization capability, positioning it as a promising solution for RAG on multi-modality documents. Our code and data are available at https://github.com/openbmb/visrag.

1 INTRODUCTION

VisRAG uses VLMs to retrieve document images and generate answers without parsing documents into text, preserving their original visual information. The paper reports end-to-end relative gains over TextRAG and stronger retrieval and generation results in its evaluations.

1 INTRODUCTION

  • RAG supplements LLMs with retrieved external knowledge to address hallucination and the difficulty of updating parametric knowledge.The introduction describes open-source frameworks such as llamaindex as facilitating RAG research and deployment.Source: §1 · p. 1
  • TextRAG parses interleaved text and figures into text units, risking errors and overlooking image information in real-world multi-modality documents.Related image-retrieval approaches often assume images and descriptive text have already been extracted and paired, unlike interleaved document pages.Source: §1 · pp. 1–2
  • VisRAG-Ret embeds queries and document images, and VisRAG-Gen answers from retrieved images while preserving their original visual format.For single-image VLMs, page concatenation and weighted selection support handling multiple retrieved documents.Source: §1 · p. 2
  • VisRAG achieves 40% relative improvement over TextRAG with MiniCPM-V 2.6 and 20% with GPT-4o as generator.The paper attributes these end-to-end gains to a cascade effect.Source: §1 · p. 2
  • VisRAG-Ret outperforms state-of-the-art text- and vision-centric retrievers, while VisRAG-Gen surpasses traditional text-based generators with open-source VLMs.With multi-image VLMs, performance gains increase as more documents are retrieved; the paper also reports better training-data efficiency, generalization, and robustness across text- and vision-centric documents.Source: §1 · p. 2

2 RELATED WORK

Prior work leaves a gap in applying RAG to real-world mixed-modality documents, while advances in VLMs and direct document-image retrieval provide relevant foundations.

Retrieval-augmented Generation (RAG)

Vision-language Models

Multi-modality Retrieval and RAG

  • Prior multimodal retrieval and RAG often rely on separately extracted and paired text and images, limiting their fit to mixed-modality documents and leaving end-to-end RAG evaluation incomplete.DSE (Ma et al., 2024) and ColPali (Faysse et al., 2024) encode document images directly for retrieval, but do not compare with text-based retrieval across in-domain and out-of-domain settings or evaluate end-to-end RAG.Source: §2 · p. 3

3 METHODOLOGY

VisRAG processes document pages as images for retrieval and generation, preserving information that text parsing can lose. Its methodology combines VLM-based retrieval and generation with curated and synthetic training and evaluation data.

3.1 PRELIMINARY: RETRIEVAL-AUGMENTED GENERATION

  • Typical RAG pre-encodes corpus units, retrieves candidates with approximate nearest-neighbor search, then prompts an LLM with the query and retrieved units to answer.Source: §3.1 · p. 4

3.2 VISRAG: VISION-BASED RETRIEVAL-AUGMENTED GENERATION

  • VisRAG uses document pages as the retrieval and generation units, processing each directly as an image with VLMs rather than parsing it into text segments.This page-based approach is intended to preserve information in the original document.Source: §3.1 · p. 4, §3.2 · p. 4

3.2.1 RETRIEVAL

  • VisRAG-Ret separately encodes queries and page images with a VLM, pools hidden states with later-token weighting, scores cosine similarity, and trains with InfoNCE.The dual-encoder paradigm follows Karpukhin et al. (2020), and the pooling approach follows Muennighoff (2022).Source: §3.2.1 · pp. 4–5

3.2.2 GENERATION

  • VisRAG-Gen supports page concatenation, weighted selection among per-page answers, and multi-image VLM input for reasoning across retrieved pages.Weighted selection combines answer-generation probability, computed as reciprocal perplexity, with normalized retrieval score; multi-image input can support single- or multi-hop questions.Source: §3.2.2 · p. 5

3.3 DATA CONSTRUCTION

  • Training data combine question-document pairs from VQA datasets with GPT-4o-generated queries from openly available PDFs; context-dependent queries are filtered, and performance is reported on source-dataset evaluation sets.Retrieval uses MRR@10 and Recall@10; generation uses relaxed exact match with a 5% error margin for numeric responses.Source: §3.3 · pp. 5–6

4 EXPERIMENTAL METHODOLOGY

The experiments evaluate VisRAG retrieval and generation against text-based baselines, using two document-text extraction methods and controlled retrieval and generation settings.

Document Parsing

  • Two extraction pipelines support TextRAG comparisons: PPOCR detects and merges text regions, while fine-tuned MiniCPM-V 2.0 extracts text directly from document images.The paper labels these methods “OCR” and “Captioner,” respectively.Source: §4 · p. 6

Retrieval Experiments

  • VisRAG-Ret, built on MiniCPM-V 2.0 with SigLIP and MiniCPM, is evaluated alongside lexical, text-embedding, and vision-retrieval baselines.Off-the-shelf comparators include BM25, bge-large-en-v1.5, NV-Embed-v2, and SigLIP.Source: §4 · p. 6
  • Out-of-domain models train on synthetic data only, while in-domain models use VQA training and synthetic data; both settings evaluate the same models.In-domain evaluation also includes ColPali, trained on the authors’ dataset using its official code and hyperparameters.Source: §4 · pp. 6–7

Generation Experiments

  • Generation experiments fix retrieval to VisRAG-Ret and compare VLMs, single-image selection or concatenation, and OCR-based text-generation baselines.They evaluate top-1, top-2, and top-3 retrieved documents, plus an Oracle condition supplying only positive documents as an upper bound.Source: §4 · p. 7

Implementation Details

  • VisRAG-Ret is fine-tuned for one epoch with in-batch negatives, batch size 128, and temperature 0.02; generation uses off-the-shelf models without fine-tuning.Training uses eight NVIDIA A100 80GB GPUs; baseline retrievers share the hyperparameters, and textual baselines use extracted text as document input.Source: §4 · p. 7

5 EVALUATION RESULTS

VisRAG improves end-to-end accuracy over TextRAG through stronger retrieval and generation, with gains across datasets. The analyses also report training-data efficiency and performance differences by document length and model design.

5.1 OVERALL PERFORMANCE

  • Out-of-domain VisRAG-Ret outperforms BM25 and bge-large and reaches 95% of NV-Embed-v2’s performance; bge-large fails on these real-document datasets despite its MTEB results.NV-Embed-v2 and bge-large are trained on millions of query-document pairs, 10× more than VisRAG’s training data.Source: §5.1 · pp. 7–8
  • With ground-truth documents, image-direct VisRAG-Gen models outperform text-only TextRAG-Gen; MiniCPM-V 2.0 scores 30% higher than MiniCPM (OCR).The comparison uses the same VisRAG retriever across generation methods.Source: Table 3 · p. 8, §5.1 · pp. 8–9
  • With matched training data, VisRAG-Ret’s gains over MiniCPM (OCR) and SigLIP are larger out of domain, and its hybrid design performs well across datasets.The gains are 13% and 20% over MiniCPM (OCR) and SigLIP out of domain, versus 4% and 9% in domain; direct VLM encoding avoids information loss from parsing, while combining visual and language components captures holistic document information.Source: §5.1 · p. 8
  • VisRAG-Ret uses 4.5KB in one 2304-dimensional vector, versus ColPali’s 256KB across 1030 128-dimensional vectors, while maintaining superior retrieval performance.The paper describes this memory efficiency as better suited to scaling to millions or billions of documents.Source: §5.1 · p. 8
  • 25% to 51%: VisRAG’s end-to-end accuracy on InfographicsVQA; across six datasets, it achieves a 40% relative accuracy increment on average.The reported gains combine more accurate document retrieval with improved answer generation from correctly retrieved documents.Source: Figure 3 · p. 9, §5.1 · p. 9
  • With noisy top-1-to-3 retrieved documents, VisRAG-Gen beats TextRAG-Gen within model series; performance rises with more pages for multi-image MiniCPM-V 2.6 and GPT-4o, but not MiniCPM-V 2.0.For single-image MiniCPM-V 2.0, weighted selection works better than page concatenation with two or three documents.Source: §5.1 · p. 9

5.2 TRAINING DATA EFFICIENCY

  • 20K examples let VisRAG match bge-large (OCR), versus about 75K for MiniCPM (OCR); later, VisRAG retains a 13% advantage over MiniCPM (OCR).NV-Embed-v2 (OCR) slightly outperforms VisRAG trained on 240K synthetic examples, but it uses millions of curated pairs and an 8B-parameter model.Source: Figure 4 · p. 9, §5.2 · pp. 9–10

5.3 PERFORMANCE ON DIFFERENT DATA SUBSETS

  • VisRAG’s relative performance over TextRAG generally improves as positive-document length decreases, while VisRAG (SigLIP) underperforms VisRAG across subsets and sometimes trails TextRAG.VisRAG beats TextRAG on most subsets; the authors suggest vision encoders help with visually oriented documents, while VisRAG’s language model better captures semantics conveyed through visual cues.Source: Figure 5 · p. 10, §5.3 · p. 10

6 CONCLUSION · A DATA CONSTRUCTION DETAILS

VisRAG uses VLMs for retrieval and generation without the parsing stage required by traditional text-based RAG, and the paper reports that it consistently outperforms text-based RAG on both stages while maintaining a simpler pipeline.

6 CONCLUSION

  • VisRAG uses VLMs for retrieval and generation, eliminating the parsing stage required by traditional text-based RAG.The authors present it as a vision-based RAG paradigm for handling multimodal documents.Source: §6 · p. 10
  • VisRAG consistently outperforms text-based RAG on retrieval and generation while maintaining a simpler pipeline.The authors hope this approach will inspire future RAG development to incorporate VLMs for multimodal documents.Source: §6 · p. 10

A.1 SYNTHETIC DATA · A.2 QUERY FILTERING · B DOCUMENT PARSING

The supplementary methods augment VisRAG’s training data with web documents and generated queries, filter context-dependent queries, and examine pipeline-based and model-based document parsing.

A.1 SYNTHETIC DATA

  • Web documents augment VisRAG’s training dataset, and Table 4 lists the sources of the collected documents.Table 4 reports statistics of the crawled documents.Source: §A.1 · p. 17, Table 4 · p. 17
  • GPT-4o generates queries based on the collected documents using a prompt that takes a document page as input.The prompt is shown in Figure 6.Source: §A.1 · p. 17, Figure 6 · p. 17

A.2 QUERY FILTERING

  • A significant portion of VQA-dataset queries are context-dependent and unsuitable for retrieval, motivating a filtering step.Source: §A.2 · p. 17
  • GPT-4o classifies queries using an instruction that includes human-annotated DocVQA samples; Figure 7 labels context-dependent queries as B.Source: §A.2 · p. 17, Figure 7 · p. 18
  • Filtering reduces context-dependent queries, but a small number remain and are reported to have minimal impact on dataset quality.Source: §A.2 · p. 17

B DOCUMENT PARSING

  • The document-parsing experiments cover two strategy categories: pipeline-based parsing and model-based parsing.Source: B DOCUMENT PARSING · p. 19

B.1 PIPELINE-BASED PARSING … E PROMPTS FOR GENERATION

The appendices describe OCR- and model-based document parsing, the models used for retrieval and generation, and the prompts and Recall@10 evaluation materials. For pipeline parsing, PPOCR methods outperform Pytesseract, while adjacent merging and layout preserving perform similarly.

B.1 PIPELINE-BASED PARSING

  • PPOCR-based parsing outperforms Pytesseract, and adjacent merging and layout preserving yield similar results; the authors use adjacent merging for “(OCR)” runs.PPOCR retains recognized text with confidence scores above 0.6; adjacent merging combines nearby text boxes, while layout preserving orders text by spatial position.Source: §B.1 · p. 19, Table 5 · p. 19

B.2 MODEL-BASED PARSING

  • MiniCPM-V 2.0 is trained for one epoch with batch size 2048 and learning rate 5e-6.Source: §B.2 · p. 20

C MODELS USED IN THIS PAPER

  • MiniCPM constructs the text-based retriever and generator baselines, SigLIP-400m the vision-based retriever baseline, and MiniCPM-V 2.0 VisRAG-Ret and VisRAG-Gen.MiniCPM-V 2.0 also serves as the document parsing model.Source: C MODELS USED IN THIS PAPER · p. 20
  • MiniCPM-V 2.6 builds VisRAG-Gen and the MiniCPM-V 2.6 (OCR) text-based generation baseline, while GPT-4o builds VisRAG-Gen and synthesizes training data.MiniCPM-V 2.6 accepts multiple images and has stronger OCR capabilities, according to the paper.Source: C MODELS USED IN THIS PAPER · p. 20

D RETRIEVAL PERFORMANCE IN RECALL@10

E PROMPTS FOR GENERATION

F CASE STUDY

Two case studies show TextRAG failing when text extraction loses a decorative-font term and when parsing loses layout information; VisRAG succeeds in both cases.

F CASE STUDY

  • In DocVQA, TextRAG fails to retrieve the document because it misses “Club Jetty” in a decorative font, while VisRAG retrieves it and answers correctly.The case compares both systems under the same setup as the paper’s end-to-end performance evaluation.Source: F CASE STUDY · p. 21, Table 8 · p. 22
  • In InfographicsVQA, both systems retrieve the document, but VisRAG uses layout information to identify Europe’s number correctly while TextRAG confuses 53% and 49%.TextRAG’s loss of layout information leaves unclear which number pertains to Europe.Source: F CASE STUDY · p. 21, Table 9 · p. 23

G ADDITIONAL RETRIEVAL AND GENERATION RESULTS

Supplementary results compare multimodal retrieval fusion with VisRAG-Ret and examine generation with more retrieved pages. They show VisRAG-Ret’s advantage over fusion, degradation as page count increases, and stronger multi-page document understanding by Qwen2-VL-7B-Instruct than MiniCPM-V 2.6.

Additional retrieval results

  • VisRAG-Ret reaches 71.49 MRR@10 out-of-domain and 77.91 MRR@10 in-domain, outperforming RRF fusion of MiniCPM OCR and SigLIP despite fusion improving over individual modality baselines.RRF combines textual information from OCR with visual clues, but remains below the unified VisRAG-Ret model.Source: G ADDITIONAL RETRIEVAL AND GENERATION… · p. 24, Table 10 · p. 24

Additional generation results

  • Generation performance degrades with more retrieved pages for MiniCPM-V 2.0 page concatenation and MiniCPM-V 2.6 multi-page input, while MiniCPM-V 2.6 is more robust to increased context.The experiments use top-6 and top-10 retrieved documents from VisRAG-Ret; open-source VLMs still struggle with multi-page reasoning and noisy retrieved data.Source: G ADDITIONAL RETRIEVAL AND GENERATION… · p. 24, Table 11 · p. 24
  • Qwen2-VL-7B-Instruct outperforms MiniCPM-V 2.6 in the evaluated tasks, indicating stronger document understanding.The generation comparisons use MiniCPM-V 2.0 with page concatenation and MiniCPM-V 2.6 with direct feeding.Source: G ADDITIONAL RETRIEVAL AND GENERATION… · p. 24, Table 11 · p. 24

H RETRIEVAL EFFICIENCY

VisRAG-Ret reduces total document processing time by bypassing parsing, while its online query-processing latency is nearly identical to MiniCPM (OCR) for text-only queries.

H RETRIEVAL EFFICIENCY

  • Latencies were averaged over PlotQA queries and documents; encoding used an NVIDIA A100 40G GPU at batch size 1, and parsing used one Intel Xeon Platinum 8350C CPU core.The evaluation measured offline document parsing and encoding, plus online query encoding and search.Source: H RETRIEVAL EFFICIENCY · p. 24
  • 58% less total document processing time for VisRAG-Ret than MiniCPM (OCR) results from bypassing the parsing stage, despite longer document encoding.Table 12 reports offline latencies per document, including parsing and encoding.Source: H RETRIEVAL EFFICIENCY · pp. 24–25, Table 12 · p. 25
  • Online query-processing latencies for VisRAG-Ret and MiniCPM (OCR) are nearly identical because queries consist solely of textual inputs.Online latency covers query encoding and search per query.Source: H RETRIEVAL EFFICIENCY · p. 25, Table 12 · p. 25

I RETRIEVAL PERFORMANCE ON TEXT RETRIEVAL BENCHMARKS

On text retrieval, VisRAG-Ret retrieves from rendered document screenshots and significantly outperforms MiniCPM (OCR), which uses textual information. The result suggests its pooling-based representation captures textual details effectively for text-heavy retrieval.

I RETRIEVAL PERFORMANCE ON TEXT RETRIEVAL BENCHMARKS

  • VisRAG-Ret using rendered screenshots significantly outperforms MiniCPM (OCR) using textual information on the BEIR text retrieval benchmark.The authors attribute this result to VisRAG-Ret’s pooling-based representation effectively capturing textual details and suiting text-heavy document retrieval.Source: Table 13 · p. 25, I RETRIEVAL PERFORMANCE ON TEXT… · p. 25
  • The evaluation rendered BEIR document texts as screenshots for VisRAG-Ret, while MiniCPM (OCR) used BEIR’s raw textual data.Screenshots were created at 800px width with 24px DejaVuSans text; their heights varied by document length.Source: I RETRIEVAL PERFORMANCE ON TEXT… · p. 25

Related papers

Loading 2410.10594v2…