PaperlayerView exact revision on arXiv

Source-linked AI summary

VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents

Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, Maosong Sun

arXiv:2410.10594v2cs.IRcs.AIcs.CLcs.CV

TL;DR

Text-based RAG cannot fully use layout and image information in multi-modality documents, and parsing can lose or distort original data. VisRAG uses VLMs to retrieve and generate directly from document images, outperforming traditional RAG in retrieval and generation with 20–40% end-to-end relative gains.

  • Problem

    Text-based RAG relies on parsed text, limiting its use of layout and image information in real-world multi-modality documents and introducing potential parsing information loss.

  • Method

    VisRAG uses a VLM-based retriever to embed document images directly and a VLM-based generator to answer from retrieved images.

  • Results

    VisRAG consistently outperforms text-based RAG in retrieval and generation, with 20–40% relative end-to-end gains over TextRAG.

  • Takeaways & Limitations

    VisRAG preserves documents in their original visual format while maintaining a simpler RAG pipeline for multi-modality documents.

Abstract

from arXiv · show

Retrieval-augmented generation (RAG) is an effective technique that enables large language models (LLMs) to utilize external knowledge sources for generation. However, current RAG systems are solely based on text, rendering it impossible to utilize vision information like layout and images that play crucial roles in real-world multi-modality documents. In this paper, we introduce VisRAG, which tackles this issue by establishing a vision-language model (VLM)-based RAG pipeline. In this pipeline, instead of first parsing the document to obtain text, the document is directly embedded using a VLM as an image and then retrieved to enhance the generation of a VLM. Compared to traditional text-based RAG, VisRAG maximizes the retention and utilization of the data information in the original documents, eliminating the information loss introduced during the parsing process. We collect both open-source and synthetic data to train the retriever in VisRAG and explore a variety of generation methods. Experiments demonstrate that VisRAG outperforms traditional RAG in both the retrieval and generation stages, achieving a 20--40% end-to-end performance gain over traditional text-based RAG pipeline. Further analysis reveals that VisRAG is efficient in utilizing training data and demonstrates strong generalization capability, positioning it as a promising solution for RAG on multi-modality documents. Our code and data are available at https://github.com/openbmb/visrag.

1 INTRODUCTION

Traditional RAG pipelines struggle with multi-modality documents because parsing can introduce errors and text-only retrieval overlooks visual information. VisRAG instead uses VLMs to retrieve and generate directly from document images, preserving original visual information and improving end-to-end performance.

  • RAG supplies retrieved information from external knowledge bases to address hallucination and knowledge-update limitations in LLM applications.
  • TextRAG parses multi-modality documents through layout recognition, OCR, and post-processing before using segmented text as retrieval units.The parsing cascade can introduce errors that negatively affect retrieval and generation.
  • TextRAG overlooks information in non-text modalities, including images, when documents interleave text and figures on the same page.
  • VLMs combine language models with vision encoders to capture visual and textual information in images and documents.Their demonstrated capabilities include describing pictures, explaining figures, and transcribing document text.
  • VisRAG uses a VLM-based retriever and generator, directly embedding document images and processing retrieved images without a parsing stage.Its retriever maps queries and document images into an embedding space, while page concatenation and weighted selection support single-image VLMs.
  • 40% relative improvement with MiniCPM-V 2.6 and 20% with GPT-4o: VisRAG outperforms TextRAG in direct end-to-end pipeline comparisons.The paper attributes these gains to improvements across the retrieval and generation cascade.

2 RELATED WORK

Related work covers text-based RAG, vision-language models, and multimodal retrieval, while Figure 1 contrasts TextRAG’s parsed-text pipeline with VisRAG’s direct use of document images.

  • Comparison of TextRAG and VisRAG: Figure 1 compares TextRAG, which uses parsed text for retrieval and generation, with VisRAG, which uses original document images through VLMs.
  • Retrieval-augmented Generation (RAG): RAG retrieves external knowledge for LLM generation, with text-based retrievers and LLM-based generators forming the typical pipeline.
  • Retrieval-augmented Generation (RAG): Prior RAG research primarily improves text retrievers through mechanisms such as generator feedback.
  • Vision-language Models: VLM research connects language models with vision encoders to support fine-grained multimodal understanding.Representative systems include CLIP, Flamingo, LLaVA, and BLIP.
  • Multi-modality Retrieval and RAG: Multimodal retrieval includes matching images to text, retrieving text-image pairs for questions, and retrieving texts answering questions about images.UniIR is described as a universal model for several such tasks.

3 METHODOLOGY

VisRAG replaces text-segment retrieval and generation with page-level visual processing by VLMs, aiming to preserve multimodal document information. Its pipeline combines a VLM retriever, several multi-page generation strategies, and datasets built from VQA and synthetic query-document pairs.

  • VisRAG framework: VisRAG uses document pages as retrieval and generation units, processing each page directly as an image without an additional parsing step.This contrasts with TextRAG, which uses parsed text segments and requires parsing for complex multimodal documents.
  • VisRAG framework: VisRAG-Ret separately encodes the text query and document-page image with a VLM, then represents each using position-weighted mean pooling over final hidden states.Later tokens receive higher weights in the pooling operation.
  • VisRAG framework: VisRAG-Ret computes query-page similarity from their embeddings and is optimized with the InfoNCE loss over positive and negative documents.The similarity score uses cosine similarity; the loss includes a temperature parameter.
  • VisRAG framework: VisRAG-Gen generates answers from retrieved pages using page concatenation, weighted selection, or VLMs designed to accept multiple images.Weighted selection combines per-page answer-generation probability with normalized retrieval score, while multi-image VLMs can reason across pages.
  • Data construction: The datasets combine VQA-derived question-document pairs across document and figure types with synthetic query-document pairs, while filtering context-dependent questions.Evaluation reports MRR@10 and Recall@10 for retrieval and relaxed exact-match answer accuracy for generation.

4 EXPERIMENTAL METHODOLOGY

The experiments compare VisRAG with text- and vision-centric alternatives under off-the-shelf, out-of-domain, and in-domain retrieval settings, then evaluate generation methods using a fixed VisRAG retriever. The setup uses explicit parsing baselines, matched training conditions, and both single- and multi-image generators.

  • Experimental setup: VisRAG is evaluated against TextRAG using OCR-based and MiniCPM-V-based captioner document-parsing methods.The OCR pipeline detects text regions with PPOCR and merges nearby boxes; the captioner directly extracts text from document images.
  • Retrieval experiments: Retrieval experiments cover off-the-shelf, out-of-domain, and in-domain settings, with VisRAG-Ret evaluated in the latter two.The retrieval model is built on MiniCPM-V 2.0, combining SigLIP as vision encoder and MiniCPM as language model.
  • Retrieval experiments: Off-the-shelf retrieval baselines include BM25, bge-large-en-v1.5, NV-Embed-v2, and SigLIP, evaluated using extracted texts or image representations.NV-Embed-v2 is described as a 7.85B-parameter state-of-the-art text embedding model.
  • Retrieval experiments: Out-of-domain models are trained only on synthetic data and evaluated on VQA datasets without in-domain supervision.The compared models include MiniCPM with OCR or captioner inputs and SigLIP.
  • Retrieval experiments: In-domain models are trained on a blend of VQA training data and synthetic data, with ColPali additionally evaluated using its official training setup.This setting tests performance when supervised labels are available.
  • Generation experiments: Generation experiments fix VisRAG-Ret and compare text-based generation with single-image and multi-image VLM generation.They evaluate page concatenation and weighted selection, including MiniCPM-V 2.0, MiniCPM-V 2.6, and GPT-4o.
  • Implementation details: VisRAG-Ret is fine-tuned for one epoch with in-batch negatives, while generation uses off-the-shelf LLMs and VLMs without fine-tuning.Baseline retrievers use the same hyperparameters, with textual baselines receiving extracted text as document-side input.

5 EVALUATION RESULTS

VisRAG improves retrieval and generation over text-based baselines while preserving document images and reducing the data needed for effective retrieval. Its end-to-end gains extend across datasets, retrieved-document settings, and document-length subsets.

  • Retrieval Performance: 13% and 20% gains over MiniCPM (OCR) and SigLIP in the out-of-domain setting exceed the corresponding 4% and 9% in-domain gains.The results indicate stronger generalization than text- and vision-centric models under matched training conditions.
  • Retrieval Performance: 4.5KB in one vector gives VisRAG-Ret much lower memory use than ColPali’s 256KB across 1030 vectors.The authors identify this representation as more suitable for scaling to millions or billions of documents.
  • Generation Performance: 30% higher performance than MiniCPM (OCR) is achieved by MiniCPM-V 2.0 when generating from ground-truth document images.Both methods use the same retriever, while VisRAG-Gen processes document images directly and TextRAG-Gen relies on extracted text.
  • Generation Performance: VisRAG-Gen consistently outperforms TextRAG-Gen within the same model series when receiving the top-1 to 3 retrieved documents.Multi-image models gain as more documents are retrieved, whereas the single-image MiniCPM-V 2.0 does not improve with additional documents.
  • End-to-end Performance: 25% to 51% overall accuracy on InfographicsVQA and a 40% average relative accuracy increment across six datasets are reported for VisRAG.The end-to-end improvement combines higher correct retrieval with higher correct generation from accurately retrieved documents.
  • Training Data Efficiency: 20K examples match bge-large (OCR), while MiniCPM (OCR) needs about 75K; VisRAG later retains a 13% performance advantage.The comparison uses different amounts of synthetic training data in the out-of-domain setting.
  • Subset Analysis: VisRAG, VisRAG (SigLIP), and TextRAG are compared across query subsets grouped by positive-document length percentiles, with TextRAG set to zero.The analysis examines relative retrieval and generation differences as document lengths vary.

6 CONCLUSION

The paper proposes VisRAG, a VLM-based RAG paradigm that directly uses document images for retrieval and generation instead of parsing them into text. Its empirical results show consistent advantages over text-based RAG with a simpler pipeline.

  • 6 CONCLUSION: VisRAG uses VLMs for retrieval and generation, eliminating the parsing stage required by traditional text-based RAG.The paradigm directly processes document images to preserve multimodal information.
  • 6 CONCLUSION: VisRAG consistently outperforms text-based RAG on retrieval and generation while maintaining a simpler pipeline.The authors position the approach as a basis for incorporating VLMs into future RAG systems for multimodal documents.
  • 6 CONCLUSION: The paper aims to inspire future RAG development for handling multi-modal documents with VLMs.This conclusion follows the reported retrieval and generation results.

A.1 SYNTHETIC DATA

The synthetic-data procedure gathers web documents, prompts GPT-4o to generate retrieval queries, and filters context-dependent VQA queries before training evaluation datasets.

  • Synthetic Query Generation: GPT-4o generates queries from additional web-crawled documents to augment VisRAG’s training dataset.The document sources are summarized in Table 4, and the generation prompt appears in Figure 6.
  • Synthetic Query Generation: The query-generation prompt presents a document image as a retrieval target and asks for questions from varied angles.The prompt frames the task as finding the target document relevant to a query.
  • Query Filtering: GPT-4o classifies VQA queries to filter context-dependent questions that are unsuitable for retrieval.The classifier uses human-annotated DocVQA samples, while a small residual of context-dependent queries may remain.
  • Document Parsing: The synthetic-data section distinguishes pipeline-based parsing from model-based parsing as two document-extraction strategies.These strategies are introduced as the paper’s categories of document parsing methods.

B.1 PIPELINE-BASED PARSING

The pipeline-based parsing setup compares OCR-oriented document extraction methods and downstream retrieval configurations. It evaluates Pytesseract, PPOCR variants, and related model choices for text-based processing.

  • Parsing pipelines: Pytesseract extracts image text in one step, while PPOCR detects, classifies, and recognizes text regions sequentially.Pytesseract provides bounding boxes, confidence scores, and orientation correction; PPOCR explicitly separates detection, orientation classification, and recognition.
  • Parsing policies: Adjacent merging combines nearby text boxes, whereas layout preserving maintains the document’s original layout.Adjacent merging uses vertical proximity within 15 pixels and horizontal alignment within 100 pixels before concatenating text.
  • Parsing comparison: PPOCR-based methods substantially outperform Pytesseract, while adjacent merging and layout preserving achieve similar retrieval results.The study therefore selects adjacent merging for its OCR runs.
  • Model-based parsing: A model-based Captioner approach uses MiniCPM-V 2.0 to transcribe document images into text, trained with image-caption and GPT-4V-generated description data.The descriptions cover DocVQA, ChartQA, SlideVQA, InfographicsVQA, TextVQA, and ArxivQA; Figure 8 shows the GPT-4V prompt.
  • Models: The pipeline uses MiniCPM as a baseline text retriever and generator, SigLIP as a vision-retriever baseline, and MiniCPM-V variants for VisRAG components.MiniCPM-V 2.0 builds VisRAG-Ret and VisRAG-Gen, while MiniCPM-V 2.6 builds VisRAG-Gen and an OCR generation baseline.

F CASE STUDY

The case studies contrast VisRAG and TextRAG on retrieval and generation failures involving decorative text and document layout. VisRAG succeeds where extracted text or layout cues are insufficient.

  • Comparison: Both cases compare VisRAG with TextRAG under the same end-to-end setup.The comparison uses the setup described for end-to-end performance evaluation.
  • DocVQA: VisRAG retrieves the relevant DocVQA document when decorative-font text prevents TextRAG from extracting the query term.The query concerns “Club Jetty”; TextRAG fails retrieval, whereas VisRAG retrieves the document successfully.
  • InfographicsVQA: VisRAG answers the InfographicsVQA case correctly by using layout information that distinguishes whether 53% or 49% pertains to Europe.Both systems retrieve the document, but TextRAG generates an incorrect answer after layout information is lost.

G ADDITIONAL RETRIEVAL AND GENERATION RESULTS

Additional experiments examine modality fusion and generation with larger retrieved contexts. VisRAG-Ret remains stronger than fused modality baselines, while generation degrades as more pages are supplied.

  • Retrieval: 71.49 out-of-domain and 77.91 in-domain are reported for VisRAG-Ret, which outperforms reciprocal-rank fusion of OCR and visual retrieval outputs.RRF combines MiniCPM (OCR) and SigLIP, but remains below VisRAG-Ret on both reported settings.
  • Retrieval: Fusing text and image modalities meaningfully improves retrieval over individual modality baselines, but does not match VisRAG-Ret.The result is presented as evidence for unified multimodal understanding in VisRAG-Ret.
  • Generation: Generation performance significantly degrades when increasing retrieved pages from top-6 to top-10 for both page concatenation and multi-page input.MiniCPM-V 2.6 is more robust than MiniCPM-V 2.0, while Qwen2-VL-7B-Instruct outperforms MiniCPM-V 2.6 in these tasks.
  • Generation: Open-source VLMs remain challenged by reasoning over multiple pages and extracting relevant information from noisy retrieved data.The additional generation experiments use MiniCPM-V 2.0, MiniCPM-V 2.6, and Qwen2-VL-7B-Instruct with the same VisRAG retriever.

H RETRIEVAL EFFICIENCY

The efficiency evaluation separates offline document processing from online query processing. VisRAG-Ret spends more time encoding documents but avoids parsing and reduces total document-processing time.

  • Measurement: The efficiency experiment measures offline parsing and encoding latency per document and online query encoding and search latency per query.Document encoding runs on an NVIDIA A100 40G GPU, while parsing runs on one Intel Xeon Platinum 8350C CPU core.
  • Offline efficiency: 58% reduction in total document processing time is achieved by VisRAG-Ret compared with MiniCPM (OCR).VisRAG-Ret requires more document-encoding time but bypasses the parsing stage used by MiniCPM (OCR).
  • Online efficiency: VisRAG-Ret and MiniCPM (OCR) have nearly identical online query-processing latency.Online latency includes query encoding and search; queries contain only textual inputs.

I RETRIEVAL PERFORMANCE ON TEXT RETRIEVAL BENCHMARKS

VisRAG-Ret is evaluated on BEIR text retrieval by retrieving rendered document screenshots, and it significantly outperforms a text-based OCR baseline. The result indicates that its pooling-based representation captures textual details for text-heavy retrieval.

  • Evaluation setup: Table 13 reports retrieval performance on BEIR subsets using NDCG@10.VisRAG-Ret performs retrieval directly on rendered document screenshots.
  • Evaluation setup: BEIR evaluates VisRAG-Ret on textual retrieval after document texts are converted into rendered screenshots.Screenshots are generated with Pillow3 at 800px width, 24px font size, and DejaVuSans.
  • Results: VisRAG-Ret significantly outperforms MiniCPM (OCR) while relying only on rendered screenshots.MiniCPM (OCR) uses textual information, whereas VisRAG-Ret uses the rendered screenshots.
  • Results: VisRAG-Ret’s pooling-based representation effectively captures textual details and suits text-heavy document retrieval.
Loading 2410.10594v2…