PaperlayerView exact revision on arXiv

Source-linked AI summary

ColPali: Efficient Document Retrieval with Vision Language Models

Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, Pierre Colombo

arXiv:2407.01449v6cs.IRcs.CLcs.CV

TL;DR

Visually rich documents challenge text-centric retrieval systems because indexing relies on lengthy extraction pipelines and queries may require visual understanding. The paper introduces ViDoRe and ColPali, which embeds document-page images with Vision Language Models and uses late interaction; ColPali largely outperforms existing methods while enabling faster indexing and low query latency.

  • Problem

    Text-centric retrieval pipelines rely on lengthy extraction processes and struggle when accurate document matching requires both textual and visual understanding.

  • Method

    The paper introduces ViDoRe for page-level visual document retrieval evaluation and ColPali, which creates multi-vector page-image embeddings with Vision Language Models for late-interaction matching.

  • Results

    ColPali largely outperforms the best existing document retrieval methods on ViDoRe while enabling faster corpus indexing and low query latency.

  • Takeaways & Limitations

    Visual document retrieval with ColPali provides a simpler, end-to-end trainable approach that circumvents several pain points of modern document retrieval applications.

  • Takeaways & Limitations

    Training comparable text-based systems would require costly and lengthy extraction pipelines over more than 100K documents, including external proprietary captioning models.

Abstract

from arXiv · show

Documents are visually rich structures that convey information through text, but also figures, page layouts, tables, or even fonts. Since modern retrieval systems mainly rely on the textual information they extract from document pages to index documents -often through lengthy and brittle processes-, they struggle to exploit key visual cues efficiently. This limits their capabilities in many practical document retrieval applications such as Retrieval Augmented Generation (RAG). To benchmark current systems on visually rich document retrieval, we introduce the Visual Document Retrieval Benchmark ViDoRe, composed of various page-level retrieval tasks spanning multiple domains, languages, and practical settings. The inherent complexity and performance shortcomings of modern systems motivate a new concept; doing document retrieval by directly embedding the images of the document pages. We release ColPali, a Vision Language Model trained to produce high-quality multi-vector embeddings from images of document pages. Combined with a late interaction matching mechanism, ColPali largely outperforms modern document retrieval pipelines while being drastically simpler, faster and end-to-end trainable. We release models, data, code and benchmarks under open licenses at https://hf.co/vidore.

1 INTRODUCTION

Document retrieval must handle visually rich pages, but text-centric ingestion pipelines are lengthy and often miss visual cues. ColPali and ViDoRe address this gap by benchmarking and retrieving directly from page images.

  • Document retrieval supports search, information extraction, and Retrieval Augmented Generation pipelines.
  • PDF retrieval commonly requires parsing or OCR, layout detection, chunking, and sometimes captioning before indexing.These stages process text and visual page objects separately.
  • ViDoRe: ViDoRe benchmarks page-level retrieval across domains, visual elements, languages, and practical settings requiring textual and visual understanding.
  • ColPali: ColPali uses Vision Language Models to index documents from visual features and match queries with late interaction.
  • ColPali: ColPali significantly outperforms other retrieval systems on ViDoRe while remaining fast and end-to-end trainable.

2 PROBLEM FORMULATION & RELATED WORK

The paper frames retrieval as a latency-sensitive page-ranking problem with separate indexing and querying phases. Existing text and vision-language approaches leave a benchmark and visual-document retrieval gap that ViDoRe targets.

  • Problem Setting: Page-level retrieval scores corpus pages against queries, with offline indexation followed by latency-sensitive online querying.
  • Problem Setting: Efficient systems should provide strong retrieval performance, fast online querying, and high-throughput corpus indexation.
  • Document Retrieval in Text Space: TF-IDF and BM25 remain simple, efficient statistical retrievers, while neural embedding models achieve state-of-the-art text retrieval performance.
  • Neural Retrievers: Bi-encoders enable fast cosine matching from independently indexed document vectors, whereas cross-encoders are slower but slightly more performant.
  • Multi-Vector retrieval via late interaction: Late interaction indexes multiple document-token vectors and compares them with individual query-token embeddings.
  • Retrieval Evaluation: Text-only systems barely exploit visual cues even though documents often use visual elements to convey information efficiently.
  • Retrieval Evaluation: No prior benchmark evaluated practical document retrieval end-to-end across document types, topics, and textual plus visual features.
  • Integrating Visual Features: Contrastive vision-language models support some OCR but often optimize visual representations insufficiently for text understanding and retrieval.

3 THE ViDoRe BENCHMARK

ViDoRe is a page-level benchmark designed for multimodal retrieval across modalities, domains, languages, and realistic industrial tasks. It evaluates both retrieval quality and operational requirements such as latency and indexing throughput.

  • Benchmark Design: ViDoRe fills the gap between natural-image and textual-passage benchmarks by evaluating document retrieval with visual features.
  • Benchmark Design: The benchmark covers page-level matching across text, figures, infographics, tables, medical, business, scientific, administrative, English, and French tasks.
  • Academic Tasks: Academic tasks repurpose visual question-answering datasets by treating questions as queries and associated pages as gold documents.
  • Practical Tasks: Practical tasks pair 1,000 pages per topic with 100 human-filtered queries generated for realistic domain-specific retrieval.
  • Evaluation Metrics: Retrieval quality uses nDCG, Recall@K, and MRR, with nDCG@5 reported as the main metric.
  • Evaluation Metrics: The benchmark also measures query latency and indexing throughput to test practical industrial requirements.
  • Assessing Current Systems: Unstructured constructs high-quality PDF text chunks through document parsing, layout detection, OCR, and related processing.
  • Assessing Current Systems: Figure 2 compares average per-page indexing latencies, using Unstructured settings for PDF Parser results.

4 LATE INTERACTION BASED VISION RETRIEVAL

ColPali adapts a Vision Language Model to produce compact multi-vector representations for text and document images. Late interaction matches each query vector to its strongest document-vector counterpart and supports differentiable contrastive training.

  • Architecture: ColPali extends PaliGemma-3B with ColBERT-style multi-vector representations for text and images.
  • Architecture: A projection layer maps language-model output embeddings to a reduced D = 128 vector space for lightweight representations.
  • Late Interaction: Query and page embeddings are multi-vector representations in a shared R^D space with separate query and document vector counts.
  • Late Interaction: Late interaction sums each query vector’s maximum dot product with the document page’s vectors.
  • Contrastive Loss: The late interaction operator is differentiable, allowing backpropagation through an in-batch contrastive loss.
  • Training: The training dataset contains 118,695 query-page pairs from academic data and synthetic VLM-generated pseudo-questions.
  • Training: Models are trained for one epoch with LoRA adapters, bfloat16 precision, an 8-GPU setup, and batch size 32.
  • Training: Five appended unused tokens provide differentiable query expansion or re-weighting.

5 RESULTS

Results show that ColPali’s retrieval quality comes from combining document-oriented training, multimodal language-model contextualization, and multi-vector late interaction, while also offering practical efficiency and interpretability benefits.

  • 5.1 PERFORMANCE (R1): ColPali combines a task-specific dataset, language-model contextualization, and multi-vector embeddings to improve document retrieval.These factors are presented as jointly achieving performance gains by leveraging text semantics and preserving visual information across image patches.
  • 5.1 PERFORMANCE (R1): BiSigLIP improves retrieval across tasks, particularly for figure retrieval on ArxivQA and table retrieval on TabFQuAD.BiSigLIP fine-tunes the textual component of SigLIP on a document-oriented dataset while retaining a single-vector bi-encoder design.
  • 5.1 PERFORMANCE (R1): ColPali’s patch-level late interaction produces a step-change improvement over BiPali and outperforms evaluated retrieval baselines across visually complex and text-centric documents.The strongest differences occur on InfographicVQA, ArxivQA, and TabFQuAD, while text-centric documents also improve across evaluated domains and languages.
  • 5.1 PERFORMANCE (R1): ColSigLIP performs poorly because its pretraining optimizes pooled representations rather than individual patch and token embeddings, creating a mismatch for late interaction.A PaliGemma variant using SigLIP image representations and PaliGemma text representations also performs severely below the original SigLIP components.
  • 5.2 LATENCIES & MEMORY FOOTPRINT: ColPali reduces indexing overhead by encoding page images end-to-end, while query latency adds about 1 ms per 1000 pages and requires 257.5 KB per page.The reported online query costs are about 30 ms for ColPali language-model encoding versus 22 ms for BGE-M3 encoding for 15-token queries.
  • 5.2 LATENCIES & MEMORY FOOTPRINT: With a pool factor of 3, hierarchical mean token pooling reduces stored vectors by 66.7% while retaining 97.8% of original retrieval performance.The text-dense Shift dataset is an outlier with less redundant patch information and greater degradation under pooling.

6 ABLATION STUDY

The ablations examine tradeoffs in visual retriever architecture, training components, task adaptation, backbone quality, and distribution shift. Results show that larger or better backbones and targeted adaptation improve retrieval, while fewer patches and some training changes reduce performance.

  • Tradeoffs between model size and the number of image patches.: −24.8 nDCG@5 results from reducing ColPali from 1024 to 512 image patches, although memory usage is much lower.ColIdefics2 with 64 patches improves by +20.1 nDCG@5 over ColPali with 512 patches, but remains −4.7 nDCG@5 below ColPali with 1024 patches and is about twice as slow.
  • Unfreezing the vision component.: −0.7 nDCG@5 follows from unfreezing and updating the vision encoder and projection layer.The passage notes that this conclusion may change with larger training datasets.
  • Impact of “query augmentation” tokens.: +9.8 nDCG@G on Shift and +6.3 nDCG@5 on TabFQuAD occur on French tasks when query augmentation tokens are removed, with no significant English change.The reported Shift metric is nDCG@G, while TabFQuAD uses nDCG@5.
  • Impact of the Pairwise CE loss.: −1.6 nDCG@5 occurs on the aggregated benchmark when in-batch negative contrastive loss replaces pairwise CE loss.Pairwise CE considers only the hardest negative sample.
  • Adapting models to new tasks.: +2.6 nDCG@5 and +5 Recall@1 on TabFQuAD follow adding 1552 French table samples, with +0.4 nDCG@5 overall and no degradation elsewhere.The added French data is the only French training data; other examples remain unchanged.
  • Better VLMs lead to better visual retrievers.: +5.3 nDCG@5 is obtained by ColQwen2-VL over ColPali, while using approximately matched memory requirements.ColQwen2-VL uses Qwen2-VL 2B and 768 image patches versus ColPali’s 1024 patches.
  • Out-of-domain generalization.: −2.2 nDCG@5 is the performance drop when training solely on DocMatix, while the model still exceeds the closest baseline by over 12 points.This evaluates performance on a fully disjoint training distribution.

7 CONCLUSIONS

The paper introduces ViDoRe for realistic retrieval of visually complex document pages and ColPali for visual-only multi-vector indexing. ColPali outperforms existing methods while supporting faster indexing, low query latency, and public release of the research artifacts.

  • 7 CONCLUSIONS: ViDoRe evaluates document retrieval in realistic settings involving visually complex documents.The benchmark is released as part of the paper’s contribution.
  • 7 CONCLUSIONS: ColPali creates high-quality multi-vector embeddings purely from visual document features using Vision-Language Models.The method is presented as a retrieval approach for visually rich documents.
  • 7 CONCLUSIONS: ColPali largely outperforms existing document retrieval methods while enabling faster corpus indexing and low querying latencies.The conclusion frames this as addressing several pain points of modern document retrieval applications.
  • 7 CONCLUSIONS: The authors publicly release the benchmark, data, codebase, models, and baselines under open licenses.The stated goal is to support transparency, reproducibility, and future work.

REPRODUCIBILITY STATEMENT

The paper documents its datasets and reproducibility resources, including benchmark task sources and the released code, models, data, and evaluation benchmark.

  • REPRODUCIBILITY STATEMENT: The authors release training data, model checkpoints, the codebase, and the complete evaluation benchmark under MIT licenses.A public ViDoRe leaderboard is also provided to support concurrent work.
  • REPRODUCIBILITY STATEMENT: DocVQA uses UCSF Industry Documents Library images with manually annotated questions and answers.This identifies the source and annotation process for the dataset.
  • REPRODUCIBILITY STATEMENT: InfoVQA uses Internet infographics collected with the query “infographics,” with manually annotated questions and answers.The dataset targets infographic-based visual question answering.
  • REPRODUCIBILITY STATEMENT: TAT-DQA is constructed from public real-world financial reports and focuses on tabular and textual content requiring numerical reasoning.Questions and answers were manually annotated by finance experts.
  • REPRODUCIBILITY STATEMENT: arXivQA uses figures extracted from arXiv publications, with questions generated synthetically using GPT-4 Vision.The dataset evaluates question answering over scientific figures.
  • REPRODUCIBILITY STATEMENT: TabFQuAD evaluates table question answering in realistic industry settings using French industrial PDF documents.Additional queries augment the existing human-annotated questions.

A.2 PRACTICAL DATASETS

The practical dataset is built from web-crawled PDF pages and VLM-generated questions, with human validation and leakage controls. The section also describes training, indexing, storage, and latency considerations.

  • Dataset construction: The dataset pipeline crawls public documents, converts PDFs into one image per page, and generates page-related queries with a VLM.The crawler gathers documents across themes, while image-based query generation targets visually rich retrieval scenarios.
  • Query generation: At most three question-answer pairs are generated per page, with questions required to be relevant and answerable from the page.The generation prompt targets user-like multimodal retrieval questions rather than questions requiring knowledge of the corpus.
  • Human validation: Every synthetic query is manually validated by four volunteer annotators for quality, relevance, consistency, and problematic content.Annotators also flag pages containing personally identifiable information or unsuitable material.
  • Storage and compression: ColPali embeddings are substantially larger than BM25 and BGE-M3 embeddings, but pooling or binary quantization can reduce storage costs by two orders of magnitude with minimal performance loss.Hierarchical mean pooling with factor 3 reduces vectors by 66.7% while retaining 97.8% of original performance.
  • Indexing and efficiency: ColPali indexes page images directly, avoiding layout detection, OCR, chunking, and captioning steps required by standard pipelines.This design can accelerate indexing despite ColPali being larger than standard retrieval encoders.

C.1 OTHER METRICS

The evaluation reports Recall@1 for comprehensive benchmark comparisons and nDCG@5 for negative-result and ablation analyses. Figure 6 visualizes patch-level similarity for a query token.

  • Comprehensive evaluation: Table 6 reports baseline and ColPali performance on ViDoRe using Recall@1, omitting text-only metrics for benchmarks containing only visual elements.The table provides the primary comprehensive evaluation across the benchmark.
  • Negative results and ablations: Table 7 reports nDCG@5 for negative results and ablations, with ColPali included as a reference.Text-only metrics are again omitted for benchmarks containing only visual elements.
  • Similarity visualization: Figure 6 shows similarity between image patches and the underlined token in a query from the Shift test set.The visualization examines which page regions align with the queried token.

D MORE SIMILARITY MAPS

ColPali’s similarity maps connect query tokens to relevant image patches and can also highlight semantically related regions. Some white patches receive high scores, which the authors associate with internal global-image computations.

  • Token-to-patch similarity: For the token “Kazakhstan,” ColPali assigns high similarity to patches containing that word and to a patch around “Kashagan,” an offshore oil field in Kazakhstan.The authors describe this as suggestive of world-knowledge capabilities.
  • White-patch activations: White patches can receive high similarity scores because the authors believe their vectors were repurposed for internal computations and stored global information from the image.This interpretation is also observed in the similarity map shown in Figure 3.
  • Model basis: ColPali extends PaliGemma into a VLM that produces ColBERT-style multi-vector representations for text and images.PaliGemma combines a SigLIP vision encoder with a Gemma-2B language decoder and processes images as vision tokens.
  • Matching mechanism: Late interaction computes fine-grained cross-modal matching through maximum similarities between text-token and image-patch embeddings.This preserves token- and patch-level interactions while using separately encoded representations.

F EXAMPLES FROM THE ViDoRe BENCHMARK

ViDoRe includes practical retrieval questions spanning energy, government, finance, healthcare, technology, and multilingual settings. The examples ask for specific facts recoverable from visually rich documents.

  • Energy: Energy examples ask for projected electricity demand, sustainable aviation fuel capacity and consumption, and photovoltaic-system savings.These queries target numerical information from energy-related pages and charts.
  • Retrieval setting: The benchmark examples require retrieving answers from pages containing text, tables, charts, or other document content rather than relying on query wording alone.The supplied examples are framed as user questions seeking information from a document corpus.
  • Healthcare and technology: Healthcare and technology examples ask about TAII outcome areas, robot activation signals, PDP architecture, the PZT formula, and the AVPU scale.These examples span policy, robotics, architecture, materials science, and clinical assessment.
  • Government and finance: Government and finance examples ask about KPMG Hazem Hassan’s strategy, EPA mandates, tax-deferred accounts, and public healthcare financing entities.The questions cover organizational strategy, regulation, financial products, and government programs.
  • Multilingual retrieval: Multilingual examples include French questions about Kazakhstan’s offshore oil production, SAF projections, and cumulative crude-oil discoveries.These queries demonstrate coverage beyond English-language retrieval.
Loading 2407.01449v6…