PaperlayerView exact revision on arXiv

Source-linked AI summary

ModernVBERT: Towards Smaller Visual Document Retrievers

Paul Teiletche, Quentin Macé, Max Conti, Antonio Loison, Gautier Viaud, Pierre Colombo, Manuel Faysse

arXiv:2510.01149v3cs.IR

TL;DR

Visual document retrieval needs retrieval-specific training rather than simply repurposed generative vision-language decoders. The paper systematically studies the training pipeline, proposes ModernVBERT, and finds that its compact retrieval variant achieves performance close to much larger models while remaining efficient.

  • Problem

    Repurposed generative vision-language decoders bottleneck visual document retrieval, while RAG systems require accurate, efficient retrieval over complex documents.

  • Method

    The paper uses controlled experiments across modality alignment, attention masking, image resolution, data mixtures, and contrastive training to develop ModernVBERT and ColModernVBERT.

  • Results

    ColModernVBERT is over 10 times smaller than ColPali yet only 0.6 nDCG@5 points below it on aggregated ViDoRe scores.

  • Takeaways & Limitations

    Retrieval-focused design with bidirectional attention, strong vision-language alignment, and late interaction can produce compact visual document retrievers with strong benchmark performance.

  • Takeaways & Limitations

    The analysis focuses on relatively small models and English, so whether its observed scaling and multilingual trade-offs persist remains unclear.

Abstract

from arXiv · show

Retrieving specific information from a large corpus of documents is a prevalent industrial use case of modern AI, notably due to the popularity of Retrieval-Augmented Generation (RAG) systems. Although neural document retrieval models have historically operated exclusively in the text space, Visual Document Retrieval (VDR) models - large vision-language decoders repurposed as embedding models which directly work with page screenshots as inputs - are increasingly popular due to the performance and indexing latency gains they offer. In this work, we show that, while cost-efficient, this approach of repurposing generative models bottlenecks retrieval performance. Through controlled experiments, we revisit the entire training pipeline, and establish a principled recipe for improving visual document retrieval models. We notably measure the impact of attention masking, image resolution, modality alignment data regimes, and late interaction centered contrastive objectives which emerge as central performance factors. Building on these insights, we release ModernVBERT, a compact 250M-parameter vision-language encoder that outperforms recent models up to 10 times larger when fine-tuned on document retrieval tasks, enabling efficient inference on cheap CPU hardware and greatly reducing latency and costs while maintaining strong performance. Models, code and data are available at https://huggingface.co/ModernVBERT.

1 INTRODUCTION

The paper argues that repurposed generative vision-language models bottleneck visual document retrieval, motivating a systematic study of retrieval-specific design choices. It introduces ModernVBERT and ColModernVBERT as compact alternatives with strong benchmark performance.

  • Motivation: RAG retrieval quality matters because first-stage recall upper-bounds generated-answer quality, while retrieval latency and indexing costs affect system efficiency.The motivation applies especially to long, complex documents such as PDFs, scientific articles, and reports.
  • Visual Document Retrieval: Visual Document Retrieval operates directly on page screenshots, avoiding OCR and layout parsing while exploiting visual cues such as tables, figures, typography, and layout.This approach is presented as a simpler alternative to text-based preprocessing pipelines.
  • Limits of Generative VLM Repurposing: Generative vision-language decoders bottleneck retrieval because their model sizes, attention patterns, image resolutions, and objectives target generation rather than retrieval.The paper frames these choices as inherited design constraints from generative use cases.
  • Contribution: Bidirectional masking, token-level objectives, image resolution, and data mixtures are identified as important visual retriever design factors.The study emphasizes stronger image–text token alignment rather than only stronger image embeddings.
  • Contribution: ModernVBERT combines retrieval-focused training insights into a small 250M multimodal encoder, while ColModernVBERT is fine-tuned for document retrieval.The released models, checkpoints, and training code are intended to support efficient retrieval research.
  • Results: ColModernVBERT matches models 10x larger on standard visual document retrieval benchmarks despite its modest size and limited training budget.Its leading performance-size tradeoff is illustrated on ViDoRe.

2 METHODOLOGY

The methodology evaluates visual retrievers through controlled, two-stage training: modality alignment followed by contrastive post-training. It varies attention masking, architecture, image processing, data mixtures, and contrastive batching to study retrieval performance.

  • Controlled Experimental Setup: The controlled attention experiment compares otherwise matched 210M-parameter transformers trained with bidirectional MLM, causal next-token prediction, or causal-to-bidirectional annealing.The checkpoints share training data, size, architecture, and scheduling, isolating attention-mask effects.
  • Model Architecture: The early-fusion architecture projects visual patch embeddings into the language-model embedding space and concatenates them with text tokens for joint processing.Figure 2 presents this visual-token and text-token integration scheme.
  • Training Procedure: The training procedure first aligns a pretrained language model with visual inputs, then uses text-image contrastive learning to improve image representations for retrieval.This biphasic design separates modality alignment from retrieval specialization.
  • Modality Alignment: Modality alignment maps visual features into the language-model embedding space while jointly adapting image and text components with LoRA.Decoder models use causal language modeling, whereas bidirectional encoders use masked language modeling over textual tokens.
  • Modality Alignment: The modality-alignment corpus emphasizes text-containing images and documents across web pages, books, and scientific papers, totaling approximately 2B text tokens.The training mixture is adjusted toward document-focused retrieval.
  • Contrastive Post-training: Contrastive post-training uses document-query and natural image-description pairs with InfoNCE loss, including in-batch and hard negatives based on query-document similarity.Task-aware batching groups documents by source to produce more homogeneous batches and richer training signals.
  • Evaluation: Evaluation aggregates ViDoRe and ViDoRe v24 nDCG@5 scores for document retrieval and includes natural-image retrieval and zero-shot classification tasks.The broader evaluation tests whether document-focused training transfers to more general visual retrieval.

3 WHAT MAKES A GREAT VISUAL RETRIEVER?

Controlled experiments identify attention masking, modality alignment, image resolution, and contrastive data mixtures as central design choices for visual document retrieval. The results show that choices benefiting document retrieval can differ from those benefiting natural-image and classification tasks.

  • Modality alignment design: Early fusion improves document retrieval regardless of the language-model objective but degrades natural-image and classification tasks relative to fine-tuned SigLIP.Scores are aggregated MIEB nDCG and Accuracy measurements.
  • Modality alignment design: Document retrieval improves consistently with more modality-alignment data and surpasses the standalone vision encoder, whereas natural-image tasks plateau past 1B tokens.The findings attribute the divergence to fine-grained image–text interactions benefiting document retrieval, while language-model processing has limited utility for high-level natural-image tasks.
  • Attention masking and training objectives: Bidirectional attention exceeds causal attention by +10.6 nDCG@5 in late interaction, while single-vector performance differs by only +1.6 nDCG@5.The comparison uses average nDCG@5 on English ViDoRe splits; causal decoders struggle to contextualize early image or text tokens for late interaction.
  • Image resolution: Higher image resolution substantially improves visual document retrieval, and high-resolution cooldown during modality alignment provides additional gains.By contrast, raising resolution tends to degrade non-document tasks.
  • Contrastive training design: Adding text-only pairs improves visual document retrieval by +1.7 nDCG@5, indicating cross-modal transfer through the jointly learned text–image embedding space.The accompanying mixture comparison reports that text-only pairs improve document retrieval but can degrade other task categories.
  • Contrastive training design: Natural-image caption pairs improve downstream performance broadly, with a smaller but consistent +0.2 nDCG@5 gain on document retrieval.The largest improvements occur on natural-image tasks, showing that contrastive data need not be exclusively document-focused.

4 BUILDING A SMALL YET MIGHTY VISUAL RETRIEVER.

ModernVBERT combines bidirectional text encoding, vision-language modality alignment, higher-resolution images, and late-interaction contrastive training into a compact visual retriever. ColModernVBERT matches much larger models while offering substantially faster CPU query encoding.

  • Training: The 250M-parameter recipe aligns bidirectional text and vision encoders with MLM, increases image resolution from 1024px to 2048px, and mixes document-query with text-only contrastive pairs.ColModernVBERT uses hard negatives and a 2/1 text-to-image ratio; BiModernVBERT is trained as a single-vector variant.
  • Results: ColModernVBERT is only 0.6 nDCG@5 points below ColPali on aggregated ViDoRe scores despite having over 10 times fewer parameters.It edges many larger single-vector repurposed VLMs but remains below the top ViDoRe models built on larger decoder VLMs.
  • Results: ColModernVBERT is 12.7 nDCG@5 points above the 176M late-interaction model ColFlor on document retrieval benchmarks.It also largely outperforms substantially larger off-the-shelf dual encoders.
  • Results: Table 3 compares ViDoRe performance, model size, and inference latency, highlighting ColModernVBERT's performance-size tradeoff among sub-1B models.Bold values indicate the best performance among sub-1B models.
  • Speed: ModernVBERT achieves more than a 7x CPU speedup over models with similar ViDoRe performance.Batched inference reaches 5000 queries per second on Nvidia H100 GPUs.

5 RELATED WORK

Visual retrieval research has repurposed generative VLMs for embeddings and increasingly uses late interaction to match query and document tokens. Recent work varies data, attention masking, resolution, and task sources, while native bidirectional vision-language alignment remains underexplored.

  • Repurposed VLMs: Recent visual retrieval work repurposes improving generative VLMs for multimodal embeddings, with language pretraining and modality alignment strongly influencing retriever performance.
  • Research Directions: Recent studies improve visual retrieval by scaling contrastive data and compute, changing attention masks, increasing image resolution, or diversifying tasks and data sources.
  • Late Interaction: Late interaction matches multiple query and document embeddings through MaxSim, enabling granular token interactions at added storage and slight matching-compute costs.

6 CONCLUSION

The paper identifies bidirectional attention, late interaction, and strong vision-language alignment as central design factors for efficient visual document retrieval. It releases a compact model and artifacts while leaving larger-scale and multilingual behavior for future study.

  • Conclusion: ColModernVBERT, a compact 250M-parameter encoder, matches models up to 10× larger on visual retrieval benchmarks.
  • Conclusion: Bidirectional attention and late-interaction retrieval show a tight synergy, while strong vision-language alignment is essential for document-centric retrieval.
  • Future Work & Limitations: The analysis targets relatively small models and focuses exclusively on English, leaving larger-scale and multilingual behavior unresolved.

ETHICS STATEMENT

The paper reports training and project-level environmental costs, alongside inference efficiency benefits from the compact model. It also notes reduced generative-output risks for the encoder while acknowledging that retrieval can propagate dataset biases.

  • Environmental Costs: Training ColModernVBERT required approximately 2,000 H100 GPU-hours, estimated at 41 kg of CO2e.All combined experiments totaled about 18k H100-hours.
  • Environmental Costs: The authors release checkpoints and training artifacts to support reuse and extension without retraining.
  • Environmental Costs: Efficient CPU query encoding keeps inference costs low and reduces barriers to adopting visual retrieval.
  • Safety and Bias: The encoder produces fixed-length embeddings rather than free-form content, reducing avenues for harmful generation, hallucination, or deceptive outputs.Retrieval systems can still propagate biases present in their underlying data.

DETAILED CONTRIBUTIONS

The project involved coordinated contributions spanning experiments, data-mixture optimization, codebase development, writing, and scientific supervision.

  • The first author designed the modality-alignment codebase and supervised most large-scale modality-alignment and contrastive-training experiments.
  • Large-scale ablations covered data mixtures and contrastive training, with responsibility for the final training run assigned within the team.
  • Senior contributors supported project framing, industry expertise, grants, reviews, and scientific leadership alongside substantial paper-writing contributions.

A.1 IMPLEMENTATION AND RESOURCES

The appendix documents training resources, similarity functions, datasets, contrastive data mixes, comparison models, and representative evaluation results.

  • A.2 SIMILARITY FUNCTIONS: Single-vector models use mean pooling for MLM-aligned encoders and EOS pooling for CLM-based models before cosine query-document scoring.
  • A.2 SIMILARITY FUNCTIONS: Multi-vector models use late interaction over token-level query and document embeddings.
  • A.1 IMPLEMENTATION AND RESOURCES: The training resources include modality-alignment datasets, NatCap’s 333k contextualized image-caption pairs, and generalist and document-focused contrastive mixes.
  • C.1 PERFORMANCE AGAINST OFF-THE-SHELF DUAL ENCODER: Early-fusion language-model training improves document retrieval over the standalone vision tower, while encoder MLM training eventually surpasses decoder CLM training.

C.3 BRIDGING THE GAP WITH LONGER CONTRASTIVE TRAINING

Longer contrastive training improves performance overall, especially on visual document benchmarks, but gains plateau or remain limited on some natural-image tasks.

  • The experiment doubles the contrastive dataset size at each step and trains until convergence, with early stopping and a one-epoch limit used to avoid overfitting.
  • More in-distribution contrastive data improves performance overall and quickly surpasses the vision-tower baseline on visual document benchmarks.
  • Scaling contrastive data narrows the gap on image captioning but produces no clear improvement in image classification.

C.3.1 OPTIMAL TEXT-TO-IMAGE RATIO FOR DOCUMENT RETRIEVAL

Increasing text-only data relative to image data consistently improves retrieval, while late-interaction gains extend beyond documents and model merging can degrade document retrieval.

  • C.3.1 OPTIMAL TEXT-TO-IMAGE RATIO FOR DOCUMENT RETRIEVAL: Increasing text-only pairs while fixing image-pair volume consistently enhances retrieval performance.
  • Late interaction improves single-vector retrieval by +20.2% (11.5 points) and matches E5-V while surpassing SigLIP on generalist retrieval.
  • Model merging is evaluated as a way to mitigate task-specific degradation while retaining contrastive-training performance.
  • Merging model results is task-dependent, with performance degrading relative to both original models in Document Retrieval.

C.4.2 CURRICULUM FOR DOCUMENT RETRIEVER CONTRASTIVE POST-TRAINING

Direct specialization on the target retrieval task is preferred over a preliminary generalist contrastive phase, which reduces final performance. The model also remains competitive for text retrieval while using image-processing strategies to control visual-token and latency costs.

  • Curriculum for document retriever contrastive post-training: A preliminary generalist contrastive phase reduces final document-retrieval performance by 0.5%, so direct target-task specialization is optimal.This harm holds regardless of whether a multi-vector approach is used.
  • Text retrieval evaluation: ColModernVBERT achieves an average NDCG@5 competitive with text-specialized single- and multi-vector models on NanoBEIR without explicit text-modality optimization.The benchmark covers multiple text-retrieval subdomains.
  • Image processing and visual tokens: Pixel shuffling compresses the roughly 17,500 visual tokens produced by a 2048x2048 image without shuffling by concatenating spatially close patch embeddings.The trade-off balances image resolution against sequence length and visual-token count.
  • Latency evaluation: Image-processing latency is evaluated across resolutions on both an L4 GPU and CPU, while text-query latency is measured on high-end CPUs and an Nvidia H100.Image latency averages 100 images; text latency uses online-style batch-size-one measurements and separate batching conventions for GPU throughput.
Loading 2510.01149v3…