PaperlayerView exact revision on arXiv

Source-linked AI summary

Nemotron ColEmbed V2: Top-Performing Late Interaction Embedding Models for Visual Document Retrieval

Gabriel de Souza P. Moreira, Ronay Ak, Mengyao Xu, Oliver Holworthy, Benedikt Schifferer, Zhiding Yu, Yauhen Babakhin, Radek Osmulski, Jiarui Cai, Ryan Chesler, Bo Liu, Even Oldridge

arXiv:2602.03992v2cs.IR

TL;DR

Visual document retrieval seeks to preserve information in complex documents without relying solely on OCR-based extraction. Nemotron ColEmbed V2 addresses this with multimodal late-interaction models and training techniques that achieve leading ViDoRe performance, while examining deployment trade-offs in latency and storage.

  • Problem

    OCR-based preprocessing for PDFs and slides can lose visual information from tables, charts, and infographics, motivating image-based visual document retrieval.

  • Method

    The paper releases 3B, 4B, and 8B VLM-based late-interaction models trained with bidirectional attention, contrastive learning, hard-negative mining, curated data, and model merging.

  • Results

    63.42 NDCG@10 ranks the 8B model first on ViDoRe V3, with a 3% improvement over second place as of February 03, 2026.

  • Takeaways & Limitations

    The models achieve top ViDoRe performance while the paper quantifies accuracy-storage trade-offs and deployment challenges associated with late interaction.

  • Takeaways & Limitations

    Late interaction requires multi-vector corpus scoring and specialized MaxSim support, potentially increasing serving latency compared with embedding-plus-cross-encoder pipelines.

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) systems have been popular for generative applications, powering language models by injecting external knowledge. Companies have been trying to leverage their large catalog of documents (e.g. PDFs, presentation slides) in such RAG pipelines, whose first step is the retrieval component. Dense retrieval has been a popular approach, where embedding models are used to generate a dense representation of the user query that is closer to relevant content embeddings. More recently, VLM-based embedding models have become popular for visual document retrieval, as they preserve visual information and simplify the indexing pipeline compared to OCR text extraction. Motivated by the growing demand for visual document retrieval, we introduce Nemotron ColEmbed V2, a family of models that achieve state-of-the-art performance on the ViDoRe benchmarks. We release three variants - with 3B, 4B, and 8B parameters - based on pre-trained VLMs: NVIDIA Eagle 2 with Llama 3.2 3B backbone, Qwen3-VL-4B-Instruct and Qwen3-VL-8B-Instruct, respectively. The 8B model ranks first on the ViDoRe V3 leaderboard as of February 03, 2026, achieving an average NDCG@10 of 63.42. We describe the main techniques used across data processing, training, and post-training - such as cluster-based sampling, hard-negative mining, bidirectional attention, late interaction, and model merging - that helped us build our top-performing models. We also discuss compute and storage engineering challenges posed by the late interaction mechanism and present experiments on how to balance accuracy and storage with lower dimension embeddings.

1. Introduction

Visual document retrieval addresses information loss from OCR-based document preprocessing by retrieving pages directly from images, while Nemotron ColEmbed V2 combines VLMs, late interaction, and contrastive training to achieve leading benchmark performance.

  • Visual document retrieval preserves tables, charts, and infographics by retrieving document pages directly from images without text extraction.
  • ViDoRe V3 evaluates visual document retrieval in complex real-world settings involving multi-type and multilingual queries across ten professional domains.
  • 63.42 NDCG@10 places nemotron-colembed-vl-8b-v2 first on the ViDoRe V3 leaderboard, 3% above second place as of February 03, 2026.
  • The models use VLM backbones, bidirectional attention, contrastive late-interaction training, curated text-only and text-image data, hard-negative mining, and model merging.
  • The released family includes 3B, 4B, and 8B models, with the 8B model first and the smaller models among the leaderboard’s top six for their sizes.

2. Background

The background motivates VLM-based late interaction as a middle ground between efficient bi-encoders and expressive but expensive cross-encoders, trained with contrastive objectives for fine-grained retrieval.

  • 2.1. Visual Document Retrieval: VLM-based late interaction combines pretrained visual-language backbones with multi-vector matching for visually rich documents.
  • 2.2. Dense Retrieval: Bi-encoders efficiently index single vectors but can miss fine-grained lexical, visual, or layout cues, whereas cross-encoders model such alignments at impractical corpus-scale cost.
  • 2.3. Late Interaction: Late interaction retains precomputable document embeddings while enabling query-token and document-token matching through MaxSim.
  • 2.3. Late Interaction: Late interaction improves expressiveness over pooling but introduces latency and storage overhead, motivating dimensionality-reduction studies for scalability.
  • 2.4. Contrastive Learning: Contrastive learning increases similarity for query-positive pairs and decreases similarity to negatives using an InfoNCE objective with a temperature parameter and similarity function.

3. Nemotron ColEmbed V2: a Family of Multimodal Late Interaction Models for Visual Document Retrieval

This section introduces the Nemotron ColEmbed V2 model family and points to a tabulated overview of its architectures.

  • Nemotron ColEmbed V2 is presented as a family of multimodal late-interaction models for visual document retrieval.
  • Table 1 provides the overview of the Nemotron ColEmbed V2 family.

3.1. llama-nemotron-colembed-vl-3b-v2 Architecture

The 3B model builds on Eagle 2 with dynamic image tiling and curated multimodal design choices, using different tile budgets for memory-efficient training and finer-grained inference.

  • Dynamic tiling supports variable-resolution images, and each tile produces 256 visual tokens for late-interaction processing.
  • The v2 model is trained with 50% more data than v1, including 115k augmented queries translated into different languages.
  • The 3B model uses an internal Eagle 2 VLM with SigLIP 2 image encoding and a Llama 3.2 3B language backbone.
  • Training uses max_input_tiles = 2 for memory efficiency because increasing it to 4 did not improve performance, while inference uses max_input_tiles = 8 for finer visual granularity.
  • Figure 2 illustrates image tiling, visual-token encoding, and late-interaction scoring in the 3B architecture.

3.2. nemotron-colembed-vl-4/8b-v2 Architectures

The 4B and 8B models use Qwen3-VL backbones with multimodal, long-context processing and architecture designed to preserve detailed visual information.

  • 3.2. nemotron-colembed-vl-4/8b-v2 Architectures: Qwen3-VL 4B and 8B provide the backbones for Nemotron ColEmbed V2 late-interaction models, supporting multimodal inputs and long context.Their architecture combines a SigLIP-2 vision encoder, a two-layer MLP vision–language merger, and Qwen3 LLMs.
  • 3.2. nemotron-colembed-vl-4/8b-v2 Architectures: Dynamic native-resolution image processing maps pages to variable-length visual token sequences.This design accommodates images with differing resolutions while retaining visual-token detail.
  • 3.2. nemotron-colembed-vl-4/8b-v2 Architectures: DeepStack injects intermediate-layer visual tokens into multiple LLM layers to enhance perceptual capability and preserve visual details.

3.3. Training data and Hyperparameters

ColEmbed V2 training uses a 500,000-sample blend from public visual-retrieval datasets, with additional sampling and translation techniques described for data preparation.

  • 3.3. Training data and Hyperparameters: 500,000 training samples were drawn from public datasets including Vidore-ColPali-Training, Wiki-SS-NQ, DocMatix-IR, VDR5, and VisRAG.
  • 3.3. Training data and Hyperparameters: The data pipeline uses hard-negative mining, cluster-based sampling, cross-lingual query translation, and two-stage training.These techniques are described in Section 3.4.
  • 3.3. Training data and Hyperparameters: Training used one epoch, learning rate 2e-6, AdamW, weight decay 0.1, batch size and gradient accumulation of 1, and two negatives per sample.
  • 3.3. Training data and Hyperparameters: LoRA reduced training time from 3h40m to 1h40m but produced slightly lower accuracy in experiments.

3.4. Key Methods for Better Performance

Nemotron ColEmbed V2 combines bidirectional attention, hard-negative filtering, balanced and multilingual data construction, architecture-specific training stages, and model merging to improve visual document retrieval.

  • 3.4.1. Modifying LLM decoder causal attention to bi-direction attention for encoders: Bidirectional attention lets each token use preceding and succeeding context, and the authors observed substantial retrieval improvement after applying it to Eagle 2 and Qwen3-VL.Decoder models otherwise use causal attention that blocks access to following tokens during prediction.
  • 3.4.2. Hard-Negative Mining: Hard-negative mining selects challenging page-image negatives while filtering likely false negatives with a 0.95 similarity threshold relative to the positive.An internal Llama-Eagle 3B VLM retrieves the top-k similar page images, and candidates below 95% of query–positive similarity are retained.
  • 3.4.3. Cluster-based Data Sampling: Uniform sampling from 14 clusters balances domains after page-image embeddings are reduced from 3072 to 50 dimensions with PCA before K-Means clustering.Positive contexts are clustered, then sampled together with associated queries and negatives to mitigate dataset imbalance.
  • 3.4.4. Cross-lingual Translation: Queries sampled from each cluster are translated into other languages with Qwen3-235B-A22 to support cross-lingual retrieval.This addresses multilingual evaluation in which English or French document corpora are queried in six languages.
  • 3.4.5. Two-stage Training: The 3B model uses two-stage textual then image-retrieval training, whereas the 4B and 8B models use single-stage image-corpus contrastive training because Qwen3-VL has strong cross-modal pre-training.
  • 3.4.6. Model Merging: Weighted model merging combines eight 3B models and four models each for 4B and 8B, varying training blends and dataloader seeds.
  • 3.4.6. Model Merging: Model merging improved accuracy by 0.8% for 3B, 1.0% for 4B, and 1.5% for 8B relative to the best individual ensemble model.The authors report that gains might scale with model size.

4. Results

Nemotron ColEmbed V2 models perform strongly across ViDoRe benchmarks, with the 8B model leading ViDoRe V3 and ranking among the top models on earlier and multilingual evaluations.

  • 4.1. Vidore V3 Leaderboard: ViDoRe V3 evaluates visual document retrieval on complex enterprise scenarios using eight public and two private tasks.The benchmark covers multi-type and multilingual queries across ten professional domains, with private-task evaluation run by MTEB maintainers.
  • 4.1. Vidore V3 Leaderboard: The 8B model ranks first on ViDoRe V3 with an NDCG@10 of 63.42, improving 3% over second place.The 4B and 3B models achieve the highest average NDCG@10 among models of their respective sizes.
  • 4.2. Vidore V1&V2 Leaderboard: On ViDoRe V1 and V2, the 8B model places second while the 3B and 4B models rank among the top four.Table 3 reports retrieval accuracy using NDCG@5 and integrates both benchmarks because ViDoRe V1 provided public in-domain training data.
  • 4.3. MIRACL-Vision benchmark: On MIRACL-Vision, Nemotron ColEmbed V2 models outperform the analyzed models, with the 8B model scoring highest for most languages.The benchmark covers popular and under-resourced languages, and the authors attribute performance to backbone pre-training and cross-lingual training examples.

5. Late-interaction Deployment Challenges

Late interaction improves visual-document retrieval accuracy but substantially increases storage and can add serving latency. Deployment therefore requires balancing retrieval quality against model size, storage, latency, and corpus-specific constraints, with dimensionality reduction offering partial relief.

  • 5.1. Accuracy Considerations: Late interaction raises ViDoRe V3 NDCG@10 by about 12% for ColEmbed 4B and 9% for ColEmbed 8B versus average pooling under matched training conditions.Table 5 compares both methods using the same architecture, training data, and hyperparameters.
  • 5.2. Retrieval Systems Considerations: Production retrieval must balance model size, embedding storage, serving latency, and reranking cost rather than optimize accuracy alone.Document embedding generation is affected mainly by model size, storage by dimensions, precision, and vector count, and reranking improves accuracy at added serving latency.
  • 5.3. Retrieval Pipelines Trade-offs: Late interaction requires much more indexing storage because it preserves multiple token embeddings; its footprint depends on sequence length, dimension, and numerical precision.Table 6 compares model size, embedding dimension, storage, and ViDoRe V3 NDCG@10 across retrieval approaches.
  • 5.3. Retrieval Pipelines Trade-offs: 8,789.1 GB versus 3.8 GB is required for late-interaction versus single-vector embeddings for one million pages, a 2312x difference.The compared models share a Llama 3.2 1B backbone, but the paper notes that training blends and image encoders also differ.
  • 5.3. Retrieval Pipelines Trade-offs: Reranking the top 50 pages raises NDCG@10 from 48.69 to 54.40, close to 55.48 from late interaction while using a small fraction of the storage.The single-vector-plus-cross-encoder pipeline avoids comparing query vectors with the whole corpus during late interaction.
  • 5.3. Retrieval Pipelines Trade-offs: Late-interaction latency comes from MaxSim comparisons against multi-vectors across the corpus and may require specialized vector-database support.A single-vector retriever followed by a cross-encoder limits early interaction to top-k candidates and therefore offers lower latency.
  • 5.4. Ablation on Embedding-size Reduction: Pipeline choice depends on use-case constraints, and smaller embeddings or lower precision are proposed ways to reduce storage without substantially sacrificing retrieval accuracy.The paper also notes that binary quantization may perform poorly at small embedding dimensions and needs further testing for late interaction.
  • 5.4. Ablation on Embedding-size Reduction: Reducing ColEmbed 8B embeddings to 512 dimensions cuts storage by 87.5% while retaining 96.02% of retrieval accuracy, and 128 dimensions retain 95.36% using 3% of storage.For 1M pages, even the 128-dimensional setting requires 184.3 GB, which may remain too high for large production corpora.

6. Conclusion

Nemotron ColEmbed V2 introduces late-interaction models for visual document retrieval and reports top performance on ViDoRe alongside multilingual evaluation. The paper highlights deployment trade-offs and recommends further work to reduce storage without materially reducing retrieval accuracy.

  • 6. Conclusion: Nemotron ColEmbed V2 achieves top performance on ViDoRe and demonstrates multilingual capabilities on MIRACL-Vision.The paper attributes accuracy gains to bidirectional attention, positive-aware hard-negative mining, cluster-based sampling, cross-lingual translation, and model merging.
  • 6. Conclusion: The release provides late-interaction models intended as a foundation for future visual document retrieval research and practical applications.The conclusion also discusses deployment challenges and quantifies accuracy-storage trade-offs through embedding-size ablations.
  • 6. Conclusion: Further research should reduce late-interaction storage through smaller dimensions, lower numerical precision, or fewer embeddings without significantly penalizing retrieval accuracy.This recommendation directly follows the paper’s deployment analysis of storage requirements and accuracy preservation.
Loading 2602.03992v2…