PaperlayerView exact revision on arXiv

Source-linked AI summary

M3DR: Towards Universal Multilingual Multimodal Document Retrieval

Adithya S Kolavi, Vyoman Jain

arXiv:2512.03514v1cs.IRcs.AIcs.CLcs.CV

TL;DR

Vision-based document retrieval remains largely English-centric, leaving retrieval across languages and scripts an open challenge. M3DR combines synthetic multilingual training data with contrastive learning and evaluates dense- and multi-vector models on multilingual retrieval. NetraEmbed reports a 152% relative improvement over ColPali-v1.3 on cross-lingual retrieval while also improving monolingual results.

  • Problem

    Existing vision-based document retrieval systems remain predominantly English-centric, while multilingual retrieval must span diverse languages and scripts.

    Source: §1 · p. 1

  • Method

    M3DR combines synthetic multilingual document data, VLM-generated queries, and contrastive training for dense-vector and ColBERT-style retrieval models.

    Source: §1 · p. 2, §3 · p. 3, §4.2.2 · p. 5

  • Results

    NetraEmbed improves over ColPali-v1.3 by 152% on cross-lingual retrieval (0.716 vs 0.284 NDCG@5) and 80% on monolingual tasks (0.738 vs 0.410 NDCG@5).

    Source: §6 · p. 8

  • Takeaways & Limitations

    M3DR demonstrates multilingual multimodal retrieval across 22 diverse languages while maintaining competitive English performance at 0.554 NDCG@5 on ViDoRe v2.

    Source: §6 · p. 8

    Rare language pairs show 10–12% degradation compared with high-resource pairs, and the evaluation is limited to document-level retrieval.

    Source: §6 · p. 8

Abstract

from arXiv · show

Multimodal document retrieval systems have shown strong progress in aligning visual and textual content for semantic search. However, most existing approaches remain heavily English-centric, limiting their effectiveness in multilingual contexts. In this work, we present M3DR (Multilingual Multimodal Document Retrieval), a framework designed to bridge this gap across languages, enabling applicability across diverse linguistic and cultural contexts. M3DR leverages synthetic multilingual document data and generalizes across different vision-language architectures and model sizes, enabling robust cross-lingual and cross-modal alignment. Using contrastive training, our models learn unified representations for text and document images that transfer effectively across languages. We validate this capability on 22 typologically diverse languages, demonstrating consistent performance and adaptability across linguistic and script variations. We further introduce a comprehensive benchmark that captures real-world multilingual scenarios, evaluating models under monolingual, multilingual, and mixed-language settings. M3DR generalizes across both single dense vector and ColBERT-style token-level multi-vector retrieval paradigms. Our models, NetraEmbed and ColNetraEmbed achieve state-of-the-art performance with ~150% relative improvements on cross-lingual retrieval.

1. Introduction

Vision-based retrieval preserves document visuals but remains English-centric, leaving multilingual retrieval unresolved. M3DR targets that gap across 22 languages and reports a 152% relative improvement over baselines on cross-lingual retrieval.

1. Introduction

  • Existing OCR pipelines can lose visual information, struggle with diverse fonts and scripts, and compound errors, especially for low-resource languages.Vision-based systems avoid OCR dependencies and preserve visual-textual information, but remain predominantly English-centric and perform poorly on multilingual content.Source: §1 · p. 1
  • The motivating question is whether universal document retrievers can maintain high performance across diverse languages without sacrificing English competitiveness.Source: §1 · p. 1
  • 152% relative improvement over baselines on cross-lingual retrieval (0.716 vs 0.284 NDCG@5) establishes state-of-the-art performance for multilingual multimodal document retrieval.The model generalizes across dense-vector and ColBERT-style multi-vector retrieval paradigms.Source: §1 · p. 1
  • M3DR trains multilingual multimodal document retrievers across 22 typologically diverse languages and script families.Its contributions include a synthetic data pipeline, the Nayana IR Bench with 23 datasets, and two 4B-parameter models.Source: §1 · pp. 1–2

2. Related Work

Prior work spans visual document understanding, vision-based retrieval, multimodal retrieval, and multilingual embeddings, but the cited approaches leave multilingual visual document retrieval incompletely addressed.

2. Related Work

  • Vision-language document understanding has advanced OCR-free processing, but the cited work primarily targets question answering and evaluates in English.Source: §2 · p. 2
  • Vision-based retrievers show strong English performance, while the cited methods do not systematically address multilingual scenarios.M3DR extends vision-based retrieval to 22 languages and evaluates transfer across linguistic and script boundaries.Source: §2 · p. 2
  • M3DR combines document-specific retrieval with multilingual support and builds on contrastive, negative-sampling, and Matryoshka training strategies.Source: §2 · pp. 2–3
  • Text-based multilingual embeddings transfer across languages but discard visual information; multimodal multilingual work includes separate text and image encoders and approaches with limited language coverage.The related-work discussion identifies fine-grained visual-textual interaction as a distinction of vision-based approaches.Source: §2 · p. 3

3. Training Data and Benchmark

The paper addresses scarce multilingual multimodal training data and evaluation with a synthetic-data pipeline and the Nayana-IR benchmark. The benchmark covers cross-lingual and monolingual retrieval across 22 languages.

3. Training Data and Benchmark

  • M3DR addresses limited training data and benchmarks through synthetic multilingual corpora, VLM-based query synthesis, and Nayana-IR standardized evaluation.Source: §3 · p. 3

3.1.1. Synthetic Parallel Corpus Generation

  • The corpus pipeline starts from 50,000 English document images and uses layout-aware translation and authentic script-specific rendering to create parallel multilingual material.It extracts text regions, visual elements, tables, and layout metadata, translates into 22 languages, and renders at 1024–2048px height.Source: §3.1.1 · p. 3, Figure 2 · p. 4

3.1.2. Query Synthesis

  • VLMs generate five query types per document, producing a training corpus of 1M document images with corresponding queries across 22 languages.The query mix is 70% questions, 15% reverse-lookup answers, and 15% text snippets.Source: §3.1.2 · p. 3, Figure 2 · p. 4

3.2. Nayana-IR Benchmark

  • Nayana-IR introduces 23 datasets for multilingual multimodal retrieval, with cross-lingual and monolingual evaluation protocols.The benchmark is designed to be separate from training data.Source: §3.2 · p. 3

3.2.1. Benchmark Structure

  • The benchmark comprises 5,870 parallel cross-lingual documents and 1,000 queries, plus 22 monolingual datasets; its total is 27,870 images and 5,400 queries.It uses BEIR-compatible formatting and reports NDCG, Recall, MAP, and MRR metrics across script families.Source: §3.2.1 · pp. 3–4

4. Methodology

M3DR supports dense-vector and token-level multi-vector retrieval, with flexible embedding dimensions and contrastive training. Its training-strategy comparison favors positive-only in-batch negatives over the tested hard-negative alternatives.

4.1.1. Single Dense Vector Model

  • The single dense-vector model pools VLM token states into one normalized embedding, while the multi-vector model retains token embeddings for fine-grained matching.The multi-vector representation uses 256 visual tokens per document image; dense embeddings use 2560 dimensions with Gemma 3 4B-it.Source: §4.1.1 · p. 4, §4.1.2 · pp. 4–5
  • Matryoshka learning trains embeddings for truncation to 768, 1536, or 2560 dimensions, enabling post-deployment storage-accuracy choices without retraining.The text specifies 70% storage reduction at 768 dimensions and 40% at 1536 dimensions.Source: §4.1.1 · p. 4

4.1.2. ColBERT-Style Multi-Vector Model

  • The ColBERT-style model uses MaxSim late interaction so each query token can match its strongest document token.It retains per-token representations rather than pooling them into one vector.Source: §4.1.2 · pp. 4–5

4.2.1. Loss Functions for Single Dense Vector Models

  • Single-vector training uses InfoNCE with in-batch negatives and can combine pairwise ranking with InfoNCE when explicit hard negatives are available.The hybrid loss uses λ = 0.5 to balance discrimination against hard negatives with in-batch diversity.Source: §4.2.1 · p. 5

4.2.2. Loss Functions for ColBERT-Style Multi-Vector Models

  • The multi-vector model applies temperature-scaled InfoNCE to MaxSim scores and uses in-batch negatives without explicit hard negatives.This training encourages query tokens to match relevant document regions while distinguishing other documents in the batch.Source: §4.2.2 · p. 5

4.3. Training Strategy

  • Positive-only training with in-batch negatives substantially outperforms document-level negatives and mined hard negatives throughout training.The alternatives use nearby pages or negatives mined using text and visual similarity.Source: Figure 3 · p. 5, §4.3 · p. 5

5. Experiments

M3DR performs strongly across multilingual retrieval tasks and languages, while supporting storage–accuracy trade-offs and both dense- and multi-vector retrieval. NetraEmbed leads the reported multilingual comparisons, with substantial storage and compute advantages over ColBERT-style retrieval.

5.1. Experimental Setup

  • The experiments compare two 4B-parameter Gemma 3 4B-IT models trained on 22 languages: dense-vector NetraEmbed and ColBERT-style multi-vector ColNetraEmbed.The baseline suite includes vision-language retrieval models and multimodal embedding models.Source: §5.1 · p. 6

5.2. Main Results

  • NetraEmbed scores 0.554 NDCG@5 on ViDoRe v2, while existing VLM-based baselines often score 0.00–0.32 NDCG@5 on cross-lingual tasks.The evaluation uses NDCG@5 as its primary metric across Nayana-IR and ViDoRe v2.Source: §5.2 · p. 6, §5.1 · p. 6

5.3. Per-Language Performance Analysis

  • NetraEmbed performs consistently across 22 languages and script families; English-centric baselines show significant drops on non-English content.The reported scaling analysis also finds progressive improvements from 6 to 22 languages.Source: §5.3 · p. 6, Figure 4 · p. 7

5.4. Matryoshka Embeddings: Efficiency-Accuracy Trade-offs

  • At 768 dimensions, NetraEmbed retains 95.0% of full-model performance (0.680 vs 0.716 NDCG@5) while reducing storage by 70% (∼3 KB vs ∼10 KB per document).The 1536-dimensional representation retains 98.6% performance with 40% storage reduction.Source: §5.4 · p. 6

5.5. Single Dense Vector vs. ColBERT-Style Multi-Vector Retrieval

  • NetraEmbed exceeds ColNetraEmbed by 12.4% on cross-lingual NDCG@5 (0.716 vs 0.637) and 10.1% on monolingual tasks (0.738 vs 0.670).Both models achieve similar ViDoRe v2 performance (0.554 vs 0.551).Source: §5.5 · p. 6
  • Single dense vectors use ∼10 KB per document versus ∼2.5 MB for 256-token ColBERT-style multi-vectors, a 250× storage reduction.The section also reports HNSW cosine similarity instead of MaxSim over token matrices and efficient scaling to billion-document corpora.Source: §5.5 · p. 6

5.2. Main Results

  • 0.716 NDCG@5 cross-lingual and 0.738 monolingual retrieval: NetraEmbed improves 152% and 80% over ColPali-v1.3 (0.284 and 0.410); ColNetraEmbed scores 0.637 and 0.670, respectively.Table 1 reports the main multilingual and English benchmark results.Source: §5.2 · p. 6, Table 1 · p. 7

6. Conclusion

M3DR targets multilingual vision-based document retrieval with a framework evaluated across 22 diverse languages. NetraEmbed reports large multilingual gains while retaining competitive English performance, alongside released research resources and identified limitations.

6. Conclusion

  • M3DR addresses multilingual vision-based retrieval across 22 typologically diverse languages, including Latin, Devanagari, Dravidian, CJK, and Arabic scripts.The framework is evaluated in an appendix ablation program that scales from 45k images across 6 languages to approximately 250k image-text pairs across 22 languages.Source: §6 · p. 8, A. Overview · p. 12
  • NetraEmbed reaches 0.716 NDCG@5 cross-lingual and 0.738 monolingual, improving 152% and 80% over ColPali-v1.3 (0.284 and 0.410), while scoring 0.554 NDCG@5 on ViDoRe v2.The conclusion reports these results as gains in multilingual retrieval without sacrificing monolingual competitiveness.Source: §6 · p. 8
  • The authors identify base-model choice, in-batch-negative BiEncoderLoss, last-token pooling, and Matryoshka truncation as key design insights.Last-token pooling outperforms mean pooling by 13+ NDCG@5 points for decoder-based VLMs; 768-dimensional truncation retains 95% performance with 70% storage reduction.Source: §6 · p. 8
  • The released Nayana-IR benchmark contains 23 datasets, ∼28K images, and ∼5.4K queries; models, code, and evaluation scripts are also available.The reported LoRA fine-tuning run for SOTA results uses 4×A100 GPUs for ∼12 hours.Source: §6 · p. 8
  • The authors describe applications in multilingual enterprise knowledge management, cross-border research, educational resource discovery, and cultural heritage preservation.They recommend deployment monitoring and ongoing evaluation and mitigation of language performance disparities.Source: §6 · p. 8
  • The authors state that rare language pairs show 10–12% degradation versus high-resource pairs and that complex tables with language-specific number formats remain challenging.They also identify passage- or region-level retrieval and scaling beyond 22 languages, including zero-shot transfer to unseen scripts, as future work.Source: §6 · p. 8

A. Overview

  • The appendix overview describes ablations of model choice, loss, pooling, embedding dimensions, model merging, retrieval architecture, and cross-lingual scaling.Experiments use ViDoRe v2, NayanaIR-CrossBench, and the 22-dataset monolingual benchmark, with NDCG@5, Recall@10, MAP@10, and MRR@10.Source: A. Overview · p. 12

B. Base Model Selection

Base-model comparisons expose a sharp divide between English-centric ViDoRe performance and cross-lingual retrieval, motivating Gemma3 as the foundation. Ablations favor simple in-batch objectives, last-token pooling for dense models, and general-purpose VLMs over OCR-pretrained models.

B. Base Model Selection

  • 59.20% ViDoRe NDCG@5 for colqwen2.5v0.2 fell to 14.26% on cross-lingual tasks, a 45-point drop that motivated selecting Gemma3 4B for its multilingual transfer.The comparison spans single-vector dense and multi-vector Col retrieval architectures.Source: B. Base Model Selection · p. 12, Figure 5 · p. 13

C. Preliminary Ablations on 6 Languages

  • On the 6-language subset of ∼45k image-text pairs, the study compared dense and Col loss functions, OCR models, and other retrieval choices before full-scale training.The dense candidates included BiEncoderLoss and three cross-entropy variants; Col candidates were standard and pairwise ColBERT loss.Source: C. Preliminary Ablations on 6 Languages · p. 13, §C.1 · p. 13

C.1.1. Dense Model Loss Functions

  • Hard negative mining made dense-model training erratic: NDCG@5 fluctuated between 46.73% and 50.48%, while Recall@10 peaked at 62.03% before declining.Its 1–3 point peak gains required careful checkpoint selection, unlike stable BiEncoderLoss convergence.Source: §C.1.1 · p. 14, Figure 6 · p. 14, Table 4 · p. 14

C.1.2. Col Model Loss Functions

  • 56.41% vs 39.53% ViDoRe NDCG@5: standard ColBERT loss outperformed pairwise loss by 17 points, while BiEncoderLoss was optimal for dense models.The datasets lacked explicitly annotated hard negatives, making in-batch negatives sufficient for discriminative learning.Source: Table 5 · p. 14, §C.1.2 · p. 15, Figure 7 · p. 15

C.2. Pooling Strategy Ablations: Last Token vs Mean Pooling

  • Last-token pooling reached 49.31% ViDoRe NDCG@5 versus 35.85% for mean pooling, a 13.5-point advantage adopted as the dense-model default.The authors also report lower inference cost because only one token representation is processed.Source: §C.2 · p. 15, Table 6 · p. 15, Figure 8 · p. 16

C.3. OCR Model Ablations

  • 0.97% NDCG@5 for pretrained Granite-Docling rose only to 4.17% after finetuning, and the base Gemma 3 4B matched that finetuned result.The authors attribute the mismatch to OCR pretraining's focus on transcription rather than semantic retrieval.Source: §C.3 · p. 16, Table 7 · p. 16

C.4. Training on 6 Languages: Initial Results

On six languages, dense Gemma3 models outperformed late-interaction models on cross-lingual retrieval, prompting the authors to pursue Gemma3-based dense and Col models. A six-language Matryoshka model incurred some benchmark degradation, leading to a test of whether broader language scaling could offset it.

C.4. Training on 6 Languages: Initial Results

  • 60.39% vs 42.22% NDCG@5: dense Gemma3 exceeded the best late-interaction model by 18 points on cross-lingual tasks, despite Col models retaining a ViDoRe advantage.The result motivated pursuing single-vector Gemma3 and multi-vector ColGemma3.Source: §C.4 · pp. 16–17, Table 8 · p. 16, Figure 9 · p. 17

D. Matryoshka Embedding Ablations

D.1. Matryoshka on 6 Languages

  • The six-language Matryoshka model lost 3 points on ViDoRe and cross-benchmarks but improved slightly on monolingual tasks, motivating a test at larger language scale.Source: §D.1 · p. 17, Table 9 · p. 17

D.2. Matryoshka on 22 Languages (Final Scale)

At full scale, Matryoshka embeddings retained most cross-lingual performance at reduced dimensions, while scaling dense training from six to 22 languages improved benchmark results. Model merging balanced ViDoRe and cross-lingual performance, and ColGemma3 strengthened cross-lingual results among multi-vector models.

D.2. Matryoshka on 22 Languages (Final Scale)

  • 73.15% vs 77.31% cross-lingual NDCG@5: 768-dimensional embeddings retained 94.6% of full performance with 70% less storage; 1536 dimensions retained 99.4%.Both primary and v2 training runs showed consistent behavior across dimensions.Source: §D.2 · pp. 17–18, Figure 10 · p. 18, Table 10 · p. 18

E. Model Merging Strategies

  • Merged models occupied a balanced middle ground: SLERP achieved 53.85% ViDoRe and 74.91% cross-lingual NDCG@5, while LinearMerge achieved 54.06% and 73.46%.The merges required no additional training; SLERP better preserved cross-lingual performance, while LinearMerge better retained ViDoRe accuracy.Source: E. Model Merging Strategies · p. 18, Figure 11 · p. 19, Table 11 · p. 19

F.1. Dense Models at Full Scale

  • Scaling dense training from 6 to 22 languages improved ViDoRe by 6.3 points, cross-lingual results by 11.2 points, and monolingual results by 11.3 points.The reported relative gains were larger on cross-lingual and monolingual benchmarks than on ViDoRe.Source: Table 12 · p. 19, Figure 12 · p. 20, §F.1 · p. 20

F.2. Multi-Vector Col Models at Full Scale

  • 63.75% cross-lingual NDCG@5 for ColGemma3 exceeded finetuned ColPali at 42.64% and ColQwen2 at 42.86%.Its three stages trained on ColPali data, finetuned on 22-language Nayana IR, and merged checkpoints.Source: Table 13 · p. 20, §F.2 · p. 20

G. Dense vs Col Architecture Comparison

Single-vector dense models led late interaction on cross-lingual accuracy and deployment efficiency, while late interaction showed slight ViDoRe and interpretability advantages. PCA visualizations trace multilingual embeddings from language-separated clusters toward semantic alignment across two, six, and 15 languages.

G. Dense vs Col Architecture Comparison

H. Cross-Lingual Embedding Convergence: PCA Analysis

  • The PCA analysis tracks whether embeddings for semantically equivalent content across languages move from separated clusters to aligned representations during Nayana IR training.Source: H. Cross-Lingual Embedding… · p. 21

H.1. Methodology

  • Checkpoints at 500 through 5066 steps were embedded in 2560 dimensions, projected to 2D with PCA, and compared across 2-, 6-, and 15-language configurations.Source: §H.1 · p. 21

H.2. 2-Language Alignment: Kannada ↔English

  • For Kannada queries and English documents, the PCA clusters largely intermixed by checkpoint 5066 after beginning in separate regions at checkpoint 500.Source: §H.2 · p. 21, Figure 13 · p. 22

H.3. 6-Language Cross-Lingual Alignment

  • Across six languages and scripts, language-specific clusters progressively merged, with near-complete cross-lingual mixing by checkpoint 5066.Source: Figure 14 · p. 22, §H.3 · p. 22
  • Matryoshka embeddings at 2560 dimensions showed the same progression from language-clustered distributions to semantically organized representations.The authors conclude that flexible dimensionality did not compromise cross-lingual alignment in this visualization.Source: §H.3 · p. 22, Figure 15 · p. 23

H.4. 15-Language Scaling

  • In the 15-language visualization, dispersed early clusters progressively converged, showing strong cross-lingual mixing by checkpoint 5066 despite diverse language families and writing systems.Source: §H.4 · pp. 22–23, Figure 16 · p. 23

H.5. Hindi ↔Kannada Detailed Analysis

Hindi and Kannada embeddings progressively align despite their different scripts. The visualizations also indicate shared semantic representations and consistent visual grounding across language queries.

H.5. Hindi ↔Kannada Detailed Analysis

  • By checkpoint 5066, Hindi and Kannada document embeddings are nearly intermixed after separating at checkpoint 500, indicating alignment across Devanagari and Kannada scripts.The passage describes gradual overlap at checkpoints 1500 and 2500 before near-perfect alignment at checkpoint 5066.Source: §H.5 · p. 23, Figure 17 · p. 24

H.6. Implications for Cross-Lingual Retrieval

  • Nayana IR training gradually collapses language-specific clusters into a shared, semantically organized single-vector space, supporting retrieval across scripts and language families.The passage attributes the gradual convergence to multilingual training with Nayana IR and describes the dataset as containing sufficient cross-lingual signal.Source: §H.6 · pp. 23–24

I. Col Model Attention Visualization

  • Col models’ MaxSim attention maps show cross-lingual text queries attending to the same visual regions, indicating language-agnostic visual grounding.Source: §I · p. 24

I.1. Experimental Setup

The experiments compare English-centric and multilingual-fine-tuned Col models using a scientific term queried in three languages. They report greater cross-lingual query consistency after multilingual fine-tuning, alongside the training setup and data coverage.

I.1. Experimental Setup

  • The consistency test queries the English scientific term “Acremonium coephenophialum” in English, Hindi, and Kannada, comparing English-centric ColPali models with Nayana-fine-tuned models.Source: §I.1 · p. 24
  • 0.109 Kannada similarity for ColGemma3-ColPaliOnly contrasts with 0.381 Hindi versus 0.335 English for ColGemma3-Finetuned, whose queries activate the same visual regions.The base-model Kannada query fails to activate the correct region, whereas fine-tuned models show consistent activation across the three languages.Source: §I.1 · p. 24, Figure 20 · p. 25, Figure 21 · p. 26

J.1. LoRA Setup

  • LoRA used rank 32, alpha 32, and dropout 0.1 on projection layers.The listed layers include down, gate, up, k, q, v, and o projections.Source: §J.1 · p. 26

J.2. Hyperparameters

  • Two epochs gave the best results; training used batch size 32, learning rate 2e-4, and 100 warmup steps.Source: §J.2 · p. 26

J.3. Compute

  • Final multilingual training used 4 to 8 A100 80 GB GPUs, taking 6 to 8 hours in total, or about 64 GPU hours.The passage reports mixed precision and DDP for final training.Source: §J.3 · p. 26

J.4. Data

  • Nayana IR spans 11 Indic scripts, 6 major European languages, 4 Asian languages, and Arabic, supporting multilingual and cross-lingual retrieval tasks.Source: §J.4 · p. 26
Loading 2512.03514v1…