PaperlayerView exact revision on arXiv

Source-linked AI summary

NanoVDR: Distilling a 2B Vision-Language Retriever into a 70M Text-Only Encoder for Visual Document Retrieval

Zhuchenyang Liu, Yao Zhang, Yu Xiao

arXiv:2603.12824v2cs.IRcs.CVcs.LG

TL;DR

VLM retrievers impose heavy query-time costs because they use the same large encoder for visually complex documents and plain-text queries. NanoVDR decouples these paths through a frozen VLM teacher and a distilled text-only student, with pointwise alignment and translated-query augmentation producing an efficient 69M-parameter model that retains 95.1% of teacher quality.

  • Problem

    VDR systems use heavyweight VLMs for both document indexing and plain-text query encoding, creating high query-time computation and GPU dependence.

  • Method

    NanoVDR distills a frozen 2B VLM’s query representations into text-only student encoders using pointwise cosine alignment and cached teacher query embeddings.

  • Results

    NanoVDR-S-Multi retains 95.1% of teacher quality and outperforms DSE-Qwen2 (2B) on ViDoRe v2 and v3 with 32× fewer parameters and 50× lower CPU query latency.

  • Takeaways & Limitations

    Translated-query augmentation addresses cross-lingual transfer as the primary bottleneck without additional document processing during training.

  • Takeaways & Limitations

    The student inherits the teacher’s quality ceiling, while offline indexing still requires the full 2B VLM and generalization beyond text-only VDR queries remains untested.

Abstract

from arXiv · show

Vision-Language Model (VLM) based retrievers have advanced visual document retrieval (VDR) to impressive quality. They require the same multi-billion parameter encoder for both document indexing and query encoding, incurring high latency and GPU dependence even for plain-text queries. We observe that this design is unnecessarily symmetric: documents are visually complex and demand strong visual understanding, whereas queries are just short text strings. NanoVDR exploits this query--document asymmetry by decoupling the two encoding paths: a frozen 2B VLM teacher indexes documents offline, while a distilled text-only student as small as 69M parameters encodes queries at inference. The key design choice is the distillation objective. Through systematic comparison of six objectives across three backbones and 22 ViDoRe benchmark datasets, we find that pointwise cosine alignment on query text consistently outperforms ranking-based and contrastive alternatives, while requiring only pre-cached teacher query embeddings and no document processing during training. Furthermore, we identify cross-lingual transfer as the primary performance bottleneck, and resolve it cheaply by augmenting training data with machine-translated queries. The resulting NanoVDR-S-Multi (DistilBERT, 69M) retains 95.1\% of teacher quality and outperforms DSE-Qwen2 (2B) on v2 and v3 with 32$\times$ fewer parameters and 50$\times$ lower CPU query latency, at a total training cost under 13 GPU-hours.

1 Introduction

NanoVDR addresses the unnecessary symmetry of VDR by using a frozen VLM for offline document indexing and a lightweight text-only student for online queries. Pointwise cosine distillation and multilingual query augmentation deliver strong quality with substantially lower resource demands.

  • Motivation: Current VDR systems use the same multi-billion-parameter VLM for document indexing and text-only query encoding, causing high query-time overhead.These systems require GPU inference even for plain-text queries and incur large index storage costs.
  • Approach: NanoVDR decouples retrieval into offline VLM document encoding and online text-only student query encoding.The student maps queries into the teacher’s embedding space without a vision module and supports single-vector cosine retrieval.
  • Distillation design: Pointwise cosine alignment directly matches student and teacher query embeddings and avoids document processing during training.The framework pre-caches teacher query embeddings instead of requiring corpus-related processing or negative sampling.
  • Efficiency: 32× fewer parameters and 50× lower query latency let NanoVDR-S outperform DSE-Qwen2 on ViDoRe v2 and v3.The model has 69M parameters and costs under 13 GPU-hours to train.
  • Cross-lingual transfer: 95.1% teacher retention is achieved by multilingual query augmentation, improving on the 92.4% retention of NanoVDR-S.The paper identifies cross-lingual transfer, rather than cross-modal transfer, as the primary bottleneck.

2 Related Work

Prior VDR systems improve visual retrieval through multi-vector, single-vector, and vision-native architectures, but generally retain substantial vision-language computation. NanoVDR instead distills a VLM into a text-only query encoder through pure spatial alignment.

  • Existing VDR architectures: ColPali uses token-level late interaction over hundreds of embeddings per document page, while DSE produces one embedding per screenshot.Both approaches use VLM-based visual document representations for retrieval.
  • Efficiency-oriented alternatives: VISTA and ModernVBERT reduce model scale but still require vision components for multimodal retrieval or both query and document encoding.VISTA has not been evaluated on document-level benchmarks, while ModernVBERT fuses a SigLIP2 vision backbone.
  • Description-based retrieval: SERVAL generates textual descriptions of document images before indexing them with a text encoder, but still requires massive VLM inference for every document.Its reported pipeline combines a 72B VLM with a 7B encoder.
  • Distillation: NanoVDR performs cross-modal distillation from a VLM teacher to a text-only student, unlike prior distillation approaches that retain image or multimodal processing.Pure spatial alignment eliminates document representations during training.

3 Methodology

NanoVDR freezes a VLM teacher for offline page indexing and trains a text-only student to produce query embeddings in the same space. Query-centric cosine alignment requires only cached teacher query embeddings and supports inexpensive multilingual extension.

  • System overview: The frozen teacher maps each document image to a single embedding, while the student maps each text query into the same d-dimensional space.Retrieval ranks pages using cosine similarity between student query and teacher document embeddings.
  • Student encoder: The student combines a pretrained text backbone, mean pooling, and a two-layer MLP projector.The teacher remains frozen throughout training.
  • Query-centric distillation: The teacher first encodes training queries in text-only mode, after which the student minimizes angular distance to those target embeddings.Matching teacher query embeddings enables retrieval against teacher document embeddings without student image inputs.
  • Training efficiency: Alignment-only training requires cached teacher query embeddings, whereas ranking objectives additionally require cached document embeddings and teacher image processing.This removes document embeddings, negative sampling, and corpus-level processing from the alignment pipeline.
  • Multilingual augmentation: Multilingual extension uses translated query text and teacher text-mode re-encoding rather than new document images or document re-encoding.Approximately 489K English queries are translated into five target languages and balanced to about 200K queries per language.

4 Experimental Setup

The evaluation spans the full 22-dataset ViDoRe benchmark, ten baselines, and three student backbones. Experiments report NDCG@5 alongside model size, scoring method, and implementation cost.

  • Benchmark: The benchmark contains 22 datasets across ViDoRe versions v1, v2, and v3, with NDCG@5 averaged separately by version.The versions contain 10, 4, and 8 datasets respectively.
  • Training data: Training aggregates 726K filtered query-document image pairs from four public sources and retains 711K training pairs after validation holdout.The validation set contains 14.5K pairs selected by stratified sampling.
  • Baselines: The baseline suite spans multi-vector VLMs, single-vector VLMs, and vision-native encoders.Ten baselines contextualize NanoVDR’s efficiency–quality tradeoff.
  • Models: NanoVDR students range from 69M to 151M parameters and use a 2048-dimensional teacher embedding space.The evaluated backbones are DistilBERT, BERT-base, and ModernBERT-base.
  • Metrics and reporting: Table 1 reports NDCG@5 ×100 for each benchmark version together with parameter count, scoring method, and teacher-retention subscripts.NanoVDR uses cosine scoring, while some baselines use token-level MaxSim scoring.
  • Implementation cost: Under 13 GPU-hours is the reported total training cost for the best NanoVDR model, including approximately one GPU-hour for teacher query pre-caching.Single-model training takes 10.1–11.7 hours on one NVIDIA H200 GPU.

5 Main Results

NanoVDR achieves competitive or superior retrieval quality while substantially reducing query-time model size, latency, and deployment cost. Its advantages combine a distilled text-only query encoder with the teacher’s single-vector retrieval architecture.

  • Retrieval performance: 82.2/60.5/43.5 NDCG@5 on v1/v2/v3 lets NanoVDR-S retain 92.4% of teacher performance with 29× fewer parameters.NanoVDR-S-Multi raises retention to 95.1% with scores of 82.2/61.9/46.5.
  • Retrieval performance: On v2 and v3, all NanoVDR variants surpass ColPali and DSE-Qwen2 despite using text-only models with 32× fewer parameters than DSE-Qwen2.NanoVDR-S-Multi reaches 46.5 NDCG@5 on v3, while NanoVDR-M reaches 62.2 on v2.
  • Query-time efficiency: 51 ms CPU query encoding makes NanoVDR-S 50× faster than DSE-Qwen2 and 143× faster than ColPali.Even NanoVDR-L completes query encoding in 109 ms.
  • Query-time efficiency: 274 MB for the NanoVDR-S checkpoint compares with 11.9 GB for ColPali and 35.1 GB for Tomoro-8B, supporting deployment without GPU memory.The reported latency measurements use a single CPU thread with batch size 1.
  • Index efficiency: Single-vector cosine scoring takes 2.5 ms per 10K documents versus 7.1 s for MaxSim, while index storage is 8.2 GB versus 264–819 GB per 1M pages.These scoring and storage advantages are inherited from the teacher’s single-vector architecture.

6 Ablation and Analysis

The ablations show that pure pointwise alignment is the strongest and simplest distillation objective, while cross-lingual coverage—not the modality gap—limits retention on non-English queries. Query-only multilingual augmentation substantially narrows this gap.

  • Distillation objectives: 54 evaluation points compare six loss configurations across three backbones and three benchmarks under identical training settings.The ablation uses 711K pairs and trains each configuration to convergence.
  • Distillation objectives: Pure alignment improves NDCG@5 over pure ranking by +1.1/+4.0/+2.5 on v1/v2/v3, with monotonic gains as alignment weight increases.The trend is consistent in the three-backbone average and across individual backbones.
  • Distillation objectives: InfoNCE decreases NDCG@5 by −10.7/−21.6/−14.1 on v1/v2/v3 relative to alignment, showing the value of preserving continuous embedding geometry.InfoNCE uses hard one-hot labels rather than the teacher’s soft ranking distribution.
  • Data efficiency: At 25% of the 711K training pairs, NanoVDR-S reaches 93%/82%/70% retention on v1/v2/v3, indicating strong diminishing returns from additional data.At 10% of the data, it reaches 79% retention on v1, corresponding to 66.7 NDCG@5.
  • Cross-lingual transfer: Retention follows language coverage: English reaches 94.3%, whereas Portuguese, absent from training, reaches 75.6%; within identical multilingual corpora, the gap is 17.4 percentage points.The correlation between training-data proportion and retention is r=+0.563, although it is not statistically significant at n=6.
  • Cross-lingual transfer: Multilingual query augmentation raises Portuguese by +9.3 NDCG, preserves v1 at 82.2, and improves multilingual v2/v3 by +1.4/+3.0 while narrowing the maximum language gap to 2.7 percentage points.After augmentation, all six languages exceed 92% retention and English shows zero regression.

7 Conclusion

NanoVDR enables efficient visual document retrieval by distilling a VLM embedding space into a text-only query encoder. Pointwise alignment and translated-query augmentation yield strong quality with substantially lower resource requirements.

  • NanoVDR performs cross-modal distillation from a VLM teacher to a text-only student for efficient visual document retrieval.The framework extends encoder asymmetry across modalities.
  • Pointwise cosine alignment consistently outperforms ranking-based and contrastive alternatives while eliminating document representations and negative sampling during training.The objective directly aligns student and teacher query embeddings.
  • Translated-query augmentation closes a 19 pp language gap without additional image processing.The query-centric alignment setup makes cross-lingual transfer the primary bottleneck.
  • 95.1% teacher-quality retention is achieved by NanoVDR-S-Multi, a 69M-parameter DistilBERT student.The model outperforms DSE-Qwen2 on ViDoRe v2 and v3 with 32× fewer parameters, 50× lower query latency, and 32–100× less index storage.

Limitations

NanoVDR’s evaluation and training scope is centered on visual document retrieval with text-only queries and multilingual benchmark coverage. Its main boundaries include dependence on the teacher’s quality, unchanged offline indexing cost, and possible translation errors.

  • Limitations: The work does not reduce offline indexing cost, which still requires the full 2B VLM teacher to encode every document image.Teacher compression and progressive indexing are identified as future directions.
  • Limitations: Evaluation is limited to first-stage visual document retrieval with text-only queries, leaving generalization to other retrieval settings open.The stated scope boundary concerns retrieval setting rather than benchmark size.
  • Limitations: Lightweight machine translation may introduce semantic shifts or struggle with domain-specific terminology in multilingual query augmentation.Finance and physics are given as examples of potentially difficult domains.
  • Evaluation scope: ViDoRe spans three progressively challenging versions, with v1, v2, and v3 covering increasingly diverse languages and evaluation datasets.The benchmark comprises 22 datasets across versions and reaches six languages in v3.

E Training Cost: Alignment vs. Ranking

Alignment-only distillation is both more accurate and cheaper than ranking-based training in this setting. Its cost advantage comes from avoiding document-image encoding and reducing cached embedding storage.

  • 24 GPU-hours are required to encode 711K document images with the 2B teacher, versus under 1 GPU-hour for text queries.This yields a 30× speedup for query encoding relative to document-image encoding.
  • Alignment-only training reduces total pre-caching cost from ∼25 to ∼1 GPU-hour and halves embedding storage from 5.8 GB to 2.9 GB.The image-encoding step is eliminated entirely.
  • Alignment-only distillation is dominant in this setting because it is both more accurate and substantially cheaper.The comparison combines the reported accuracy advantage with the measured cost reduction.

F Data Efficiency

NanoVDR-S gains most of its data-efficiency benefit from the first quarter of training data, but multilingual evaluation requires greater data diversity. Performance saturation therefore differs sharply across benchmark versions.

  • At 25% of training data, NanoVDR-S reaches 93%/82%/70% retention on v1/v2/v3, compared with 98%/93%/87% using all data.The 25% setting corresponds to 178K of the 711K training pairs.
  • v1 reaches 93% retention with 25% data, while v3 requires 100% data to reach 87%.The difference mirrors the finding that multilingual evaluation demands greater data diversity.
  • At 10% of the data, NanoVDR-S achieves 66.7 on v1, corresponding to 79% retention.This setting uses 71K training pairs.
  • Portuguese is consistently the lowest-retention language across all eight v3 datasets because it is entirely absent from training data.Table 14 provides the per-language retention breakdown across all 22 ViDoRe datasets.

H Residual Factors Beyond Language

Beyond language, retention varies with teacher quality and corpus size, while embedding alignment alone is not predictive. Query-only multilingual augmentation addresses the identified cross-lingual bottleneck and raises teacher retention.

  • Corpus and teacher factors: Teacher NDCG@5 correlates most strongly with student retention (r=+0.607), indicating that better-separated teacher embeddings facilitate distillation.Noisy or ambiguous teacher rankings can propagate errors.
  • Corpus and teacher factors: Corpus size correlates negatively with retention (r = −0.566), because larger candidate sets require finer-grained query discrimination.Larger corpora contain more confusable candidates for top-k retrieval.
  • Alignment and retrieval: Mean student–teacher query cosine similarity shows near-zero correlation with retention (r=+0.094), so coordinate matching alone does not predict retrieval success.Retention depends on whether relevant documents remain among the student’s nearest neighbors.
  • Cross-lingual augmentation: The augmentation appends 777,649 translated queries to 711,603 original training pairs, yielding 1,489,252 combined training pairs.English queries are sampled from four sources and translated toward 200K per target language.
Loading 2603.12824v2…