PaperlayerView exact revision on arXiv

Source-linked AI summary

DistilVDR: A Compact End-to-End Visual Document Retriever via Dual-Student Distillation

Zhuchenyang Liu, Ziyi Wang, Yao Zhang, Yu Xiao

arXiv:2608.10636v1cs.IRcs.CLcs.CV

TL;DR

Visual document retrieval is costly because strong systems are large, while prior compression approaches do not provide a compact end-to-end single-vector retriever. DistilVDR independently distils asymmetric query and document encoders from one vision-language teacher, reaching a better quality-and-cost operating point than reproduced small multi-vector retrievers.

  • Problem

    Strong visual document retrievers require substantial memory and indexing cost, while prior compression routes do not yield a compact end-to-end single-vector system.

  • Method

    DistilVDR independently cosine-distils asymmetric query and document encoders from one 8B vision-language teacher for single-vector dense retrieval.

  • Results

    61.74 average NDCG@5 on ViDoRe v1+v2+v3, with a better quality-and-cost operating point than reproduced small multi-vector retrievers.

  • Takeaways & Limitations

    The two variants provide a compact retrieval operating point combining competitive quality with substantially smaller indexing footprint and faster indexing.

  • Takeaways & Limitations

    The evaluation lacks a same-architecture, same-data, compute-matched contrastive-from-scratch control isolating distillation’s contribution.

Abstract

from arXiv · show

Visual document retrieval (VDR) is dominated by multi-billion-parameter models that are slow to index at full corpus scale and expensive to serve. Prior compression routes either train a smaller multi-vector encoder from scratch or distil only the query side; neither yields a compact single-vector retriever end-to-end. We present DistilVDR, a 524M end-to-end VDR system distilled bilaterally from a single 8B vision-language teacher under a pointwise cosine alignment loss. All supervision comes from the frozen teacher's embedding space, which was itself trained with relevance supervision, so the student objective needs no relevance labels, negative sampling, or contrastive term. We match VDR's text-query and image-document input asymmetry with an asymmetric encoder-only student that concentrates visual capacity on the document side and keeps the query side at 70M parameters. We release two variants that share the same encoders and training and differ only in the document encoder's visual-tile budget: DistilVDR-HiRes attains 61.74 average NDCG@5 on ViDoRe v1+v2+v3 (86.9% of the 8B teacher) and leads every reproduced sub-1B baseline on the high-resolution-sensitive v3 benchmark, while DistilVDR-Fast attains 59.98 at a 3 times smaller visual-token budget. Both variants store one million documents in a 15.6 times smaller index than the strongest sub-1B multi-vector baseline and index the corpus an order of magnitude faster. The code is available at https://github.com/Ryenhails/NanoVDR.

1 Introduction

Visual document retrieval is costly because leading systems use billions of parameters, large memory, and expensive indexing, while existing compact approaches retain multi-vector overhead or leave the document encoder large. DistilVDR addresses this gap with a 524M end-to-end single-vector retriever distilled bilaterally from an 8B teacher, with HiRes reaching 61.74 average NDCG@5 on ViDoRe v1+v2+v3.

  • Motivation: Top-performing VDR systems span 2B–8B parameters, require more than 16 GB of GPU memory per document, and cost tens of GPU-hours to index one million documents.Examples include Qwen3-VL-Embedding-8B and Tomoro-ColQwen3-8B.
  • Limitations of prior work: From-scratch compact retrievers retain multi-vector storage and late-interaction costs, while single-vector variants lose quality by 17.6 NDCG@5 on ViDoRe v1 and 20.3 on v2.The gaps are reported for BiModernVBERT relative to ColModernVBERT.
  • Limitations of prior work: Existing distillation produces a small query encoder but leaves the document encoder at teacher scale, so NanoVDR still retains the 2B teacher in its document path.Consequently, million-document indexing remains a tens-of-GPU-hours operation and deployment must host the teacher.
  • DistilVDR: DistilVDR is a 524M end-to-end single-vector retriever using bilateral cosine alignment to cached embeddings from one 8B vision-language teacher and a 70M text-only query tower.The system also reproduces twelve released retrievers under one evaluation and profiling pipeline.
  • Results: 61.74 average NDCG@5 is achieved by DistilVDR-HiRes on ViDoRe v1+v2+v3, corresponding to 86.9% of the 8B teacher.The released variants differ only in the document encoder’s visual-tile budget.

2 Related Work

This section situates DistilVDR against three adjacent areas: dense-text retrieval distillation, universal multimodal embedders, and OCR-free document understanding. The comparisons emphasize DistilVDR’s fixed text-query/image-document setting and retrieval-oriented representations.

  • Distillation in dense text retrieval: Dense-text retrieval commonly combines hard-negative mining, retrieval-oriented pretraining, or cross-encoder-to-bi-encoder distillation; asymmetric reranker-to-bi-encoder teaching is the closest text-side analogue.The passage identifies this analogue as differing from DistilVDR in two ways, but the supplied text does not specify them.
  • Universal multimodal embedders: Universal multimodal embedders convert generative vision-language models into embedding models, whereas DistilVDR prioritizes depth in one subdomain at sub-1 B scale.VDR’s fixed text-query/image-document input shape lets both encoders remain small.
  • OCR-free document understanding: OCR-free document-understanding models process document images directly, avoiding OCR error compounding, but target generation or extraction rather than nearest-neighbour retrieval.The cited examples include Donut, Pix2Struct, UDOP, LayoutLM, and LayoutLMv3.

3 Method

DistilVDR uses asymmetric dual-student distillation to produce normalized single-vector query and document embeddings in a shared teacher space for dense retrieval. Its document student allocates substantially more capacity to visual encoding, while both students independently match frozen-teacher targets without relevance labels, negatives, or contrastive training.

  • Retrieval formulation: The system ranks document images with dot-product scores between L2-normalized query and document vectors, which are computed once during indexing.A text query encoder and image document encoder map both modalities into the same R^k space.
  • Dual-student architecture: A frozen 8B vision-language teacher provides 4,096-dimensional targets, while separate query and document students independently reproduce them for deployment.Both students project into the teacher’s output space and are L2-normalized; the teacher is not used at retrieval time.
  • Asymmetric encoders: The asymmetric students comprise a 454M-parameter document encoder and a 70M-parameter query encoder tailored to image-document and text-query inputs.The document encoder combines visual and text-backbone components, whereas the query encoder uses DistilBERT-base followed by pooling and projection.
  • Document visual processing: A fixed per-document tile cap, aspect-ratio-matched tiling, and a native-resolution thumbnail control visual-token budgets and distinguish the Fast and HiRes variants.The design addresses the tradeoff between losing small text at low resolution and exceeding the contextual encoder’s sequence capacity.
  • Distillation objective: Each student is trained independently with cosine alignment to cached, normalized teacher embeddings, with no contrastive term, hard negative, or relevance label in the student objective.The query and document distillations never share a forward pass, keeping training fully decoupled and independently parallelisable.

4 Experiments

Experiments on the full 22-dataset ViDoRe suite show that DistilVDR variants outperform reproduced sub-1B baselines while preserving strong accuracy against larger models. Fast provides the highest throughput and compact single-vector indexing, whereas HiRes improves performance on high-resolution-sensitive v3.

  • Retrieval quality: 61.74 average NDCG@5 is reported for HiRes and 59.98 for Fast across ViDoRe v1+v2+v3.The suite contains 22 datasets spanning English, multilingual, and professional-domain benchmarks.
  • Retrieval quality: 8.73 points is HiRes’s lead over colSmol-500M at 53.01 average, while Fast leads by 6.97 points.Both variants exceed every other reproduced sub-1B baseline by more than these margins.
  • Retrieval quality: 47.07 versus 43.66 is HiRes’s v3 score against Fast, with HiRes leading the next-best sub-1B retriever by 13.55 points.Fast trades v3 quality for a 3× smaller visual-token budget, while the variants remain within 1.5 points on v1 and v2.
  • Gap analysis: 6.03 NDCG@5 points come from replacing the teacher’s document side with the student, compared with 4.69 points from replacing the query side.The two side losses do not simply add to the full 9.31-point teacher–HiRes gap because of interaction between student errors.
  • Efficiency: 99.04 docs/sec and 2.10 GB peak VRAM make Fast the highest-throughput variant, roughly 18× faster than the 8B teacher.HiRes runs 7× faster than the teacher, while both variants use single-vector indexes and scoring substantially smaller than multi-vector baselines.

5 Ablations

The ablations show that visual resolution, training-data scale, and embedding dimensionality materially affect retrieval quality and index cost, while query-backbone changes trade modest quality for latency and parameter savings. A one-epoch contrastive refinement adds no NDCG@5 improvement over cosine distillation.

  • Visual tile budget: 5.12 average NDCG@5 is lost by removing tiling versus HiRes, while HiRes gains 1.76 average points and 3.41 on v3 over Fast at 3× the visual tokens.The 0-tile control uses a single 448×448 view; Fast and HiRes are the deployed variants.
  • Training-data scale: Quality rises monotonically with training-data scale and saturates above 75%, with the full 1.20 M-image mixture worth about 5 average points over quarter-scale training.The document encoder is retrained on uniform subsamples of the mixture.
  • Output dimension: 3.19 average NDCG@5 and 4.77 on v3 are lost when the target is truncated to 768 dimensions, despite shrinking the index 5.3× from 16.4 GB to 3.07 GB per million documents.The 4096-d default is retained because the teacher’s Matryoshka-trained prefixes remain valid embeddings.
  • Query backbone: 67.40 average NDCG@5 makes ModernBERT-base 1.04 points stronger than DistilBERT-base, but it costs 2.2× the parameters and 5.1× the query latency.Query encoding at B = 1 takes 2.07 ms for DistilBERT-base versus 10.61 ms for ModernBERT-base; DistilBERT-base is adopted.
  • Contrastive refinement: No NDCG@5 improvement occurs at any tested γ when contrastive supervision is added; InfoNCE drifts from −0.12 at γ = 0.5 to −0.50 at γ = 2.0, while KL stays within ±0.14.All configurations start from the same distilled checkpoint, so the ablation tests refinement only and supports pure cosine alignment as the default.

6 Conclusion

DistilVDR is a 524 M end-to-end visual document retriever distilled from a single 8 B vision-language teacher through an asymmetric encoder-only design. Its HiRes and Fast variants provide improved quality-cost efficiency at the 500 M scale, with smaller indexes and faster indexing than reproduced small multi-vector retrievers.

  • 6 Conclusion: 524 M DistilVDR independently cosine-distils both encoders from one 8 B vision-language teacher in an asymmetric encoder-only architecture.The system is end-to-end and uses bilateral distillation.
  • 6 Conclusion: 86.9 % teacher retention makes HiRes the leader among reproduced sub-1 B baselines on ViDoRe v3, while Fast retains 84.4 % with a 3× smaller visual-token budget.The variants share encoders and training and differ only in the document-side visual-tile budget.
  • 6 Conclusion: 15.6 times smaller index footprints and order-of-magnitude faster indexing establish a better quality-and-cost operating point than any reproduced released small multi-vector retriever.The comparison was measured end to end under one evaluation and one profiling pipeline at the 500 M scale.

Limitations

The evaluation does not isolate distillation from other training and architecture choices, and ViDoRe does not cover several production-relevant settings. Methodologically, student representations remain bounded by weaknesses in the frozen teacher’s embedding space.

  • Experimental limitations: Reported comparisons use deployable official releases rather than retrained, budget-matched baselines, so performance margins cannot be attributed to a single factor.The teacher, 1.20 M-image mixture, and architecture differ from the baselines; the isolating control is not reported in the supplied passage.
  • Dataset limitations: ViDoRe covers three difficulty levels, six languages, and eight professional domains, but excludes enterprise layouts, degraded scans, and production query distributions.Consequently, the reported quality may not transfer directly to those deployment settings.
  • Methodological limitations: Both student towers reproduce a frozen teacher’s embeddings, so systematic weaknesses of Qwen3-VL-Embedding-8B propagate to DistilVDR and the student cannot exceed its teacher on the replaced side.Table 2 is consistent with this limitation because no student-side substitution improves on the teacher×teacher oracle.

A ViDoRe Benchmark Details

ViDoRe comprises three progressively challenging benchmark versions evaluated here as a full 22-dataset suite. Across v1–v3, the benchmark expands multilingual coverage, annotation sophistication, and domain breadth.

  • Benchmark scope: The full ViDoRe evaluation covers 22 datasets spanning three progressively challenging benchmark versions.Table 6 records each dataset’s corpus language, query languages, and query source.
  • v1: v1 contains 10 English- and French-language datasets, but state-of-the-art models now exceed 90 NDCG@5, indicating saturation.Five datasets use human-authored queries from visual QA benchmarks, while five use Claude-3 Sonnet-generated queries over curated document collections.
  • v2: v2 introduces 4 datasets with queries in up to four languages and uses blind contextual generation to reduce extractive bias and require cross-document reasoning.Its fourth dataset provides expert human annotations over the same ESG corpus.
  • v3: v3 provides 8 public datasets across six languages, combining LLM-synthesised queries with expert review and page-level relevance rankings.Queries also include boundingbox annotations and multilingual translations; domains span enterprise scenarios from finance to physics.

B Training Data Composition and Preprocessing · C Baselines

Training uses a three-group public-data mixture, with document-side supplements deduplicated against base and evaluation corpora and teacher embeddings cached before student training. Baselines span sub-1B and 2–3B single- and multi-vector retrievers.

  • B Training Data Composition and Preprocessing: The document encoder mixture combines three groups of publicly released datasets, with the full per-source and query-encoder breakdown given in Table 7.
  • B Training Data Composition and Preprocessing: The 711 K NanoVDR base mixture includes VisRAG-Synthetic, VisRAG-InDomain, ColPali training data, and multilingual VDR-Multilingual data.
  • B Training Data Composition and Preprocessing: 1.49 M paired queries combine the same 711 K base queries with 778 K Helsinki-NLP MarianMT translations for the query encoder.
  • B Training Data Composition and Preprocessing: 454 K images from fourteen Racineai domain-specific collections supplement the document side after perceptual-hash deduplication against base and ViDoRe v1/v2/v3 corpora.
  • B Training Data Composition and Preprocessing: 31.7 K finance-domain document images and DocReRank/FinHNQue-FinanceHardNegative queries form a third supplement, deduplicated against base and evaluation corpora.
  • B Training Data Composition and Preprocessing: Perceptual-hash distance 0 removes only exact visual duplicates; deduplicated images are dynamically tiled, while teacher embeddings are precomputed and cached as float32 arrays.
  • C Baselines: Sub-1B baselines include colSmol-256M, colSmol-500M, SauerkrautLM-ColLFM2-450M, ColModernVBERT, SigLIP2-L, and BiModernVBERT.
  • C Baselines: At 2–3B, baselines include DSE-Qwen2, Qwen3-VL-Embedding-2B, and ColPali v1.3, spanning single-vector and multi-vector retrieval.

D Training Recipe · E Dynamic Tiling Algorithm · F Training Loss Curves

The training recipe is shared by both document-encoder variants, which differ only in their visual-tile budget, while dynamic tiling fits documents within the student’s context window. Training losses for both document variants and the shared query encoder decrease smoothly to low plateaus with small train/validation gaps.

  • D Training Recipe: Both DistilVDR-Fast and DistilVDR-HiRes use the same hyperparameters and hardware under the cosine alignment objective, differing only in Tmax.Per-epoch loss curves for the three students are reported as a convergence diagnostic.
  • E Dynamic Tiling Algorithm: The dynamic tiling rule selects an aspect-ratio-matched grid from G = {(p, q) : p, q ∈Z≥1, nmin ≤ p · q ≤nmax}.The image is resized to the selected grid, split into non-overlapping tiles, and may receive one thumbnail.
  • E Dynamic Tiling Algorithm: Ties in aspect-ratio distance favor the higher-resolution grid when the original image area is large, preserving global layout context through an optional thumbnail.The thumbnail is appended at the encoder’s native tile resolution.
  • E Dynamic Tiling Algorithm: The document encoder uses nmin = 1, nmax = 6, tile size s = 448, and an enabled thumbnail.These settings define the visual-token budget used by the document encoder.
  • E Dynamic Tiling Algorithm: 7168 patch tokens is the maximum document sequence length, computed as (nmax + 1) · s2/p2 = 7 · 1024, fitting ModernBERT-base’s 8192-token context without truncation.The calculation uses InternViT-300M’s patch size p = 14.
  • F Training Loss Curves: All train and validation cosine-alignment loss curves decrease smoothly to a low plateau for DistilVDR-HiRes, DistilVDR-Fast, and the shared query encoder.The curves show only a small train/validation gap, indicating steady-state fits to teacher targets within the allotted epoch budget.

G Efficiency Benchmark Protocol

The efficiency benchmark fixes hardware, software, attention-backend, batch-size, warmup, synchronization, and storage conditions across query, document, VRAM, index-size, and CPU-scoring measurements. A conservative eager-attention check additionally reports throughput and VRAM for both DistilVDR variants.

  • Hardware and software stack: The reported stack uses one NVIDIA H200 GPU with 141 GB HBM3e, an Intel Xeon Platinum 8480+ node with 32 CPU cores and 128 GB RAM, PyTorch 2.5, CUDA 12.6, and bfloat16 encoders.BiModernVBERT is instead evaluated in float32 under its released configuration because bf16 mean-pooling causes a several-NDCG drop.
  • Attention backend: Every encoder uses a flash-attention-family kernel; baselines and ModernBERT access it through PyTorch sdpa, while InternViT uses its native flash-attention path by default.Explicitly setting flash_attention_2 changes throughput by under ±1 docs/sec.
  • Conservative attention-backend reference: 31.91 docs/sec is DistilVDR-Fast’s throughput with eager InternViT attention at B = 8, using 5.73 GB VRAM.Under the same conservative configuration, DistilVDR-HiRes reaches 9.27 docs/sec with 11.56 GB VRAM, and Fast remains faster than every sub-1 B multi-vector baseline.
  • Measurement protocol: Query latency averages 20 queries at B = 1 after 3 warmup queries, with cuda.synchronize() around the forward call and tokenisation included.Document throughput and peak VRAM use B = 8 on a 100-page synthetic corpus after 3 warmup forward passes, while multi-vector baselines fully form indexed per-document tensors.
  • Measurement protocol: CPU scoring latency averages 20 single-threaded runs over 10 000 synthetic candidates, using dot products for single vectors and MaxSim over 32 query tokens for multi-vectors.The benchmark evaluates each method in its appropriate storage format.
Loading 2608.10636v1…