ColNanoVDR: Document-Free Query Distillation for Multi-Vector Visual Document Retrieval via Optimal Transport

• About 95% of teacher NDCG@5 is retained; ColQwen3.5 is 26× faster on one CPU thread, and OTW reads 12.6× less cached data than score distillation.Figure 1 compares ViDoRe v3 NDCG@5 with CPU query throughput and shows cached teacher data read during training.Source: Figure 1 · p. 1, §1 · p. 2
• NanoVDR's document-free query-embedding approach targets single-vector retrievers, while multi-vector teachers lead the best single-vector system by 15–19 NDCG@5 points on ViDoRe v3.Source: §1 · p. 1
• Score distillation encodes every training page; for ColVec1.1, one million training pairs are estimated to require about 2 TiB of cached page tokens.Source: §1 · p. 2
• OTW aligns differently sized query-token sets by entropic optimal transport without token correspondence, learns student-token weights, and bounds MaxSim score differences on every page.The training objective uses teacher query tokens without encoding pages.Source: §1 · p. 2

• Index-side compression reduces or merges document tokens, while single-vector conversion and compact native retrievers are alternative overhead-reduction directions.Native compact retrievers rebuild both encoders and require re-indexing collections.Source: §2 · pp. 2–3
• Late-interaction distillation has matched sampled query-document scores (Huang and Chen, 2024) or ranking distributions (Clavié, 2024; Takehi et al., 2025), but reads documents during training.Source: §2 · p. 3
• NanoVDR distills a single-vector query embedding (Liu et al., 2026b); ColNanoVDR supplies a document-free counterpart for late interaction, where token sets differ in size and lack correspondence.Source: §2 · p. 3
• Unlike prior token-level alignment of patches and words, ColNanoVDR transports the retrieval-score token embeddings themselves, making alignment cost a bound on page-wise score difference.Its query-context weights are student-side transport marginals learned without relevance labels or documents.Source: §2 · p. 3

3 Method

• The method replaces only the teacher’s query encoder; the teacher’s document encoder and page index remain unchanged.Teacher query tokens are computed once and cached for training.Source: §3 · p. 3

3.1 Queries as weighted token sets

• Each query is a weighted set of unit-norm tokens scored by the weighted average of their best matches to page tokens.Uniform weights recover standard MaxSim up to division by query length; the teacher uses uniform weights and the student learns weights.Source: §3.1 · p. 4

3.2 Aligning token sets with optimal transport

• Entropic optimal transport softly aligns student and teacher tokens without token correspondence, minimizing their average cosine disagreement.Entropy spreads weight over plausible partners, making the alignment plan suitable for gradient training.Source: §3.2 · p. 4

3.3 Learned token weights and the OTW objective

• A query-conditioned linear head predicts each student token’s weight, and the OTW loss trains the encoder, projection, and head using query tokens alone.The weight head receives no direct supervision; training requires neither pages nor a document cache.Source: §3.3 · p. 5

3.4 Why aligning queries suffices

• OTW’s alignment cost upper-bounds the student–teacher MaxSim score difference on every non-empty finite page of unit-norm tokens.This includes pages absent from training, providing a sufficient condition for retrieval fidelity.Source: §3.4 · p. 5

3.5 Architecture, solver, and inference

• A text-only student projects and normalizes token states into the teacher’s embedding width, then scales tokens by learned weights for MaxSim inference.The teacher is discarded at inference, and the weighted tokens work with the unchanged index and scoring kernel.Source: Figure 3 · p. 4, §3.5 · pp. 5–6

4.1 Setup

• Five multi-vector teachers are distilled into Ettin students, with objectives compared under identical data and optimization and NDCG@5/retention evaluated on ViDoRe v1–v3.The main student is Ettin-150M, a 149M-parameter model with its heads.Source: §4.1 · p. 6

4.2 Main results

• Across five teachers, 149M students retain about 99% of teacher NDCG@5 on v1 and 93.0%–96.5% on v2/v3; Ettin-32M retains 87% on v3.Retention rises monotonically from Ettin-32M to Ettin-400M, with gains concentrated on v2 and v3.Source: §4.2 · p. 7, Figure 4 · p. 7
• 87 ms/query on one CPU thread makes ColNanoVDR 26× faster than ColQwen3.5; OTW reads 12.6× less cached teacher data than score distillation.Per-epoch training speed is comparable, so the saving is in storage and reads rather than computation.Source: §4.2 · p. 7, Table 2 · p. 7

4.3 Comparison of training objectives

• NDCG@5: OTW matches Listwise KL on v1 and v3 for ColQwen3.5 and leads on all three benchmarks for Tomoro-ColQwen3-8B, by 5.0 points on v3.InfoNCE trails both methods on both teachers.Source: §4.3 · p. 8, Table 3 · p. 8
• Without learned weights, OT-uniform trails OTW by 4.7–9.2 points on Tomoro-ColQwen3-8B, while Coverage trails OTW on every benchmark for both teachers.A second-stage weight head closes Coverage’s gap on ColQwen3.5; OTW reaches the same result in one stage.Source: §4.3 · p. 8

4.4 Deployment with index compression

• At 9× fewer index vectors, ViDoRe v3 retention falls only from 95.8% to 95.3% when teacher and student use the same pooled index.The authors conclude that query-encoder and index-compression savings compose.Source: §4.4 · p. 8, Table 4 · p. 8

5 Conclusion

• ColNanoVDR applies document-free, query-side distillation to multi-vector VDR using OTW, which aligns query-token sets with learned weights and bounds MaxSim score differences on every page.Source: §5 · p. 8
• About 95% of teacher NDCG@5 is retained by 149M text-only students distilled from five teachers, with query encoding up to 26× faster on one CPU thread.Source: §5 · p. 8
• 12.6× less cached teacher data is read by OTW than score distillation while matching its performance.Source: §5 · p. 8

Limitations

• Every configuration is a single fixed-seed run, so the study reports no variance.The objective comparison covers only ColQwen3.5 and Tomoro-ColQwen3-8B; ordering under the other three teachers is untested.Source: Limitations · p. 9
• Training uses paired queries, and the English-centric students are evaluated on predominantly English ViDoRe queries without language-disaggregated results.Training on unpaired query logs is not exercised, and results are not broken down by language.Source: Limitations · p. 9
• Compression affects only the query encoder: the teacher’s document index and storage remain unchanged, so search-cost reduction comes only through shorter query encoding.Source: Limitations · p. 9
• The theoretical guarantee is sufficient rather than necessary and is about eight times loose on measured students; it does not distinguish student weightings.Score distillation, which does not optimize the guarantee, retrieves comparably.Source: Limitations · p. 9

A Proofs for Section 3

• The appendix fixes notation, proves Theorem 1 and its ranking consequence, compares chordal and cosine bounds, analyzes the solver’s returned plan, and derives free weights and inference scoring.Source: A Proofs for Section 3 · p. 11

A.1 Setting and notation

• Token embeddings lie on the teacher’s unit sphere, and a page is a non-empty finite set whose best-match function takes the maximum token similarity.The appendix represents student and teacher token sets as discrete probability measures; teacher token weights are uniform in the stated setting, though the results allow arbitrary teacher weights.Source: §A.1 · p. 11
• Transport plans couple the student and teacher measures with prescribed marginals, and the appendix uses both chordal distance and cosine cost between tokens.Source: §A.1 · p. 11

A.2 Proof of Theorem 1

• The transport cost bounds the student–teacher weighted MaxSim score difference on every page, and the entropic training loss upper-bounds the optimal cosine transport cost.The proof uses the page best-match function’s Lipschitz property and expresses score differences through aligned token pairs.Source: §A.2 · pp. 11–12, Figure 5 · p. 12

A.3 Consequence for rankings

• A student preserves the teacher’s order for two pages when the teacher’s score margin exceeds twice the 1-Wasserstein token distance, and therefore also twice the cosine transport cost.Source: §A.3 · p. 13

A.4 Chordal and cosine bounds

• The chordal-distance bound implies a cosine-cost bound; the entropic plan’s linear transport cost is bounded above by optimal cosine transport, while its entropy term is not used.The entropic value including entropy can be negative and then provides no bound.Source: §A.4 · p. 13

A.5 The plan returned by the solver

• The solver’s returned plan matches teacher marginals exactly but may miss student marginals by a truncation residual, which degrades the score bound gracefully.Source: §A.5 · p. 13

A.6 Free student weights and the coverage loss

• With free student weights, transport assigns each teacher token’s mass to its nearest student token as entropy vanishes, and each student receives the teacher mass it covers.If several student tokens tie as nearest neighbors, the limiting mass is split evenly; coverage is the limiting objective but uses implicit weights unavailable at inference.Source: §A.6 · pp. 13–14

A.7 Weighted scoring at inference

• Scaling student token vectors by their positive learned weights realizes the weighted score with ordinary MaxSim, and the resulting mean score ranks pages identically.The weights must remain in the query-vector norms rather than being renormalized away.Source: §A.7 · p. 14

B Implementation Details

• The implementation appendix covers cached teacher query tokens, student optimization, the Sinkhorn solver, training data, and evaluation protocol.Source: B Implementation Details · p. 14

B.1 Teacher caches and query tokenization

• ColVec1.1’s estimated page cache is about 2.0 TiB per million training pairs, versus roughly 45 GiB of query cache per million queries.On training queries, ColQwen3.5 averages 28.6 teacher tokens and 17.1 student tokens per query.Source: §B.1 · p. 15

B.2 Student architecture and optimization

• Each student uses a bias-free projection to the teacher’s width with L2 normalization, plus a linear head that predicts one token weight.Runs use AdamW for 10 epochs, effective batch size 1,024, and a one-cycle cosine learning-rate schedule; all results use seed 42.Source: §B.2 · p. 15

B.3 Sinkhorn solver

• The log-domain Sinkhorn solver enforces transport marginals, while gradients use one recomputed update pair rather than backpropagating through all 50 iterations.The one-step scheme avoids differentiating through the full solver; the returned plan satisfies the column constraint exactly and the row constraint up to a truncation residual.Source: §B.3 · p. 16

B.4 Training data

• The 711,603 base training pairs combine VisRAG synthetic and in-domain data, five-language VDR queries, and the ColPali training set; 777,649 translated variants reuse base pages.The base sources contribute 32.9%, 13.2%, 38.6%, and 15.3%, respectively.Source: §B.4 · p. 16

B.5 Evaluation

• NDCG@5 is averaged over 10 ViDoRe v1, 4 v2, and 8 v3 datasets, and student rankings use weighted mean-MaxSim.Reference systems and teachers are public HuggingFace checkpoints.Source: §B.5 · p. 16

C.1 Query-encoding cost

• On a single-thread CPU, query-encoding throughput is measured on the same node using float32 and batch size 1; GPU timings use one H200 in bf16.Reported timing is the median over 20 queries after three warmup runs and excludes MaxSim scoring; OTW trains per epoch at a speed comparable to Listwise KL and InfoNCE.Source: §B.2 · p. 15, §C.1 · p. 17

C.2 Supervision cost

• Document-free objectives read only query tokens, while score distillation reads 12.6 times as much cached teacher data: 342.7 GiB versus 27.3 GiB.Including translated variants, the page cache is 11.6 times the query cache; the query-only cache is 8.0% of the total cache.Source: §C.2 · p. 17
• Encoding the training pages for ColQwen3.5 took 23 H200 GPU-hours, with page tokens accounting for 96% of tokens written; page-side encoding is repeated for each new teacher.For three teachers cached without pages, encoding 1,489,252 training and translated queries took 1.0–1.7 hours each, including model loading.Source: §C.2 · p. 17

C.4 Index compression

• Hierarchical pooling clusters each page’s tokens independently and replaces each cluster with a normalized mean; retention compares student and teacher NDCG@5 on the same pooled index.Pooling is applied offline without touching either encoder; the described pool factors yield 587 vectors per page at f = 3 and 195 at f = 9.Source: §C.4 · pp. 17–18

D.1 Weighted score distillation

• Weighted Listwise KL scores 90.9/60.9/54.9 on v1/v2/v3, versus 90.9/61.2/55.0 for uniform-weight Listwise KL, and is no higher on any benchmark.The weight head does not transfer OTW’s gain to score distillation.Source: §D.1 · p. 18

D.2 Two-stage weight heads

• Training a weight head for two epochs on a frozen coverage student reproduces jointly trained OTW within 0.1 points on ColQwen3.5.This two-stage alternative takes 10 + 2 epochs; OTW remains the one-stage default, training embeddings and weights end to end in 10 epochs.Source: Table 8 · p. 18, §D.2 · p. 18

D.3 Protocol for inference-time analyses

• Inference-time analyses freeze token embeddings and replace only the weights; teacher-derived alternatives use teacher query tokens and are not deployable.Numbers are NDCG@5 over the complete benchmark under the Table 1 protocol.Source: §D.3 · p. 18

D.4 Weight interventions

• Replacing learned weights with uniform weights costs 1.6/3.5/3.5 NDCG@5 points for the 149M student and 1.4/1.3/1.9 for the 395M student; pruning dead tokens changes almost nothing.The fixed embeddings were trained with learned weights, indicating they are adapted to those weights; placing all mass on dead tokens collapses retrieval.Source: Table 9 · p. 19, §D.4 · p. 19

D.5 Oracle weights and the role of ε

• For ε ≤ 0.1, hard, soft, and learned weights lie within 0.1 points of one another, and hard and soft assignments beat uniform weights by 1 to 2 points.Soft assignments degrade toward uniform as ε rises past 0.2; training at ε = 0.02 or 0.1 stays within a point of the default 0.05 on every benchmark.Source: Table 10 · p. 19, §D.5 · p. 19

E Empirical Check of the Bound

• The analysis uses 4,735 evaluation queries spanning ViDoRe v1, v2, and v3, and compares student and teacher scores over every page in each corresponding corpus.The reported bound measurements are medians over queries.Source: E Empirical Check of the Bound · pp. 19–20
• The chordal bound W1 and the training-setting bound hold on every tested query, but W1 exceeds the worst-case discrepancy it certifies by 7.3 to 8.3 times for document-free students.W1 ranges from 0.57 to 0.66, below the trivial bound of 2 but not tight.Source: Table 11 · p. 20, E Empirical Check of the Bound · p. 20
• Among document-free objectives, W1 orders students like average NDCG@5 except for one v2 benchmark-level exception; document-dependent students can score accurately while remaining far from the teacher’s measure.OTW has the highest Spearman correlation with the teacher’s document scores among document-free objectives.Source: E Empirical Check of the Bound · p. 20