Source-linked AI summary
Generative Late-Interaction Embeddings For Visual Document Retrieval
Mohamed Eltahir, Talal Aloushan, Rose Khairoalsendi, Jana Shata, Mohammed Alhassan, Leen Alrehaili, Tanveer Hussain, Naeemullah Khan
TL;DR
Visual late-interaction retrieval is accurate because it preserves many token-level vectors, but their storage cost makes aggressive compression difficult. GLIE exploits their low-dimensional spherical geometry to store a few learned vectors, retrieve cheaply, and regenerate full embeddings for exact rescoring; it retains 79% of uncompressed quality at 1.0 KB per page and 91% at 4 KB while outperforming prior baselines across ViDoRe v1 and v2.
Problem
Late-interaction visual retrieval stores about 1,000 vectors per page, while existing compression methods degrade under aggressive budgets or require encoder retraining.
Method
GLIE learns k spherical vectors from normalized centroids as a lightweight index and decodes them into the full page embedding set for exact rescoring of top candidates.
Results
GLIE retains 79% of uncompressed retrieval quality at 1.0 KB per page and 91% at 4 KB, beating every prior baseline across ViDoRe v1 and v2 subsets at every budget.
Takeaways & Limitations
Reconstructing evidence on demand rather than sampling it opens a separate storage-efficiency axis whose main design surface is the decoder.
Takeaways & Limitations
At four vectors per page, perfect decoding of the same code and shortlist would reach 0.782 versus GLIE’s 0.657, leaving decoder headroom.
Abstract
from arXiv · showhide
Late-interaction retrieval is the state-of-the-art for visual document search, but it pays for its accuracy in storage. Existing compression methods retain a subset or local average of the N~1,000 vectors per page. Under aggressive storage budgets, however, these methods degrade sharply, and alternatives require retraining the encoder. Investigating this degradation across three encoders, we found two consistent properties: the vectors lie exactly on the unit sphere and concentrate near a manifold of intrinsic dimension five to six. This geometry yields two insights. First, standard k-means centroids fall inside the sphere, causing systematic underestimation of MaxSim scores. Normalizing them to the surface is a free correction worth up to +0.093 nDCG@5 over raw centroids. Second, because the page manifold has few degrees of freedom, the full set of vectors can be regenerated from only a few. To this end, we introduce Generative Late-Interaction Embeddings (GLIE): k << N vectors per page learned from the normalized centroids to serve as both a lightweight index and a basis for regenerating the page's full embedding set. At query time, search runs exclusively on these k vectors, and a decoder expands only the top candidates back to all N vectors for exact rescoring. At four vectors per page on ViDoRe v1, GLIE retains nearly 80% of the uncompressed system's nDCG@5, against 70% for the best prior post-hoc method. These results use a 415K-parameter network fitted in under three GPU-minutes on just a thousand training pages. At a matched training budget, fine-tuning the encoder does not reach even the training-free stage of GLIE, and the full system beats it at every budget. These patterns hold across a second encoder and ViDoRe v2. By reconstructing evidence on demand rather than sampling it, GLIE opens a new axis for storage-efficient retrieval, with the decoder as its main design surface.
1 INTRODUCTION
GLIE treats visual-document embeddings as low-dimensional spherical objects rather than subsets to compress, combining normalized centroids with generative reconstruction. It enables cheap retrieval over a few stored vectors and exact rescoring after expanding only top candidates.
- Motivation: The geometry suggests replacing subsampling with a compact description of the page manifold’s few degrees of freedom.The approach targets the stored object itself rather than only reducing the number or precision of stored vectors.
- Geometry of the stored object: The page’s 1,031 vectors concentrate near a five-to-six-dimensional manifold on the unit sphere across ten evaluation corpora and two further encoders.One additional encoder uses a 3,072-dimensional embedding space.
- Spherical anchoring: +0.093 nDCG@5 is gained by projecting k-means centroids back onto the unit sphere, correcting their systematic MaxSim underestimation at zero cost.Standard centroids lie inside the sphere because they are Euclidean means of unit vectors.
- Generative Late-Interaction Embeddings (GLIE): GLIE begins from normalized clustering, then refines the code from the full token set so scoring-relevant structure averaged away by centroids can be encoded.The decoder regenerates evidence that a k-vector sample cannot represent.
- Generative Late-Interaction Embeddings (GLIE): GLIE stores k vectors per page and regenerates all N on demand, using stored-vector retrieval followed by top-L expansion and exact rescoring.This replaces extractive subset sampling with a generative codec and asymmetric inference pipeline.
2 RELATED WORK
Prior work reduces late-interaction storage by merging, pruning, quantizing, retraining representations, or shrinking dimensions and scoring costs. GLIE differs by asking what the stored vector set is geometrically and encoding that structure directly.
- Post-hoc reduction of the stored set: Post-hoc methods merge, prune, or combine document vectors, while reported near-lossless results use substantially more than GLIE’s most aggressive budgets.Light-ColPali retains 93.6% of nDCG@5 at merging factor 49, with about twenty-one vectors per page.
- Small budgets via retraining: Retraining methods learn fixed or nested document representations but require abandoning the public checkpoint and re-encoding the corpus.MetaEmbed reports training with 32 H100 GPUs for 30 hours.
- Orthogonal axes: Quantization, pruning at query time, dimension reduction, and multi-vector sketches act on storage or scoring axes orthogonal to vector-count compression.These approaches can compose with reducing the number of vectors per page.
- Motivation: GLIE’s motivating distinction is that existing approaches do not treat the stored set as a low-dimensional manifold on the sphere.The paper positions geometry-aware coding as a separate alternative to training-free reduction and encoder retraining.
3 GENERATIVE LATE-INTERACTION EMBEDDINGS
GLIE represents each page with a small spherical code learned from normalized clustering, then regenerates full token vectors only for shortlisted candidates. Its design uses page geometry, MaxSim-aware training, and anchored decoding to preserve retrieval evidence while reducing search cost.
- 3.1 WHAT THE STORED OBJECT IS: Intrinsic dimension medians stay near five to six across three encoders despite a 24× ambient-dimension range, while page vectors lie on the unit sphere.ColPali has median intrinsic dimension 4.9 in 128 dimensions; the other encoders measure 5.1 and 6.1 in 3,072 dimensions.
- 3.1 WHAT THE STORED OBJECT IS: Normalized k-means corrects centroids’ systematic MaxSim underestimation, improving full-benchmark nDCG@5 by +0.031 to +0.093 as k varies.The improvement shrinks with larger k because tighter clusters have means closer to the sphere.
- 3.4 ZERO-INITIALIZED REFINEMENT THAT READS THE PAGE: GLIE learns k vectors from normalized centroids while reading the full token set, so the code can recover scoring-relevant structure that cluster means discard.The encoder remains frozen; zero-initialized refinement starts exactly at normalized clustering, and the 415K-parameter refiner and decoder run once per page at indexing time.
- 3.5 ANCHORED GENERATIVE READ-OUT: The decoder expands k stored vectors into N unit vectors with exact anchors, ensuring regenerated MaxSim is never lower than code MaxSim while constraining children around their anchors.Each cluster receives count-proportional slots, and generated children remain within approximately 37° of their anchor.
- 3.6 TWO-STAGE INFERENCE: GLIE replaces fixed subset scoring with on-demand evidence reconstruction: every page is retrieved using k vectors, then only the top-L candidates are decoded and exactly rescored.This asymmetric pipeline makes the expensive full representation available only where reranking requires it.
- 3.3 OBJECTIVES: The training objective separately makes the code rank pages and the regenerated set behave as a page, using MaxSim, listwise, and anti-invention constraints rather than reconstruction error.Reconstruction error is omitted because pulling children toward cluster means would collapse the extreme points that MaxSim uses.
4 EXPERIMENTS
Across ViDoRe v1 and v2, GLIE consistently outperforms prior post-hoc baselines under aggressive storage budgets, while transferring across training settings and a second encoder.
- Main results: 79% of uncompressed retrieval quality at 1.0 KB per page and 91% at 4 KB, with GLIE beating every prior baseline across all ViDoRe v1 and v2 subsets and budgets.TAT-DQA margins range from +0.039 to +0.054 at k ≤8.
- Main results: On ViDoRe v1, GLIE’s margin is about +0.04 through k ≤8, halves at k = 16, and decays to noise by k = 32.The reported mean margin is +0.039 below k = 8 versus +0.010 above it.
- Cascade limits: At k = 4 on ViDoRe v1, the oracle gap separates decode fidelity, 0.657 to 0.782, from shortlist recall, 0.782 to 0.836.Increasing L from 5 to 100 raises the oracle from 0.705 to 0.822 while GLIE moves from 0.647 to 0.660, identifying decoding as the larger headroom.
- Matched training: At matched training source and budget, encoder fine-tuning fails to reach free normalized clustering, while the frozen-backbone codec beats it by +0.074 to +0.132 at all six budgets.The comparison uses the ViDoRe v1 evaluation and the matched-training setup summarized in Table 5.
- Storage: At k = 4, storage falls from 257.8 KB to 1,040 bytes per page, shrinking one million pages from 258 GB to 1.0 GB.Because the backbone is frozen, changing the storage budget affects cached embeddings without re-encoding the corpus.
- Transfer: On ColQwen2, GLIE retains 82% of uncompressed quality at k = 4 and wins at five of six budgets against spherical anchoring alone.The stronger encoder leaves less room for learned improvement, with margins of +0.015 below k < 8 and +0.013 above.
5 ABLATION STUDY
Ablations identify spherical anchoring as the dominant component, while learned codes and generative read-outs provide additional gains mainly at aggressive budgets.
- Component decomposition: +0.093 at k = 4 to +0.030 at k = 64 comes from spherical anchoring, the largest single component at every budget.The correction is free and reflects re-projecting centroids onto the unit sphere.
- Component decomposition: The learned code adds +0.044 to +0.016 through k = 16, while the generative read-out peaks at +0.016 at k = 4 on ViDoRe v1.On ViDoRe v2, the read-out contribution does not decay through k = 64.
- Decoder capacity: Decoder capacity changes nDCG@5 by at most 0.009 across a seventy-fold parameter range, with the smallest decoder best at k = 4.The tested range spans 184K to 13M parameters and one to three blocks.
6 CONCLUSION
GLIE treats storage reduction as a geometric problem: page vectors lie on a low-dimensional unit-sphere manifold, enabling a few anchors and a shared decoder to regenerate evidence on demand. The method opens a new storage axis, but decoder quality and the information capacity of k vectors remain limiting factors.
- 415K parameters and 1,000 training pages suffice because few page degrees of freedom can be represented by a few anchors and one shared decoder.The method targets the storage problem through geometric structure rather than simply storing fewer or smaller vectors.
- At four vectors per page, perfect decoding of the same code and shortlist would reach 0.782 versus 0.657, leaving decoder quality as the immediate improvement target.The shortlist oracle measures reconstruction headroom for the same stored code and candidate shortlist.
- Extending evaluation to more encoders, combining GLIE with quantized storage, and testing video late interaction are identified as promising next steps.The proposed extensions target generality, complementary footprint reduction, and settings with larger, more redundant token sets.
A IMPLEMENTATION DETAILS
The implementation uses compact attention and MLP components, cosine-decayed AdamW training with hard negatives, and a shortlist cascade for inference. Storage uses bfloat16 vectors with lightweight per-cluster metadata, and each budget fits within minutes on one A100.
- The refiner and decoder together contain 415K parameters, using one four-head cross-attention layer, a width-256 two-layer MLP, and α = 0.75.Per-page k-means uses two restarts.
- Inference keeps a shortlist of L = 20, stores bfloat16 vectors with fp16 norms and uint16 counts, and fits one budget in 2.7 minutes on an NVIDIA A100 80GB.Three random seeds are used, and checkpoints are selected separately for single-stage and cascade deployment.
B EVALUATION PROTOCOL DETAILS
The evaluation excludes test queries and pages from learned components, covers all queries across the ViDoRe subsets, and uses de-duplicated page corpora with query-less distractors. Results are compared with the strongest training-free baseline under each subset and budget.
- 3,943 queries are evaluated without learned components seeing any ViDoRe test query or page, including 1,663 TAT-DQA queries.The evaluation covers all queries of each subset.
- Each subset corpus contains unique pages deduplicated by image identity while retaining query-less pages as distractors.The pipeline reproduces the official ColPali v1.3 leaderboard average.
- Margins are reported against the strongest training-free baseline selected separately for each subset and storage budget, while learned rows average three runs.
C LIGHT-COLPALI REPRODUCTION
The Light-ColPali reproduction fine-tunes a merged, renormalized representation through a fresh LoRA adapter under a matched small training budget. Its results are intended to measure what that budget buys, rather than reproduce the published method's much larger training regime.
- Light-ColPali merges post-projector embeddings by clustering, renormalizes cluster means, and fine-tunes them end to end with a fresh rank-32 LoRA adapter.The reproduced adapter has 13.3M trainable parameters and is trained separately for each budget.
- The reproduction trains for five epochs on 4,000 query-page pairs with batch 8, gradient accumulation 4, and 10% warmup.It uses the pairwise loss from the ColPali training recipe.
- The reproduction is a matched-small-budget comparison: the published method trains on 130K queries for about 72 GPU-hours per budget and reports results only at k ≳16.The reproduced runs use about 1.5 GPU-hours per budget and are evaluated zero-shot across ten ViDoRe v1 subsets.
D PER-SUBSET RESULTS
Per-subset results are reported for ViDoRe v1 and v2 using nDCG@5 over all queries, with learned methods averaged across three seeds where applicable.
- ViDoRe v1 per-subset results are split across two tables, while ViDoRe v2 results are presented in a single table.
- All reported per-subset results use nDCG@5 over all queries, with graded nDCG@5 specified for ViDoRe v2.
- The learned-row results in the ViDoRe v1 tables are three-seed means, and each table identifies the best method per column in bold.
- The uncompressed row defines the ceiling and does not compete in the per-subset comparisons.