PaperlayerView exact revision on arXiv

Source-linked AI summary

EigenLI: Spectral Approximations to Late Interaction

Archish S, Sabyasachi Basu, Ankit Garg, Ravishankar Krishnaswamy, Kirankumar Shiragur

arXiv:2609.07561v1cs.IRcs.LG

TL;DR

Multi-vector retrieval offers strong retrieval quality but retains high indexing and storage costs because documents contain many token vectors. EigenLI compresses documents into spectral subspaces, and its performance varies by model while its single-vector form outperforms MUVERA in the reported experiments.

  • Problem

    Multi-vector models use many token-level vectors, leaving indexing and storage costs high despite prior efficiency work.

    Source: §1 · p. 1

  • Method

    EigenLI uses each document’s top-k spectral subspace and a query-projection scoring function, with an exact high-dimensional single-vector form called EigenLI-SV.

    Source: §1.1.1 · p. 2, §3 · p. 4, §3.1 · p. 5

  • Results

    At k = 32, EigenLI improves over Ward for ColBERTv2 and ColQwen3, is comparable on AnswerAI-ColBERT-small, but underperforms Ward on GTE-ModernColBERT; EigenLI-SV substantially outperforms MUVERA.

    Source: §5 · pp. 10–11

  • Takeaways & Limitations

    EigenLI-SV’s single-vector encodings can use highly optimized approximate-nearest-neighbor systems.

    Source: §1.1.2 · p. 3

    EigenLI-SV’s dimension scales quadratically with token-vector dimension, making it substantially more expensive for models with larger token embeddings.

    Source: §5 · p. 11

Abstract

from arXiv · show

Late-interaction models such as ColBERT achieve strong effectiveness by representing each document with many token-level vectors, but this expressivity leads to large indexing cost, storage footprints and expensive MaxSim scoring. We show that late-interaction representations exhibit an intrinsic low-rank structure: document token embeddings concentrate in a low-dimensional subspace that preserves most of the retrieval signal. Leveraging this observation, we introduce EigenLI, a spectral approximation framework that compresses late-interaction representations via document-specific low-dimensional subspaces. Unlike clustering or pooling methods, EigenLI identifies the dominant eigendirections of each document and uses them to construct reduced interaction representations. Empirically, $k$-EigenLI with $k \le 32$ outperforms k-means and Ward clustering based pooling methods on ColBERTv2 and AnswerAI-ColBERT-small; GTE-ModernColBERT exhibits a different tradeoff at $k=32$, where clustering methods perform better. The same spectral construction also yields EigenLI-SV, an ANN-compatible single-vector representation derived from the second-order summary of the reduced structure. Across multiple datasets and all three text models, EigenLI-SV consistently outperforms comparable single-vector surrogates such as MUVERA.

1 Introduction

EigenLI compresses document token vectors into document-specific spectral subspaces, motivated by the cost of multi-vector retrieval. Its results vary by model, while EigenLI-SV provides a single-vector alternative that outperforms MUVERA in the reported experiments.

1 Introduction

  • Multi-vector retrieval uses token-level vectors and MaxSim, but the resulting indexing and storage costs motivate reducing document-vector counts.Visual-document retrieval can use around 1000 vectors per document, further motivating smaller representations.Source: §1 · pp. 1–2

1.1 Our Contributions

  • EigenLI represents each document with the top-k eigenvectors of its second-moment matrix and scores query tokens by their squared projection mass onto that subspace.This replaces a set of token vectors with a subspace representation rather than a smaller set of representative vectors.Source: §1.1.1 · p. 2

1.1.2 EigenLI-SV

  • On 13 BEIR development sets, ColBERTv2 EigenLI-SV at k = 32 improves nDCG@10 over MUVERA by 69.6% under the geometric mean of per-dataset performance ratios.Its 8256-dimensional encoding is smaller than MUVERA’s 10240-dimensional encoding.Source: §1.1.2 · p. 3

1.1.3 Model and Domain Coverage

  • At k = 32, EigenLI improves over Ward for ColBERTv2 and ColQwen3, is comparable on AnswerAI-ColBERT-small, and underperforms Ward on GTE-ModernColBERT.For ColBERTv2, arithmetic-mean relative nDCG@10 improvements are 8.0% over Ward; for ColQwen3, they are 3.4%.Source: §1.1.3 · p. 3

2 Related Work

Prior work compresses multi-vector representations through pruning, pooling, or training-based approaches; EigenLI instead uses document-specific spectral subspaces and a new scoring function.

Compression strategies

  • Training-free compression includes heuristic or learned pruning and clustering-based pooling; Ward pooling performs well on average, while training-based methods learn pooling, memory-token, or clustering strategies.The cited work reports Ward pooling outperforming k-means on average across BEIR datasets.Source: §2 · p. 3

Spectral compression

  • Unlike standard PCA uses on single-vector embeddings [44], EigenLI applies a document-specific top-k eigenspace to reduce the number of vectors and adds a retrieval scoring function.The paper notes that representing documents as subspaces has prior literature, but distinguishes its use cases, methodology, and techniques.Source: §2 · pp. 3–4

3 Method: k-EigenLI

k-EigenLI replaces each document’s token-vector set with a document-specific top-k eigenspace. EigenLI-SV captures the resulting score in a single-vector representation, and the paper relates the subspace score to MaxSim under a residual-bound condition.

3 Method: k-EigenLI

  • k-EigenLI compresses each document using the top-k eigenspace of its token-vector second-moment matrix.The eigenspace is document-specific, rather than a set of token representatives obtained by clustering.Source: §3 · p. 4

3.1 EigenLI-SV: A Single-Vector Derivative of k-EigenLI

  • EigenLI-SV encodes the k-EigenLI score in a single vector of dimension d(d + 1)/2, independent of k.For d = 128, the encoding has 8,256 dimensions and is mathematically score-equivalent to the compressed multivector; implementation metrics agree within 10^-3 numerical tolerance.Source: §3.1 · p. 5

3.2 Relating MaxSim to our score function

  • Under a per-token projection-residual bound, the authors’ analysis suggests that documents scoring highly under MaxSim also score highly under the subspace function.The analysis assumes each document token is within δ of its projection onto the top-k subspace.Source: §3.2 · p. 7
  • A preliminary eigenvalue-weighted alternative consistently underperformed the subspace-based scoring technique.The authors note that their chosen scoring function does not use the eigenvalues.Source: §3.2 · p. 7

4 Experiments

The experiments evaluate EigenLI across text and multimodal retrieval benchmarks against clustering-based pooling and single-vector baselines. Results vary by model for multi-vector compression, while EigenLI-SV outperforms MUVERA across the evaluated BEIR datasets.

Experimental setup

  • Evaluation covers MS MARCO passages, BEIR, and ViDoRe-v3, comparing multi-vector compression with k-means++ and Ward and single-vector representations with MUVERA.The BEIR evaluation includes three text models; ViDoRe-v3 uses ColQwen3 VL 4B, and the single-vector comparison excludes ColQwen3 because its 51,360-dimensional encoding is prohibitive.Source: §4 · pp. 7–8

Performance of k-EigenLI

  • EigenLI-32 improves over k-means++ and Ward across Recall, nDCG, and MRR on ColBERTv2 and AnswerAI-ColBERT-small, while GTE-ModernColBERT trails both on nDCG and MRR.For GTE-ModernColBERT, EigenLI is competitive with k-means++ at deeper Recall cutoffs.Source: Table 1 · p. 8, §4 · p. 8

Comparison with MUVERA

  • 78.9%, 178.1%, and 156.0% are the arithmetic-mean nDCG@10 improvements of EigenLI-SV over MUVERA for ColBERTv2, AnswerAI-ColBERT-small, and GTE-ModernColBERT, respectively.The latter two comparisons use centered MUVERA; ColBERTv2 uses the original representation.Source: Table 3 · p. 9, §4 · p. 9

Performance on anisotropic models and centering

  • Centering substantially improves MUVERA across all model-level macro averages, while EigenLI-32 changes little on AnswerAI-ColBERT-small and declines mildly on GTE-ModernColBERT.Centered MUVERA remains below uncentered EigenLI-32 in nDCG@10.Source: §4 · p. 9, Table 4 · p. 10

Compatibility with vector quantization

  • EigenLI-SV-32 is evaluated with FAISS IndexPQ at 1, 2, 4, and 8 bits per dimension, reporting relative Recall and nDCG losses against full-MaxSim results.The reported dataset set includes ArguAna, FiQA, SCIDOCS, and SciFact; entries report Recall loss at @10, @100, and @1000.Source: §4 · p. 10, Table 5 · p. 10

Compression cost

  • Across 39 model–dataset pairs, k-means++ takes 6.81× longer than EigenLI and Ward takes 17.46× longer under arithmetic-mean time ratios.The corresponding geometric-mean ratios are 6.47× and 16.89×.Source: §4 · p. 10

5 Discussion, Limitations, and Future Directions

EigenLI uses low-dimensional spectral subspaces to compress document representations, but its effectiveness depends on model geometry. The discussion identifies practical evaluation and scalability gaps, alongside open questions about when spectral or clustering-based compression is preferable.

Discussion

  • At k = 32, EigenLI improves over Ward on ColBERTv2 and ColQwen3, is comparable on AnswerAI-ColBERT-small, and underperforms on GTE-ModernColBERT.EigenLI-SV also substantially outperforms MUVERA at comparable representation dimensionality.Source: §5 · p. 10

Limitations

  • A large-scale EigenLI pipeline using EigenLI-SV, potentially quantization, ANN retrieval, and full-MaxSim reranking has not been compared against optimized Ward-pooling and PLAID systems.The proposed comparison should consider effectiveness, latency, index size, construction cost, and memory usage.Source: §5 · pp. 10–11
  • EigenLI-SV dimension grows quadratically with token-vector dimension, while the current quantization experiments are limited in scope.The paper identifies sketching, dimensionality reduction, learned projections, and a more systematic quantization study as directions for improving scalability.Source: §5 · p. 11

Future Directions

  • The geometric properties determining when EigenLI or Ward is preferable remain open, as does whether combining them could improve robustness.The paper also asks whether EigenLI can be modified so effectiveness increases monotonically with the number of retained directions.Source: §5 · p. 11
  • GTE-ModernColBERT underperforms Ward, and EigenLI can decline substantially as k rises from 8 to 32 on datasets such as DBPedia-Entity.The authors describe this as highly degenerate low-rank structure and identify more robust spectral compression as future work.Source: §5 · p. 11
  • The paper asks whether retrieval-relevant document ordering, rather than uniform score approximation, could explain EigenLI-SV’s empirical advantage over MUVERA.It also raises whether MUVERA could exploit low-rank structure, and what other mathematical structure in token-level embeddings might support compression.Source: §5 · pp. 11–12

A Detailed ViDoRe-v3 Results

The ViDoRe-v3 appendix presents detailed nDCG results across its datasets.

A Detailed ViDoRe-v3 Results

  • Table 12 reports nDCG across ViDoRe-v3 datasets.The supplied table caption does not specify result values or a winning method.Source: Table 12 · p. 18
  • The reported results cover multiple ViDoRe-v3 datasets.The caption does not identify the dataset names.Source: Table 12 · p. 18
  • The table is scoped to nDCG rather than another retrieval metric.No metric values are provided in the caption.Source: Table 12 · p. 18

B Detailed ColBERTv2 BEIR Results

The ColBERTv2 BEIR appendix provides detailed nDCG and MRR results, complementing recall results under the same evaluation protocol and compression budgets.

B Detailed ColBERTv2 BEIR Results

C Detailed Anisotropic-Model BEIR Results

The appendix compares centered and uncentered representations for AnswerAI-ColBERT-small and GTE-ModernColBERT, with only MUVERA using centered representations in the full comparisons.

C Detailed Anisotropic-Model BEIR Results

Related papers

Loading 2609.07561v1…