Source-linked AI summary
Multi-Vector Index Compression in Any Modality
Hanxiang Qin, Alexander Martin, Rohan Jha, Chunsheng Zuo, Reno Kriz, Benjamin Van Durme
TL;DR
Multi-vector late-interaction retrieval is costly for long multimodal documents because computation and storage grow with document length. The paper evaluates fixed-budget, query-agnostic compression and introduces attention-guided clustering, which consistently outperforms the other compression methods across modalities and can exceed uncompressed performance on some benchmarks.
Problem
Late-interaction retrieval needs fixed-budget representations because computation and storage costs grow linearly with multimodal document length.
Method
The paper compares sequence resizing, memory tokens, hierarchical pooling, and attention-guided clustering for query-agnostic multi-vector index compression.
Results
AGC consistently outperforms other compression methods across modalities and maintains 97% of uncompressed performance at nDCG@10.
Takeaways & Limitations
AGC provides robust fixed-budget compression across modalities, with competitive or improved retrieval performance relative to uncompressed indexing.
Takeaways & Limitations
SeqResize can leave tokens unused, MemTok can collapse distinct features, and H-Pool’s greedy merging is vulnerable to noisy outliers in multimodal data.
Abstract
from arXiv · showhide
We study efficient multi-vector retrieval for late interaction in any modality. Late interaction has emerged as a dominant paradigm for information retrieval in text, images, visual documents, and videos, but its computation and storage costs grow linearly with document length, making it costly for image-, video-, and audio-rich corpora. To address this limitation, we explore query-agnostic methods for compressing multi-vector document representations under a constant vector budget. We introduce four approaches for index compression: sequence resizing, memory tokens, hierarchical pooling, and a novel attention-guided clustering (AGC). AGC uses an attention-guided mechanism to identify the most semantically salient regions of a document as cluster centroids and to weight token aggregation. Evaluating these methods on retrieval tasks spanning text (BEIR), visual-document (ViDoRe), and video (MSR-VTT, MultiVENT 2.0), we show that attention-guided clustering consistently outperforms other parameterized compression methods (sequence resizing and memory tokens), provides greater flexibility in index size than non-parametric hierarchical clustering, and achieves competitive or improved performance compared to a full, uncompressed index. The source code is available at: github.com/hanxiangqin/omni-col-press.
1 Introduction
Multimodal retrieval must index increasingly large and varied collections, but multi-vector representations become prohibitively costly as document length grows. The paper introduces four compression methods, including attention-guided clustering, and evaluates them across modalities, finding AGC strongest across learned compression settings.
- Motivation: Multimodal collections require scalable indexing, but storing multi-vector representations creates severe storage costs.A single video may require 10MB, implying an estimated 140 Petabytes for 14 billion videos.
- Motivation: Late-interaction computation and storage grow linearly with document length, creating a barrier for multimodal documents that may contain thousands of tokens.The paper also reports that many such token representations are underutilized in late interaction.
- Approach: The paper adapts sequence resizing, memory tokens, and hierarchical pooling, while introducing attention-guided clustering to address redundancy and noise.AGC uses learnable universal query tokens to select centroids and weight cluster aggregation.
- Findings: AGC is reported as the strongest compression technique across four retrieval tasks and three modalities, with new state-of-the-art results on ViDoRe and MSR-VTT.The evaluated tasks are BEIR, ViDoRe, MSR-VTT, and MultiVENT 2.0.
- Findings: Compression training improves over an uncompressed multi-vector index on ViDoRe and MSR-VTT.The paper attributes this finding to reducing redundancy and noise in multimodal inputs.
2 Related Work
Prior work compresses multi-vector indices through dimensionality reduction, token reduction, pooling, projection, clustering, and quantization. Attention-based compression distinguishes query-aware pruning, which is incompatible with precomputed retrieval indices, from query-agnostic importance estimation, where a gap remains for this setting.
- Multimodal Retrieval: Multimodal retrieval benchmarks include text-only BEIR, natural-language video retrieval datasets, and the multilingual MultiVENT 2.0 benchmark.These benchmarks measure retrieval across text, video, and audiovisual settings.
- Multi-Vector Index Compression: Multi-vector indices can be compressed along vector dimension, sequence length, token selection, pooling, projection, clustering, and quantization axes.The paper focuses on reducing the number of sequence tokens.
- Attention-based Compression: Query-aware token pruning uses prompt attention, whereas retrieval indexing requires query-agnostic document representations computed before the query is known.Query-agnostic approaches instead use self-attention scores or learnable parameters to estimate token importance.
3 Preliminaries
The paper formulates multimodal late-interaction retrieval as ranking documents using fixed-budget multi-vector representations generated without access to the future query. Compression therefore seeks high retrieval accuracy under a constant storage budget while preserving information relevant to unknown queries.
- Retrieval: Retrieval ranks documents by relevance to a text query over a collection containing one or more modalities.The collection may contain text, audio, and visual content.
- Late Interaction: ColBERT-style late interaction sums each query token’s maximum dot-product similarity with document vectors.The operation compares each query embedding against the document representation and aggregates the maxima.
- Problem Formulation: Multimodal document compression generates representations optimized for scalable late-interaction retrieval under constraints.The document mapping produces an optimized representation rather than retaining the original sequence unchanged.
- Problem Formulation: A document mapping transforms each document into m vectors, where m is fixed independently of the document’s original length.The mapping may combine direct encoding with parametric or heuristic compression.
- Problem Formulation: Because indexing occurs before the query is known, document compression must be query-agnostic while preserving information likely to matter for future queries.Parameterized mappings use learnable weights, and their objective is to maximize retrieval accuracy within the fixed storage budget.
4 Multi-Vector Compression
The section compares three established multi-vector compression methods and hierarchical pooling, describing how each produces a fixed-budget representation and where multimodal data exposes their limitations.
- Sequence Resizing: SeqResize projects encoder outputs along the sequence dimension to produce a fixed-token compressed representation.A two-layer MLP performs the resizing after representations are padded or truncated to a fixed length.
- Memory Tokens: MemTok appends learnable memory tokens, whose final encoder states form the document representation.The document and memory tokens are jointly processed with bidirectional self-attention, after which non-memory positions are discarded.
- Hierarchical Pooling: H-Pool iteratively merges similar vectors and replaces each resulting cluster with its mean without requiring compression-specific training.Ward-linkage agglomerative pooling continues until the target number of clusters remains, with optional protected tokens concatenated afterward.
- Limitations: Under fixed token budgets, SeqResize may leave tokens unused, MemTok can smooth over distinct features, and both may encode redundant multimodal signals.The cited limitations include modeling failure, information collapse, and inefficient allocation to silent, static, or repetitive content.
- Limitations: H-Pool removes redundancy but greedy iterative merging remains vulnerable to noisy multimodal outliers.Its heuristic merging strategy is modality-agnostic, but the cited limitation is sensitivity to noisy outliers.
5 Attention-Guided Clustering
Attention-Guided Clustering compresses documents by selecting salient token centroids with universal queries, assigning remaining tokens to them, and aggregating each cluster with saliency-weighted averages.
- Attention-based Centroid Selection: AGC uses learned universal query tokens to estimate token importance without relying on a specific user query.The universal queries probe document content, and their attention weights are aggregated into saliency scores.
- Attention-based Centroid Selection: AGC selects the top-m tokens by saliency as cluster centroids for a target compression budget m.The centroids are chosen from the document tokens with the largest saliency scores.
- Clustering: Hard clustering assigns every remaining token to its most cosine-similar centroid, preserving related context while reducing redundancy.Unlike iterative agglomerative merging, AGC anchors clusters around globally salient centroids and keeps distinct concepts separated through hard assignment.
- Weighted Aggregation: Weighted aggregation forms each compressed vector by averaging assigned document tokens according to their saliency.This weighting reflects unequal information density in multimodal data while retaining continuous token contributions for gradient flow.
6 Experiments
Across text, visual-document, and video retrieval, AGC provides the strongest or most transferable compressed-index performance, often retaining or exceeding uncompressed retrieval quality. Experiments also show that multimodal full indices are costly and underutilized, while attention-guided aggregation better preserves salient information across compression budgets.
- AGC maintains 97% of uncompressed model performance at nDCG@10 across modalities.
- AGC is the only compression method to outperform the full-index base model on MSR-VTT R@1.
- Only about 1% of 1.3 million unique document tokens are active during a single evaluation pass, indicating substantial full-index underutilization.
- At a 32-token budget, MemTok and AGC maintain stable BEIR performance while compressing documents by around 80%.
- On ViDoRe, AGC and H-Pool significantly outperform SeqResize and MemTok, while AGC is more stable across domains.
- Every compression method achieves a new MSR-VTT state of the art, even with only 5 vectors per document.
- Audio sampling limited MultiVENT 2.0 evaluation, and the full model index could not be built because of compute requirements.
- AGC performance improves with larger token budgets and transfers effectively to unseen compression ratios after training at one ratio.
7 Conclusion
The paper introduces AGC for fixed-budget multi-vector compression and finds it robust across modalities, compression ratios, domains, and model specifications. It also identifies static budget allocation as a limitation and proposes content-aware allocation as future work.
- AGC combines attention-based centroid selection, clustering, and weighted aggregation to maximize the utility of a fixed document token budget.
- AGC consistently outperforms the other compression methods across modalities and achieves state-of-the-art performance on MSR-VTT.
- AGC remains strong and robust across varied compression ratios, domains, and model specifications.
- The fixed budget is applied statically, or potentially linearly for H-Pool, rather than in proportion to a document’s inherent informational content.
- Future work could allocate compression budgets according to document informational content using lightweight features such as document token utilization.