PaperlayerView exact revision on arXiv

Source-linked AI summary

Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Framework

Yibo Yan, Mingdong Ou, Yi Cao, Xin Zou, Jiahao Huo, Shuliang Liu, James Kwok, Xuming Hu

arXiv:2602.19549v2cs.CLcs.CVcs.IR

TL;DR

Multi-vector VDR improves visual document retrieval but creates prohibitive storage and computation overhead, while pruning and merging alone impose different compression–fidelity trade-offs. PRUNE-THEN-MERGE addresses this with adaptive pruning followed by hierarchical merging, extending near-lossless compression and maintaining strong performance at aggressive rates.

  • Problem

    Multi-vector VDR offers strong retrieval performance but incurs prohibitive overhead, while existing pruning and merging methods imperfectly balance compression and feature fidelity.

  • Method

    PRUNE-THEN-MERGE adaptively prunes low-information patches and then hierarchically merges the remaining semantically rich embeddings.

  • Results

    10 percentage points: across 29 VDR datasets, the method extends DocPruner’s near-lossless compression range from 50-60% to 60-70% and outperforms baselines at high compression rates.

  • Takeaways & Limitations

    PRUNE-THEN-MERGE provides a practical framework for compressing multi-vector representations while retaining near-lossless retrieval performance at aggressive compression rates.

  • Takeaways & Limitations

    The pruning stage depends on the base LVLM’s internal attention as a proxy for patch importance, and the framework uses predefined adaptation and merging hyperparameters.

Abstract

from arXiv · show

Visual Document Retrieval (VDR), which aims to retrieve relevant pages within vast corpora of visually-rich documents, is of significance in current multimodal retrieval applications. The state-of-the-art multi-vector paradigm excels in performance but suffers from prohibitive overhead, a problem that current efficiency methods like pruning and merging address imperfectly, creating a difficult trade-off between compression rate and feature fidelity. To overcome this dilemma, we introduce Prune-then-Merge, a novel two-stage framework that synergizes these complementary approaches. Our method first employs an adaptive pruning stage to filter out low-information patches, creating a refined, high-signal set of embeddings. Subsequently, a hierarchical merging stage compresses this pre-filtered set, effectively summarizing semantic content without the noise-induced feature dilution seen in single-stage methods. Extensive experiments on 29 VDR datasets demonstrate that our framework consistently outperforms existing methods, significantly extending the near-lossless compression range and providing robust performance at high compression ratios.

1 Introduction

VDR retrieves relevant pages from visually rich document corpora using textual and visual cues, with multi-vector retrieval improving fine-grained matching but incurring substantial efficiency costs. PRUNE-THEN-MERGE combines adaptive pruning and hierarchical merging to preserve performance under compression.

  • VDR retrieves relevant document pages from large corpora using queries that combine textual and visual cues.
  • Multi-vector retrieval represents each page with patch-level embeddings, enabling fine-grained late-interaction matching for complex document content.
  • Multi-vector models face prohibitive storage and computational overhead because each page may require hundreds or thousands of vectors.
  • Pruning removes less informative patches but can decline sharply at high compression, while merging reduces vectors but may dilute distinctive features.
  • PRUNE-THEN-MERGE first filters low-information patches adaptively, then hierarchically merges the remaining semantically rich vectors.
  • 10 percentage points: PRUNE-THEN-MERGE extends DocPruner’s near-lossless compression range from 50-60% to 60-70%.
  • At compression rates of 80% and above, PRUNE-THEN-MERGE consistently outperforms all baselines and avoids pruning-only performance cliffs.

2 Related Work

VDR evolved from OCR-based retrieval toward LVLM-based single-vector and patch-level multi-vector representations. Efficiency work focuses on pruning or merging embeddings, each retaining distinct limitations that motivate a hybrid approach.

  • OCR-based VDR indexes extracted text but can discard layout and nontextual information needed for complex documents.
  • Single-vector LVLM representations preserve visual integrity but often miss fine-grained details needed for precise matching.
  • Patch-level retrieval represents documents and queries as sets of embeddings and computes relevance through late interaction such as MaxSim.
  • Multi-vector storage costs O(Np ×D) per page, compared with O(D) for single-vector models.
  • Pruning discards redundant patches, whereas merging combines multiple patches into fewer vectors; pruning can drop sharply at high rates and merging can dilute salient features.

3 Methodology

PRUNE-THEN-MERGE compresses multi-vector VDR representations by adaptively pruning low-information patches before hierarchically merging the remaining semantic embeddings. This decomposition reduces storage and online computation while aiming to preserve retrieval fidelity by limiting noise-induced feature dilution.

  • Framework overview: PRUNE-THEN-MERGE is a query-agnostic offline framework that combines adaptive pruning with hierarchical merging of document patch embeddings.The framework first filters low-information patches, then compresses the remaining semantically rich patches.
  • Stage 1: Adaptive Pruning: Adaptive pruning uses LVLM attention as a proxy for patch importance and retains patches whose document-specific scores exceed an adaptive threshold.The threshold is based on the mean and standard deviation of patch importance scores; if no patch survives, the highest-scoring patch is retained.
  • Stage 2: Hierarchical Merging: Hierarchical agglomerative clustering compresses the filtered embeddings into fewer representatives by normalizing them, computing cosine distances, and forming target-sized clusters.Each cluster is summarized by a centroid, producing semantically rich compressed representations.
  • Retrieval and efficiency: The compressed document representation reduces the original embedding count, lowering MaxSim computation, online cost, and offline storage requirements.Retrieval computes relevance over the final compressed embedding set, where the reduced patch count is much smaller than the original.
  • Theoretical motivation: The framework models compression as an information-filtering stage followed by redundancy reduction, approximating an otherwise intractable Information Bottleneck objective.Pruning targets low-information noise, while merging addresses semantic redundancy through a compact representation.
  • The synergistic advantage: Pruning before merging reduces centroid bias from noise and yields lower distortion relative to the true signal than merging the full patch set directly.Noise vectors can pull naive centroids away from the signal center, whereas pre-filtering produces centroids that are largely unbiased by noise.

4 Experiments and Analysis

Experiments evaluate PRUNE-THEN-MERGE across multiple VDR benchmarks, models, and compression settings. The framework preserves retrieval quality at high compression and generally outperforms pruning-only and merging-only baselines.

  • Experimental setup: Evaluation covers six VDR benchmarks totaling 29 datasets and integrates ColQwen2.5, ColNomic, and Jina-v4 as base models.
  • Baselines: The benchmark compares uncompressed base models with random, adaptive pruning, and merging-based baselines.
  • VDR performance comparison: PRUNE-THEN-MERGE maintains near-lossless performance up to approximately 70% pruning across 16 datasets from four major VDR benchmarks.
  • VDR performance comparison: At approximately 84–87% pruning on ViDoRe-V1 with ColQwen2.5, PRUNE-THEN-MERGE reaches nDCG@5 0.86, while DocPruner falls to 0.77.
  • Generalization to complex settings: The framework generalizes across nine JinaVDR languages, achieving overall nDCG@5 0.52 at 84% compression versus DocPruner’s 0.46 at 80%.
  • Generalization to complex settings: On REAL-MM-RAG, PRUNE-THEN-MERGE reaches nDCG@5 0.65 at 84% compression with ColQwen2.5, surpassing DocPruner at 0.56 and Sem-Cluster at 0.61.
  • Variant analysis: At approximately 80% compression on ViDoRe-V2, the full framework achieves nDCG@5 0.55, compared with 0.51 for its adaptive pruning-only variant.
  • Efficiency analysis: Across three base models, storage decreases 54.60% overall and 58.88% for ColQwen2.5, while average nDCG@5 decreases 0.45% and encoding latency rises from 0.46s to 0.69s.

5 Conclusion

The paper presents PRUNE-THEN-MERGE as a two-stage compression framework for efficient multi-vector VDR. Experiments show that it extends near-lossless compression and preserves performance at aggressive compression rates.

  • PRUNE-THEN-MERGE combines adaptive pruning with hierarchical merging in a first-refine-then-compress framework.
  • Experiments demonstrate an extended near-lossless compression range and superior performance at aggressive compression rates.

Limitations

The framework has two stated limitations concerning the reliability of attention-based patch importance and reliance on predefined compression hyperparameters.

  • The pruning stage depends on the base LVLM’s internal attention being a reliable proxy for patch importance.The authors plan to investigate query-independent metrics such as gradient-based importance.
  • The framework relies on predefined adaptation and merging factors to balance compression and performance.The authors plan more automated, data-driven parameter selection based on document complexity and layout.

A Illustrative Examples

The paper situates visual document retrieval within multi-vector retrieval, benchmarks, and efficiency-oriented compression. It presents PRUNE-THEN-MERGE as a two-stage workflow that filters low-information patches before compressing the remaining embeddings.

  • A Illustrative Examples: Figure 11 contrasts attention-based and random pruning, showing why selective removal helps at low ratios but both methods lose critical content at extreme ratios.At 10% pruning, attention-based pruning removes non-informative whitespace more effectively; at 90%, both methods cause catastrophic content loss.
  • A Illustrative Examples: Multi-vector retrieval represents each document as patch-level embeddings and uses MaxSim for fine-grained token-to-patch matching.This paradigm improves matching precision but introduces substantial storage and computation costs.
  • A Illustrative Examples: ColPali extends late-interaction retrieval to visual documents by generating multi-vector patch embeddings directly from document images.This supports matching query representations against specific image patches in visually rich documents.
  • A Illustrative Examples: The paper evaluates diverse visual-document benchmarks spanning ViDoRe, JinaVDR, Real-MM-RAG, and other multilingual or multimodal settings.The benchmark illustrations include ViDoRe-V1, ViDoRe-V2, JinaVDR, and Real-MM-RAG.
  • A Illustrative Examples: PRUNE-THEN-MERGE first adaptively removes low-information patches and then hierarchically merges the semantically richer remainder.The framework formalizes offline compression followed by MaxSim-based online retrieval, with patch embeddings and attention weights as inputs.

D.3 Analysis of Synergistic Gain

The analysis argues that pruning followed by merging reduces distortion more effectively than merging raw patch sets. Its rationale is to remove low-information noise before estimating centroids or quantizing the remaining signal.

  • D.3 Analysis of Synergistic Gain: Sequential pruning and merging yields a synergistic distortion reduction relative to a naive single-stage merge.The comparison is defined with respect to the true signal set Dsig.
  • D.3 Analysis of Synergistic Gain: A naive merge over the full patch set produces biased centroids because noise shifts the estimated center of mass away from the signal.The proof sketch identifies this shift as a bias term in the representation.
  • D.3 Analysis of Synergistic Gain: Pruning first makes the remaining set approximately signal-only, allowing centroids to better estimate the true signal distribution.The framework treats centroids computed on the nearly noise-free set as approximately unbiased estimators.
  • D.3 Analysis of Synergistic Gain: The compression pipeline forms the Markov chain D → D′ → D′′, so information cannot increase across the two stages.The stated goal is to keep both information inequalities close to equality through information-preserving pruning and efficient rate-distortion compression.

E.6 MMLongBench-Doc Benchmark

MMLongBench-Doc evaluates long-context multimodal document understanding using lengthy, information-dense PDFs and expert-annotated questions. Its design tests both page localization and reasoning across multiple pages.

  • E.6 MMLongBench-Doc Benchmark: MMLongBench-Doc contains 1,082 expert-annotated questions across 135 PDF documents averaging 47.5 pages and more than 21,000 tokens.The benchmark targets long-context, multimodal document understanding rather than primarily single-page tasks.
  • E.6 MMLongBench-Doc Benchmark: Its questions require evidence from diverse sources within lengthy documents.This structure makes the benchmark suitable for testing information retrieval and long-context reasoning together.
  • E.6 MMLongBench-Doc Benchmark: The benchmark includes single-page, cross-page, and unanswerable questions to test localization, multi-page reasoning, and hallucination behavior.The reported proportions are 45.7% single-page, 33.7% cross-page, and 20.6% unanswerable questions.
  • E.6 MMLongBench-Doc Benchmark: The implementation section reports empirical hyperparameter searches for representative baseline performance trade-offs.This methodological note accompanies the benchmark evaluation setup.

F.1 Pruning-based Methods

The experiments compare random, attention-based, similarity-aware, and merging baselines with PRUNE-THEN-MERGE across compression settings. Results show strong performance at high compression, especially for complex or weaker-language settings, while gains vary by base model.

  • F.1 Pruning-based Methods: Random pruning uniformly discards a specified fraction of patches and serves as a lower bound for uninformed compression.At least one patch is retained to avoid empty document representations.
  • F.1 Pruning-based Methods: Attention-plus-similarity pruning combines attention importance with cosine similarity to the global embedding before applying a dynamic threshold.The composite score balances importance and representativeness through a weighting factor.
  • F.1 Pruning-based Methods: The two-stage adaptive baseline first identifies important patches by attention and then removes patches redundant with selected pivots.This combines attention filtering with similarity-based deduplication.
  • F.1 Pruning-based Methods: Hierarchical clustering merges semantically related embeddings into centroid vectors, while sequence and 2D pooling provide alternative merge-only baselines.The clustering reduction is controlled by a merging factor, whereas 2D pooling uses spatial patch layout.
  • F.1 Pruning-based Methods: 89% compression preserves nDCG@5 of 0.82 for Japanese documents with ColNomic, compared with 0.76 for Sem-Cluster.For Chinese documents, the method maintains 0.95 nDCG@5 at 89% compression, while Sem-Cluster reaches 0.94.
  • F.1 Pruning-based Methods: At 83% compression on Hindi, PRUNE-THEN-MERGE achieves 0.46 nDCG@5 versus DocPruner’s 0.40.On Dutch, it maintains the baseline score of 0.68 up to 82% compression, while DocPruner drops to 0.56.
  • F.1 Pruning-based Methods: At 84% compression with ColQwen2.5, PRUNE-THEN-MERGE reaches 0.65 nDCG@5 versus 0.61 for Sem-Cluster.The reported advantage is model-dependent and is not observed equally across Jina-v4 and ColNomic.

G.5 Variant Analysis

Variant analyses show that adaptive, document-specific pruning outperforms fixed-ratio or global-threshold alternatives, while PRUNE-THEN-MERGE is especially advantageous for semantically complex datasets and maintains stronger performance at high compression.

  • Adaptive pruning: At approximately 55% pruning, adaptive methods reach nDCG@5 0.56, versus 0.52 for attention-threshold-nfp and 0.51 for attention-ratio.Adaptive thresholding adjusts pruning to each document's content density, unlike fixed thresholds or ratios.
  • Dataset-dependent gains: The performance gap widens on human-annotated esg_human_labeled_v2, a dataset described as having higher semantic complexity.The paper attributes this pattern to conceptual human-generated queries requiring more robust semantic representations.
  • Cross-model comparisons: Figure 12 compares nDCG@5 across Jina-v4, ColQwen2.5, and ColNomic, distinguishing adaptive from non-adaptive and pruning from merging methods.Solid lines indicate adaptive methods, dashed lines non-adaptive methods; circular nodes indicate pruning and square nodes merging.
  • Cross-dataset comparisons: Figures 13 and 14 extend the same performance comparison to ViDoSeek and MMLongBench-Doc across the three base models.These figures use the same visual distinctions for adaptation and method type.
  • Dataset-dependent gains: At approximately 80% compression on esg_human_labeled_v2, PRUNE-THEN-MERGE scores 0.60 nDCG@5, compared with 0.54 for adaptive-pruning-only.The gap is narrower on economics_reports_v2 at the same compression level: 0.51 versus 0.49.
Loading 2602.19549v2…