PaperlayerView exact revision on arXiv

Source-linked AI summary

Do All Visual Tokens Matter Equally? Object-Evidence Preserving Token Merging for Vision-Language Retrieval

Suhyeong Park, Junha Jung, Jungwoo Park, Jaewoo Kang

arXiv:2607.04605v2cs.IRcs.AIcs.CLcs.CV

TL;DR

Multi-vector vision-language retrieval is expensive because dense image tokens increase storage and scoring costs, while compression can remove query-relevant object evidence. SaMer uses object-aware merging to preserve that evidence and, with K=64, removes over 93% of image-side tokens while improving retrieval on Flickr30K and MSCOCO.

  • Problem

    Dense image-side tokens make late-interaction retrieval expensive, while query-dependent compression can remove or mix object- and region-level evidence needed for matching.

  • Method

    SaMer merges post-projector image tokens into K representatives using feature-spatial assignments, with training-time object annotations discouraging cross-instance mixing and projection-only adaptation.

  • Results

    With K = 64, SaMer removes more than 93% of image-side tokens, reduces ColPali storage by 16.09×, and improves R@1 on Flickr30K and MSCOCO.

  • Takeaways & Limitations

    Efficient multi-vector retrieval benefits from preserving query-selectable object evidence, not merely reducing visual token count.

Abstract

from arXiv · show

Multi-vector vision-language retrieval preserves fine-grained visual evidence through maximum-similarity late interaction, but dense image-side tokens make storage and scoring expensive. Existing token compression methods reduce this cost, yet they can remove or collapse object- and region-level evidence that future query tokens may need to select. We propose SaMer, an object-aware token merging framework that compresses image-side post-projector tokens into $K$ representative centroids while preserving the original late-interaction interface. SaMer uses object annotations only during training as a merge prior to discourage cross-instance mixing, requires no ground-truth bounding boxes or detectors at inference time, and adapts only the shared projection layer with frozen vision and language backbones. With $K=64$, SaMer removes more than 93% of image-side tokens and reduces ColPali storage by $16.09\times$, while improving R@1 on Flickr30K and MSCOCO. These gains arise because object-aware merging preserves query-selectable object evidence that pruning or feature-only pooling can remove or collapse. SaMer also outperforms compression baselines and shows stronger phrase-level grounding, suggesting that efficient multi-vector retrieval depends not only on reducing token count, but on preserving the evidence future query tokens need to select.

1 Introduction

SaMer frames visual-token compression for late-interaction retrieval as preserving query-selectable object and region evidence while reducing storage and scoring costs. It uses training-time object annotations as a merge prior, requires no annotations or detector at inference, and achieves substantial compression with improved retrieval performance.

  • 1 Introduction: Late-interaction retrieval compares each query token with all image-side tokens, preserving object-, attribute-, and relation-level evidence but increasing storage and scoring costs.Multi-vector retrievers may store hundreds to over a thousand image-side token embeddings per image.
  • 1 Introduction: Query-dependent patch importance makes retrieval compression different from generic visual-token reduction, because different queries may require different object or attribute evidence.Pruning or merging can remove evidence needed by future MaxSim queries.
  • 1 Introduction: SaMer compresses post-projector visual tokens into K normalized weighted centroids using feature-spatial soft assignment while preserving the original late-interaction interface.Its object-aware design targets object- and region-level semantic evidence for future MaxSim queries.
  • 1 Introduction: Training-time object annotations guide token grouping as a merge prior, discouraging cross-instance merging without requiring ground-truth bounding boxes or an object detector at inference.SaMer performs annotation-free feature-spatial merging and remains pluggable into existing multi-vector retrievers.
  • 1 Introduction: K = 64 removes more than 93% of image-side tokens, reduces ColPali image-side storage by 16.09×, and improves retrieval performance.The paper also evaluates whether phrase-level evidence remains concentrated inside annotated phrase regions.

2 Related Works

Related work spans contrastive and multimodal-pretraining approaches, late-interaction multi-vector retrieval, and visual-token reduction. SaMer differs from prior reduction methods by compressing post-projector retrieval embeddings used for multi-vector indexing and late-interaction scoring.

  • Vision-language retrieval: Contrastive and multimodal-pretraining models provide strong image-text representations but can underrepresent fine-grained objects, attributes, and regions.Examples include ALIGN, AL-BEF, BLIP, and BLIP-2.
  • Late-interaction retrieval: Late-interaction retrievers such as ColPali and ColQwen2 preserve visual evidence with multiple embeddings but incur storage and scoring costs.The cost arises from storing and scoring many visual token embeddings per candidate.
  • Visual token reduction: Prior token-reduction methods mainly target transformer computation, whereas SaMer compresses post-projector retrieval embeddings stored in multi-vector indexes and used for late-interaction scoring.Related methods include LLaVA-PruMerge, LVPruning, VisionZip, ZipVL, LiteLVLM, and ToMe.

3 Preliminaries

Multi-vector vision-language retrieval represents images and text as token sets in a shared retrieval space and scores them through MaxSim late interaction. This fine-grained matching makes the visual-token count a storage and computation bottleneck, motivating representative-token merging.

  • Token representations: Images and text queries are represented as sets of N visual tokens and M query tokens projected into the same D-dimensional retrieval space.The visual tokens are V(I) = {v_i}^N and query tokens are Q(q) = {q_j}^M.
  • MaxSim scoring: MaxSim late interaction matches each query token to its most similar visual token, enabling fine-grained matching of objects, attributes, and regions.The query-image score aggregates these per-query-token maximum similarities.
  • Efficiency bottleneck: Every query token compares with every visual token, making N the bottleneck for retrieval storage and computation.The comparison pattern preserves fine-grained evidence but scales with the number of image-side tokens.
  • Efficiency bottleneck: Full-token retrieval requires O(ND) storage per image and O(MND) dot-product cost per query-image pair.Reducing image-side post-projector tokens directly lowers both costs while preserving the encoder and late-interaction interface.
  • Compression strategy: Token merging aggregates visual tokens into representative tokens, whereas token pruning discards evidence before future queries are known.The distinction motivates compression methods that reduce token count without abandoning late-interaction retrieval.

4 Methods

SaMer compresses post-projector image tokens into K representatives while preserving the original late-interaction retrieval interface. It combines feature-spatial soft assignment with a training-only object-aware merge prior and adapts only the shared projection layer.

  • Token compression: SaMer assigns post-projector visual tokens to K ≪ N representatives, whose merged embeddings replace stored image tokens while MaxSim scoring remains unchanged.This directly reduces index storage and query-time similarity computation without modifying the vision encoder or retrieval scoring function.
  • Token compression: Feature-spatial distance combines feature similarity with spatial coherence, discouraging semantically similar tokens from distant regions from merging too aggressively.Each representative maintains feature and spatial centroids initialized from uniformly spaced image-grid seeds.
  • Token compression: Soft assignment forms each representative as an ℓ2-normalized weighted centroid, aggregating visual tokens rather than discarding them as pruning does.The compressed representation is scored with late interaction over merged tokens.
  • Object-aware training: During training, object annotations provide a weak merge prior that penalizes assignments mixing different object instances, while the final merged tokens still use soft assignment.The prior is estimated with stop-gradient hard assignments and then added to the feature-spatial distance, lowering cross-instance assignment weights.
  • Inference and adaptation: At inference, SaMer uses bbox-free feature-spatial soft assignment, caches K merged visual tokens, and retrieves with the same late-interaction scoring rule.Token merging remains nonparametric, while gradients from compressed MaxSim adaptation update only the shared projection layer.

5 Experiments

SaMer is evaluated with retriever-only experiments across natural-image, compositional-image, and document-image benchmarks, using K = 64 image-side tokens. It improves retrieval most consistently on natural-image and compositional-image tasks, where preserving object- and region-level evidence is important.

  • Evaluation setup: SaMer is evaluated on Flickr30K, MSCOCO, ImageCoDe, and DocVQA using retriever-only evaluation.Flickr30K is treated as in-domain because SaMer is adapted only on the Flickr30K-Entities training split; MSCOCO is cross-dataset.
  • Result interpretation: SaMer is most effective on natural-image retrieval, where redundant visual tokens coexist with query-specific object and region evidence.The results support preserving query-selectable object evidence rather than retaining only globally salient or visually similar patches.
  • Natural-image retrieval: On Flickr30K, SaMer raises ColPali R@1 from 77.0 to 82.4 and ColQwen2 R@1 from 73.6 to 79.3.It also exceeds H-Pool, whose R@1 is 73.7 with ColPali and 71.8 with ColQwen2.
  • Natural-image retrieval: On MSCOCO, SaMer improves ColPali R@1 from 47.4 to 51.6 and outperforms H-Pool, SAP, and HPC.These results extend the Flickr30K trend to a cross-dataset natural-image benchmark.
  • Compositional-image retrieval: With ColPali on ImageCoDe, SaMer improves R@1 from 5.4 to 5.9 and nDCG@10 from 13.2 to 14.4.ImageCoDe requires distinguishing visually similar images using subtle object, attribute, and relation cues.

6 Analysis of SaMer

SaMer’s gains persist under controlled projection-only adaptation because its object-aware merge rule better preserves retrieval-relevant evidence. Analysis identifies K = 64 as a favorable quality–compression operating point and shows improved grounding and efficiency.

  • Adaptation versus merge design: Under the same frozen backbones, projection-only updates, and K = 64 budget, SaMer achieves the best R@1 and nDCG@10 among adapted compression methods.This indicates the improvement is not explained by projection-only adaptation alone.
  • Compression budget analysis: On Flickr30K, R@5 rises from 95.8 at K = 32 to 96.3 at K = 64, then reaches only 96.8 at K = 512.MSCOCO follows the same pattern, increasing from 75.0 at K = 32 to 75.5 at K = 64 and 76.1 at K = 512.
  • Compression budget analysis: At K = 64, SaMer lies above H-Pool, SAP, and HPC on both Flickr30K and MSCOCO, indicating a better quality-budget trade-off.K = 64 is used throughout the main experiments.
  • Merge component study: The object-aware merge prior raises Flickr30K R@1 to 82.4 and MSCOCO R@1 to 51.6, whereas spatial coherence alone provides little benefit.Feature-only merging changes Flickr30K R@1 from 80.7 to 80.4, while MSCOCO R@1 remains nearly unchanged.
  • Phrase-level grounding: SaMer raises BoxMass from 41.3 to 54.2, surpassing the full-token baseline of 51.8, while improving RegionHit to 68.3 and CoverageIoU to 16.4.These results show more concentrated phrase-level relevance under compression.
  • Efficiency: For ColQwen2, SaMer reduces storage by 3.73× on Flickr30K and 1.99× on MSCOCO, while improving QPS by 1.9× and 3.1×, respectively.It also reduces MaxSim comparisons by 3.7× and 2.0×, respectively.

7 Conclusion · A Centroid Construction Variants

SaMer is an object-aware token merging framework that compresses post-projector image tokens into representative centroids while preserving query-selectable evidence. Among centroid construction variants, soft assignment performs best across Flickr30K and MSCOCO by using assignment confidence to preserve retrieval-relevant evidence.

  • 7 Conclusion: SaMer targets efficient multi-vector vision-language retrieval by compressing image-side post-projector tokens into K representative centroids.Its design prioritizes preserving query-selectable evidence rather than merely reducing token count.
  • 7 Conclusion: Object annotations are used only during training to discourage cross-instance merging, with no bounding boxes or object detectors required at inference.The object-aware prior shapes training-time merging without adding inference-time detection requirements.
  • 7 Conclusion: SaMer frames image-side compression as evidence preservation, emphasizing whether future queries can select visual evidence rather than token-count reduction alone.This motivation connects token merging to the requirements of multi-vector retrieval.
  • A Centroid Construction Variants: The mean variant averages assigned tokens without additional normalization, whereas the normalized variant applies normalization after centroid construction.These variants differ in whether normalization follows centroid formation.
  • A Centroid Construction Variants: The medoid variant preserves an existing token by selecting the token closest to the cluster center, while the object-weighted variant emphasizes tokens associated with object regions.The object-weighted approach changes token contributions during centroid construction by assigning higher weights to object-region tokens.
  • A Centroid Construction Variants: Soft assignment, used by SaMer, performs best across all metrics on both Flickr30K and MSCOCO.It outperforms hard mean or medoid-style representatives and the object-weighted variant.
  • A Centroid Construction Variants: Soft assignment preserves retrieval-relevant evidence by allowing tokens to contribute to clusters according to assignment confidence.Its advantage over object-weighted construction suggests object information is more effective when shaping merge assignment than only reweighting tokens afterward.

B Implementation Details

SaMer modifies only image-side post-projector representations, using fixed main-experiment compression and temperature settings while adapting the shared projection layer with frozen vision and language backbones. Object annotations construct a training-only merge prior; inference uses annotation-free feature-spatial soft assignment.

  • Implementation Details: SaMer uses K = 64 merged image tokens, γ = 0.1 spatial weighting, and τ = τs = 0.07 temperatures in main experiments.These settings correspond to retrieval and soft-assignment temperatures, respectively.
  • Implementation Details: During compression-aware adaptation, the vision encoder and language backbone remain frozen while only the shared projection layer is updated.
  • Implementation Details: Object annotations are used only during training to assign background or object-instance labels for constructing the object-aware merge prior.Tokens inside multiple boxes receive the smallest-box label.
  • Implementation Details: At inference, SaMer discards object labels and bounding boxes and performs annotation-free feature-spatial soft assignment.

C Grounding Metrics

The grounding metrics test whether compressed tokens preserve phrase-level visual evidence by evaluating relevance concentration, weak localization, and spatial coverage against annotated phrase regions. BoxMass, RegionHit, and CoverageIoU are higher-is-better measures that jointly assess evidence preservation after aggressive compression.

  • Metric setup: Grounding metrics compare a normalized relevance map over image/grid locations with a ground-truth phrase box mask.They evaluate relevance concentration, weak localization, and spatial coverage.
  • BoxMass: BoxMass measures whether phrase-specific relevance is concentrated inside the annotated object region.Higher BoxMass indicates that more queried visual evidence falls inside the correct object region.
  • RegionHit: RegionHit measures whether the model localizes the target phrase at all using thresholded high-relevance regions across thresholds from 0.1 to 0.9.Its low IoU threshold targets weak phrase-region localization rather than segmentation-quality prediction.
  • CoverageIoU: CoverageIoU measures the degree of spatial agreement between the high-relevance region and the target box, rather than only whether relevance reaches it.Together, the three higher-is-better metrics evaluate phrase-level evidence preservation after aggressive visual token compression.

D Merge Component Study · E Merging vs. Pruning Comparison · F Qualitative Grounding Examples

SaMer’s object-aware merge penalty improves retrieval and grounding over feature-only merging, while merging preserves more query-selectable evidence than pruning. Qualitative examples show that SaMer concentrates relevance more consistently on target regions than competing compression methods.

  • D Merge Component Study: The merge component study reports all retrieval metrics on Flickr30K and MSCOCO alongside object-consistency metrics.The main-paper compact table presents representative retrieval and grounding metrics, while the expanded table reports the complete set.
  • D Merge Component Study: The spatial term alone does not substantially improve over feature-only merging, whereas the object-aware penalty consistently improves retrieval and grounding.The component study reports this trend across the full retrieval and object-consistency metrics.
  • E Merging vs. Pruning Comparison: 73.5 and 45.1 are SaMer’s training-free Flickr30K and MSCOCO R@1 values, versus 68.6 and 68.8 and 40.5 and 41.1 for random and spatial pruning.In the training-free setting, SaMer incurs a smaller retrieval drop than both pruning baselines.
  • E Merging vs. Pruning Comparison: Merging preserves aggregated visual representatives, whereas pruning removes visual evidence before the query is known.The passage identifies this difference as the reason merging is more effective for multi-vector retrieval.
  • F Qualitative Grounding Examples: Compression baselines often spread relevance to nearby objects or background regions, including surrounding clothing, road areas, or non-target people.These methods may recover part of the target region but do not consistently isolate it.
  • F Qualitative Grounding Examples: SaMer more consistently concentrates grounding relevance inside the target region than competing compression methods.Figure 4 compares single-vector baselines, full multi-vector retrievers, SaMer variants, and compression methods applied to multi-vector retrievers.

G Efficiency Measurements · H SaMer Algorithm

SaMer’s efficiency measurements show that visual-token reduction lowers both theoretical MaxSim comparisons and measured retrieval latency. Its algorithm uses object annotations only during training, then performs bbox-free feature-spatial merging and compressed late-interaction retrieval at inference.

  • G Efficiency Measurements: Detailed FP16 storage estimates for one million images use embedding dimension D = 128 to quantify image-side storage reduction.The measurements provide the storage basis for reported reduction ratios.
  • G Efficiency Measurements: Detailed retrieval comparisons report total MaxSim operations, latency, and query throughput, with parenthetical improvements over the corresponding full multi-vector retriever.MaxSim Ops denotes total token-level similarity comparisons.
  • G Efficiency Measurements: Reducing visual tokens lowers both the theoretical MaxSim comparison count and measured retrieval latency.These measurements support the efficiency summary in the main paper.
  • H SaMer Algorithm: During training, box labels define an instance inconsistency penalty inside SaMer’s soft assignment, while inference discards the object prior.This separates object-aware supervision from deployment-time compression.
  • H SaMer Algorithm: SaMer adapts only the shared projection layer while keeping the vision encoder and language backbone frozen.The algorithm specifies projection-only adaptation using frozen fv and ft.
  • H SaMer Algorithm: Training initializes K representatives with feature and spatial centroids, assigns box labels by token coordinates, and applies an instance penalty before soft merging.The procedure computes feature-spatial distances, hard assignments, label distributions, and object-aware soft assignments.
  • H SaMer Algorithm: At inference, SaMer constructs image-side representations with bbox-free feature-spatial soft assignment and caches K merged tokens in the retrieval index.The inference procedure extracts and projects visual tokens before compression and compressed late-interaction scoring.
  • H SaMer Algorithm: Qualitative grounding examples show SaMer concentrating phrase relevance inside annotated regions, whereas compression baselines often spread relevance to nearby objects or background.The comparison includes single-vector baselines, full multi-vector retrievers, and compression methods applied to multi-vector retrievers.
Loading 2607.04605v2…