PaperlayerView exact revision on arXiv

Source-linked AI summary

Your Embedding Model is SMARTer Than You Think

Jianrui Zhang, Hyun Jung Lee, Sukanta Ganguly, Tae-Eui Kam, Donghyun Kim, Yong Jae Lee

arXiv:2605.24938v1cs.IRcs.AIcs.CV

TL;DR

Single-vector multimodal retrievers are efficient but compress away localized evidence, while multi-vector alternatives often require costly training and may neglect global summaries. SMART unlocks latent token-level retrieval from standard single-vector models by combining late interaction over hidden states with pooled scoring, either inference-only or through lightweight post-training. It consistently improves retrieval across modalities and can convert a SoTA single-vector model into a multi-vector variant better than SoTA pretrained counterparts.

  • Problem

    Single-vector retrievers compress multimodal sequences into one global representation, limiting access to localized evidence, while multi-vector methods require training and may omit global pooled signals.

  • Method

    SMART applies late-interaction MAXSIM to non-pooling hidden states while retaining the pooled score, with optional lightweight adapter or hybrid-objective finetuning.

  • Results

    SMART consistently improves retrieval across diverse modalities and can convert a SoTA single-vector embedding model into a multi-vector variant better than SoTA pretrained counterparts.

  • Takeaways & Limitations

    SMART provides a zero-training-cost inference enhancement and an efficient training pathway for recovering fine-grained localized evidence in multimodal dense retrieval.

  • Takeaways & Limitations

    SMART is studied for dense retrieval, is not beneficial as an inference-only tool for more global classification tasks, and limited compute restricted LamRA-Ret training to the visdoc subset.

Abstract

from arXiv · show

Multimodal retrieval relies heavily on single-vector retrievers, which compress rich, sequential token sequences into one single global representation. While efficient, they discard fine-grained, local evidence critical for dense retrieval tasks. Multi-vector approaches were introduced as a solution, but they strictly require training and many ignore the necessity of a globally summarizing representation. To address this, we introduce SMART, a framework that unlocks the latent multi-vector capabilities of standard single-vector models. We first demonstrate that standard contrastive training on the pooled embedding implicitly shapes the retrieval geometry of preceding hidden states via gradient flow. By applying direct late-interaction over these frozen hidden states during inference, SMART acts as a plug-and-play upgrade that consistently improves performance across diverse modalities, improving even the state-of-the-art models further on MMEB-V2. We also reveal SMART's superior performance, as simple lightweight post-training not only saves time and compute, but also brings forth further improvement on Visual Document retrieval, allowing a single-vector model to outperform SoTA multi-vector counterparts. Ultimately, SMART offers both a highly efficient inference enhancement and a powerful finetuning technique for multimodal retrieval. We open source our code and weights at https://github.com/HanSolo9682/SMART.

1 Introduction

Single-vector multimodal retrievers offer efficient global matching but compress away localized evidence, while existing multi-vector alternatives require costly training and may omit global pooled signals. SMART reuses hidden states from standard single-vector models for late interaction, with inference-only and lightweight post-training variants.

  • Motivation: Single-vector architectures efficiently support global similarity matching by compressing multimodal hidden-state sequences into one pooled embedding.This compression creates a capacity bottleneck for fine-grained retrieval.
  • Motivation: Multi-vector methods address the bottleneck but typically require task-specific finetuning or learnable tokens, increasing training and memory costs.Some also emphasize local matching without preserving the global pooled readout.
  • SMART: SMART exploits localized semantic evidence retained in non-pooling hidden states, enabling an existing single-vector model to support token-level matching.The framework preserves the model’s global compatibility signal while recovering local information.
  • SMART: SMART provides a training-free plug-and-play upgrade that combines late interaction with pooled scoring and improves retrieval across complex tasks and backbones.The method is reported to improve even SoTA Qwen3-VL-Embedding models.
  • SMART: Lightweight adapters or hybrid-objective finetuning further improve multi-vector retrieval while avoiding training a dedicated multi-vector model from scratch.These variants reuse the single-vector backbone and its hidden states.

2 Related Work

Related work establishes single-vector models as efficient unified retrievers and multi-vector models as a response to their capacity limits. SMART differs by applying late interaction directly to existing hidden states without requiring full multi-vector training.

  • Single-Vector Embedding Models: Modern multimodal retrievers build unified representation spaces for diverse modalities, but single-vector models leave fine-grained non-pooled hidden-state information unused.SMART applies to off-the-shelf single-vector models to reuse that information.
  • Multi-Vector Embedding Models: Multi-vector architectures use late interaction for localized matching but commonly require full-scale task-specific training, adapters, or learnable tokens.SMART can instead operate entirely at inference time.
  • Multimodal Retrieval Benchmarks: MMEB-V2 provides a broad evaluation setting spanning diverse multimodal dense-retrieval tasks, alongside domain-specific visual-document and image-retrieval benchmarks.The paper evaluates SMART on MMEB-V2 because of its broad task coverage.

3 SMART

SMART reuses non-pooling hidden states from single-vector retrievers for token-level late interaction while retaining the original pooled similarity signal. Its design follows from gradient flow through the pooled representation and combines global compatibility with local evidence.

  • Pooled supervision reaches non-pooling hidden states: Contrastive training supervises the pooled embedding, but attention and residual pathways place preceding non-pooling hidden states on the loss’s gradient path.This indirect supervision encourages those states to organize in a geometry compatible with cosine-based token-level retrieval.
  • Single-to-Multi Adaptation for Retrieval Transformers: SMART adds a token-level late-interaction readout over existing hidden states without training a new multi-vector retriever.The original backbone and pooled readout remain unchanged.
  • Single-to-Multi Adaptation for Retrieval Transformers: SMART combines global pooled compatibility and local token-level evidence by simple addition in the shared final-layer cosine geometry.Unit weighting keeps the method hyperparameter-free, and the hybrid score favors candidates supported by both signals.
  • Single-to-Multi Adaptation for Retrieval Transformers: The late-interaction score matches each query token to its most similar candidate token, measuring local query coverage in candidate hidden states.SMART uses valid non-pooling tokens and normalized final-layer hidden states for this readout.
  • Direct Late Interaction over Hidden States: The controlled benchmark pairs visually identical reports whose local code–marker assignments differ, isolating whether retrieval detects the queried binding.Each query specifies one local binding, while the hard negative preserves layout, codes, colors, and shapes but reassigns the codes.

4 Experiments

Experiments show that SMART improves multimodal retrieval without training and can be further strengthened through lightweight post-training. Across backbones and domains, it recovers local evidence while retaining global compatibility, including gains over native multi-vector systems.

  • 4 Experiments: In controlled local-binding retrieval, late interaction reaches 56.8%, outperforming jina-embeddings-v4 at 50.9% and Colpali at 48.7%.The hybrid score can weaken in this adversarial setting because aggregate similarity conflicts with the local-binding decision.
  • 4 Experiments: Training-free SMART consistently improves retrieval across diverse MMEB-V2 domains and backbones without parameter updates or finetuning.The reported gains include VLM2Vec-V2.0, Qwen3-VL-Embedding-2B, and Qwen3-VL-Embedding-8B.
  • 4 Experiments: +0.51 points: SMART raises Qwen3-VL-Embedding-8B’s MMEB-V2 average from 78.83% to 79.34%.
  • 4 Experiments: SMART improves Visual Document Retrieval across all four tested backbones and substantially boosts Video Retrieval for three reported models.Video gains are +1.37% for VLM2Vec, +2.01% for Qwen3-VL-Embedding-2B, and +1.42% for Qwen3-VL-Embedding-8B.
  • 4 Experiments: A lightweight readout-only adapter improves over inference-only SMART by about 1 point on both Qwen3-VL-Embedding-2B and 8B.The adapter freezes the backbone and trains only a linear token-level readout while preserving the pooled score.
  • 4 Experiments: The SMART adapter lets Qwen3-VL-Embedding-2B outperform jina-embeddings-v4 by 0.34 points after 1 hour and 50 minutes of training.
  • 4 Experiments: Converting a trained single-vector model with SMART improves its average from 72.60 to 74.18 using inference-only scoring, a roughly 1.6-point gain.
  • 4 Experiments: Qualitative analyses show SMART correcting globally plausible retrieval errors by matching fine-grained image or document details through token-level evidence.

5 Conclusion

SMART recovers localized evidence from multimodal single-vector models through late interaction while retaining global compatibility. It supports efficient inference-time enhancement and lightweight training, improving retrieval performance across diverse settings.

  • SMART unlocks localized representations by applying late-interaction MAXSIM to hidden states shaped by pooled contrastive training.The framework also uses a hybrid scoring objective that combines localized representations with the global summary signal.
  • Inference-only SMART consistently improves accuracy across diverse modalities and scales to state-of-the-art Qwen3-VL-Embedding architectures.
  • Lightweight auxiliary training further improves performance by jointly optimizing non-pooling hidden states for late interaction and the global summary token.This converts a state-of-the-art single-vector embedding model into a multi-vector variant better than state-of-the-art pretrained counterparts.
  • SMART provides a computationally efficient pathway for recovering fine-grained localized evidence from existing multimodal architectures.

Limitations

SMART is evaluated for dense retrieval rather than broader global tasks. The authors report no inference-only benefit for classification and defer some training experiments because of limited compute.

  • SMART is not beneficial as an inference-only tool for global tasks such as classification.
  • Limited compute restricted training of the LamRA-Ret models to the visual-document subset.The authors leave broader exploration of these aspects to future work.

A Toy Dataset

The toy benchmark isolates local binding retrieval by pairing documents with identical global content but permuted code–marker assignments. Its results show that late interaction recovers local evidence better than pooled scoring in this adversarial setting.

  • The benchmark pairs dense visual documents with identical layouts, codes, and marker descriptors but permuted local code–marker bindings.This construction removes easy global cues and makes the positive document identifiable through local binding evidence.
  • 40 document pairs with 25 bindings each produce 1000 local-binding queries, evaluated by ranking each positive against its paired hard negative.
  • 31.9% pairwise accuracy from the original single-vector score rises to 56.8% with late interaction alone.The result indicates that non-pooling hidden states retain local binding evidence.
  • 42.6% pairwise accuracy from the hybrid score exceeds pooled scoring but remains below late interaction alone because the pooled signal is intentionally misleading.

B Applicable Task Categorization

SMART targets dense corpus-level retrieval tasks requiring fine-grained alignment, while its use is constrained for classification, VQA, visual grounding, video moments, and some composed-image settings.

  • Applicable dense retrieval tasks: SMART is designed for image, visual-document, and video retrieval where semantic complexity requires fine-grained query–target alignment.
  • Excluded or constrained tasks: Classification and VQA provide low-entropy targets that are often compressible into a single vector, so SMART yields no benefit and token interaction can introduce noise.
  • Excluded or constrained tasks: Image visual grounding is omitted from open-corpus retrieval because valid distractor crops and spurious correlations can penalize semantically correct matches.
  • Excluded or constrained tasks: Video moment retrieval requires holistic spatiotemporal abstraction, whereas SMART’s token-level inference lacks explicit temporal reasoning for high-level actions.
  • Task-specific adjustment: For composed image retrieval, masking query vision tokens prevents misleading visual alignments and significantly improves retrieval accuracy.

C Visualization

SMART’s visualizations show that late interaction recovers localized evidence missed by pooled single-vector retrieval. Token-level similarity highlights semantically meaningful regions supporting these corrections.

  • Qualitative examples: SMART retrieves correct candidates when single-vector retrieval selects globally plausible images but misses localized visual details.The examples include broad similarities such as stone buildings, castles, towers, and monastery-like structures.
  • Qualitative examples: Late interaction complements the pooled score by matching query evidence against corresponding candidate-image regions.
  • Token-level visualization: For each selected query token, SMART identifies the top-5 candidate-image tokens using hidden-state cosine similarity.MAXSIM uses the highest-scoring token, while other high-similarity tokens provide nearby supporting evidence.

D Ablation for Hybrid Scoring

The ablation supports SMART’s hybrid scoring design, which combines trained late interaction with the pooled anchor. This combination achieves the strongest reported performance and approaches the cited state-of-the-art average.

  • Hybrid scoring: 79.10 overall performance: LamRA-Hybrid achieves the highest result by combining trained MAXSIM interaction with the pooled single-token anchor.
  • Hybrid scoring: 6.5-point average improvement: LamRA-Hybrid outperforms the original single-vector baseline.
  • Comparison with SoTA: 79.10 vs. 79.27 average: hybrid LamRA virtually matches Qwen3-VL-Embedding-2B’s cited SoTA performance.
Loading 2605.24938v1…