Source-linked AI summary
MARS: What Retrieval Signals Are Hidden in Multimodal Large Language Models for Text-Video Retrieval?
Uicheol Jung, Juyoung Hong, Geuntaek Lim, Yukyung Choi
TL;DR
Text-video retrieval needs representations that preserve fine-grained cues, but MLLM embedders often compress inputs into a single final-layer token. MARS combines hidden states across decoder layers into multiple adaptive slots, compares corresponding slots, and uses hard-negative-aware specialization; across four benchmarks, it achieves state-of-the-art results in direct retrieval and reranking.
Problem
Single-token final-layer MLLM embeddings can compress diverse video-text cues and limit fine-grained retrieval of videos with similar scenes, actions, and temporal patterns.
Method
MARS fuses hidden states from multiple decoder layers into adaptive representation slots, compares corresponding text-video slots, and applies hard-negative-aware slot specialization.
Results
MARS achieves state-of-the-art results across four text-video retrieval benchmarks in both direct similarity-based retrieval and reranking settings.
Takeaways & Limitations
Multi-layer fusion, adaptive slots, and hard-negative-aware specialization provide complementary gains for fine-grained retrieval.
Takeaways & Limitations
The evaluated benchmarks may not fully reflect open-domain scenarios with substantially longer videos, noisy descriptions, or diverse user queries.
Abstract
from arXiv · showhide
Text-video retrieval requires representations that can distinguish videos with similar scenes, actions, and temporal patterns. Recent multimodal large language models have been adapted as embedding models, but they often represent each input using a single token from the final layer. This can compress diverse video-text cues into a single vector and limit fine-grained retrieval. To address this limitation, we propose MARS, a multi-layer and multi-slot embedding framework for text-video retrieval. MARS constructs multiple adaptive representation slots by combining hidden states from different decoder layers, compares corresponding text and video slots, and aggregates their similarities for retrieval. To better handle confusing candidates, we further introduce a hard-negative-aware slot specialization objective that encourages the slots to capture discriminative matching cues. Experiments on four text-video retrieval benchmarks show that MARS achieves state-of-the-art results in both direct similarity-based retrieval and reranking settings. Ablation studies and analyses demonstrate that multi-layer fusion, multiple slots, and hard-negative-aware slot specialization provide complementary gains. Code is available at https://github.com/sejong-rcv/MARS.
1 Introduction
MARS addresses the information bottleneck of single-token, final-layer MLLM embeddings by combining multi-layer evidence with multiple adaptive slots for fine-grained text-video retrieval. It achieves state-of-the-art results across four benchmarks in both direct retrieval and reranking settings.
- Motivation: Single-token extraction from the final layer creates an information bottleneck for videos containing objects, actions, and temporal dynamics.These multi-granular cues must be preserved to distinguish visually similar scenes.
- Motivation: Useful retrieval signals can be distributed across decoder layers rather than concentrated in the final layer.Aggregating multi-layer evidence therefore provides an alternative to relying solely on the final output.
- MARS: MARS constructs adaptive representation slots by fusing hidden states from multiple decoder layers with slot-specific weights.Different slots can emphasize different layer-wise evidence while preserving complementary cues for matching.
- MARS: MARS compares corresponding slots separately before aggregating their similarities, rather than collapsing all cues into one global vector.This design targets fine-grained discriminability for candidates with similar scenes or semantics.
- Evaluation: MARS is validated on four benchmarks and analyzed through multi-layer evidence aggregation, complementary slot-wise matching, and slot specialization.The analyses examine how these design choices contribute to performance gains.
2 Related Work
Prior video retrieval methods extend CLIP or video foundation models through temporal and fine-grained modeling, often using heavy cross-modal scoring. MARS instead uses an efficient dual-encoder structure while addressing the limited use of MLLM hidden representations.
- Video Retrieval: CLIP-based video representations commonly use frame aggregation, temporal modeling, and fine-grained alignment.Video foundation models also improve retrieval through large-scale pre-training.
- Video Retrieval: Many recent video retrieval gains rely on heavy cross-modal decoders or reranking.These methods differ from direct similarity-based dual-encoder retrieval.
- MARS: MARS follows an efficient dual-encoder structure in which videos and texts are encoded independently and ranked by representation similarity.It avoids additional pairwise scoring during direct retrieval.
- MLLM Embedders: MLLM-based multimodal embedders predominantly derive embeddings from a single token such as the final or EOS token.MARS revisits this extraction strategy for text-video retrieval.
3 Method
MARS formulates text-video retrieval around multiple adaptive slots rather than a single embedding, fusing layer-wise hidden states and matching corresponding text-video slots. Its training combines contrastive alignment, slot diversity regularization, and hard-negative-aware specialization.
- MARS framework: MARS represents each text and video with multiple adaptive slots constructed from hidden states across decoder layers.Each slot uses learnable slot-specific weights to fuse layer-wise hidden states, preserving complementary fine-grained cues.
- Representation token prompting: Adaptive representation tokens are appended to prompted text and video inputs, with preceding hidden states used as slot representations.The tokens are shared across modalities, and each token corresponds to one representation slot.
- Slot-wise matching: Corresponding normalized text and video slots are compared with cosine similarity, then their scores are uniformly aggregated into the final retrieval score.Uniform slot weights β_m = 1/M provide a simple aggregation that performs consistently across benchmarks.
- Hard-negative-aware slot specialization: Hard-negative-aware specialization selects the most discriminative slot to separate each positive pair from a confusing negative.The hinge loss increases separation when the positive-negative gap falls below the margin δ, while varying selected slots encourage specialization.
- Training objective: MARS trains with symmetric contrastive alignment, slot diversity regularization, and hard-negative-aware slot specialization.The diversity term penalizes high similarity between slots from the same input, encouraging distinct retrieval-relevant information.
4 Experiments
MARS improves text-video retrieval through multi-layer evidence aggregation, multiple complementary slots, and hard-negative-aware specialization, while retaining efficient dual-encoder retrieval. Across benchmarks, it delivers strong retrieval performance, generalizes across backbones and modalities, and benefits from moderate slot counts.
- Main Results: MARS achieves the strongest direct similarity-based retrieval performance across four benchmarks, with 67.0 T2V mR@1 and 64.5 V2T mR@1.With DSL score calibration, MARS* reaches 71.6 T2V mR@1 and 72.0 V2T mR@1.
- Main Results: MARS-R achieves the best T2V mR@1 of 73.2 when candidate-level reranking is applied and improves over BLiM in average T2V and V2T performance.
- Ablation Studies: Multi-layer fusion raises average R@1 from 61.2 to 63.3, multiple slots to 65.1, and hard-negative-aware specialization to the full-model best of 65.7.The ablation evaluates eight T2V/V2T settings; diversity regularization adds a modest further gain before specialization.
- Ablation Studies: Average R@1 peaks at 77.7 with M = 4, up from 75.6 at M = 2, but drops when more slots are used.The results indicate that a moderate number of slots is preferable to excessive slots.
- Slot Analysis: Learned slots remain distinct: mean off-diagonal top-1 agreement is 0.537 on DiDeMo, 0.429 on ActivityNet, 0.206 on LSMDC, and 0.419 on MSR-VTT.Off-diagonal slot cosine similarities also remain generally small, indicating separation in embedding space.
- Computational Cost Analysis: MARS has the lowest online cost across all four benchmarks while obtaining the highest average T2V R@1 of 67.0.After offline video encoding, it encodes each text query once and compares against stored video representations.
- Generalization: Across three MLLM backbones, MARS consistently improves both T2V and V2T R@1 across all four benchmarks.The evaluated backbones are VideoLLaMA3-7B and Qwen2-VL at 7B and 2B scales.
5 Conclusion
MARS is a multi-layer and multi-slot embedding framework for text-video retrieval with MLLMs. Across four benchmarks, it achieves strong retrieval performance, while ablations show complementary gains from its main design components.
- MARS combines multi-layer fusion, adaptive representation slots, and hard-negative-aware specialization for text-video retrieval with MLLMs.
- Experiments on four benchmarks show strong retrieval performance while retaining a direct similarity-based retrieval structure.
- Ablations and analyses confirm that multi-layer fusion, adaptive representation slots, and hard-negative-aware specialization provide complementary gains.
- The findings suggest that richer representation extraction from MLLMs is promising for fine-grained retrieval.
Limitations
The evaluation does not fully cover open-domain and broader multimodal retrieval settings. MARS also focuses on video-level retrieval rather than temporal grounding or moment-level retrieval.
- The benchmarks may not fully reflect open-domain scenarios with substantially longer videos, noisy descriptions, or diverse user queries.
- MARS’s applicability to audio-language and 3D-language retrieval remains unexplored, despite demonstrated generalization to text-image retrieval.
- The study focuses on video-level retrieval and does not explicitly address temporal grounding or moment-level retrieval.
A Dataset Details
The experiments use four text-video retrieval datasets with different text construction and evaluation protocols. DiDeMo and ActivityNet use concatenated paragraph-level captions, while LSMDC and MSR-VTT use caption-level training samples.
- DiDeMo: DiDeMo uses concatenated captions for paragraph-video retrieval, with 8,381 training pairs and 1,003 test pairs.Each training video has 3.9 captions on average.
- ActivityNet Captions: ActivityNet Captions constructs paragraph-level queries by concatenating captions, with 10,009 training pairs and 4,917 test pairs.Its videos are longer and contain multiple event-level captions per video compared with DiDeMo.
- LSMDC: LSMDC follows the standard setup and uses a 1,000-sample test split, with 101,055 processed training samples.
- MSR-VTT: MSR-VTT uses the standard 1K-A protocol, with 9,000 training videos, 20 captions per video, and 180,000 training text-video pairs.The test split contains 1,000 videos, and each caption is treated as an individual text input during training.
B Implementation Details
MARS is implemented on VideoChat-Flash-Qwen2-7B with adaptive representation tokens and layer fusion across decoder hidden states. Training uses parameter-efficient adaptation, dataset-specific slot counts, and shared video-processing settings.
- Model architecture: MARS uses VideoChat-Flash-Qwen2-7B, combining a UMT-L vision encoder, multimodal projection layer, and Qwen2 language model with 28 decoder layers.
- Model architecture: Adaptive representation tokens are inserted for video and text inputs, and their hidden states from all decoder layers are used for layer fusion.
- Model configuration: The model uses four slots for DiDeMo, ActivityNet, and LSMDC, and three slots for MSR-VTT.
- Parameter-efficient training: LoRA adapts language-model attention and MLP projections plus the multimodal projection layer, while pretrained parameters remain frozen.Trainable parameters also include adaptive representation tokens and slot-wise layer-fusion weights.
- Training configuration: The training-settings table reports shared settings alongside dataset-specific slots, epochs, learning rates, and effective batch sizes.
- Training configuration: All experiments sample 16 frames per video and use gradient checkpointing with training on two NVIDIA A100 40GB GPUs.
C Slot Aggregation Analysis
MARS compares uniform, learnable, and MaxSim slot aggregation strategies, with uniform aggregation yielding the strongest average retrieval performance.
- C Slot Aggregation Analysis: Uniform aggregation preserves contributions from all slots, while learnable aggregation weights slots and MaxSim selects the strongest slot-level similarity.The ablation compares these three strategies as alternatives for combining corresponding slot similarities.
- C Slot Aggregation Analysis: 65.7 average R@1 across eight T2V/V2T settings makes uniform aggregation stronger than learnable aggregation at 65.4 and MaxSim at 65.1.Uniform aggregation assigns equal contribution to every slot and avoids additional slot-weighting parameters.
D Additional Analysis of Adaptive Representation Slots
Additional analyses examine adaptive slot geometry using representation slots and uniformly aggregated corresponding slot-wise similarities.
- D Additional Analysis of Adaptive Representation Slots: The additional analyses extend the main slot analysis by examining geometry from complementary perspectives.They use the same multi-slot representation and uniform aggregation setup as the main method.
- D Additional Analysis of Adaptive Representation Slots: Figure A1 reports linear CKA and effective rank to characterize relationships among adaptive slot embedding matrices.These diagnostics provide structural and dimensional information about slot representations.
D.1 Slot Geometry Diagnostics
Slot geometry diagnostics use linear CKA and effective rank to examine structural similarity and embedding-space usage across adaptive representation slots.
- D.1 Slot Geometry Diagnostics: Linear CKA measures sample-level structural similarity between slot embedding matrices, complementing pairwise cosine similarity between individual vector directions.Across benchmarks, CKA-based structural similarity varies, indicating that slot distinctness can depend on dataset characteristics.
- D.1 Slot Geometry Diagnostics: Effective rank reflects how broadly each slot uses available embedding dimensions, computed from the entropy of normalized squared singular values.Higher effective rank indicates use of a broader set of embedding directions.
- D.1 Slot Geometry Diagnostics: Different slots can have different effective ranks, indicating that they do not necessarily use the embedding space in the same way.Together with retrieval-side analyses, these diagnostics suggest distinct representation patterns among slots.