PaperlayerView exact revision on arXiv

Source-linked AI summary

VidVec: Unlocking Video MLLM Embeddings for Video-Text Retrieval

Issar Tzachor, Dvir Samuel, Rami Ben-Ari

arXiv:2602.08099v1cs.CVcs.AI

TL;DR

MLLM embedders have lagged behind Video Foundation Models on video retrieval, motivating a focused study of MLLM representations for this task. VidVec combines intermediate-layer embeddings with calibrated head scoring and text-only caption-to-summary alignment, achieving state-of-the-art retrieval without visual fine-tuning.

  • Problem

    MLLM embedding methods have shown promise for vision tasks, but video retrieval remains behind dedicated Video Foundation Models and lacks strong video-focused results.

  • Method

    VidVec analyzes MLLM layers, uses intermediate embeddings and a calibrated likelihood-scoring head, then applies text-only caption-to-summary alignment.

  • Results

    VidVec’s complete approach achieves state-of-the-art results across multiple video retrieval benchmarks without training the retrieval system beyond text.

  • Takeaways & Limitations

    The results indicate that large multimodal models can support training-free and data-efficient adaptation to embedding-based video retrieval.

  • Takeaways & Limitations

    The optimization relies solely on text, treating dense video captions as textual proxies for the underlying videos.

Abstract

from arXiv · show

Recent studies have adapted generative Multimodal Large Language Models (MLLMs) into embedding extractors for vision tasks, typically through fine-tuning to produce universal representations. However, their performance on video remains inferior to Video Foundation Models (VFMs). In this paper, we focus on leveraging MLLMs for video-text embedding and retrieval. We first conduct a systematic layer-wise analysis, showing that intermediate (pre-trained) MLLM layers already encode substantial task-relevant information. Leveraging this insight, we demonstrate that combining intermediate-layer embeddings with a calibrated MLLM head yields strong zero-shot retrieval performance without any training. Building on these findings, we introduce a lightweight text-based alignment strategy which maps dense video captions to short summaries and enables task-related video-text embedding learning without visual supervision. Remarkably, without any fine-tuning beyond text, our method outperforms current methods, often by a substantial margin, achieving state-of-the-art results across common video retrieval benchmarks.

1. Introduction

Video retrieval remains a difficult and underdeveloped setting for MLLM embedders, which trail dedicated Video Foundation Models. VidVec addresses this gap by combining intermediate-layer representations, calibrated head scoring, and text-only alignment to achieve strong retrieval without visual fine-tuning.

  • Motivation: Video retrieval remains behind dedicated Video Foundation Models, while existing video-focused MLLM methods underperform on standard benchmarks.The paper identifies video as a primary challenge because of richer content and temporal dynamics.
  • VidVec approach: Intermediate MLLM layers can provide markedly more effective video–text retrieval representations than the final layer.VidVec systematically assesses hidden representations and uses appropriate intermediate layers for zero-shot retrieval.
  • VidVec approach: A calibrated MLLM head enables zero-shot pairwise scoring, turning off-the-shelf Video MLLMs into competitive video–text retrievers.The head is used as a calibrated likelihood scorer for reranking candidates.
  • VidVec approach: Text-only in-context optimization maps dense video captions to short summaries, enabling task-related embedding learning without visual supervision.The strategy uses text-to-text supervision rather than video–text contrastive training.
  • Results: Without fine-tuning beyond text, VidVec achieves state-of-the-art performance on common video retrieval benchmarks.The paper reports this outcome across multiple benchmarks.

2. Related Work

Prior work largely relies on dual encoders, video-specialized models, or multimodal post-training for retrieval. VidVec instead studies MLLM readout layers and text-only summarization as efficient routes to video–text alignment.

  • Vision–language embeddings for retrieval: Dual-encoder vision–language models align image and text embeddings contrastively but are less natural for interleaved multimodal and instruction-conditioned retrieval.These methods remain highly effective for image–text retrieval.
  • Video–text retrieval and video representation learning: Video retrieval methods commonly use video-specialized encoders, temporal modeling, and large-scale video–text supervision.VidVec contrasts this paradigm with MLLM-based embedding extraction.
  • MLLMs as embedders: MLLM embedding methods use hidden-state extraction, prompting, contrastive training, or multimodal post-training to produce retrieval representations.Examples include E5-V, LamRA, VLM2Vec, and VLM2Vec-V2.
  • VidVec’s position: VidVec differs by making video–text retrieval central, analyzing readout layers, and learning alignment from text-only dense-caption–to–summary mappings.The approach avoids visual supervision and video–text contrastive training.

3. Method

VidVec formulates video–text retrieval with MLLM embeddings, then combines cosine-similarity retrieval, calibrated-head reranking, and text-only caption-to-summary alignment. The method uses intermediate hidden states for zero-shot representations and Dual-Softmax training for optimized embeddings without visual supervision.

  • Problem Formulation: VidVec extracts d-dimensional query and candidate embeddings from an MLLM and ranks candidates by cosine similarity.A second stage reranks the top-K candidates using the model’s relevance score.
  • Embedding Extraction: The MLLM architecture converts video into visual tokens through a vision encoder and projector before joint processing with text tokens.The embedding is taken from the hidden state immediately preceding a dedicated <emb> token under Explicit One-word Limitation prompting.
  • MLLM Head Reranking: The calibrated MLLM head reranks top-K candidates by the likelihood of affirmative Yes/yes tokens in a binary relevance prompt.This requires K independent forward passes and orders candidates by decreasing Srank.
  • In-Context Optimization: Task-oriented text mappings materially affect performance, so VidVec fine-tunes lightweight LoRA layers to map dense video descriptions to short summaries.The mapping uses video descriptions while avoiding access to visual data; summaries serve as compact textual anchors for video encoding and retrieval.
  • Training Objective: Dual-Softmax Loss symmetrically combines text-to-video and video-to-text match distributions to emphasize pairs with high mutual confidence.VidVec performs this optimization solely in text space, treating dense captions as textual proxies for videos.
  • Overall Pipeline: The method combines intermediate-layer <emb-1> extraction for VidVec-ZS, calibrated-head reranking, and text-to-text token optimization for VidVec-O.Standard retrieval encodes query and gallery items and ranks them by embedding similarity.

4. Zero-shot Layer-wise Analysis

The layer-wise analysis finds that retrieval-relevant information emerges in intermediate and late intermediate MLLM layers, while early layers contribute little. Performance generally improves with depth but peaks before the final layer, with task-aware prompting further improving zero-shot retrieval.

  • Motivation: 14.3% MSR-VTT R@1 is obtained from the off-the-shelf final <emb-1> representation before layer-wise analysis.This establishes a comparatively low baseline for direct hidden-state extraction.
  • Layer-wise Results: Several mid- to late-stage intermediate layers achieve markedly stronger zero-shot MSR-VTT Recall@1 than early layers across evaluated MLLM backbones.The analysis includes Qwen-VL generations, VideoLLaMA3-7B, and Keye-VL-8B.
  • Prompting: A task-aware prefix requesting the main subjects, appearance, setting, and activity produces a noticeable improvement in off-the-shelf MLLM video retrieval.The prefix is adopted for subsequent zero-shot experiments.

5. Evaluation

VidVec is evaluated against MLLM embedders and Video Foundation Models on standard video–text retrieval benchmarks. Its zero-shot, text-only optimized, and reranked variants consistently outperform prior MLLM embedders and achieve state-of-the-art performance on most evaluated benchmarks.

  • Evaluation setup: Evaluation covers four standard video–text retrieval benchmarks, using established splits without training on benchmark training sets.
  • Zero-shot performance: VidVec-ZS uses intermediate-layer embeddings and reranking without further training, outperforming prior MLLM embedders on MSR-VTT, VATEX, and DiDeMo.Recall@1 gains are +3.1%, +7.7%, and +9.4%, respectively.
  • Zero-shot performance: Off-the-shelf VideoLLaMA3 already contains well-aligned video–text embeddings internally, according to the zero-shot results.
  • Optimized embeddings: VidVec-O consistently surpasses existing MLLM embedders in both retrieval directions, including V2T Recall@1 gains of +6.5% on VATEX and +9.4% on DiDeMo.The optimized model uses text-based in-context optimization to elicit aligned video–text embeddings.
  • Reranking: The calibrated reranker provides larger improvements than text-based optimization, and combining it with VidVec-O yields state-of-the-art Text-to-Video performance.
  • Comparison with VFMs: Against Video Foundation Models, VidVec achieves state-of-the-art performance on most benchmarks in both T2V and V2T retrieval, despite far less training data.It improves by +1.2% on MSRVTT V2T and +1.2% on MSVD T2V, but underperforms on selected VATEX and DiDeMo directions.

6. Ablation Study

The ablation study compares textual data choices for optimization on MSRVTT. Short video-caption pairs improve over prior NLI-based text, while in-context optimization performs best using video-related text alone.

  • Ablation design: The ablation evaluates alternative text-to-text mapping strategies and a Qwen-2-VL backbone on MSRVTT Recall@1.
  • Textual data choice: Short video-caption pairs improve performance over prior NLI-based text, while in-context optimization achieves the best performance.The optimization uses detailed video captions and aligned short summaries.
  • Experimental condition: The ablation experiments omit dual-softmax to isolate the effect of the examined component.

7. Summary

VidVec shows that off-the-shelf video MLLMs can support effective video–text retrieval through intermediate representations, calibrated scoring, and text-only alignment. The approach achieves strong and state-of-the-art results while highlighting the potential for training-free and data-efficient adaptation.

  • VidVec uses intermediate representations and the model head for calibrated scoring to obtain strong retrieval performance without training.
  • Its text-to-text in-context optimization improves multimodal alignment without visual supervision.
  • The complete approach achieves state-of-the-art results across multiple video retrieval benchmarks and significantly outperforms recent trained MLLM embedders.
  • The results highlight large multimodal models’ potential for training-free and data-efficient adaptation to embedding-based tasks.

Societal Impact

The paper states that it advances machine learning and reports no societal consequences requiring specific emphasis. The appendix lists benchmark details, additional dataset evaluations, and implementation details.

  • The work’s stated goal is to advance the field of Machine Learning.
  • The authors identify no potential societal consequences that require specific highlighting.
  • Appendix contents: The appendix includes benchmark details, evaluations on more datasets, and additional implementation details.

A. Benchmark Datasets

The evaluation covers five video–text retrieval benchmarks with varied dataset sizes, caption structures, and evaluation protocols. Additional results compare VidVec with existing MLLM embedders on MSR-VTT and DiDeMo.

  • Benchmark coverage: The benchmarks are MSR-VTT, MSVD, VATEX, DiDeMo, and ActivityNet.Dataset statistics are summarized in Table 6.
  • Dataset protocols: MSR-VTT contains 10,000 videos and 200,000 captions, evaluated on the standard 1k-A test split.
  • Dataset protocols: MSVD contains 1,970 videos split into 1,200/100/670 train, validation, and test videos, with any video-associated caption treated as positive for V2T.
  • Dataset protocols: VATEX provides 25,991 training, 3,000 validation, and 6,000 test videos, while evaluation uses 1,500 validation videos with English captions.
  • Dataset protocols: ActivityNet contains 20,000 videos and uses concatenated descriptions for video–paragraph retrieval on the ‘val1’ split.

B.2. ActivityNet Evaluation

Additional evaluations examine ActivityNet, optimization strategies, backbone generalization, and calibrated retrieval procedures. VidVec also shows a 5.1% relative gain over fine-tuned InternVideo2-6B on ActivityNet, while in-context optimization improves Qwen2-VL substantially.

  • ActivityNet evaluation: 5.1% relative gain over fine-tuned InternVideo2-6B is reported for VidVec on ActivityNet retrieval.The ActivityNet results are reported separately because VideoLLaMA3 included ActivityNet data during generative training.
  • Optimization ablation: In-context optimization achieves the best MSR-VTT Recall@1 performance among the compared textual optimization approaches.Brief video-description pairs improve over prior NLI-based textual data.
  • Backbone generalization: In-context optimization improves Qwen2-VL from 44.7 to 47.2 on MSR-VTT-T2V-R@1, only 0.2 points below VideoLLaMA3.
  • Implementation and evaluation: The method uses LoRA token optimization with a batch of 288 pairs and applies dual-softmax calibration using fixed temperatures for both retrieval directions.Dual-softmax takes softmaxes over both columns and rows of the similarity matrix.

E. Limitations

VidVec’s text-based optimization is constrained by the quality and coverage of video captions, while reranking adds inference cost and remains limited to a simple pairwise scoring strategy.

  • Textual supervision: Caption quality and coverage bound VidVec’s effectiveness, especially for fine-grained visual details and long-range temporal dependencies absent from text.
  • Inference cost: Reranking requires additional forward passes over top-K candidates, which may limit applicability for large K values.
  • Reranking design: The reranking stage uses simple pairwise MLLM-head scoring, leaving more advanced reranking strategies for future work.
Loading 2602.08099v1…