Source-linked AI summary
X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrieval
Satya Krishna Gorti, Noel Vouitsis, Junwei Ma, Keyvan Golestan, Maksims Volkovs, Animesh Garg, Guangwei Yu
TL;DR
Text-video retrieval is challenging because texts describe only parts of videos, while common text-agnostic pooling can include irrelevant visual information. X-Pool uses cross-modal attention to select text-relevant frames and achieves state-of-the-art results across three benchmarks, with up to 12% relative Recall@1 improvement.
Problem
Videos contain more information than texts typically describe, but existing methods often aggregate entire videos without directly reasoning about text-relevant sub-regions.
Method
X-Pool applies learned scaled dot product attention between a text and video frames, producing a text-conditioned aggregated video representation.
Results
X-Pool achieves state-of-the-art results across MSR-VTT, MSVD, and LSMDC, including a 12% relative improvement in Recall@1 over CLIP4Clip-seqTransf on LSMDC.
Takeaways & Limitations
Text-conditioned pooling highlights frames relevant to the query while suppressing distracting visual cues, and is more robust to diverse video content.
Takeaways & Limitations
Top-k text-conditioned pooling requires task- and instance-specific tuning of k, while selecting frames may require reasoning beyond cosine similarity.
Abstract
from arXiv · showhide
In text-video retrieval, the objective is to learn a cross-modal similarity function between a text and a video that ranks relevant text-video pairs higher than irrelevant pairs. However, videos inherently express a much wider gamut of information than texts. Instead, texts often capture sub-regions of entire videos and are most semantically similar to certain frames within videos. Therefore, for a given text, a retrieval model should focus on the text's most semantically similar video sub-regions to make a more relevant comparison. Yet, most existing works aggregate entire videos without directly considering text. Common text-agnostic aggregations schemes include mean-pooling or self-attention over the frames, but these are likely to encode misleading visual information not described in the given text. To address this, we propose a cross-modal attention model called X-Pool that reasons between a text and the frames of a video. Our core mechanism is a scaled dot product attention for a text to attend to its most semantically similar frames. We then generate an aggregated video representation conditioned on the text's attention weights over the frames. We evaluate our method on three benchmark datasets of MSR-VTT, MSVD and LSMDC, achieving new state-of-the-art results by up to 12% in relative improvement in Recall@1. Our findings thereby highlight the importance of joint text-video reasoning to extract important visual cues according to text. Full code and demo can be found at: https://layer6ai-labs.github.io/xpool/
1. Introduction
Text-video retrieval must compare a query with the video frames that best match its meaning, because videos contain more diverse content than individual texts. X-Pool addresses this mismatch through text-conditioned cross-modal attention over video frames.
- Text-video retrieval learns a similarity function between texts and videos, commonly by embedding both modalities into a joint latent space.
- Texts usually describe only subsets of a video's frames, and different captions can match different scenes within the same video.
- Text-agnostic mean-pooling or self-attention can encode visual information unrelated to the query and potentially reduce retrieval performance.
- X-Pool lets each text attend to its most semantically similar frames and produces a video representation conditioned on those frames.
- X-Pool achieves state-of-the-art results on MSR-VTT, MSVD, and LSMDC, while improving robustness to videos with diverse content.
2. Related Work
Related work typically combines pretrained language and visual experts or uses joint language-vision models for text-video retrieval. Existing aggregation methods, including CLIP4Clip variants, do not directly match text with its most relevant video sub-regions.
- Many text-video retrieval systems use pretrained language and video experts, consolidating their streams through late fusion.
- Some single-stream models jointly embed text-video pairs through BERT-like architectures for early cross-modal fusion.
- CLIP-based methods demonstrate that joint language-vision representations transfer effectively toward video retrieval.
- CLIP4Clip provides mean-pooling, self-attention, and multimodal-transformer aggregation, but none directly matches text to its most relevant video sub-regions.
3. Problem Statement
Text-video retrieval ranks videos for text queries and texts for video queries using a learned similarity function. Videos are represented as sequences of sampled frames, while texts are represented as tokenized word sequences.
- The model learns a scalar similarity function s(t, v) that assigns higher scores to relevant text-video pairs than irrelevant pairs.
- Text-to-video retrieval ranks a video index set for a given query text according to text-video similarity.
- Video-to-text retrieval is the complementary task of ranking texts for a given video.
- A video is represented as a temporal sequence of F sampled image frames with resolution H × W.
- A text is represented as a sequence of tokenized words.
4. Methodology
X-Pool builds text-conditioned video representations by letting a text attend to semantically relevant frames instead of aggregating every frame text-independently. It extends this insight from top-k pooling to learned cross-modal attention that weights frame values and supports retrieval similarity learning.
- Bootstrapping From Joint Text-Image Models: Joint text-image encoders provide a basis for embedding texts and video frames into a shared latent space for retrieval.CLIP supplies the text embedding directly, while frame embeddings are temporally aggregated into a video embedding.
- Gap: Text-Agnostic Pooling: Text-agnostic mean-pooling, self-attention, and LSTM aggregation can encode visual information unrelated to the input text, especially in content-diverse videos.Scene transitions, abrupt cuts, occlusions, and distractors make relevance vary across a video's frames.
- Our Model: X-Pool: The model computes cosine similarity between the conditioned video embedding and text embedding, then applies symmetric text-to-video and video-to-text cross-entropy losses over batch pairs.Matching text-video pairs are positives and other pairwise combinations within the batch are negatives.
- Key Insight: Text-Conditioned Pooling: Text-conditioned pooling directly reasons between a query text and video frames to extract relevant sub-regions while suppressing noisy or misleading cues.The resulting video embedding is denoted z_v|t and is compared with the text embedding using cosine similarity.
- Key Insight: Text-Conditioned Pooling: Top-k pooling selects the frames with the highest text-frame cosine similarity and ignores lower-similarity frames.This simple text-conditioned baseline already produces a significant improvement over text-agnostic temporal pooling.
- Our Model: X-Pool: X-Pool replaces fixed top-k selection with learned scaled dot product attention from a projected text query to projected frame keys.Attention weights aggregate separately projected frame values, followed by a fully connected layer and residual connection; the model learns how much information to extract for each text-video pair.
5. Experiments
Experiments evaluate X-Pool on three benchmark datasets using standard text-to-video retrieval metrics, compare it with prior methods, and test text-conditioned pooling through retrieval, robustness, and qualitative analyses. X-Pool outperforms text-agnostic pooling baselines and remains effective as video content diversity increases.
- Benchmark datasets and metrics: Experiments evaluate MSR-VTT, MSVD, and LSMDC using Recall@1, Recall@5, Recall@10, Median Rank, and Mean Rank.MSR-VTT uses 7k-Train and 9k-Train splits; MSVD and LSMDC results are evaluated on their test splits.
- Benchmark results: Across all datasets and metrics, X-Pool outperforms works using text-agnostic pooling, including methods based on multiple video modalities.The comparisons include MSR-VTT-9K, MSR-VTT-7K, MSVD, and LSMDC results.
- Benchmark results: 12% relative improvement in Recall@1 is achieved on LSMDC over CLIP4Clip-seqTransf, while improvements reach 5% on MSR-VTT and over 2% on MSVD.The LSMDC task is described as more challenging because movie-scene descriptions are more ambiguous.
- Robustness to content diversity: X-Pool remains robust as scene transitions increase, whereas mean-pooling significantly deteriorates under greater video content diversity.The robustness experiment uses t2v Median Rank on augmented MSR-VTT test videos containing added scene transitions.
- Top-k experiments: The optimal top-k value varies widely across text-video pairs, making the hyperparameter difficult to select generally.X-Pool is introduced to address this drawback while retaining the text-conditioned pooling motivation.
6. Conclusion
The work challenges text-agnostic video pooling and presents text-conditioned pooling through X-Pool, a cross-modal attention model for text-video retrieval.
- Text-agnostic video pooling has drawbacks for text-video retrieval, motivating an alternative framework for text-conditioned pooling.
- X-Pool models cross-modal attention between a text and video frames.
- X-Pool learns to attend to the frames most relevant to a given text.
- The text-conditioned attention makes the model substantially more robust to video content diversity.
A. Video-to-Text Retrieval Results
The appendix reports video-to-text retrieval results separately for MSR-VTT-9K, MSVD, and LSMDC.
- Table A1 reports video-to-text retrieval results on the MSR-VTT-9K dataset.
- Table A2 reports video-to-text retrieval results on the MSVD dataset.
- Table A3 reports video-to-text retrieval results on the LSMDC dataset.
B. Number of Frames Experiment
The frame-count experiment uses 12 sampled frames by default and examines how changing the number of sampled frames affects MSR-VTT-9K performance.
- 12 sampled frames are used by default for training and inference.
- 6 frames produce worse performance, likely because important information is missing at that scale.
- Performance saturates as the number of frames increases.
C. Online Inference in a Large-Scale Production System
Because X-Pool produces text-conditioned video embeddings, complete index-video embeddings cannot be pre-computed for online text-to-video retrieval. A candidate-generation and reranking strategy is proposed for large-scale systems.
- Text-conditioned embeddings prevent complete pre-computation of the video index for unknown online queries.
- A production system can mean-pool pre-computed X-Pool frame embeddings to retrieve P candidates, then rerank them with text-conditioned attention.
- The proposed two-stage procedure reduces complexity from O(T V) to O(T P + V) when P << V.