A CLIP-Hitchhiker's Guide to Long Video Retrieval

1 Introduction

• Query-scoring without learned parameters outperforms CLIP-initialized temporal aggregation methods by a large margin on Recall@1 and achieves state-of-the-art retrieval on three datasets.Figure 1 compares MSR-VTT with ActivityNet and Condensed Movies; the text reports state-of-the-art results on ActivityNet Captions, MSR-VTT, and Condensed Movies.Source: Figure 1 · p. 2, §1 · p. 2
• Long-form videos can contain redundant frames alongside brief informative events, making mean-pooling a poor fit for their temporal structure.The paper gives the example of a long video in which a few seconds capture a salient event.Source: §1 · p. 2
• The proposed approach predicts frame relevance and uses the scores for a weighted mean of frame embeddings.The paper investigates query-scoring, self-attention scoring conditioned on video information, and joint-attention scoring conditioned on both video and query.Source: §1 · pp. 2–3
• The paper presents the method as an improved mean-pooling baseline for comparing temporal aggregation methods and reports improved CLIP performance on long-video classification.The classification evaluation uses the Charades dataset.Source: §1 · p. 3

2 Related Work

• Large-scale image-text representations, including CLIP, have strong image capabilities, while available video-text pretraining data is much smaller.The text contrasts image-text datasets with video-text datasets and motivates adapting pretrained image-text models to long-form video.Source: §2 · p. 3
• Earlier video-text retrieval work aggregated features from classification networks, explored additional modalities, and addressed aggregation over full-video duration.Approaches included vector quantization and audio features.Source: §2 · p. 3
• CLIP-based temporal modeling has performed comparably to or worse than mean frame embeddings, while temporal transformers typically require vast training data.The mean representation also cannot intelligently aggregate multiple events over a long video.Source: §2 · p. 4
• SCSampler selects salient clips for classification using a separately learned, cheaper model, whereas this work weights frame samples online during training.The selected clips in SCSampler are fed to a classification network whose scores are averaged.Source: §2 · p. 4
• The paper studies weighted-mean aggregation and shows that query-guided linear weighting can achieve state-of-the-art performance without additional learning.The joint image-text representation allows the text query to guide temporal aggregation.Source: §2 · p. 4

3 Temporal Aggregation of Image-Text Representations

• Query-frame relevance scores weight the original frame embeddings into one video representation, addressing the limits of learned temporal aggregation and uniform mean-pooling.The scores are frame-query similarities, normalized with a softmax whose temperature controls how concentrated the weights are.Source: §3 · pp. 4–5
• As temperature approaches zero, the representation approaches the most relevant frame; as it grows, the weights become uniform and recover mean-pooling.Intermediate temperatures balance emphasizing relevant frames with retaining the video's temporal content.Source: §3 · p. 5

3.1 Alternative Scoring Methods

• Self-attention predicts query-independent frame scores, while joint-attention also inputs the text query; both produce scalar weights rather than using attention outputs as the video representation.Joint-attention performs cross- and self-attention between query and frame embeddings; the final aggregation remains a linear combination of the original frame embeddings.Source: Self-Attention · pp. 5–6, Joint-Attention Scoring · p. 6, Figure 2 · p. 6
• Query-scoring has no learned parameters, while query-conditioned aggregation increases retrieval complexity because each video representation depends on the query.A two-stage ranking can use mean embeddings for the full ranking and apply query scoring only to top-ranked results.Source: Complexity · pp. 6–7

3.2 Alternative Aggregation Methods

• Hard top-K averages selected frames equally and excludes the rest, whereas soft query-scoring generally retains some information from every frame.Averaging frame-level similarity logits is comparable in zero-shot evaluation, but hard top-K performs worse when fine-tuning.Source: Hard Top-K · p. 7, Averaging per-frame logits rather than… · p. 7

3.3 Video-to-text retrieval and video classification

• The methods can also support video classification by framing classification as video-to-text retrieval.In that setting, storing video embeddings is unnecessary for text retrieval, and the number of action-label queries tends to be small.Source: §3.3 · p. 7

4.1 Downstream Datasets

• The evaluation covers MSR-VTT, Condensed Movies, and ActivityNet retrieval, plus multilabel Charades classification, across videos ranging from 15 to 180 seconds on average.MSR-VTT averages 15 seconds, Condensed Movies 132 seconds, ActivityNet 180 seconds, and Charades 30 seconds.Source: §4.1 · p. 8

4.2 Experiment Protocol

• Experiments use CLIP ViT-B/16, finetune end-to-end, sample 16 training frames and 120 uniformly sampled test frames, and use τ = 0.1 for query-scoring.The model is finetuned with Adam at learning rate 5e-7; the test-frame count applies unless otherwise specified.Source: §4.2 · pp. 8–9

4.3 Results

• Query-scoring achieves state-of-the-art text-to-video retrieval on MSR-VTT, ActivityNet, and Condensed Movies, outperforming prior CLIP aggregation methods with millions of learned parameters.The method has a single parameter; the tables report MSR-VTT and Condensed Movies/ActivityNet comparisons.Source: Comparison to the state of the art · p. 9, Table 2 · p. 9, Table 3 · p. 10
• On Charades, query-scoring matches ActionCLIP and improves on CLIP4CLIP seqTransf and mean pooling without ActionCLIP’s temporal modeling, 320-frame averaging, or prompt templates.Test-time normalization is discussed separately from temporal modeling; dual softmax requires access to all queries at test time.Source: Comparison to the state of the art · pp. 9–10, Table 4 · p. 10

4.4 Ablation study

• Alternative scoring methods perform strongly even without query information; temporal self-attention scores better on ActivityNet, while concise Condensed Movies descriptions afford the greatest benefits.The authors suggest that dense ActivityNet paragraphs make most frames relevant to the query, so removing noisy frames may suffice without query-level information.Source: §4.4 · pp. 10–11, Table 5 · p. 11
• Feature averaging beats score averaging in zero-shot and finetuning settings; hard top-K is comparable in zero-shot, while soft scoring features are optimal for finetuning.The authors note that smoother frame selection appears more favorable for training.Source: §4.4 · p. 11, Table 6 · p. 11
• Performance for both baseline and query-scoring improves as test frames increase, with diminishing gains after 1 fps; temperatures from 0.05 to 0.15 are consistently optimal.The query-scoring advantage over the baseline also grows with more frames; very large temperatures may cause drifting errors during learning.Source: §4.4 · pp. 11–12, Figure 3 · p. 12

Why are weighted-mean frame embeddings so effective?

• Query-scoring improves single-frame representations after finetuning, and its highest-scoring frames are more relevant to the text query than its lowest-scoring frames.Mean-pooled 16-frame embeddings occupy locations disjoint from single-frame embeddings, and a linear classifier readily distinguishes them.Source: Why are weighted-mean frame embeddings… · pp. 12–13, Table 8 · p. 13, Table 7 · p. 13

• The proposed methods mean-weight frame embeddings, with and without query information, to pick out salient frames.Source: §5 · p. 14
• The method provides a strong baseline, outperforming prior work across four datasets, including more complex temporal modelling.Source: §5 · p. 14
• Future work could address the scarcity of long-form video data by combining self-supervised learning with textual supervision.Source: §5 · p. 14

• The full split contains 6,513 training, 497 validation, and 2,990 test videos.Source: A MSR-VTT Full Split · p. 15
• Query-scoring outperforms all prior work on the MSR-VTT full split except in mean rank.The authors suggest the mean-rank exception could be attributable to smaller amounts of training data.Source: Table 9 · p. 15, A MSR-VTT Full Split · p. 15
• The table compares text-to-video retrieval methods, including CLIP-backbone methods with different temporal aggregation approaches and reported aggregation parameters.Source: Table 9 · p. 15

• Finetuning on Condensed Movies shows a pronounced query-scoring boost over hard top-k and mean-pooling.Source: B Extended Comparison of Aggregation… · p. 15, Table 10 · p. 15
• The reported alternatives to query-scoring are hard top-k and mean-pooling.Source: B Extended Comparison of Aggregation… · p. 15
• The reported comparison evaluates text-to-video retrieval when finetuning on Condensed Movies.Source: Table 10 · p. 15