CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval

• Video-text retrieval supports efficient video search and multimodal visual-language understanding.Source: §1 · p. 1
• Feature-level methods benefit from large-scale video-text pretraining but rely on frozen, off-the-shelf video features that prevent learning from propagating into the encoder.Source: §1 · p. 1
• CLIP4Clip transfers image-text pretrained CLIP to video-text retrieval, adds three similarity-calculation mechanisms, and trains end-to-end on raw video frames.Source: §1 · p. 2
• One single image is far from enough for video encoding in video-text retrieval.Source: §1 · p. 2
• Post-pretraining CLIP4Clip on a large-scale video-text dataset improves performance, especially for zero-shot prediction.Source: §1 · p. 2
• Mean pooling without new parameters is preferred for small datasets, whereas parameterized temporal modeling such as self-attention is preferred for large datasets.Source: §1 · p. 2
• The study also examines hyperparameters and reports the best setting.Source: §1 · p. 2

• Transformer-based video encoders were emerging for action classification, motivating study of transformer backbones for multimodal video-text retrieval.Source: §2 · p. 2
• CLIP demonstrated that large-scale image-text supervision can learn fine-grained visual concepts transferable to retrieval.Source: §2 · p. 2
• Early video-text retrieval methods used intensive cross-modal fusion, while recent pretrained models improved zero-shot and fine-tuned retrieval.Source: §2 · p. 2
• CLIP4Clip uses two single-modal encoders and a similarity calculator, processing sampled video frames as patch embeddings and comparing them with text representations.Source: Figure 1 · p. 3
• Unlike concurrent CLIP-based zero-shot work, CLIP4Clip investigates similarity mechanisms and continues pretraining on large-scale video-language data.Source: §2 · p. 3

• The task learns a similarity function that ranks videos for captions or captions for videos, assigning higher scores to relevant video-text pairs.Source: §3 · p. 3
• CLIP4Clip uses sampled video frames, a video encoder, a CLIP text encoder, and a similarity calculator in an end-to-end pixel-based framework.The video backbone is pretrained CLIP ViT-B/32, while the caption representation comes from CLIP's 12-layer Transformer text encoder.Source: §3 · p. 3, §3.1 · p. 3, §3.2 · p. 4

Video Encoder

• The video encoder compares 2D linear projection, which embeds frame patches independently, with 3D linear projection, which embeds patches across time.The 2D projection ignores temporal information among frames; the 3D projection is investigated to enhance temporal feature extraction.Source: §3.1 · p. 4

Similarity Calculator

• The similarity calculator includes parameter-free mean pooling, sequential LSTM or Transformer encoding, and tight Transformer-based video-caption interaction.Sequential encoding incorporates temporal information before mean pooling, whereas the tight type predicts similarity through a linear layer after multimodal interaction.Source: §3.3 · pp. 4–5

Training Strategy

• Training uses symmetric cross-entropy over B × B video-text similarities, combining video-to-text and text-to-video losses.Source: §3.4 · p. 5
• Frames are sampled uniformly at one frame per second, and CLIP4Clip is post-pretrained on about 380k Food and Entertaining videos from Howto100M.The sampling strategy balances information richness with computational and memory costs, while post-pretraining transfers temporal knowledge from video-text data.Source: §3.4 · p. 5

4.1 Datasets

• The evaluation covers MSR-VTT, MSVD, LSMDC, ActivityNet, and DiDeMo using text-to-video or video-paragraph retrieval settings.The datasets differ in size, annotations, and retrieval formulation; ActivityNet and DiDeMo use video-paragraph retrieval.Source: §4.1 · pp. 5–6

4.2 Metrics

• The study reports R@1, R@5, and R@10, with R@50 additionally used for ActivityNet, alongside median and mean rank.Higher recall is better, whereas lower median and mean rank are better.Source: §4.1 · pp. 6–7

4.3 Comparison to the State of the Art

• CLIP4Clip achieves state-of-the-art results on all five datasets by a large margin compared with the listed baselines.The results include comparisons against methods using different training datasets and end-to-end settings.Source: Table 1 · p. 6, Table 2 · p. 7, Table 3 · p. 7
• Parameter-free mean pooling performs best on smaller training settings, whereas sequential Transformer modeling performs better when more training data is available.On MSR-VTT, -meanP is best for Training-7K and -seqTransf for Training-9K; sequential models also perform better on LSMDC, while parameter-free modeling is best on MSVD.Source: §4.3 · pp. 8–9

4.4 Ablation Study

• Performance increases substantially from one to six frames, and the reported setting uses 12 frames, batch size 128, and learning rate 1e-7.The study also finds that fine-tuning all CLIP transformer encoder layers is preferable and that overly large learning rates hurt performance.Source: §4.4 · p. 9

4.5 Post-pretraining on Video Dataset

• Post-pretraining on Howto100M-380k improves both zero-shot and fine-tuned retrieval, with a much larger increase for zero-shot prediction.Fine-tuning the post-pretrained model improves LSMDC and MSVD and achieves approximately comparable results on MSR-VTT.Source: §4.5 · p. 9
• The authors identify larger video-text pretraining datasets as a future direction and report that 3D linear temporal extraction underperforms 2D linear extraction.They attribute the 3D result to discrepant initialization because CLIP was trained with 2D linear layers.Source: §4.5 · p. 10, §4.7 · p. 10

• CLIP4Clip uses pretrained CLIP as its backbone for video clip retrieval from frame-level input.Source: §5 · p. 10
• CLIP4Clip achieves state-of-the-art results on MSR-VTT, MSVC, LSMDC, ActivityNet, and DiDeMo.Source: §5 · p. 10
• Image features can promote video-text retrieval, but one image is insufficient for video encoding.Source: §5 · p. 10
• Post-pretraining an image-text pretrained CLIP on video-text data can further improve video-text retrieval performance.Source: §5 · p. 10
• 3D patch linear projection and sequential similarity are identified as promising retrieval approaches, while CLIP use is learning-rate sensitive.Source: §5 · p. 10

• Tables A1–A3 present CLIP4Clip video-to-text retrieval results on MSR-VTT, LSMDC, MSVD, ActivityNet, and DiDeMo.Source: Table A1 · p. 13
• Table A1 distinguishes MSR-VTT Training-7K and Training-9K splits and identifies pre-training and training datasets through the TrainD column.Source: Table A1 · p. 13
• Table A1 marks end-to-end training from raw video with a check mark in the E2E column.Source: Table A1 · p. 13
• Table A2 identifies MSVD and WIT training data, includes CC3M plus WebVid-2M as CW, and names its baseline methods.Source: Table A2 · p. 13
• Tables A3–A5 specify dataset abbreviations, training-data sources, end-to-end indicators, and listed baselines for LSMDC, ActivityNet, and DiDeMo.Source: Table A3 · p. 13, Table A4 · p. 13, Table A5 · p. 14