Source-linked AI summary
CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, Tianrui Li
TL;DR
Video-text retrieval requires models that can exploit visual-language knowledge while representing temporal video content efficiently. CLIP4Clip transfers pretrained CLIP to end-to-end retrieval, studies similarity and temporal modeling choices, and reports state-of-the-art results across five datasets. The authors also identify dataset-dependent modeling preferences and note that larger-scale pretraining remains future work.
Problem
Video-text retrieval needs effective video representations, while transferring image-text pretrained knowledge raises questions about frame sufficiency, temporal modeling, post-pretraining, and hyperparameter sensitivity.
Method
CLIP4Clip transfers CLIP to end-to-end video-text retrieval from frame-level inputs and evaluates parameter-free, sequential, and tight similarity calculators.
Results
CLIP4Clip achieves state-of-the-art results on MSR-VTT, MSVC, LSMDC, ActivityNet, and DiDeMo.
Takeaways & Limitations
Mean pooling is preferred for small datasets, while parameterized temporal modeling such as self-attention is preferred for large datasets.
Takeaways & Limitations
The authors identify pretraining on an even larger dataset as future work.
Abstract
from arXiv · showhide
Video-text retrieval plays an essential role in multi-modal research and has been widely used in many real-world web applications. The CLIP (Contrastive Language-Image Pre-training), an image-language pre-training model, has demonstrated the power of visual concepts learning from web collected image-text datasets. In this paper, we propose a CLIP4Clip model to transfer the knowledge of the CLIP model to video-language retrieval in an end-to-end manner. Several questions are investigated via empirical studies: 1) Whether image feature is enough for video-text retrieval? 2) How a post-pretraining on a large-scale video-text dataset based on the CLIP affect the performance? 3) What is the practical mechanism to model temporal dependency between video frames? And 4) The Hyper-parameters sensitivity of the model on video-text retrieval task. Extensive experimental results present that the CLIP4Clip model transferred from the CLIP can achieve SOTA results on various video-text retrieval datasets, including MSR-VTT, MSVC, LSMDC, ActivityNet, and DiDeMo. We release our code at https://github.com/ArrowLuo/CLIP4Clip.
1 Introduction
Video-text retrieval matters for efficient video search and multimodal understanding, but existing approaches trade off raw-video learning against computational cost. CLIP4Clip transfers CLIP to end-to-end video retrieval and empirically studies frame representation, post-pretraining, temporal modeling, and hyperparameters.
- Video-text retrieval supports efficient video search and multimodal visual-language understanding.
- Feature-level methods benefit from large-scale video-text pretraining but rely on frozen, off-the-shelf video features that prevent learning from propagating into the encoder.
- CLIP4Clip transfers image-text pretrained CLIP to video-text retrieval, adds three similarity-calculation mechanisms, and trains end-to-end on raw video frames.
- One single image is far from enough for video encoding in video-text retrieval.
- Post-pretraining CLIP4Clip on a large-scale video-text dataset improves performance, especially for zero-shot prediction.
- Mean pooling without new parameters is preferred for small datasets, whereas parameterized temporal modeling such as self-attention is preferred for large datasets.
- The study also examines hyperparameters and reports the best setting.
2 Related Works
Related work spans transformer-based video backbones, text-supervised visual representation learning, and video-text retrieval systems. CLIP4Clip extends this trajectory by transferring CLIP and continuing pretraining on video-language data while investigating similarity mechanisms.
- Transformer-based video encoders were emerging for action classification, motivating study of transformer backbones for multimodal video-text retrieval.
- CLIP demonstrated that large-scale image-text supervision can learn fine-grained visual concepts transferable to retrieval.
- Early video-text retrieval methods used intensive cross-modal fusion, while recent pretrained models improved zero-shot and fine-tuned retrieval.
- CLIP4Clip uses two single-modal encoders and a similarity calculator, processing sampled video frames as patch embeddings and comparing them with text representations.
- Unlike concurrent CLIP-based zero-shot work, CLIP4Clip investigates similarity mechanisms and continues pretraining on large-scale video-language data.
3 Framework
CLIP4Clip frames video-text retrieval as learning similarity between sampled video frames and captions, using CLIP-based encoders and several temporal or multimodal similarity mechanisms. The framework trains end-to-end on pixels, compares parameter-free, sequential, and tight similarity calculation, samples frames uniformly, and post-pretrains on video-text data.
- The task learns a similarity function that ranks videos for captions or captions for videos, assigning higher scores to relevant video-text pairs.
- CLIP4Clip uses sampled video frames, a video encoder, a CLIP text encoder, and a similarity calculator in an end-to-end pixel-based framework.The video backbone is pretrained CLIP ViT-B/32, while the caption representation comes from CLIP's 12-layer Transformer text encoder.
- Video Encoder: The video encoder compares 2D linear projection, which embeds frame patches independently, with 3D linear projection, which embeds patches across time.The 2D projection ignores temporal information among frames; the 3D projection is investigated to enhance temporal feature extraction.
- Similarity Calculator: The similarity calculator includes parameter-free mean pooling, sequential LSTM or Transformer encoding, and tight Transformer-based video-caption interaction.Sequential encoding incorporates temporal information before mean pooling, whereas the tight type predicts similarity through a linear layer after multimodal interaction.
- Training Strategy: Training uses symmetric cross-entropy over B × B video-text similarities, combining video-to-text and text-to-video losses.
- Training Strategy: Frames are sampled uniformly at one frame per second, and CLIP4Clip is post-pretrained on about 380k Food and Entertaining videos from Howto100M.The sampling strategy balances information richness with computational and memory costs, while post-pretraining transfers temporal knowledge from video-text data.
4 Experiments
CLIP4Clip is evaluated across five video-text retrieval datasets using standard retrieval metrics, ablations, and comparisons with prior methods. It achieves state-of-the-art results across all five datasets, while experiments examine similarity mechanisms, temporal modeling, hyperparameters, and video post-pretraining.
- 4.1 Datasets: The evaluation covers MSR-VTT, MSVD, LSMDC, ActivityNet, and DiDeMo using text-to-video or video-paragraph retrieval settings.The datasets differ in size, annotations, and retrieval formulation; ActivityNet and DiDeMo use video-paragraph retrieval.
- 4.2 Metrics: The study reports R@1, R@5, and R@10, with R@50 additionally used for ActivityNet, alongside median and mean rank.Higher recall is better, whereas lower median and mean rank are better.
- 4.3 Comparison to the State of the Art: CLIP4Clip achieves state-of-the-art results on all five datasets by a large margin compared with the listed baselines.The results include comparisons against methods using different training datasets and end-to-end settings.
- 4.3 Comparison to the State of the Art: Parameter-free mean pooling performs best on smaller training settings, whereas sequential Transformer modeling performs better when more training data is available.On MSR-VTT, -meanP is best for Training-7K and -seqTransf for Training-9K; sequential models also perform better on LSMDC, while parameter-free modeling is best on MSVD.
- 4.4 Ablation Study: Performance increases substantially from one to six frames, and the reported setting uses 12 frames, batch size 128, and learning rate 1e-7.The study also finds that fine-tuning all CLIP transformer encoder layers is preferable and that overly large learning rates hurt performance.
- 4.5 Post-pretraining on Video Dataset: Post-pretraining on Howto100M-380k improves both zero-shot and fine-tuned retrieval, with a much larger increase for zero-shot prediction.Fine-tuning the post-pretrained model improves LSMDC and MSVD and achieves approximately comparable results on MSR-VTT.
- 4.5 Post-pretraining on Video Dataset: The authors identify larger video-text pretraining datasets as a future direction and report that 3D linear temporal extraction underperforms 2D linear extraction.They attribute the 3D result to discrepant initialization because CLIP was trained with 2D linear layers.
5 Conclusion
CLIP4Clip uses pretrained CLIP for end-to-end video clip retrieval and reports state-of-the-art results across five datasets. The empirical studies identify image features, post-pretraining, similarity design, and learning-rate sensitivity as important factors.
- CLIP4Clip uses pretrained CLIP as its backbone for video clip retrieval from frame-level input.
- CLIP4Clip achieves state-of-the-art results on MSR-VTT, MSVC, LSMDC, ActivityNet, and DiDeMo.
- Image features can promote video-text retrieval, but one image is insufficient for video encoding.
- Post-pretraining an image-text pretrained CLIP on video-text data can further improve video-text retrieval performance.
- 3D patch linear projection and sequential similarity are identified as promising retrieval approaches, while CLIP use is learning-rate sensitive.
A Video-to-Text Retrieval
The paper reports video-to-text retrieval results for CLIP4Clip across MSR-VTT, LSMDC, MSVD, ActivityNet, and DiDeMo. The accompanying tables distinguish dataset splits, training data, end-to-end training, and baseline methods where specified.
- Tables A1–A3 present CLIP4Clip video-to-text retrieval results on MSR-VTT, LSMDC, MSVD, ActivityNet, and DiDeMo.
- Table A1 distinguishes MSR-VTT Training-7K and Training-9K splits and identifies pre-training and training datasets through the TrainD column.
- Table A1 marks end-to-end training from raw video with a check mark in the E2E column.
- Table A2 identifies MSVD and WIT training data, includes CC3M plus WebVid-2M as CW, and names its baseline methods.
- Tables A3–A5 specify dataset abbreviations, training-data sources, end-to-end indicators, and listed baselines for LSMDC, ActivityNet, and DiDeMo.