Source-linked AI summary
CLIP2Video: Mastering Video-Text Retrieval via Image CLIP
Han Fang, Pengfei Xiong, Luhui Xu, Yu Chen
TL;DR
Video-text retrieval must capture both spatial-temporal video content and multimodal video-language interactions, despite the complexity and noise of video-text pretraining data. CLIP2Video transfers CLIP image-language knowledge through separate temporal-difference and temporal-alignment modules, and reports new retrieval-accuracy records on major benchmarks. The approach is evaluated on datasets including MSR-VTT, MSVD, and VATEX, with VATEX results following an English-annotation validation protocol.
Problem
Video-text retrieval requires spatial-temporal video representations and multimodal video-language interaction, while video-text datasets are complex and noisy compared with image-text pretraining data.
Method
CLIP2Video transfers CLIP image-language pretraining to video-text retrieval by separating image-text learning from temporal modeling with Temporal Difference and Temporal Alignment Blocks.
Results
CLIP2Video reports new retrieval-accuracy records on MSR-VTT, MSVD, and VATEX, with thorough ablation studies attributing improvements to its divided concept.
Takeaways & Limitations
The paper presents a CLIP-based two-module framework that captures fine-frame motion and re-aligns video and language tokens for video-text retrieval.
Takeaways & Limitations
VATEX evaluation follows an HGR validation protocol using English annotations because test annotations are inaccessible.
Abstract
from arXiv · showhide
We present CLIP2Video network to transfer the image-language pre-training model to video-text retrieval in an end-to-end manner. Leading approaches in the domain of video-and-language learning try to distill the spatio-temporal video features and multi-modal interaction between videos and languages from a large-scale video-text dataset. Different from them, we leverage pretrained image-language model, simplify it as a two-stage framework with co-learning of image-text and enhancing temporal relations between video frames and video-text respectively, make it able to train on comparatively small datasets. Specifically, based on the spatial semantics captured by Contrastive Language-Image Pretraining (CLIP) model, our model involves a Temporal Difference Block to capture motions at fine temporal video frames, and a Temporal Alignment Block to re-align the tokens of video clips and phrases and enhance the multi-modal correlation. We conduct thorough ablation studies, and achieve state-of-the-art performance on major text-to-video and video-to-text retrieval benchmarks, including new records of retrieval accuracy on MSR-VTT, MSVD and VATEX.
1. Introduction
Video-text retrieval must represent both spatial content and temporal relationships while aligning videos with language. CLIP2Video transfers CLIP image-language semantics and adds temporal modules, achieving new retrieval-accuracy records on major benchmarks.
- Motivation: Video-text retrieval requires visual feature representation and multimodal interaction between videos and languages.These challenges involve modeling spatial and temporal dimensions together and jointly embedding video and text.
- Motivation: The paper separates video-language learning into image-text multimodal learning and temporal relationships between video frames and video-text.This division assigns spatial semantics to image-text pretraining and temporal structure to video-specific modeling.
- Method: CLIP2Video transfers CLIP image-language semantics to video-text retrieval through a Temporal Difference Block and a Temporal Alignment Block.TDB enhances temporal interaction between frames, while TAB aligns video clips and contextual words in a shared space.
- Method: The Temporal Difference Block adds adjacent-frame differences to video features to simulate motion change, while the Temporal Alignment Block strengthens video-phrase correlation.The two blocks address temporal relationships in video frames and video-language respectively.
- Results: CLIP2Video reports new retrieval-accuracy records on MSR-VTT, MSVD, and VATEX, supported by thorough ablation studies.The authors attribute the large improvements to the paper’s divided-concept design.
2. Related Work
Prior video-language work learns spatial-temporal video representations and cross-modal embeddings, often using large video-text datasets. Such datasets are harder to collect and noisier than image-text pretraining data, motivating alternative transfer strategies.
- Video Representation Learning: Video representation learning has primarily used 2D or 3D spatial-temporal convolutional networks, with newer work adapting transformer-based image encoders.Examples include SlowFast’s two pathways and ViViT or time-transformer variants of ViT.
- Video-language Learning: Video-language learning uses large-scale visual-language pairs, including HowTo100M, to learn spatio-temporal features through multimodal training.Methods also investigate weakly supervised learning from instructional and social-media videos.
- Video-language Learning: Video-text datasets are more complex and noisier than image-text pretraining datasets, making video pretraining models difficult to leverage fully.The passage specifically notes ambiguity between text semantics and video content in instructional videos.
- Video-Text Retrieval: Video-text retrieval methods have progressed from intensive cross-modal fusion toward pretrained models that jointly embed videos and language.Recent pretrained approaches have reported noticeable zero-shot retrieval results.
3. Methodology
CLIP2Video transfers CLIP’s image-language semantics to video-text retrieval by separating spatial multimodal learning from temporal modeling. Its Temporal Difference and Temporal Alignment Blocks encode motion and align video-text tokens in a shared space.
- Overview: CLIP2Video initializes video-text retrieval with CLIP and separates spatial image-text learning from temporal relationships in videos and video-language.The framework maps video and text into a joint embedding space while retaining CLIP’s pretrained semantic representation.
- Video representation: The video encoder applies ViT to each frame, concatenates frame embeddings, and uses a temporal transformer to model interactions across frames.ViT supplies frame-level spatial representations, while the temporal transformer incorporates cross-frame information.
- Temporal Difference Block: The Temporal Difference Block transforms differences between adjacent frame embeddings into difference-enhanced tokens inserted between frames.Attention and sigmoid transformation encode successive-frame subtraction, after which frame and difference tokens enter the temporal transformer.
- Temporal Alignment Block: The Temporal Alignment Block aggregates video and word tokens around shared centers to align their contextual representations before similarity computation.Shared centers jointly organize modal features, while normalized similarities determine cluster weights and aligned center embeddings.
- Temporal Alignment Block: Motion-sensitive frame tokens are re-sampled at a larger frame rate, encoded with the temporal difference block, and concatenated with frame features for alignment.This sampling highlights motion changes and complements the semantically coherent frame sequence when re-adjusting motion-related center weights.
- Training objective: Training uses symmetric cross-entropy over global and aligned video-text representations, with cosine similarity used during matching.The model computes loss from both global and aligned representations and combines the resulting similarity signals at inference.
4. Experiments
Experiments evaluate CLIP2Video on three video-text retrieval benchmarks using standard retrieval metrics and ablations of its temporal modules. The method achieves state-of-the-art results, while temporal difference and alignment designs improve representation and retrieval performance.
- 4.1. Experimental Settings: Experiments cover video-to-text and text-to-video retrieval on MSR-VTT, MSVD, and VATEX.MSR-VTT is evaluated under 1k-A and full protocols; VATEX follows a validation protocol with English annotations.
- 4.1. Experimental Settings: Evaluation reports Recall at rank K, median rank, and mean rank, with higher recall and lower ranks indicating better performance.
- 4.2.1. Effects of Temporal Difference Block: Temporal transformer aggregation outperforms mean pooling by interacting with and aggregating video frames.The temporal difference block further inserts difference-enhanced frame information to represent motion changes.
- 4.2.1. Effects of Temporal Difference Block: Difference-level attention improves temporal modeling over directly inserting frame subtraction or encoding correlations only with an MLP.The ablation states that explicit interaction before insertion avoids damaging the frame representation and significantly improves performance.
- 4.2.2. Effects of Temporal Alignment Block: Temporal alignment improves video-to-text retrieval, with transformer-enhanced frame embeddings redistributing emphasis toward motion-related temporal centers.Adding TDB to the alignment design produces the best reported performance in the alignment ablation.
- 4.3. Comparison with Other Methods: The method achieves state-of-the-art results on MSR-VTT, MSVD, and VATEX, including new retrieval-accuracy records.The comparison discusses both video-to-text and text-to-video retrieval and reports further gains from temporal modeling, including on small datasets.
5. Conclusion
CLIP2Video reframes video-text retrieval around separate spatial image-text learning and temporal video relations, using two temporal blocks to transfer CLIP. It significantly improves retrieval across multiple benchmarks, setting new records on MSR-VTT, MSVD, and VATEX.
- Conclusion: CLIP2Video divides video-language learning into image-text multi-modal learning and temporal relationships between video frames and video-text.This macroscopic decomposition separates spatial and temporal aspects of the task.
- Conclusion: The network transfers an image-language pretrained model using Temporal Difference and Temporal Alignment Blocks.These blocks capture fine-frame motion and re-align video and language tokens, respectively.
- Conclusion: CLIP2Video significantly improves performance on several text-video retrieval benchmarks.The conclusion reports improvements across multiple benchmarks rather than a single evaluation setting.
- Conclusion: The approach establishes new retrieval records on MSR-VTT, MSVD, and VATEX.The cited conclusion names all three benchmarks but does not provide numerical scores here.