Source-linked AI summary
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding
Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, Christoph Feichtenhofer
TL;DR
VideoCLIP addresses the challenge of learning fine-grained video-text associations for zero-shot transfer across end tasks. It pre-trains a unified model with temporally overlapping positives and retrieval-mined hard negatives, achieving strong performance across tasks and sometimes surpassing supervised approaches.
Problem
Straightforward objectives and randomly sampled aligned clips can fail to learn the fine-grained video-frame and word-token associations required by zero-shot video-text tasks.
Method
VideoCLIP pre-trains a unified video-text representation by contrasting temporally overlapping positive clips with hard negatives obtained through retrieval of similar videos.
Results
VideoCLIP outperforms prior work across diverse downstream tasks without downstream supervision and can match or surpass fully supervised approaches in some cases.
Takeaways & Limitations
Fine-grained association learning through overlapped alignment and retrieval-augmented negatives supports zero-shot transfer across multiple video-text understanding tasks.
Takeaways & Limitations
Zero-shot transfer lacks task-specific knowledge for pairing clip lengths and can mix objects with similar color or shape, while fine-tuning still provides gains.
Abstract
from arXiv · showhide
We present VideoCLIP, a contrastive approach to pre-train a unified model for zero-shot video and text understanding, without using any labels on downstream tasks. VideoCLIP trains a transformer for video and text by contrasting temporally overlapping positive video-text pairs with hard negatives from nearest neighbor retrieval. Our experiments on a diverse series of downstream tasks, including sequence-level text-video retrieval, VideoQA, token-level action localization, and action segmentation reveal state-of-the-art performance, surpassing prior work, and in some cases even outperforming supervised approaches. Code is made available at https://github.com/pytorch/fairseq/tree/main/examples/MMPT.
1 Introduction
VideoCLIP pre-trains a unified transformer for zero-shot video-text understanding by learning fine-grained associations with temporally overlapping positives and retrieval-mined hard negatives. It transfers directly across diverse video-text tasks and can outperform supervised approaches in some settings.
- Model and objective: The approach uses a unified Transformer trained with a contrastive objective on video-text clips.It is designed to support both global sequence-level and local clip/token-level tasks.
- Training innovations: Temporally overlapping video-text clips improve alignment quality and quantity compared with exact timestamp matching.The clips can have varying lengths, accommodating cases where multiple or longer text clips align better with a video clip.
- Training innovations: Retrieval-augmented pre-training forms batches from similar video clusters to provide harder negatives than sampling only within a video.The method alternates between retrieving similar video clusters and using them for pre-training.
- Zero-shot video-text understanding: VideoCLIP targets zero-shot transfer across video-text tasks without task-specific labels or fine-tuning.The model is directly applied to text-video retrieval, VideoQA, action localization, and segmentation.
- Results: VideoCLIP performs strongly across downstream tasks, including outperforming existing zero-shot and fully supervised methods on YouCook2 text-video retrieval.The reported result uses no labels from downstream tasks.
2 Related Work
Related work frames zero-shot transfer as a major pre-training direction spanning generative and discriminative models. Multimodal video-text systems use unimodal encoders, cross-encoders, or single crossmodal encoders, with fine-grained video-text association remaining a central challenge.
- Pre-training for zero-shot transfer: Zero-shot transfer research includes generative models that use task examples and prompts as context for language-model responses.This line developed alongside discriminative models for task-agnostic transfer.
- Pre-training for zero-shot transfer: Discriminative zero-shot models learn a joint video-text space through similarity search, ranking, or contrastive learning.These models can later transfer to particular tasks.
- Multimodal video-text pre-training: Multimodal video-text pre-training includes unimodal-encoder approaches for retrieval and cross-encoder approaches for modality fusion.Examples include retrieval-oriented systems and models such as HERO and UniVL.
- Multimodal video-text pre-training: Single crossmodal encoders concatenate vision and text sequences but limit zero-shot transfer capability, especially for retrieval tasks.The related-work discussion identifies fine-grained video-text association as the key challenge for diverse end tasks.
3 VideoCLIP Pre-training
VideoCLIP pre-trains unified video and text Transformers for zero-shot transfer by learning fine-grained associations through overlapping positives and retrieval-based hard negatives.
- Video and Text Transformers: VideoCLIP uses separate trainable video and text Transformers to encode projected video tokens and embedded text tokens.Video features come from a frozen CNN followed by a trainable MLP, while text tokens use BERT-style embedding lookup.
- Video and Text Transformers: Average pooling produces global video and text representations while encouraging token-level features for localization and segmentation.The method uses average pooling instead of the [CLS] token.
- Contrastive Loss: Contrastive learning minimizes symmetric video-to-text and text-to-video InfoNCE losses over positive and batch-induced negative pairs.Positive text clips overlap the video clip, while other text clips in the batch form negatives.
- Positive Pair Construction: Overlapping positives sample text first, center a video clip within its timestamp boundary, and grow the clip to a random duration up to ∼32 seconds.This avoids relying only on exact temporal alignment, which can produce semantically low-relevance positives and misleading negatives.
- Hard Negative Mining: Retrieval-augmented pre-training mines semantically similar clips from other videos as harder negatives than intra-video sampling.The approach retrieves clusters of mutually similar videos and samples overlapped video-text pairs from them for training.
6 end
Retrieval-augmented training alternates latent-space video retrieval with contrastive training to construct batches of mutually similar videos.
- Retrieval Augmented Training: VideoCLIP alternates retrieval and training in each epoch as a two-stage retrieval-augmented training process.The retrieval stage computes features, builds an index, and forms clusters before the training stage samples pairs.
- Retrieval Stage: Each video’s global feature is computed by averaging embeddings from all its video-text clips before dense-index retrieval.The appendix ablation reports this is better than using a video’s starting clip alone.
- Cluster Construction: The method builds a dense index over video features and retrieves clusters by sampling videos from the 2k nearest neighbors of random videos.Sampling from 2k neighbors is intended to make videos within each cluster mutually closer.
4 Zero-shot Transfer to End Tasks
VideoCLIP transfers its learned representation directly to retrieval, VideoQA, localization, and segmentation tasks without target-task labels, including rejection for an unseen Outside label.
- Zero-shot Transfer: Zero-shot transfer evaluates text-video retrieval, VideoQA, action localization, and action segmentation using learned video-text similarities.The tasks test similarity at sequence, frame, and token levels.
- Action Segmentation: Action segmentation includes an Outside label absent from pre-training, making rejection of unsupported tokens task-dependent.The setting resembles open-set recognition or out-of-domain intent detection.
- Action Segmentation: For each video token, the method assigns the highest-scoring label when its similarity exceeds γ and otherwise predicts Outside.The threshold γ is estimated from the maximum intra-label dot product because zero-shot transfer provides no training or validation data for tuning it.
5 Experiments
VideoCLIP is evaluated for zero-shot transfer across retrieval, VideoQA, action segmentation, and action localization, using several established datasets and no target-task labels. It performs strongly against supervised and zero-shot baselines, while ablations show that retrieval-augmented training and temporally overlapping positives are important.
- Evaluation Setup: VideoCLIP is evaluated across text-video retrieval, VideoQA, action segmentation, and action step localization on multiple datasets.The evaluation uses Youcook2, MSR-VTT, DiDeMo, COIN, and CrossTask, with zero-shot transfer applied without fine-tuning on target labels.
- Main Results: On Youcook2, zero-shot VideoCLIP outperforms existing zero-shot methods and traditional supervised methods, while fine-tuning reaches state-of-the-art.Its zero-shot performance is close to supervised pre-training baselines.
- Main Results: On MSR-VTT retrieval and DiDeMo retrieval, VideoCLIP improves over prior methods but shows a larger zero-shot-to-supervised gap on MSR-VTT.On MSR-VTT, fine-tuning reaches state-of-the-art R@1 without ImageNet or large-scale Instagram data used by Support Set.
- Main Results: Zero-shot VideoCLIP outperforms most supervised methods on MSR-VTT VideoQA and reaches the best performance after fine-tuning.The results similarly suffer from domain shift between HowTo100M and MSR-VTT.
- Main Results: Zero-shot VideoCLIP outperforms all supervised methods on COIN action segmentation without using COIN labels.Fine-tuning provides an additional ∼10% accuracy gain, indicating room for improvement.
- Ablation Study: Removing retrieval-augmented training drops R@1 by over 4%, while exact-alignment positives cause another 4% drop; together, both techniques yield a ∼50% relative recall improvement.Using the MIL-NCE objective performs significantly worse than VideoCLIP’s objective, at 16.1 versus 22.7 R@1.
- Ablation Study: Sharing video and text Transformer backbones decreases performance only slightly, while sampling from nearest neighbors produces higher-quality video clusters.The ablations support joint backbone use and the retrieval strategy used by VideoCLIP.
- Ablation Study: Using only the [CLS] token prevents VideoCLIP from exploiting token-level information and yields worse performance.This design choice is relevant to token-level tasks such as action localization and action segmentation.
6 Conclusion
VideoCLIP pre-trains a video-text model for zero-shot transfer to tasks requiring fine-grained video-language associations. It outperforms prior work across varied tasks without downstream supervision, while fine-tuning still provides gains.
- VideoCLIP targets zero-shot transfer to end tasks requiring fine-grained associations between video and language.
- Its training objective contrasts temporally overlapping positive pairs with hard negatives obtained through nearest-neighbor retrieval.
- VideoCLIP outperforms prior work across varied tasks without downstream supervision and is competitive or better than some fully supervised methods.
- Fine-tuning the pretrained model still produces additional gains.
Code
The paper makes its code and models available for VideoCLIP.
- Code and models are made available to support use of VideoCLIP.
- The availability statement concerns the paper’s implementation resources.
- Readers are directed to the release for the provided materials.
A Supplementary Material for VideoCLIP
The supplementary material expands the paper’s experimental and implementation documentation. It also adds ablations and configuration analyses.
- The supplement provides additional experimental setups for each end task.
- It specifies model hyperparameters and details how VideoCLIP is trained.
- It reports extra ablations and analyses of different VideoCLIP configurations.
A.1 End Task Setup Details
The end-task setups evaluate VideoCLIP on retrieval, question answering, action segmentation, and action-step localization using established video datasets and task-specific procedures.
- Text-Video Retrieval: Text-video retrieval on Youcook2 and MSR-VTT uses video and text Transformers to compute text-to-video similarities.
- Text-Video Retrieval: Youcook2 contains 2K cooking videos and provides 9,473 training and 3,305 test clip-text pairs after filtering.
- Multiple-choice VideoQA: MSR-VTT evaluation samples 1K clip-text pairs for testing, while multiple-choice VideoQA selects the candidate answer with maximum video-text similarity.
- Action Segmentation: COIN action segmentation uses 778 textual segment labels and sliding 32-second windows with a 16-second step, averaging logits for overlapping tokens.
- Action Step Localization: Action-step localization compares token-level video representations with latent representations of textual step labels.