Source-linked AI summary
Support-set bottlenecks for video-text representation learning
Mandela Patrick, Po-Yao Huang, Yuki Asano, Florian Metze, Alexander Hauptmann, João Henriques, Andrea Vedaldi
TL;DR
Noise contrastive learning can push apart semantically equivalent video-text samples, potentially harming semantic retrieval. The paper adds cross-instance captioning, reconstructing each caption from a weighted support set of other video representations. The resulting model achieves state-of-the-art text-to-video and video-to-text retrieval performance on three datasets.
Problem
Contrastive learning treats every nonmatching sample as a negative, even when samples are semantically equivalent, which can degrade semantic video retrieval.
Method
Cross-instance captioning reconstructs a caption from a learnable weighted mixture of other videos’ representations, using the support set as a bottleneck for shared semantics.
Results
The model achieves state-of-the-art text-to-video and video-to-text retrieval performance on three datasets.
Takeaways & Limitations
Cross-instance captioning encourages videos with similar captions to share representations, including when contrastive loss pushes them apart.
Takeaways & Limitations
The approach is demonstrated for text-to-video retrieval and requires reasonable conditional generators when generalized to other contrastive-learning settings.
Abstract
from arXiv · showhide
The dominant paradigm for learning video-text representations -- noise contrastive learning -- increases the similarity of the representations of pairs of samples that are known to be related, such as text and video from the same sample, and pushes away the representations of all other pairs. We posit that this last behaviour is too strict, enforcing dissimilar representations even for samples that are semantically-related -- for example, visually similar videos or ones that share the same depicted action. In this paper, we propose a novel method that alleviates this by leveraging a generative model to naturally push these related samples together: each sample's caption must be reconstructed as a weighted combination of other support samples' visual representations. This simple idea ensures that representations are not overly-specialized to individual samples, are reusable across the dataset, and results in representations that explicitly encode semantics shared between samples, unlike noise contrastive learning. Our proposed method outperforms others by a large margin on MSR-VTT, VATEX and ActivityNet, and MSVD for video-to-text and text-to-video retrieval.
1 INTRODUCTION
The paper argues that contrastive learning can separate semantically related videos and introduces cross-instance captioning to encourage shared representations. The method improves video representation learning beyond a strong contrastive baseline.
- 1 INTRODUCTION: Noise contrastive learning treats every nonmatching sample as a negative, separating representations that may be semantically equivalent.This can degrade downstream semantic video retrieval.
- 1 INTRODUCTION: Instance captioning provides only a modest benefit because it remains sample-specific and discourages concept sharing.Its goal is to generate a caption describing one specific video rather than another video.
- 1 INTRODUCTION: Cross-instance captioning reconstructs text from a learnable weighted mixture of support-set video representations selected through online batch-wise comparisons.The limited support set acts as a bottleneck that encourages extraction of shared semantics.
- 1 INTRODUCTION: The support-set bottleneck can associate videos sharing similar captions even when contrastive loss pushes them apart.This promotes representation sharing across semantically related samples.
- 1 INTRODUCTION: Adding cross-instance captioning sensibly improves already strong video representation results and significantly surpasses the authors’ state-of-the-art baseline.The improvement is reported for the combined learning approach.
2 RELATED WORKS
Prior work broadly distinguishes generative and discriminative self-supervised objectives, with contrastive learning especially successful for image and video representations. This paper situates its framework within multimodal representation learning and large-scale pretraining.
- 2 RELATED WORKS: Self-supervised learning obtains pretext-task signals from unlabelled data, using generative or discriminative objectives.Examples include visual colorization, rotation, clustering, masked language modeling, and next-word prediction.
- 2 RELATED WORKS: Discriminative methods learn representations by differentiating input samples, while generative methods attempt to reconstruct their input.Contrastive loss is a representative discriminative objective.
- 2 RELATED WORKS: Transformer architectures and pretraining on large textual datasets have produced transferable representations that can be fine-tuned across downstream tasks.Subsequent work extended these ideas to image and multimodal image-text representations.
- 2 RELATED WORKS: The proposed framework combines contrastive video-text alignment with generative reconstruction from support-set video embeddings selected by attention.The generative objective is designed to promote concept sharing between instances.
- 2 RELATED WORKS: Large-scale pretraining, including HowTo100M, has been used to improve video-text representations for retrieval, question-answering, and captioning.HowTo100M is described as a large-scale instructional video dataset.
3 METHOD
The method combines contrastive learning with generative objectives that reconstruct captions from weighted video representations, encouraging semantic sharing across instances.
- The framework learns video and text embeddings for paired corpus samples, with semantically similar instances intended to be nearby.
- Contrastive learning brings paired video-text embeddings together and separates mismatched pairs, treating every sample as its own class.
- Cross-instance captioning reconstructs each caption from a learned weighted combination of other videos in a support-set.
- Support-set weights form batch-wise attention computed from video embeddings with a temperature-scaled softmax.
- Identity, full-support, hybrid, and cross-captioning variants differ in which videos can contribute to caption reconstruction.
- The full training objective sums contrastive and cross-captioning losses, while retrieval uses the separately encoded video and text representations.
4 EXPERIMENTS
Experiments evaluate model components, learning objectives, support-set size, and retrieval performance across several video-text datasets. Cross-instance captioning improves retrieval, with attention analyses indicating that the bottleneck encourages matching semantically related videos rather than memorizing isolated video-caption pairs.
- Evaluation: The study evaluates text-to-video and video-to-text retrieval on MSR-VTT, ActivityNet, VATEX, and MSVD, with and without HowTo100M pre-training.
- Ablations: 2.9% and 7.0% gains in text-to-video R@1 and R@5 come from R(2+1)D-34 features over image-frame features, while concatenating both yields MdR 3.0%.
- Ablations: Over 4% higher R@1 results from learned multi-head temporal attention than mean- or max-pooling baselines.
- Ablations: The reconstruction bottleneck is sensitive to support-set size, with both smaller and very large sets degrading retrieval performance.
- State-of-the-art comparison: 45.9% and 82.4% recall at 1 and 5 on VATEX exceed recent state-of-the-art methods by 9%.
- State-of-the-art comparison: 4% higher recall at 1 than MMT is achieved on ActivityNet, while HowTo100M pre-training further increases performance across the board.
- Qualitative analysis: Attention concentrates on one or two related videos, such as another musical-performance clip, rather than distributing weight across the batch.
5 CONCLUSION
The conclusion argues that contrastive learning can separate semantically equivalent videos and captions, and proposes cross-instance captioning to encourage shared representations. The resulting model achieves state-of-the-art retrieval performance on three datasets, while broader generalization depends on learning suitable conditional generators.
- Contrastive learning may pull apart semantically equivalent videos and captions, potentially hindering downstream retrieval performance.
- Cross-instance captioning encourages videos sharing similar captions to move closer in representation space.
- The model achieves state-of-the-art text-to-video and video-to-text retrieval on three datasets.
- Generalization beyond text-to-video retrieval requires reasonable conditional generators for the relevant modality or data stream.
6 APPENDIX
The appendix documents the transformer pooling head, encoder and decoder components, training costs, and inference throughput. The model combines motion and appearance features with learned temporal aggregation for video-text retrieval.
- Architecture: The transformer pooling head contains a pre-encoder, multi-head self-attention, and feed-forward layer.
- Computational cost: HowTo100M pre-training takes around 160 GPU hours on eight NVIDIA V100 GPUs for 20 epochs.
- Inference: Inference encodes approximately 250–300 videos per second and 200–250 text queries per second.
6.2 EXPERIMENT DETAILS
Training details vary with dataset length and scale, while shared optimization settings include a max-margin loss, temperature, Adam optimization, gradient clipping, and dropout.
- Optimization: The max-margin loss uses α = 0.2 and the temperature is T = 0.1.
- Optimization: Adam optimization starts at learning rate 5 · 10^-5, with gradients greater than 0.2 clipped during training.
- Optimization: Dropout is 0.3 for all datasets except ActivityNet, where it is 0.0.
- Dataset-specific training: Batch size is 64 for MSR-VTT, ActivityNet, and VATEX, while epoch counts and truncation lengths are adjusted to dataset characteristics.
6.3 DATASET DETAILS
The experiments use diverse video-text datasets spanning large-scale instructional pretraining, multilingual retrieval, dense temporal captions, and video descriptions.
- Pretraining dataset: HowTo100M provides 1.2 million YouTube instructional videos and more than 100 million ASR-defined clips for pretraining.
- Evaluation datasets: MSR-VTT contains 10,000 videos with 20 descriptions each and supports retrieval evaluation on the 1k-A testing split.Captioning is evaluated on a separate standard split containing 2,990 videos.
- Evaluation datasets: VATEX contains 34,911 multilingual videos; experiments use 25,991 training videos and English descriptions from balanced validation and testing splits.Each video has 10 English and 10 Chinese descriptions, but only English annotations are used.
- Evaluation datasets: ActivityNet Dense Caption contains densely annotated temporal segments from 20K YouTube videos and uses 10K training and 5K val1 videos.Segment descriptions are concatenated into video-paragraphs for retrieval and captioning.
- Evaluation datasets: MSVD contains 80K English descriptions for 1,970 YouTube videos, with roughly 40 sentences per video and a 1,200/100/670 train-validation-test split.
6.4 VIDEO CAPTIONING EXPERIMENTS
Captioning is evaluated with standard text-generation metrics on MSR-VTT, VATEX, and ActivityNet, while zero-shot retrieval is also assessed across four datasets.
- Captioning evaluation: Captioning performance is measured using BLEU4, METEOR, ROUGE-L, and CIDEr on MSR-VTT, VATEX, and ActivityNet.
- Captioning results: The paper reports captioning performance on MSR-VTT, VATEX, and ActivityNet through dedicated benchmark tables.
- Zero-shot retrieval: Zero-shot retrieval is evaluated on MSR-VTT, VATEX, ActivityNet, and MSVD after pretraining on HowTo100M.Results are described as reasonable on MSR-VTT and MSVD but not strong on VATEX and ActivityNet because of a significant domain gap.
6.6 ACTION RECOGNITION
Action recognition uses fixed visual backbones with either average pooling or a transformer pooling head, and the study also reports repeated VATEX finetuning results.
- Feature extraction: Action recognition features concatenate ResNet-152 and R(2+1)D-34 outputs into a 2560-D per-second video representation.
- Pooling: Video-level representations use either average pooling at 2560-D or a proposed transformer pooling head at 1024-D.The transformer pooling head is pretrained on HowTo100M with a cross-captioning objective.
- Finetuning stability: Across three VATEX finetuning runs, the model shows low variance and consistently beats the state of the art.
- Evaluation: Action-recognition comparisons on HMDB-51 and UCF-101 report average Top-1 accuracy across all three folds against supervised and self-supervised methods.
6.8 ADDITIONAL QUALITATIVE RESULTS
Qualitative Text→Video retrieval examples on MSR-VTT, VATEX, and ActivityNet usually place the correct video among the top retrieved results.
- Retrieval examples: The qualitative retrieval figure shows top-3 Text→Video results and similarities for MSR-VTT, VATEX, and ActivityNet.For each text query, only one correct video is marked in green among the displayed results.
- Observed behavior: In most cases, the model successfully retrieves the correct videos marked in green for the text queries.