Source-linked AI summary
End-to-End Learning of Visual Representations from Uncurated Instructional Videos
Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, Andrew Zisserman
TL;DR
Large-scale narrated videos could replace costly manual annotation for learning video representations, but their narration is frequently misaligned with the visual content. The paper introduces MIL-NCE to learn from multiple candidate narrations and reports strong representations learned from HowTo100M across four tasks and eight datasets, outperforming self-supervised and several fully supervised baselines.
Problem
Manual video annotation is costly and hard to scale, while narrated instructional videos provide weak supervision because their descriptions are often misaligned with the video.
Method
MIL-NCE uses multiple candidate positive video-narration pairs in a noise-contrastive objective to learn joint video and text representations from uncurated narrated videos.
Results
The learned representations outperform self-supervised and fully supervised representations across downstream video-understanding tasks.
Takeaways & Limitations
Strong video representations can be learned from scratch using uncurated narrated videos without manually annotated video or image data.
Abstract
from arXiv · showhide
Annotating videos is cumbersome, expensive and not scalable. Yet, many strong video models still rely on manually annotated data. With the recent introduction of the HowTo100M dataset, narrated videos now offer the possibility of learning video representations without manual supervision. In this work we propose a new learning approach, MIL-NCE, capable of addressing misalignments inherent to narrated videos. With this approach we are able to learn strong video representations from scratch, without the need for any manual annotation. We evaluate our representations on a wide range of four downstream tasks over eight datasets: action recognition (HMDB-51, UCF-101, Kinetics-700), text-to-video retrieval (YouCook2, MSR-VTT), action localization (YouTube-8M Segments, CrossTask) and action segmentation (COIN). Our method outperforms all published self-supervised approaches for these tasks as well as several fully supervised baselines.
1. Introduction
Manual video annotation is costly, difficult, and poorly scalable, motivating learning from narrated instructional videos despite their weak and noisy supervision. The paper introduces MIL-NCE to learn video representations from uncurated narration and evaluates them broadly against self-supervised and supervised baselines.
- Manual video annotation is cumbersome, expensive, and difficult because action vocabulary and temporal intervals are ambiguous.
- Narrated instructional videos provide large-scale supervision, but spoken descriptions are often temporally misaligned, incomplete, irrelevant, or affected by speech-recognition errors.
- The paper proposes MIL-NCE, a training loss designed to address visually misaligned narrations in uncurated instructional videos.
- The learned representations are evaluated across four video-understanding tasks and eight datasets, including recognition, retrieval, localization, and segmentation.
- The representations outperform fully supervised baselines on several tasks and improve over self-supervised approaches on HMDB-51 and UCF101 without fine-tuning.
- MIL-NCE enables end-to-end learning of video representations from scratch using unlabelled, uncurated narrated videos from HowTo100M.
2. Related work
Prior work learns visual-language representations from metadata, annotated descriptions, or narrated videos, while MIL-NCE addresses the challenge of exploiting noisy narration without manually annotated visual data. Its design uses multiple candidate positives rather than selecting only one.
- MIL-NCE considers multiple positive video-narration candidates instead of explicitly selecting a single positive sample per bag during training.
- Many visual-language methods rely on manually annotated image or video descriptions, or on representations pretrained with manually labelled datasets.
- Narrated-video approaches use ASR transcripts or raw speech to supervise tasks including detection, captioning, classification, summarization, and retrieval.
- MIL methods have been applied to weakly supervised video tasks such as action classification, localization, anomaly detection, and object tracking.
3. Leveraging Uncurated Instructional Videos
The section formulates learning joint video–text representations from short narrated clips and introduces MIL-NCE to handle temporal misalignment by aggregating candidate matches against sampled negatives.
- 3. Leveraging Uncurated Instructional Videos: The model receives 3.2-second video clips and narrations of at most 16 words, then learns a joint embedding space from raw video pixels and text.Video and text embeddings should be close for semantically similar pairs and distant otherwise.
- 3.1. A simple joint probabilistic model: The method uses a probabilistic joint embedding model in which video and narration mappings produce vectors in the same d-dimensional space, with pair probability proportional to an exponentiated dot product.The video mapping is a CNN over a fixed-length clip, while the text mapping transforms words into one vector.
- 3.2. Learning from uncurated data: MIL-NCE: Around 50% of HowTo100M clip–narration pairs are estimated to be misaligned because speech may describe events before or after they occur.This temporal misalignment makes representation learning harder than learning from manually aligned labels.
- 3.2. Learning from uncurated data: MIL-NCE: MIL-NCE addresses misalignment by treating nearby narrations as multiple positive candidates and seeking at least one matching narration for each video.The candidate narrations occur close in time within the same video, increasing the chance that spoken words correlate with the visual content.
- 3.2. Learning from uncurated data: MIL-NCE: Unlike MIL-SVM, this multiple-instance extension does not explicitly select a single positive sample during training and can symmetrically support multiple candidate videos for one narration.The training set therefore contains sets of candidate positive pairs rather than only single pairs.
- 3.2. Learning from uncurated data: MIL-NCE: Because maximum likelihood requires an intractable normalization constant over all video–narration pairs, training instead uses noise-contrastive estimation.NCE directly optimizes the unnormalized model.
- 3.2. Learning from uncurated data: MIL-NCE: MIL-NCE replaces a single positive pair score with the sum of scores over candidate positives and maximizes this against negatives for the corresponding sample.The objective is estimated using a specific negative set for each sample.
4. Experiments
The experiments ablate MIL-NCE design choices and evaluate representations across recognition, retrieval, localization, and segmentation benchmarks. The learned representations outperform self-supervised and several fully supervised alternatives across diverse tasks.
- Evaluation scope: The evaluation covers five downstream task groups across eight datasets, including action recognition, text-to-video retrieval, action step localization, and action segmentation.The datasets span HMDB-51, UCF-101, Kinetics-700, YouCook2, MSR-VTT, CrossTask, COIN, and YouTube-8M Segments.
- Ablation studies: Adding positive candidates significantly improves over single-instance learning, with the best results obtained using 3 to 5 candidates.Using too many candidates increases the chance of sampling irrelevant contextual narrations, so the paper fixes the number at five.
- Ablation studies: MIL-NCE outperforms Max+NCE and Attn+NCE on five of six ablation tasks.The comparison tests max-pooling and attention-based alternatives for handling multiple positive clip-caption candidates.
- Ablation studies: Jointly sampling video and narration negatives performs best overall, while the simple language model is more consistent across five tasks than the alternatives tested.The language-model comparison includes LSTM, GRU, Transformer, and NetVLAD variants; replacing word2vec with BERT also worsens performance.
- Comparison to the state-of-the-art: The learned video representation outperforms prior self-supervised methods on UCF-101, matches AVTS on HMDB-51, and generalizes without fine-tuning.It also outperforms fully supervised counterparts on COIN and YouTube-8M Segments, including CBT on COIN.
- Comparison to the state-of-the-art: The model improves Kinetics-700 recognition by 4% over random initialization and 1.4% over ImageNet-pretrained I3D, while outperforming prior work on YouCook2, MSR-VTT, and CrossTask without manual supervision.The retrieval evaluation is zero-shot with respect to YouCook2 and MSR-VTT annotated videos.
5. Conclusion
The paper learns visual representations from uncurated instructional videos without manually annotated video or image data. MIL-NCE addresses narration–video misalignment, producing representations that outperform self-supervised and fully supervised representations on downstream tasks.
- MIL-NCE enables learning visual representations from uncurated instructional videos without manually annotated video or image datasets.
- The approach applies a multiple instance learning strategy derived from noise contrastive estimation to narration–video misalignment.
- The resulting representations outperform self-supervised and fully supervised representations on downstream tasks.
- The authors suggest that MIL-NCE may apply to other multiple instance learning problems where representation learning is important.
Appendix overview
The appendix supplies technical details for ablation-study baselines and visualizes the model architecture used in the work.
- Section A provides technical details about the baselines introduced in Table 2 for the ablation studies.
- Section B visualizes the model architecture used in the work.
A. Max+NCE and Attn+NCE baselines
The appendix describes two NCE baselines for selecting positive video–narration pairs: max pooling and cross-modal soft attention. It also references the architecture and BERT-input results.
- Max+NCE: Max+NCE combines standard max-pool multiple instance learning with the NCE loss.
- Max+NCE: Max+NCE selects the best positive candidate pair among all pairs in P_i according to the model.
- Attn+NCE: Attn+NCE selects positive candidate pairs through a cross-modal soft-attention mechanism between clips and narrations.
- Attn+NCE: The attention functions f_a and g_a share parameters with the video and text embedding functions except for their final Linear layers.
- Attn+NCE: The Attn+NCE objective gives the model a separate selection mechanism for positive candidate pairs.
- Figure 4 illustrates the video model f and text model g, while Table 6 reports results using BERT vectors instead of Word2Vec.
B. Model architecture
Figure 4 illustrates the video embedding model f and text embedding model g used in the main paper.
- Figure 4 provides an illustration of the video model f and text model g used in the main paper.
C. Pretrained BERT
The ablation replaces Word2Vec with BERT base while keeping the rest of the text model unchanged. The BERT-based input processes 16 words into 16 vectors of dimension 768.
- BERT base replaces the Word2Vec module in the text model.
- The model processes a sequence of 16 input words into 16 output vectors.
- The BERT output vectors have dimension 768, compared with the original 300-dimensional word vectors.