Source-linked AI summary
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, Josef Sivic
TL;DR
Manually captioned video-text datasets are expensive, subjective, and difficult to scale. The paper instead learns a joint embedding from automatically transcribed narrations in HowTo100M, achieving strong results on instructional data and transfer after fine-tuning to generic videos and movies.
Problem
Manually captioned video-text datasets are expensive, subjective, and difficult to scale despite the need for large paired datasets.
Method
The paper builds HowTo100M from narrated instructional videos and trains a joint text-video embedding using automatically transcribed clip narrations.
Results
The HowTo100M embedding outperforms prior models on instructional-video tasks and transfers to generic YouTube videos and movies after fine-tuning.
Takeaways & Limitations
Large-scale narrated instructional videos can provide useful supervision for video-language representations across instructional and non-instructional domains.
Takeaways & Limitations
The clip-caption pairs are weakly aligned and noisy, with only 51% of inspected pairs containing a visibly matching object or action.
Abstract
from arXiv · showhide
Learning text-video embeddings usually requires a dataset of video clips with manually provided captions. However, such datasets are expensive and time consuming to create and therefore difficult to obtain on a large scale. In this work, we propose instead to learn such embeddings from video data with readily available natural language annotations in the form of automatically transcribed narrations. The contributions of this work are three-fold. First, we introduce HowTo100M: a large-scale dataset of 136 million video clips sourced from 1.22M narrated instructional web videos depicting humans performing and describing over 23k different visual tasks. Our data collection procedure is fast, scalable and does not require any additional manual annotation. Second, we demonstrate that a text-video embedding trained on this data leads to state-of-the-art results for text-to-video retrieval and action localization on instructional video datasets such as YouCook2 or CrossTask. Finally, we show that this embedding transfers well to other domains: fine-tuning on generic Youtube videos (MSR-VTT dataset) and movies (LSMDC dataset) outperforms models trained on these datasets alone. Our dataset, code and models will be publicly available at: www.di.ens.fr/willow/research/howto100m/.
1. Introduction
Learning joint text-video representations is important for communicating about visual content, but existing manually captioned datasets are expensive, subjective, and too small to scale. HowTo100M instead uses narrated instructional videos as abundant paired supervision for broad video-language tasks.
- Contribution: The learned embedding transfers to instructional and non-instructional text-video datasets.
- Motivation: Joint text-video embeddings place semantically related text fragments and video clips near each other in a shared space.This representation supports tasks including retrieval, action localization, captioning, and question answering.
- Motivation: Millions of paired clips and captions may be needed for generic video-language representations because scenes and descriptions vary widely.
- Research gap: Manually annotated datasets contain only tens to hundreds of thousands of pairs and are expensive, difficult to scale, and subjective.
- Approach: HowTo100M collects 136 million narrated clips from 1.22 million instructional videos covering more than 23,000 tasks.Each clip is paired with an automatically transcribed narration.
2. Related work
Prior video-language work largely depends on costly manually captioned datasets, while instructional-video research often uses only limited predefined labels. This paper combines automatically transcribed narration with web-scale data to learn and release a broadly transferable video-language embedding.
- Vision, language and speech: Existing joint vision-language embeddings typically rely on medium-scale datasets whose video captions require substantial human annotation effort.
- Learning from instructional videos: Instructional-video datasets support learning task steps, reference resolution, action segmentation, and object-action states.
- Learning from instructional videos: Related instructional-video methods commonly extract only a small number of predefined labels from automatically generated transcriptions.
- Dataset scale: HowTo100M differs from earlier web-instructional datasets through unprecedented variety and size, spanning more than 23,000 tasks and 136 million clips.
- Contribution: The released embedding outperforms prior instructional-video methods without fine-tuning and generalizes to non-instructional videos with fine-tuning.
3. The HowTo100M dataset
HowTo100M is built from narrated instructional YouTube videos selected around physical tasks, with subtitle lines aligned to video intervals as weak clip-caption pairs. Its scale comes with noisy, incomplete, and sometimes duplicated data.
- Data collection: HowTo100M contains 1.22 million YouTube instructional videos spanning activities such as cooking, hand crafting, personal care, and gardening.Subtitles may be manually written, generated by speech recognition, or translated automatically.
- Data collection: The collection starts from WikiHow activities and filters for physical-action verbs, yielding 23,611 visual tasks.
- Data collection: Videos are retrieved with “how to” task queries and restricted to English-subtitled results.
- Data quality: Videos are deduplicated by YouTube ID, but repeated uploads or edited re-uploads may remain.
- Clip-caption pairing: Each subtitle line becomes a caption paired with the video interval in which it is uttered.Subtitle lines need not be complete sentences.
- Data quality: 51% of 400 inspected clip-caption pairs contained at least one captioned object or action visibly present in the clip.Narration can be unrelated, temporally misaligned, incomplete, unpunctuated, or grammatically incorrect.
4. Text-video joint embedding model
The model learns nonlinear mappings that embed video and caption features into a shared space, where cosine similarity reflects whether a caption describes a clip. It trains this embedding with a margin-based ranking loss and intra-video negative sampling.
- Shared embedding space: The model maps video and caption features into a common d-dimensional space using functions f and g.The learned embeddings are designed so matching video-caption pairs have high cosine similarity and mismatched pairs have low similarity.
- Shared embedding space: Nonlinear embedding functions combine linear transformations with context gating to model multiplicative feature interactions.The gating function modulates the linear-layer output with values between 0 and 1.
- Training objective: The model is trained with a bidirectional margin-based ranking loss over negative caption-clip pairs.The two loss terms separately sample a negative caption and a negative video clip, using margin δ = 0.1.
- Training objective: Half of the negative pairs come from the same original video, while the other half come from different YouTube videos.This intra-video strategy encourages attention to clip-relevant details rather than background features.
5. Experiments
Experiments evaluate the HowTo100M embedding for step localization and clip retrieval across instructional, generic YouTube, and movie-video domains. Results show benefits from large-scale narrated-video pretraining, intra-video negatives, and target-domain fine-tuning.
- Evaluation setup: The evaluation covers CrossTask step localization and clip retrieval on YouCook2, MSR-VTT, and LSMDC.These datasets span instructional, cooking, generic YouTube, and movie videos.
- Negative sampling: Intra-video negative caption-video pairs improve performance over negatives sampled from other YouTube videos.The improvement is particularly significant on the finer-grained YouCook2 and CrossTask datasets.
- Scale matters: Increasing HowTo100M training data steadily improves CrossTask localization and LSMDC, MSR-VTT, and YouCook2 retrieval without observed saturation.The tested subsets range from 15k to the full dataset of 1.22M videos.
- Comparison with state-of-the-art: HowTo100M pretraining significantly outperforms prior methods on CrossTask step localization, including the reported fully supervised upper bound on average.The improvement is consistent across tasks except Make Meringue; Add Oil to a Car rises from 6.4% to 30.7% recall.
- Comparison with state-of-the-art: The off-the-shelf HowTo100M model outperforms models trained directly on MSR-VTT and improves further after MSR-VTT fine-tuning.State-of-the-art performance is attained with only 20% of MSR-VTT samples.
- Cross-dataset fine-tuning evaluation: Fine-tuning HowTo100M on LSMDC improves over a model trained directly on LSMDC, despite the substantial domain gap between movies and instructional videos.Across YouCook2, MSR-VTT, and LSMDC, HowTo100M pretraining followed by target-dataset fine-tuning yields the best results among compared pretraining choices.
6. Conclusion
HowTo100M provides more than 130M clips from 1.2M narrated web videos, enabling joint text-video embedding training without manual annotation.
- HowTo100M contains more than 130M video clips extracted from 1.2M narrated web videos of people performing complex visual tasks.
- The data collection method is fast, scalable, and requires no manual annotation.
- The learned embedding outperforms models trained on smaller, carefully annotated video-description datasets.
Overview of Appendix
The appendices provide dataset details, ranking-loss implementation details, and an analysis of positive-pair sampling during training.
- Appendix A presents additional details about the HowTo100M dataset.
- Appendix B gives practical implementation details for the ranking loss.
- Appendix C analyzes the sampling strategy for positive pairs during training.
A. Additional details of the HowTo100M dataset
HowTo100M is organized around the WikiHow3 task hierarchy and includes transcribed narrations whose nouns and verbs are analyzed.
- HowTo100M spans 23,611 tasks organized using the WikiHow3 hierarchy.
- The dataset hierarchy includes twelve top-level categories and their subcategories, with underlying tasks and videos visualized in Figure 8.
- Transcribed narrations often describe video content and are analyzed for noun and verb frequencies.
B. Ranking loss implementation details
The ranking-loss implementation samples clips from multiple videos, uses all other batch pairs as negatives, and reweights intra- and inter-video contributions.
- The mini-batch contains b = kv clip-caption pairs sampled from v unique videos, with v = 32 and k = 64 in practice.
- Every sampled mini-batch pair serves as a negative anchor, so N(i) = B \ {i}.
- The negative-example proportions depend on the numbers of sampled videos and pairs per video, v and k.
- The method studies high- and low-scoring clip-caption pairs according to whether captions visually match their corresponding clips.
- The weighting function fixes the intra-video to inter-video negative-sampling ratio at p to 1 − p regardless of v and k.
C. Sampling strategy for positive pairs
The authors address noisy video-caption pairs in HowTo100M by comparing max-pool and random positive-pair sampling strategies. Positive-pair selection does not improve retrieval, suggesting the model tolerates substantial caption noise.
- Motivation: Narrations can fail to describe their associated clips, creating noisy positive video-caption pairs during training.The sampling strategy is motivated by caption-video mismatches in narrated instructional videos.
- Max-pool sampling: The max-pool strategy ranks each video’s N clip-caption pairs by similarity and retains the highest-scoring fraction determined by rate r.At r = 0.5, the method keeps the highest-scoring half of the pairs; r = 1.0 retains all pairs.
- Results: r = 1.0 produces the best video clip retrieval results, so selecting only high-scoring positive pairs does not improve performance.Lowering r also reduces the number of triplet losses computed per mini-batch.
- Results: Random sampling at r = 0.5 performs similarly to no max-pool sampling, indicating robustness to noisy positive pairs rather than sensitivity to the sampling rule.The authors attribute this robustness to the model’s shallow architecture and training on a large amount of data.