Source-linked AI summary
Large-scale weakly-supervised pre-training for video action recognition
Deepti Ghadiyaram, Matt Feiszli, Du Tran, Xueting Yan, Heng Wang, Dhruv Mahajan
TL;DR
Fully supervised video datasets are small and expensive to construct, motivating evidence about large-scale web-video pre-training for action recognition. The paper uses over 65 million hashtag-labeled public videos and studies label spaces, temporal features, and clip selection. Despite noisy labels and temporal localization, the resulting models significantly improve action-recognition results across Kinetics, EPIC-Kitchens, and Something-Something.
Problem
Fully supervised video datasets contain only a few hundred thousand videos and fewer than a thousand domain-specific labels, limiting evidence about scalable video pre-training.
Method
The paper pre-trains video models on over 65 million public user-generated videos using associated hashtags as weak labels, while varying label spaces and temporal dataset properties.
Results
Large-scale pre-training significantly improves action-recognition results on Kinetics, EPIC-Kitchens, and Something-Something despite label-space and temporal-localization noise.
Takeaways & Limitations
Large-scale noisy weak supervision can produce strong video feature representations and improve transfer across multiple action-recognition datasets.
Takeaways & Limitations
The datasets and exact hashtags cannot be made public, so other research groups cannot reproduce the results at this time.
Abstract
from arXiv · showhide
Current fully-supervised video datasets consist of only a few hundred thousand videos and fewer than a thousand domain-specific labels. This hinders the progress towards advanced video architectures. This paper presents an in-depth study of using large volumes of web videos for pre-training video models for the task of action recognition. Our primary empirical finding is that pre-training at a very large scale (over 65 million videos), despite on noisy social-media videos and hashtags, substantially improves the state-of-the-art on three challenging public action recognition datasets. Further, we examine three questions in the construction of weakly-supervised video action datasets. First, given that actions involve interactions with objects, how should one construct a verb-object pre-training label space to benefit transfer learning the most? Second, frame-based models perform quite well on action recognition; is pre-training for good image features sufficient or is pre-training for spatio-temporal features valuable for optimal transfer learning? Finally, actions are generally less well-localized in long videos vs. short videos; since action labels are provided at a video level, how should one choose video clips for best performance, given some fixed budget of number or minutes of videos?
1. Introduction
The paper studies whether large-scale weakly supervised web-video pre-training can improve action recognition despite label and temporal noise. It examines label-space design, video-versus-image pre-training, and the choice of short or long clips under fixed data budgets.
- Challenges: Video action recognition faces temporal noise because action labels often cover only a small portion of real-world videos.This differs from web images, which are typically selected for a relevant moment.
- Label space: The study compares noun, verb, verb-object, and target-overlapping label spaces because valid verb-object actions can be rare or unobserved.It asks how label count and diversity affect transfer learning.
- Temporal pre-training: The paper investigates whether video-specific spatio-temporal pre-training is necessary beyond strong image features and how clip length affects pre-training.Longer videos contain more frames, while shorter videos may contain more relevant frames.
- Large-scale weak supervision: Over 65 million public user-generated videos provide weak supervision through noisy social-media hashtags.The training data is intended to scale beyond expensive fully supervised annotation.
- Large-scale weak supervision: Large-scale weak supervision improves learned video features despite label and temporal noise.The reported benefit holds across models with varied capacities.
2. Related Work
The related work frames weak supervision as a practical response to the difficulty of obtaining exhaustive image and video annotations. This paper studies its transfer-learning benefits when both labels and temporal localization are noisy.
- Learning from weak supervision: Weak supervision commonly uses object labels and other metadata because exhaustive annotations for image and video tasks are difficult to gather.Prior work has used Internet content and associated text such as queries, hashtags, or comments as supervisory signals.
3. Weak-supervision of video models
The paper constructs weakly supervised video datasets by associating public hashtagged videos with seed action labels, then varies label sources and dataset properties for pre-training studies. It also documents dataset availability and implementation choices.
- Dataset construction: Public videos are collected through hashtags generated from seed action labels and assigned the corresponding seed labels.For “catching a fish,” the pipeline generates original, stemmed, and permuted hashtag phrases before downloading matching videos.
- Seed label sets: The source datasets use Kinetics actions, ImageNet nouns, Kinetics and VerbNet verbs, and verb-noun concatenations as seed label sets.The primary IG-Kinetics dataset uses 400 Kinetics action labels and contains 359 labels with sufficient videos.
- Reproducibility: The source datasets and exact hashtags are not publicly available, preventing other research groups from reproducing the results at this time.This is an author-acknowledged limitation of the dataset construction.
- Evaluation data: The target experiments include Kinetics, EPIC-Kitchens, and other action-recognition benchmarks, while aggressive deduplication removes potential source-target overlaps.Approximately 29K videos were removed from IG-Kinetics during deduplication.
- Models and objective: Pre-training uses R(2+1)D-d models with depths 18, 34, 101, and 152, and converts multi-label hashtag data into a randomly assigned multiclass objective.The resulting training objective uses softmax activations with cross-entropy loss.
4. Experiments
Experiments evaluate large-scale weakly supervised video pre-training across data scale, model capacity, label-space design, and label diversity. Results show strong transfer from noisy web videos, benefits from tail-preserving sampling, and task-dependent effects of label overlap and label-space size.
- Amount of pre-training data: Performance improves log-linearly with pre-training data size, indicating that more web videos yield better transferable feature representations despite label and temporal noise.The experiments use IG-Kinetics subsets from 500K to 65M videos.
- Amount of pre-training data: Tail-preserving sampling consistently outperforms random sampling, with performance saturating around 10M–19M videos.The strategy retains all tail-class videos while subsampling head classes.
- Effect of model capacity: Increasing model capacity improves Kinetics transfer by 3.9% after pre-training, but performance saturates from R(2+1)D-101 to R(2+1)D-152.For EPIC-Kitchens, a performance drop is observed at the largest capacity, with limited long-range reasoning beyond 32 frames noted as a possible factor.
- Nature of pre-training labels: Target tasks benefit most from source label sets whose verbs or nouns overlap with the target labels, while Sports-1M performs poorly across target tasks.IG-Noun benefits EPIC-Kitchens noun prediction, and IG-Verb improves verb prediction by at least 1.2% in both cases.
- Number of pre-training labels: A larger combined verb-noun label space does not necessarily improve transfer because IG-Verb+Noun is highly skewed and sparse.For Kinetics, fc-only accuracy improves by about 9% before saturating at 2700 labels, whereas full-finetuning improves by about 1% until 1350 labels.
4.3. Exploring the temporal dimension of video
The paper studies how temporal noise, video duration, and pre-training modality affect transfer learning. Short videos improve localization, while longer videos can provide greater diversity; direct clip-based pre-training substantially outperforms image-based alternatives.
- Effect of temporal noise: Short-5M significantly outperforms long-center-5M, indicating that short videos provide better temporal localization.Both datasets contain similarly long per-video content, but long-center-5M is drawn from longer videos and has greater temporal noise.
- Effect of temporal noise: Long-5M outperforms short-5M by 3.2%, suggesting that greater content diversity can offset temporal noise when the video-count budget is fixed.Thus, longer videos may benefit transfer learning when the total number of videos is fixed.
- Effect of temporal noise: Short-5M significantly outperforms long-500K under a fixed video-duration budget, favoring more short videos over fewer long videos.Long-center-5M and long-500K perform similarly, while short-5M also exceeds long-center-5M, consistent with better localization in short videos.
- Frame- vs. clip-based pre-training: 71.7% accuracy from direct R3D clip pre-training exceeds the best inflated model by 4.2%, showing the value of modeling temporal structure.Pre-training on 250M weakly supervised images produced no further improvement, whereas direct video-clip pre-training achieved the largest gain.
4.4. Comparisons with state-of-the-art
The paper compares its large-scale video pre-training approach with prior state-of-the-art methods across challenging action-recognition benchmarks. It reports substantial gains on Kinetics and Something-V1, while emphasizing that the compared systems differ in architectures, data, inputs, and evaluation strategies.
- Comparison scope: The comparisons span models and datasets that vary in architecture, pre-training data, input modalities, clip and frame sizes, and evaluation strategy.The paper notes that many prior state-of-the-art systems also use more complex, optimized architectures.
- Kinetics: 3.6% is the reported margin by which the approach outperforms existing methods on Kinetics with full fine-tuning.Using salient-clip sampling further yields 82.8% top-1 accuracy.
- Something-V1: 51.6% top-1 accuracy on Something-V1 improves over the state-of-the-art by 2.1% using only RGB input.Compared with other RGB-only approaches, the boost is 3.4%.
5. Discussion
Large-scale weakly supervised pre-training learns strong video representations despite noisy labels and imperfect temporal localization, improving action recognition across three public datasets. The study also identifies unresolved trade-offs in label sparsity and temporal coverage.
- 5. Discussion: Pre-training on tens of millions of noisy videos significantly improves action recognition on Kinetics, EPIC-Kitchens, and Something-Something.The models also show significant gains over Kinetics and Sports-1M pre-training.
- 5. Discussion: Sampling from the joint verb-noun distribution performs relatively poorly, likely because its plausible combinations create a large, sparse label space.The discussion links this skew to low-shot learning challenges and suggests data augmentation, non-parametric methods, and label-granularity optimization.
- 5. Discussion: A reported result uses RGB+flow and an ensemble of models, so it reflects a combined system rather than a single-model RGB configuration.
- 5. Discussion: State-of-the-art comparisons are reported separately for EPIC-Kitchens and Something-Something, with IG-Kin. denoting IG-Kinetics.The table captions also define SE as short-edge scaling.
- 5. Discussion: Temporal localization presents competing benefits: short videos offer better localization, whereas long videos provide greater diversity.The authors identify rigorous analysis of this trade-off as an open direction for improved data construction.
A.1 Dataset construction
The IG-Verb-Noun dataset constructs action labels by combining canonicalized verbs and nouns and expands hashtag matching across canonicalized and non-canonicalized forms.
- A.1 Dataset construction: IG-Verb-Noun uses the combined canonicalized forms of verbs and nouns as class labels.
- A.1 Dataset construction: Relevant hashtags include all possible canonicalized and non-canonicalized forms of the verb and noun.
- A.1 Dataset construction: For burning and candle, the hashtag set includes eight order and inflection variants.Examples include burncandle, candleburning, burningcandles, and candleburn.
A.2 Video deduplication
The deduplication pipeline detects source videos overlapping target datasets through frame-level visual features and locality-sensitive hashing, including scaled and partially overlapping content.
- A.2 Video deduplication: Videos are decoded at 16 fps, resized to 112 × 112, and represented by 64-dimensional Census Transform features per frame.
- A.2 Video deduplication: Locality-sensitive hashing measures feature-match overlap relative to source-video length between source and target videos.
- A.2 Video deduplication: The frame-level Census Transform is robust to scale and can detect target content appearing within only a small portion of a source video.
- A.2 Video deduplication: About 20K IG-Kinetics-65M videos were identified as potentially overlapping Kinetics, EPIC-Kitchens, or Something-Something-v1.Manual inspection found that some high-match source videos were true duplicates.
A.3 Model architectural details
The architectural and ablation settings specify R(2+1)D variants, model capacity, label-space comparisons, and data-sampling strategies for transfer-learning experiments.
- A.3 Model architectural details: R(2+1)D-101 and R(2+1)D-152 use bottleneck blocks with 1 × 1 × 1 convolutions before and after factorized convolutions.This variation from shallower models is used for computational feasibility.
- A.3 Model architectural details: The data-amount ablation uses Kinetics transfer learning with R(2+1)D-34, 8-frame inputs, batch size 16, and random or tail-preserving sampling.
- A.3 Model architectural details: The model-capacity ablation uses IG-Kinetics-65M pre-training with R(2+1)D, 32-frame inputs, and batch size 6.
- A.3 Model architectural details: The label-count ablations use IG-Verb+Noun or IG-Kinetics source labels for Kinetics transfer learning with R(2+1)D-34 and 8-frame inputs.Both settings use batch size 16 per GPU.
A.4 Hyper parameters
This section reports the full-finetuning hyperparameter conventions and organizes ablation and state-of-the-art comparison settings across three target datasets.
- Full-finetuning training length is defined as the number of complete passes through the target dataset.For Kinetics, learning rates follow linear scaling with gradual warmup; EPIC-Kitchens and Something-something-v1 use no warmup.
- Kinetics experiments include settings for frame-versus-clip pre-training ablations and state-of-the-art comparisons.The state-of-the-art comparison uses 32-frame input clips and a batch size of 6 per GPU.
- EPIC-Kitchens experiments vary pre-training-data amount and model capacity, alongside state-of-the-art comparisons.The data-amount ablation uses R(2+1)D-34 with 8-frame clips and batch size 16, while the capacity and comparison settings use 32-frame clips and batch size 6.
- Something-something-v1 has a dedicated state-of-the-art comparison configuration using 32-frame input clips and batch size 6 per GPU.