Source-linked AI summary
YouTube-8M: A Large-Scale Video Classification Benchmark
Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, Sudheendra Vijayanarasimhan
TL;DR
Large-scale video classification lacked a broad benchmark comparable to image datasets, especially one covering diverse visual entities beyond actions. YouTube-8M constructs such a dataset with machine-generated Knowledge Graph labels, curated visual entities, and downloadable pre-computed frame features. The resulting dataset supports competitive models and transfer learning, including a new state-of-the-art result on ActivityNet.
Problem
Existing video benchmarks are restricted mainly to action and sports classes, unlike diverse image benchmarks such as ImageNet.
Method
YouTube-8M builds a multi-label video dataset from YouTube topic annotations, filters entities for visual recognizability, extracts compressed frame features, and provides labels and features for download.
Results
Features learned on YouTube-8M generalize to Sports-1M and ActivityNet, including a new state-of-the-art result on ActivityNet.
Takeaways & Limitations
YouTube-8M provides a large-scale resource for video classification and representation learning with broad topical coverage and pre-computed features.
Takeaways & Limitations
The automatically generated labels have 78.8% precision but only 14.5% recall against human raters, leaving many labels missing.
Abstract
from arXiv · showhide
Many recent advancements in Computer Vision are attributed to large datasets. Open-source software packages for Machine Learning and inexpensive commodity hardware have reduced the barrier of entry for exploring novel approaches at scale. It is possible to train models over millions of examples within a few days. Although large-scale datasets exist for image understanding, such as ImageNet, there are no comparable size video classification datasets. In this paper, we introduce YouTube-8M, the largest multi-label video classification dataset, composed of ~8 million videos (500K hours of video), annotated with a vocabulary of 4800 visual entities. To get the videos and their labels, we used a YouTube video annotation system, which labels videos with their main topics. While the labels are machine-generated, they have high-precision and are derived from a variety of human-based signals including metadata and query click signals. We filtered the video labels (Knowledge Graph entities) using both automated and manual curation strategies, including asking human raters if the labels are visually recognizable. Then, we decoded each video at one-frame-per-second, and used a Deep CNN pre-trained on ImageNet to extract the hidden representation immediately prior to the classification layer. Finally, we compressed the frame features and make both the features and video-level labels available for download. We trained various (modest) classification models on the dataset, evaluated them using popular evaluation metrics, and report them as baselines. Despite the size of the dataset, some of our models train to convergence in less than a day on a single machine using TensorFlow. We plan to release code for training a TensorFlow model and for computing metrics.
1. INTRODUCTION
YouTube-8M addresses the lack of broad, large-scale video benchmarks by providing a general multi-label dataset with diverse visually recognizable entities, pre-computed features, and baseline experiments. Its scale also supports transfer learning to other video benchmarks.
- Existing video benchmarks largely focus on action and sports classes, unlike ImageNet’s diverse and general object/entity coverage.
- YouTube-8M introduces general multi-label video classification using labels relevant to a video’s frames rather than restricting the task to actions.
- The dataset spans activities, objects, scenes, and events, with entities selected using YouTube popularity and human ratings of visualness.
- The benchmark supplies compressed frame features extracted at one frame per second with an Inception network, enabling downloadable representation-learning experiments.
- More than 8 million videos provide over 500,000 hours of video across 4,800 classes, establishing the benchmark’s scale and diversity.
2. RELATED WORK
YouTube-8M is positioned within the progression from small image and video datasets to much larger benchmarks. It expands video classification beyond action-focused datasets through broader labels, substantially greater scale, and pre-computed features.
- Image understanding progressed from small labeled datasets toward large benchmarks such as ImageNet and SUN.
- Video benchmarks progressed from small clips to datasets such as Sports-1M, YFCC-100M, and FCVID, but remained limited in scale or scope.
- YouTube-8M provides a large-scale benchmark focused on the main themes of videos rather than only action categories.
- Its 4,800 Knowledge Graph entities represent a substantial increase in annotation diversity compared with fewer than 500 categories across other datasets.
- The dataset contains over 8 million labeled videos and more than 500,000 hours of video.
- Pre-computed state-of-the-art features are available for 1.9 billion video frames.
3. YOUTUBE-8M DATASET
YouTube-8M is a large-scale, multi-label video benchmark built from visually curated Knowledge Graph entities and compressed frame-level features. Its 8,264,650 videos and 4,800 classes support broad video understanding, while noisy and incomplete annotations create a missing-label challenge.
- Dataset scope: YouTube-8M identifies key topical themes in videos using Knowledge Graph entities rather than restricting classification to actions or sports.The dataset targets multi-label video classification, including activities, objects, scenes, and events.
- Feature processing: Frame features were extracted from Inception representations at one frame per second, truncated to six minutes, then compressed with PCA, whitening, and quantization.The compression reduces data size by a factor of 8, while training on the uncompressed data increases evaluation metrics by less than 1%.
- Dataset scale: 8,264,650 videos span 4,800 classes, with an average of 1.8 classes per video and train, validation, and test partitions.The dataset retains entities with at least 200 videos and publishes features for all splits but labels only for training and validation.
- Dataset statistics: The dataset organizes entities into 24 top-level categories and exhibits a video-count distribution that somewhat follows a natural Zipf distribution.The category mapping is released for completeness but is not used during training.
- Human-rated evaluation: Automatically generated annotations achieve 78.8% precision and 14.5% recall against human raters, making missing labels a central evaluation consideration.Models are evaluated primarily on the noisy validation partition and also on a smaller human-rated set.
4.1 Models from Frame Features
Frame-level models assign video labels to sampled frames and aggregate frame predictions for video-level classification. The section compares average-pooling models, DBoF, LSTMs, and transfer-learning strategies.
- Frame-Level Models and Average Pooling: Video-level labels are assigned to frames because no information specifies label localization or prominence within videos.The approach samples 20 random frames per video, producing about 120 million frame instances.
- Frame-Level Models and Average Pooling: Frame-level probabilities are averaged across test-video frames to produce one score per entity.The method trains 4800 independent one-vs-all classifiers and aggregates their frame scores during inference.
- Deep Bag-of-Frames: DBoF projects each selected frame through shared fully connected parameters, then max-pools sparse codes into a fixed-length video representation.Additional hidden and classification layers produce video-level predictions.
- Deep Bag-of-Frames: DBoF is trained end-to-end with SGD, using logistic loss for logistic outputs and cross-entropy loss for softmax outputs.Backpropagated gradients train the projection layer as a discriminative representation of the input frame features.
- LSTMs: LSTM models are limited to the precomputed frame features, because the authors do not have access to raw video frames.The best reported configuration uses two 1024-unit layers, with a 60-second gradient horizon during training.
- Transfer Learning: Transfer to ActivityNet uses the concatenated final-frame LSTM outputs without fine-tuning the LSTM layers.The authors describe this as more robust to overfitting than traditional fine-tuning for the smaller ActivityNet dataset.
4.2 Video level representations
The paper constructs task-independent fixed-length video representations by aggregating frame features, enabling standard classifiers, compact storage, and potential domain adaptation. It compares simple statistics with Fisher Vector and VLAD alternatives.
- Motivation: Fixed-length video features allow standard classifiers such as logistic regression, SVM, and mixture of experts to operate across videos.Their fixed dimensionality removes the need to handle variable-length video representations.
- Motivation: Video-level representations reduce training data size by a few orders of magnitude through compact whole-video encoding.The representation is extracted from frame-level features before classifier training.
- Motivation: Unsupervised video-level representations are less specialized to current labels and may generalize better to new tasks or video domains.The paper identifies this as a benefit for domain adaptation.
- Alternative Representations: Fisher Vector and VLAD representations of similar dimensionality did not achieve competitive results, so the paper leaves stronger compact variants for future work.The limitation concerns the compared alternative representations rather than fixed-length video representations generally.
- First, second order and ordinal statistics: The representation combines per-dimension means, standard deviations, and top-5 ordinal statistics from the frame features.For each of 1024 feature dimensions, the method retains first- and second-order information plus the five highest values.
- Normalization: Global normalization subtracts the feature mean, then applies PCA decorrelation and whitening before classifier training.The resulting features are approximately multivariate Gaussian with zero mean and identity covariance, supporting more balanced SGD updates.
4.3 Models from Video Features
Models trained on video-level representations use independent online classifiers, including logistic regression, online SVMs, and mixture-of-experts models. The section emphasizes scalable optimization and controlled mixture-size trade-offs.
- Scalable Classifiers: Independent binary classifiers are trained for each label using online learning because batch optimization is infeasible at the dataset scale.The authors use Adagrad for model updates on compact video-level features.
- Logistic Regression: Logistic regression models each entity probability as a sigmoid applied to the entity-specific weight vector and video feature.The classifier parameters minimize total training log-loss.
- Hinge Loss: Online SVMs use ±1 labels and hinge loss for binary entity classification.The hinge-loss max function creates a discontinuous first derivative, so subgradient updates can slow convergence.
- Mixture of Experts: Mixture-of-experts models combine a softmax over experts with sigmoid entity-existence probabilities, including a dummy non-existence state.This structure models both expert selection and the probability that an entity is present.
- Mixture of Experts: Mixture-of-experts training distributes independent label classifiers across multiple machines using Adagrad with batch size 32.The reported learning rate is 1.0.
- Mixture of Experts: Performance increases by 0.5%-1% on all metrics when mixtures grow from 1 to 2 and then 4, while parameters increase twofold or fourfold.The authors select 2 mixtures as a compromise between performance and model size.
5. EXPERIMENTS
The experiments compare frame-level, pooled-feature, mixture-of-experts, and recurrent models on YouTube-8M, then test whether learned representations transfer to Sports-1M and ActivityNet. Video-level aggregation and transfer learning outperform naive frame-level prediction, with gains varying by metric and dataset.
- Frame-level models with Inception features and averaged predictions perform poorly, showing that video-level prediction cannot be reduced to frame-level classification.
- Mean pooling of frame features followed by hinge loss or logistic regression improves video-level accuracy over naive averaging of frame-level predictions.
- DBoF with logistic classification yields 2% absolute gains in Hit@1 and PERR over mean feature pooling with single-layer logistic regression.
- LSTM generally performs best except on mAP, where one-vs-all binary MoE classifiers perform better, likely because rare-class convergence affects mAP.
- On Sports-1M, YouTube-8M features are competitive, and pre-training improves LSTM performance by approximately 1% on all metrics.
- On ActivityNet, transferred features outperform training on ActivityNet alone across all metrics, with the best feature up to 80% better than the cited non-learning and CNN-feature baseline.
6. CONCLUSIONS
YouTube-8M is presented as a large-scale benchmark and resource for video representation learning, combining a broad visual vocabulary with computationally accessible features. Experiments show strong baseline performance, reduced need for complex temporal modeling when frame features are strong, and transfer to existing benchmarks.
- Dataset and infrastructure: YouTube-8M addresses large-scale video understanding by providing labeled videos and preprocessed frame-level features that can support training on a single machine within one day.The dataset processes over 50 years of video and nearly 2 billion frames from more than 8 million videos.
- Modeling and evaluation: Mixture-of-experts models outperform logistic regression and SVMs and remain competitive with LSTM and DBoF approaches on the dataset.The experiments model entity probabilities from video-level representations aggregated from frame-level features.
- Modeling and evaluation: When frame-level features are strong, simpler video-level modeling can reduce the need for more sophisticated approaches that directly use frame-level information.This conclusion follows from comparisons involving mixture-of-experts, LSTM, and DBoF models.
- Transfer learning: Features learned on YouTube-8M generalize to Sports-1M and ActivityNet, including a new state-of-the-art result on ActivityNet.The transfer-learning experiments are used to illustrate the dataset's usefulness on existing video benchmarks.