Source-linked AI summary
The Kinetics Human Action Video Dataset
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, Andrew Zisserman
TL;DR
Human-action classification lacked a large, varied dataset suitable for training deep networks and benchmarking architectures. The paper introduces Kinetics, evaluates standard video models, and finds that performance is much lower than on UCF-101 and on par with HMDB-51, while preliminary analyses find little evidence of classifier bias across sensitive axes.
Problem
Human-action classification lacked a large, varied dataset capable of supporting deep-network training and meaningful architecture benchmarking.
Method
The paper constructs the Kinetics dataset from distinct YouTube videos, creates roughly 10-second candidate clips, and benchmarks three standard video-classification architectures.
Results
Standard models perform much lower than on UCF-101 and on par with HMDB-51, while preliminary analyses find little evidence of classifier bias across sensitive axes.
Takeaways & Limitations
Kinetics provides a substantially larger and more varied benchmark that permits training large models such as 3D ConvNets from scratch.
Takeaways & Limitations
Because clips are not exhaustively annotated when multiple actions occur, top-5 evaluation is more suitable than top-1 classification accuracy.
Abstract
from arXiv · showhide
We describe the DeepMind Kinetics human action video dataset. The dataset contains 400 human action classes, with at least 400 video clips for each action. Each clip lasts around 10s and is taken from a different YouTube video. The actions are human focussed and cover a broad range of classes including human-object interactions such as playing instruments, as well as human-human interactions such as shaking hands. We describe the statistics of the dataset, how it was collected, and give some baseline performance figures for neural network architectures trained and tested for human action classification on this dataset. We also carry out a preliminary analysis of whether imbalance in the dataset leads to bias in the classifiers.
1. Introduction
The introduction presents Kinetics as a large, diverse, high-quality dataset created to address the limitations of existing human action benchmarks and support deep-learning research in video classification. It uses short, action-containing YouTube clips for classification and aims to enable comparisons among new neural-network architectures.
- Research applications: Kinetics is intended to facilitate research by supporting deep-network training from scratch and serving as a challenging benchmark for comparing architectures.The authors hope it will enable architectures using multiple information streams and attention, while exposing tradeoffs such as static versus motion prediction.
- Dataset scope: The dataset targets large-scale, high-quality human action classification across a diverse range of actions rather than temporal localization.It includes short clips of around 10s containing the action, with no untrimmed videos.
- Motivation and problem: Kinetics was introduced because existing human action datasets were too small and insufficiently varied for current deep-learning classification models.HMDB-51 and UCF-101 had become standard benchmarks, but their usefulness was expiring because they could not adequately support training and testing modern models.
- Dataset diversity: Each clip comes from a different YouTube video, increasing variation in performers, action execution, clothing, pose, age, framing, and viewpoint.This contrasts with datasets such as UCF-101, where multiple clips can come from the same video and performer.
2. An Overview of the Kinetics Dataset
Kinetics is a large-scale dataset of human-focused actions spanning person, person-person, and person-object interactions, including fine-grained actions requiring temporal reasoning. It contains 400 classes and 306,245 YouTube clips of around 10 seconds, with non-exhaustive action annotation.
- Action classes: The dataset covers person, person-person, and person-object actions, including fine-grained classes that require temporal reasoning to distinguish.Examples include drawing, shaking hands, washing dishes, and different types of swimming.
- Action classes: Classes are organized into several non-exclusive parent-child groupings, including Music, Personal Hygiene, Dancing, and Cooking.The full class list and groupings are provided in the appendix.
- Statistics: 400 classes contain 400–1150 clips each, totaling 306,245 unique-video clips of around 10s, split into training, validation, and testing partitions.Training has 250–1000 videos per class, validation has 50, and testing has 100; clips have variable resolution and frame rate.
- Annotation: A clip may contain several actions but is usually assigned to only one class, so the dataset does not provide exhaustive annotation.Examples include texting while driving, Hula hooping while playing ukulele, and brushing teeth while dancing.
3. How the Dataset was Built
Kinetics was built by matching YouTube video titles to a curated action list, temporally locating candidate actions, and extracting approximately 10-second clips. Human labeling through Amazon Mechanical Turk then verified the action content before further dataset cleanup.
- Dataset cleanup: A clip was accepted after at least three of five confirmations, while deduplication, overlap checks, and denoising completed the cleanup pipeline.Deduplication ensured that only one clip came from each video and that clips did not share common video material.
- Action-list curation: The action list combined classes from existing action datasets, motion-capture sources, and observations because no single listing offered suitable visual actions at this scale.The cited passage identifies ActivityNet, HMDB, UCF101, MPII Human Pose, and ACT among the contributing datasets.
- Candidate retrieval: Videos were obtained from YouTube by matching video titles with the Kinetics action list.
- Temporal positioning: Image classifiers identified likely action positions within candidate videos, after which clips were created by taking 5 seconds on either side of each position.This produced 10-second clips, except when the position was within 5 seconds of a video boundary, resulting in shorter clips.
- Human verification: Amazon Mechanical Turk workers judged whether each clip showed the named human action using a dedicated labeling interface.The task was chosen because of the platform’s large pool of high-quality workers and asked whether a human performed the action.
4. Benchmark Performance
The benchmark compares ConvNet+LSTM, Two-Stream, and 3D ConvNet baselines to indicate Kinetics’ difficulty relative to UCF-101 and HMDB-51. Results show substantially lower performance on Kinetics than UCF-101, while class errors often involve fine-grained actions and appearance distinctions.
- Benchmark design: Three baselines—ConvNet+LSTM, Two-Stream, and 3D ConvNet—are compared on Kinetics, UCF-101, and HMDB-51 to indicate dataset difficulty.The study is not intended as a thorough search for the best Kinetics architecture.
- Benchmark results: Kinetics performance is far lower than UCF-101 performance, indicating that the two datasets have different difficulty levels.HMDB-51 performance is worse than Kinetics, despite having little training data and a test set designed to challenge appearance-centered methods.
- Benchmark results: The 3D ConvNet performs poorly across datasets, especially UCF-101 and HMDB-51, but approaches the other models more closely on Kinetics because of its larger training set.Unlike the other baselines, the parameter-rich 3D ConvNet is not pre-trained on ImageNet.
- Class difficulty: Eating, dancing, and body-part-centered classes are among the hardest Kinetics categories under the Two-Stream model.Examples include distinguishing foods such as hotdogs, chips, and doughnuts, as well as “massaging feet” and “shaking head”.
- Class confusion: The top class confusions mostly involve fine-grained distinctions, including long jump versus triple jump and burger versus doughnuts.Confusion between swing dancing and salsa dancing also raises questions about the Two-Stream model’s motion modeling.
5. Conclusion
The Kinetics dataset is an order of magnitude larger than prior datasets of its type, supports training large models from scratch, and yields lower standard-model performance than UCF-101 but performance on par with HMDB-51. Preliminary analysis found little evidence of classifier bias across sensitive axes such as gender, while recognizing that thorough analysis remains future work.
- Dataset contribution: Kinetics contains an order of magnitude more videos than previous datasets of its type and permits large models such as 3D ConvNets to be trained from scratch.The authors also describe data-collection and quality-assurance procedures.
- Baseline performance: Standard existing models perform much lower on Kinetics than on UCF-101, with performance on par with HMDB-51.These comparisons summarize baseline performance on the dataset.
- Bias analysis: Preliminary imbalance analysis found little evidence that classifiers exhibit bias along sensitive axes such as gender.The authors characterize this as a complex area requiring further attention and leave thorough analysis for future work with social scientists and critical humanists.
- Release: Trained baseline models will be released in TensorFlow for uses including generating features for new action classes.The release is intended to support reuse of the baselines beyond the reported experiments.
A. List of Kinetics Human Action Classes
This section lists the human action classes in the dataset, with each class followed by its number of video clips. The classes span everyday activities, object interactions, sports, dancing, and musical performance.
- Class list: Each listed action is followed by its clip count in parentheses.The section explicitly defines the bracketed number as the number of clips for that action class.
- Class list: Examples include answering questions (478), arranging flowers (583), assembling computer (542), and cleaning gutters (598).These entries illustrate everyday activities and human-object interactions represented among the classes.
- Class list: The list covers sports and physical activities including bench pressing (1106), canoeing or kayaking (1146), rock climbing (1144), and skiing crosscountry (477).The examples show substantial variation in clip counts across physical-action classes.
- Class list: Musical-performance classes include playing guitar (1135), playing cello (1081), playing saxophone (916), and playing trumpet (989).The listed instruments span string, wind, and brass performance categories.
B. List of Parent-Child Groupings
The parent-child groupings provide a nonexclusive, noncomprehensive guide to related human action classes. They span everyday activities, sports and movement, interpersonal interactions, creative pursuits, and technology use.
- Creative and domestic activities: 12 arts-and-crafts classes include arranging flowers, blowing glass, brush painting, carving pumpkin, and making jewelry.The grouping also includes drawing, getting a tattoo, knitting, spray painting, and weaving basket.
- Sports, movement, and performance: Sports and movement are organized into groupings such as 25 ball sports, 20 land-mobility actions, 10 water-mobility actions, and 29 music-related actions.Examples include dribbling basketball, riding a bike, swimming backstroke, playing guitar, and singing.
- Creative and domestic activities: 22 cooking classes and 17 eating-and-drinking classes cover food preparation, consumption, bartending, and tasting.Examples include baking cookies, cooking chicken, making pizza, drinking beer, eating burger, and tasting food.
- Technology and object use: Five electronics classes capture technology use through assembling computer, playing controller, texting, using computer, and using remote controller (not gaming).The grouping distinguishes these actions from broader activities such as playing games and paper-related tasks.