Source-linked AI summary
A Short Note about Kinetics-600
Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, Andrew Zisserman
TL;DR
The paper addresses the challenge of scaling Kinetics beyond its initial 400 classes and 400 clips per class. It introduces Kinetics-600 through revised class sourcing and candidate matching, including multilingual queries, and reports baseline and challenge results. The winning challenge error improved from 12.4% in 2017 to 11.0% in 2018.
Problem
Scaling Kinetics toward ImageNet-like size was limited by finding relevant candidate videos and classes with many candidates.
Method
The paper constructs Kinetics-600 by revising class sourcing and candidate matching within the existing collection pipeline, using multiple English and Portuguese queries and weighted ngram matching.
Results
11.0% challenge error in 2018 improved from 12.4% in 2017, using the criterion averaging Top-1 and Top-5 error.
Takeaways & Limitations
Kinetics-600 is 50% larger than Kinetics-400 and advances the project toward 1000 classes with 1000 video clips per class.
Takeaways & Limitations
Some Kinetics-400 clips were reassigned into Kinetics-600 test and validation splits, making evaluation of Kinetics-400-pretrained models non-ideal.
Abstract
from arXiv · showhide
We describe an extension of the DeepMind Kinetics human action dataset from 400 classes, each with at least 400 video clips, to 600 classes, each with at least 600 video clips. In order to scale up the dataset we changed the data collection process so it uses multiple queries per class, with some of them in a language other than english -- portuguese. This paper details the changes between the two versions of the dataset and includes a comprehensive set of statistics of the new version as well as baseline results using the I3D neural network architecture. The paper is a companion to the release of the ground truth labels for the public test set.
1. Introduction
Kinetics-600 scales the Kinetics action dataset toward ImageNet-like size while preserving its core clip-collection principles. It expands coverage and provides public test labels alongside a held-out evaluation set.
- The project addresses earlier bottlenecks in finding relevant candidate videos and classes with many candidates.Kinetics-400 relied on simple query variations, which often exhausted candidate pools or produced low yields.
- Kinetics-600 expands the dataset from 400 to 600 classes and from around 300k to around 500k video clips.These represent increases of 50% in classes and 60% in clips.
- Kinetics-600 retains 10-second YouTube clips, variable resolution and frame rate, and distinct source videos within each action class.
- The release includes a standard test set with public labels and a held-out test set whose labels are not released.The paper recommends the standard test set for ordinary reporting and the combined sets for Activity-Net challenge comparisons.
- The dataset’s video URLs and temporal intervals are available from the Kinetics website.
2. Data Collection Process
Kinetics-600 keeps the original collection pipeline but changes class sourcing and candidate-video matching to scale candidate pools. Multiple multilingual queries and weighted ngram matching support broader and more robust retrieval.
- Data Collection Process: The collection pipeline comprises class sourcing, candidate matching, clip selection, human verification, and quality filtering.Mechanical Turk workers verify sampled 10-second clips, followed by deduplication and selection of higher-quality classes and clips.
- Data Collection Process: The main process differences between Kinetics-400 and Kinetics-600 concern class sourcing and candidate YouTube-video matching.
- Class sourcing: Kinetics-600 sourced many classes from Google’s Knowledge Graph and YouTube search autocomplete, supplementing earlier dataset- and author-based sourcing.
- Candidate video matching: Kinetics-400’s class-name matching often exhausted candidate pools or yielded too few candidates, while metadata descriptions proved of little use because of spam.
- Candidate video matching: Multiple manually created English and Portuguese queries produced larger candidate pools and also promoted greater cultural, ethnic, and geographic diversity.Examples include “folding paper,” “origami,” and “dobrar papel.”
- Candidate video matching: Weighted ngram representations combine video metadata and related-video titles with standard title matching to produce multilingual similarity scores.Unlike earlier binary matching, this approach did not run out of candidates, although selected-candidate yield was lower for smaller similarities.
3. From Kinetics-400 to Kinetics-600
Kinetics-600 is an approximate superset of Kinetics-400, retaining most classes while adding, renaming, splitting, or removing others. Evaluation has a limited overlap between dataset splits that makes transfer evaluation non-ideal.
- 368 of the original 400 classes are exactly the same in Kinetics-600, with more examples.
- The remaining classes include renamed categories and classes split or removed because of strong overlap with other categories.Examples include “dying hair” becoming “dyeing hair.”
- “Hugging” became “hugging baby” and “hugging (not baby),” while “opening bottle” was divided into wine and non-wine categories.
- Clips from 30 Kinetics-400 validation classes entered the Kinetics-600 test set, and some training clips entered its validation set.Consequently, evaluating Kinetics-600 models pretrained on Kinetics-400 is not ideal, although the paper expects little practical difference.
4. Benchmark Performance
The paper benchmarks an RGB-only I3D model trained from scratch on Kinetics-600. Performance is reported across standard and held-out evaluations, with challenge winners improving over the earlier Kinetics challenge.
- Baseline setup: The baseline is an I3D model using standard RGB videos without optical flow, trained from scratch on Kinetics-600.Training used SGD with momentum, while hyperparameters were selected on validation data.
- Reported metrics: Table 2 reports validation accuracy, test accuracy, combined Test+Held-out accuracy, and the Activity-Net challenge metric.The table uses RGB inputs without test-time augmentation.
- Baseline results: 71.7 top-1 accuracy was achieved on the test set, compared with 69.7 on Test+Held-out.The paper states that the held-out test set is harder than the regular test set.
- Baseline results: 68.4 top-1 accuracy was the corresponding Kinetics-400 result, while the paper reports several possible factors behind the difference.These include more training examples, removal of some ambiguities, and a smaller batch size caused by using fewer GPUs.
- Challenge results: 11.0% was the winning-system challenge error in 2018, down from 12.4% in 2017.The challenge criterion averages Top-1 and Top-5 error.
5. Conclusion
Kinetics-600 is 50% larger than Kinetics-400 and advances the project toward a 1000-class, 1000-clip-per-class action-classification dataset. The release also explains dataset-collection changes and provides test annotations for both versions.
- Kinetics-600 is 50% larger than the original Kinetics-400 dataset.
- The dataset represents another step toward producing an action-classification dataset with 1000 classes and 1000 video clips for each class.
- The paper explains differences in data collection between the 2017 dataset version and Kinetics-600.
- Test-set annotations for Kinetics-400 and Kinetics-600 were released, eliminating the need to submit results to an external evaluation server.
A. List of New Human Action Classes in Kinetics-600
The appendix lists human-action classes that were new to Kinetics-600 or renamed from Kinetics-400. Examples span activities involving objects, exercise, sports, movement, and everyday actions.
- Other examples cover making paper aeroplanes, roasting marshmallows, throwing water balloons, and walking through snow.
- It includes exercise and sports actions such as mountain climber, playing field hockey, swimming front crawl, and swinging a baseball bat.
- The list includes object- and household-related actions such as opening refrigerators, using power drills, and changing wheels.
- The listed activities also include social and communication actions such as hand-clapping games and talking on a cell phone.