Source-linked AI summary
A Short Note on the Kinetics-700 Human Action Dataset
Joao Carreira, Eric Noland, Chloe Hillier, Andrew Zisserman
TL;DR
Kinetics-700 extends the Kinetics human-action dataset to support training and evaluating neural architectures on diverse video actions. The paper describes its expanded, curated dataset and reports baseline and challenge performance, with Kinetics-700 appearing harder than earlier versions.
Problem
The project aims to provide a large-scale, curated dataset covering diverse human actions for training and exploring neural network architectures for video understanding.
Method
The dataset is built through class sourcing, candidate YouTube matching, 10-second clip selection, human verification, and quality curation, with standard validation and held-out test sets.
Results
Kinetics-700 increases the dataset from 600 to 700 classes and from around 500k to around 650k clips, while its test accuracy is 57.3 versus 68.4 on Kinetics-400 and 71.7 on Kinetics-600.
Takeaways & Limitations
Kinetics-700 provides a larger benchmark toward the goal of an action-classification dataset with 1000 classes, while results indicate a more difficult task than earlier releases.
Takeaways & Limitations
The collection process is not well suited to action classes that progress over time, and the newly collected test set may introduce some train–test distribution shift.
Abstract
from arXiv · showhide
We describe an extension of the DeepMind Kinetics human action dataset from 600 classes to 700 classes, where for each class there are at least 600 video clips from different YouTube videos. This paper details the changes introduced for this new release of the dataset, and includes a comprehensive set of statistics as well as baseline results using the I3D neural network architecture.
1. Introduction
Kinetics-700 extends the curated Kinetics action dataset to 700 classes while retaining its YouTube-based, independently sourced clip design and adding separate validation and held-out test sets.
- Kinetics-700 increases the dataset from 600 to 700 classes, retaining all but three Kinetics-600 classes.
- 600 or more clips per action class increase the dataset by 30%, from around 500k to around 650k clips.
- Clips last 10 seconds, have variable resolution and frame rate, and come from different YouTube videos within each action class.
- The release provides a labeled standard validation set and a held-out test set whose labels are not released.
2. Data Collection Process
Kinetics-700 preserves the five-stage Kinetics collection pipeline while expanding class sourcing, multilingual candidate matching, clip selection, human verification, and curation.
- Data Collection Process: The collection pipeline comprises action class sourcing, candidate video matching, candidate clip selection, human verification, and quality analysis and filtering.Curation includes clip de-duplication and selecting higher-quality classes and clips.
- Data Collection Process: About 3% of existing Kinetics-600 videos are deleted or unlisted each year, requiring classes to be topped up alongside new-class collection.
- 2.1. Action class sourcing: Kinetics-700 adds fine-grained and imaginative classes, partly sourced from recent action datasets and replacing some general classes with subclasses.Examples include replacing “picking fruit” with “picking apples” and “picking blueberries.”
- 2.2. Candidate video matching: Query texts are separated from class names, translated into French, Portuguese, and Spanish, and used in all four languages to obtain candidate videos.The authors report adequate machine-translation quality, although ambiguity sometimes occurred.
- 2.2. Candidate video matching: Multilingual queries increased geographic representation: Latin American clips rose from 3% in Kinetics-400 to 8% in Kinetics-700.More than 50% of clips came from North America, while Africa increased from 0.8% to 1%.
- 2.2. Candidate video matching: Candidate videos are matched using multilingual weighted n-gram metadata representations combined with title matching, producing a larger candidate pool while lower similarities reduce human-verification yield.
- 2.4. Human verification: Human verification checks whether clips contain the target action, while a crowdsourced final annotation stage cleans class thumbnails in panels of 16 elements.The final stage addresses potentially confusing or multimodal classes.
- 2.4. Human verification: High-yield classes are generally included in the release, whereas only a small proportion of low-yield classes survives.
3. From Kinetics-600 to Kinetics-700
Kinetics-700 is an approximate superset of Kinetics-600, with nearly all classes retained but limited overlap between their test and earlier splits affecting safe evaluation directions.
- From Kinetics-600 to Kinetics-700: 597 of 600 Kinetics-600 classes are exactly the same in Kinetics-700, although clips may be replaced after source videos are deleted.
- From Kinetics-600 to Kinetics-700: One class was renamed, while “chopping vegetables” and “picking fruit” were split into multiple subclasses.
- From Kinetics-600 to Kinetics-700: Under 3% of Kinetics-700 test clips overlap with Kinetics-600 train, validation, test, or held-out test splits.
- From Kinetics-600 to Kinetics-700: Models trained on Kinetics-600 can largely be evaluated on the Kinetics-700 test set, but training on Kinetics-700 before evaluating Kinetics-600 is not safe.
4. Benchmark Performance
The I3D-RGB baseline achieved similar top-1 accuracy on Kinetics-700 validation and test sets, while challenge results show substantial room beyond the baseline.
- 58.7 top-1 accuracy was obtained on validation and 57.3 on test with the RGB I3D baseline.The authors report that the two sets are similarly hard.
- Figure 1 ranks the 20 easiest and 20 hardest Kinetics-700 classes by class accuracy from the I3D-RGB model.
- 29.3% average Top-1 and Top-5 error was obtained by the single RGB I3D baseline in the 2019 Kinetics challenge.
- 17.9% error was obtained by the top 2019 challenge team, compared with 29.3% for the RGB I3D baseline.
5. Conclusion
The paper presents Kinetics-700 as a substantially expanded action-classification dataset and another step toward a 1,000-class dataset.
- Kinetics-700 contains 30% more clips than Kinetics-600 and more than twice as many clips as Kinetics-400.
- The release represents another step toward producing an action-classification dataset with 1,000 classes.
A. List of New Human Action Classes in Kinetics-700
This appendix lists Kinetics-700 classes absent from Kinetics-600 or renamed for the new release.
- The appendix identifies classes that were not in Kinetics-600 or that were renamed.
- Examples of listed classes include listening with headphones, bouncing ball (not juggling), and swimming with dolphins.
- Additional examples include carving wood with a knife and pulling espresso shot.
- Other listed classes include pretending to be a statue, letting go of balloon, and being in zero gravity.
- The appendix also lists walking with crutches, putting wallpaper on wall, playing american football, and shooting off fireworks.
B. List of Low Yield Classes
Low-yield classes were ranked by the probability that candidate clips received positive votes from at least three human annotators, and most were excluded from the final dataset.
- Yield is the probability that a candidate clip was voted positive for a class by three or more human annotators.
- Most low-yield classes were not included in the final dataset, with bold indicating the classes that were included.
- 0.0033 was the yield for adding fish to aquarium, the lowest-yield class listed.
- 0.0034 was the yield for getting inside balloon, while dropping plates had a yield of 0.0057.
- 0.0106, 0.0112, and 0.0114 were the yields for grooming (person), boarding train, and buying fast food, respectively.
13. Piling coins up 0.0114
The results list ranked human-action classes with scores ranging from 0.0116 to 0.0212 across ranks 14–35. The entries cover activities including looking through a telescope, entering or exiting buildings, and playing underwater frisbee.
- Ranks 14–19 include looking through telescope 0.0116, breaking aquarium 0.0118, using a crowbar 0.0127, instant messaging 0.0133, and getting into a car 0.0134.
- Ranks 21–27 include getting out of a car 0.0153, entering building 0.0177, signing document 0.0179, cutting in line 0.0179, and waiting at crossing 0.0179.
- Ranks 31–33 include unloading the trunk of a car 0.0198, setting up fish tank 0.0198, and cutting squares 0.0201.
- Rank 35 is playing underwater frisbee 0.0212, the highest score among the listed entries.