Source-linked AI summary
The AVA-Kinetics Localized Human Actions Video Dataset
Ang Li, Meghana Thotakuri, David A. Ross, João Carreira, Alexander Vostrikov, Andrew Zisserman
TL;DR
The paper addresses the limited diversity of densely annotated AVA videos by creating AVA-Kinetics, which applies AVA-style annotations to Kinetics clips. It describes the dataset construction and evaluates a baseline model, finding improved AVA validation performance with additional Kinetics training data. The resulting dataset combines detailed person-level labels with broader visual diversity for video research.
Problem
AVA has detailed multi-label annotations but restricted visual diversity, whereas Kinetics has broad diversity but only one label per clip.
Method
The authors apply AVA-style bounding-box and atomic-action annotation to selected Kinetics-700 clips, using detected-person key-frames and human verification.
Results
+5.26 mAP is obtained on the original AVA validation set when adding Kinetics training data with ground-truth boxes, while detected-box evaluation improves by +3.96 mAP.
Takeaways & Limitations
AVA-Kinetics provides a richer training set and a test set intended to better reflect model generalization, with applications including multi-task and transfer learning.
Abstract
from arXiv · showhide
This paper describes the AVA-Kinetics localized human actions video dataset. The dataset is collected by annotating videos from the Kinetics-700 dataset using the AVA annotation protocol, and extending the original AVA dataset with these new AVA annotated Kinetics clips. The dataset contains over 230k clips annotated with the 80 AVA action classes for each of the humans in key-frames. We describe the annotation process and provide statistics about the new dataset. We also include a baseline evaluation using the Video Action Transformer Network on the AVA-Kinetics dataset, demonstrating improved performance for action classification on the AVA test set. The dataset can be downloaded from https://research.google.com/ava/
1. The AVA-Kinetics Dataset
AVA-Kinetics combines AVA’s detailed person-level action annotations with Kinetics’ broad video diversity by applying AVA-style annotation to Kinetics clips. The section describes the annotation workflow and training-set sampling strategy.
- AVA-Kinetics combines AVA-style human action and localization annotations with many Kinetics-700 videos.
- AVA provides person-level multi-label annotations and bounding boxes, while Kinetics provides diverse clips with one dominant action label.
- Data annotation process: Each Kinetics clip receives one annotated key-frame selected using person-detection confidence, with missing boxes manually added when necessary.
- Data annotation process: At least three raters propose actions from a two-second clip centered on the key-frame, retaining labels verified by a majority.
- Data annotation process: Training annotation prioritizes 27 poorly performing AVA classes and matches them to 115 relevant Kinetics classes, while validation and test clips are fully annotated.
2. Data Statistics
AVA-Kinetics substantially expands the available training data while preserving long-tailed class distributions, increasing video diversity, and revealing relationships between Kinetics and AVA labels. Its person-level statistics remain broadly similar to AVA, though Kinetics contains smaller person boxes.
- Sample distribution: Kinetics adds substantial samples to most AVA classes, with especially large gains for “listen to,” while both datasets remain long-tailed.One sample is a bounding box paired with one action label.
- Video diversity: Kinetics provides far more unique videos per class: AVA reaches at most 235, whereas top Kinetics classes exceed 10,000.Half of Kinetics classes have over 300 unique video clips; the comparison uses a log-scale plot.
- Class correlation: NPMI ranks associations between Kinetics and AVA classes, with higher values indicating stronger correlations.The analysis computes NPMI from the co-occurrence of both label types on Kinetics videos.
- Class correlation: Examples show that “dancing gangnam style” relates to “dance,” “listen,” and “watch,” while “stir” relates to cooking activities such as scrambling eggs and making tea.The cooking activities require stirring in practice, supporting the observed reversed correlations.
- Person distribution: Person-count distributions are roughly similar, averaging 1.5 boxes per AVA frame and around 1.2 per Kinetics frame.Most key-frames contain one detected person, while frames with more than five are rare.
- Person distribution: Most person boxes are small relative to the image, and Kinetics videos tend to contain smaller boxes than AVA videos.Box areas are normalized relative to a 1x1 square image.
3. Benchmarking Results
The Video Action Transformer is evaluated on ground-truth and detected person boxes across AVA, Kinetics, and combined AVA-Kinetics splits. Adding Kinetics data improves AVA validation performance and benefits nearly all action classes, with the strongest overall gains on combined training data.
- Overall performance with groundtruth boxes: +5.26 mAP is obtained on the original AVA validation set when adding Kinetics training data.
- Overall performance with groundtruth boxes: Training on the full AVA-Kinetics set produces the best performance on the full AVA-Kinetics validation set.
- Overall performance with detected boxes: +3.96 mAP is achieved on AVA validation when using automatically detected boxes at test time with AVA-Kinetics training.
- Per-class performance: Person-pose classes perform relatively better, person-object interactions are hardest, and “watch,” “cut,” “hand shake,” “jump/leap,” and “swim” improve notably.
- Performance improvement vs. data increase: Only the “enter” class declines with added Kinetics data, by 0.76% mAP, while all other classes improve.
4. Conclusions
The paper presents AVA-Kinetics as a crossover dataset combining AVA’s detailed person-level labels with Kinetics’ broad visual diversity. It positions the dataset as a resource for studying multi-task and transfer learning in video.
- AVA-Kinetics combines AVA’s detailed multi-label annotations with Kinetics’ much larger visual diversity.
- The dataset annotates one frame from each Kinetics video with AVA boxes and labels, providing a richer training set and a test set intended to better reflect model generalization.
- The dataset can support research on multi-task learning using AVA and Kinetics labels and on transfer learning between Kinetics and AVA.
A. Fully annotated Kinetics classes
The paper fully annotates all training examples from 115 Kinetics classes selected for their relevance to difficult AVA action classes. These classes were chosen to expand coverage of challenging actions.
- 115 Kinetics classes related to the hardest AVA classes were hand-selected for complete training-set annotation.
- The selection targets AVA classes with poor prior recognition performance, including actions such as swimming, pushing, watching TV, throwing, climbing, and listening.
- The selected classes include multiple fine-grained variants of swimming, throwing, climbing, listening, singing, and other human activities.