Source-linked AI summary
AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions
Chunhui Gu, Chen Sun, David A. Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, Cordelia Schmid, Jitendra Malik
TL;DR
Action-recognition datasets often lack dense, person-level spatiotemporal labels for fine-grained atomic actions. AVA addresses this with exhaustive annotations across diverse movie segments and a temporal localization baseline, achieving strong results on existing benchmarks but only 15.6% mAP on AVA. The dataset’s realistic complexity therefore provides a demanding benchmark for future video-understanding methods.
Problem
Existing datasets commonly use sparse annotations, composite actions, or short clips, limiting fine-grained spatiotemporal action recognition.
Method
AVA densely labels 80 atomic actions for people at 1 Hz across 15-minute movie segments, linking people over time and using an I3D-based tubelet localization model.
Results
The method achieves state-of-the-art performance on JHMDB and UCF101-24, while obtaining 15.6% mAP on AVA.
Takeaways & Limitations
AVA’s realistic scene and action complexity exposes the difficulty of action recognition and motivates new approaches for video understanding.
Takeaways & Limitations
The movie-based data does not reflect the true distribution of human activity because film depictions are biased.
Abstract
from arXiv · showhide
This paper introduces a video dataset of spatio-temporally localized Atomic Visual Actions (AVA). The AVA dataset densely annotates 80 atomic visual actions in 430 15-minute video clips, where actions are localized in space and time, resulting in 1.58M action labels with multiple labels per person occurring frequently. The key characteristics of our dataset are: (1) the definition of atomic visual actions, rather than composite actions; (2) precise spatio-temporal annotations with possibly multiple annotations for each person; (3) exhaustive annotation of these atomic actions over 15-minute video clips; (4) people temporally linked across consecutive segments; and (5) using movies to gather a varied set of action representations. This departs from existing datasets for spatio-temporal action recognition, which typically provide sparse annotations for composite actions in short video clips. We will release the dataset publicly. AVA, with its realistic scene and action complexity, exposes the intrinsic difficulty of action recognition. To benchmark this, we present a novel approach for action localization that builds upon the current state-of-the-art methods, and demonstrates better performance on JHMDB and UCF101-24 categories. While setting a new state of the art on existing datasets, the overall results on AVA are low at 15.6% mAP, underscoring the need for developing new approaches for video understanding.
1. Introduction
AVA introduces a person-centric dataset of atomic actions, densely annotated in space and time across diverse movie footage. Its fine-grained, exhaustive design targets realistic action complexity and motivates stronger localization methods.
- Dataset design: AVA labels each person with a pose action and possibly multiple object or person interactions at 1 Hz.Annotations use short temporal segments centered on keyframes, allowing movement cues to disambiguate actions.
- Dataset design: The vocabulary contains 80 atomic visual actions defined at fine temporal granularity rather than high-level behavioral episodes.Atomic categories have clearer visual signatures than coarse intentional or goal-directed descriptions.
- Dataset design: The dataset covers 430 movies through 15-minute intervals, with every person labeled and linked across consecutive keyframes.This produces short temporal sequences while preserving diverse activity representations.
- Dataset design: Exhaustive annotation preserves realistic long-tailed action frequencies instead of imposing an artificially balanced label distribution.Movie selection also avoids retrieving clips through explicit action keywords.
- Benchmarking: The proposed tubelet-localization approach achieves state-of-the-art performance on JHMDB and UCF101-24, but reaches only 15.6% mAP on AVA.The authors use I3D convolutions to classify tubelets with temporal information.
- Benchmarking: AVA was released publicly as a benchmark for spatiotemporal action recognition.Its realistic complexity is intended to expose difficulties hidden by simpler datasets.
2. Related work
Existing action datasets typically use short clips, sparse or temporal-only localization, or composite action labels. AVA instead combines dense spatiotemporal annotations with atomic, person-specific actions over long movie segments.
- Existing datasets: Traditional action-classification datasets use short, manually trimmed clips containing a single action.They are designed for fully supervised whole-clip forced-choice classification.
- Existing datasets: Temporal-localization datasets use untrimmed videos with multiple actions but generally lack spatial localization.AVA differs by densely annotating each subject over 15-minute clips.
- Existing datasets: Earlier spatiotemporal datasets annotate short videos with fewer clips and actions, usually using composite rather than atomic categories.AVA expands the scale, duration, and action granularity of this setting.
- Existing datasets: Still-image action datasets lack motion cues and cannot represent composite events as sequences of atomic actions.AVA provides video data for modeling such sequences.
3. Data collection
AVA is constructed through a multi-stage pipeline that selects diverse movie segments, localizes and links people, and densely verifies their atomic action labels. The process combines automated proposals with human annotation to support broad coverage and high recall.
- Pipeline: The annotation pipeline has five stages: vocabulary generation, movie and segment selection, bounding-box annotation, person linking, and action annotation.These stages cover both dataset sampling and spatiotemporal labeling.
- Vocabulary: The vocabulary emphasizes generality, atomicity, and exhaustivity, favoring generic actions with clear visual signatures.Action classes are typically independent of the interacted object.
- Movie selection: Movies are sampled without action-keyword filtering or forced label balancing, although film depictions do not reflect the true distribution of human activity.The selection targets an international and varied collection of films.
- Movie selection: Each selected movie contributes a 15-minute segment partitioned into 897 overlapping 3-second segments with a 1-second stride.The fixed interval increases the number and diversity of movies under the annotation budget.
- Action annotation: Annotators label each subject separately with one required pose action and optional object and person interactions.The interface supports up to seven action labels per person.
- Person localization: Bounding boxes combine high-precision Faster R-CNN proposals with manual additions to ensure full recall.This hybrid design reduces manual effort while retaining missed people.
- Person linking: Adjacent-frame boxes are matched using person embeddings and the Hungarian algorithm, then human-verified into 81,000 tracklets.Tracklets range from a few seconds to a few minutes.
- Action annotation: Action annotation uses proposal and verification stages, with three independent annotators and a two-annotator verification threshold for ground truth.The staged process is designed to improve recall for a large vocabulary.
4. Characteristics of the AVA dataset
AVA is characterized by diverse atomic actions, rich spatial and temporal variation, multiple labels per person, and meaningful action transitions across linked segments.
- AVA contains atomic actions whose appearance varies across contexts and requires fine-grained or temporal distinctions such as “clinking glass” versus “drinking” and “opening” versus “closing”.Within a class such as “open,” the object and surrounding context can vary substantially.
- The dataset’s action annotations roughly follow Zipf’s law, producing many examples of common actions and fewer examples of memorable actions.This distribution reflects exhaustive labeling rather than artificial class balancing.
- Multiple labels are common: every person box has one pose label, while 28% include a person-object interaction and 67% include a person-person interaction.The annotation design captures concurrent actions rather than mutually exclusive labels.
- Linking people across segments enables analysis of temporal action structure, including likely transitions for the same person and simultaneous action pairs across people.Examples include “look at phone” → “answer phone” and “ride” ↔ “drive”.
- AVA includes substantial bounding-box variation in size and aspect ratio, with many small boxes that are difficult to detect and a modal aspect ratio of 2:1.Variation reflects both camera zoom and pose.
5. Action Localization Model
The action localization model combines keyframe-based region proposals with temporal tube classification, using I3D to preserve multi-frame information for challenging action recognition.
- The model is motivated by the difficulty of clips containing multiple, small, or subtly different actions, where background scenes provide insufficient cues.These conditions distinguish AVA from simpler action-recognition settings.
- I3D is introduced to model temporal context that is lost when multiple frames are concatenated only at the first layer.The architecture replaces 2D convolutions with 3D convolutions.
- The implementation extracts 3D feature maps from I3D while using a 2D ResNet-50 keyframe model for region proposal generation.ROI pooling is extended to 3D for the spatio-temporal features.
- The approach uses Faster R-CNN to detect and regress region proposals on RGB keyframes, then classifies spatio-temporal tubes with two-stream I3D convolutions.This design combines spatial proposal generation with RGB and optical-flow temporal modeling.
- Because AVA labels are not mutually exclusive, the model uses a separate binary Sigmoid loss for each class and links frame detections into action tubes.Video-level scores are computed by averaging over the resulting tubes.
6. Experiments and Analysis
The model performs strongly on established benchmarks but poorly on AVA, where longer temporal context helps and classification remains the main challenge.
- The benchmark uses 60 classes with at least 25 validation and test instances, comprising 210,634 training, 57,371 validation, and 117,441 test examples.Unless otherwise stated, results are trained on the training set and evaluated on validation.
- The 3D two-stream model achieves state-of-the-art performance on UCF101 and JHMDB, outperforming established baselines on frame-mAP and video-mAP.Table 3 reports the comparison at IoU 0.5.
- 15.6% frame-mAP is obtained on the AVA validation set, with 12.3% video-mAP at 0.5 IoU and 17.9% at 0.2 IoU.The AVA test set reaches 14.7% frame-mAP.
- Ablation study: AVA benefits from more than 10 input frames and larger temporal windows, with longer context helping more than on JHMDB and UCF101.Combining RGB and optical flow also improves performance over a single modality.
- Ablation study: More training data generally improves performance, but scene-correlated or low-diversity categories can outperform better-sampled categories.“Swimming” and “fall down” perform well with fewer examples, whereas “touching” and “smoking” remain difficult.
- Ablation study: Nearly 60% separates localization from end-to-end detection performance on AVA, compared with less than 15% on JHMDB and UCF101.The reported gap indicates that AVA’s main difficulty lies in action classification rather than localization.
7. Conclusion
AVA provides fine-grained spatiotemporal atomic-action annotations over diverse movie segments and a baseline method, but its substantially lower performance than on standard benchmarks highlights the dataset’s difficulty and motivates new approaches.
- AVA provides spatiotemporal annotations of atomic actions at 1 Hz over diverse 15-minute movie segments.
- The proposed method outperforms the current state of the art on standard benchmarks and serves as a baseline.
- Performance is significantly lower on AVA than on UCF101 or JHMDB, underscoring the need for new action-recognition approaches.
- Future work includes modeling more complex activities from AVA’s atomic actions.
Appendix
The appendix presents additional quantitative information and examples for the AVA dataset and its action-detection approach.
- The appendix provides quantitative information and examples for the AVA dataset and the proposed action-detection approach.
8. Additional details on the annotation
The annotation process combines detected person boxes with manual additions and uses a two-stage proposal-and-verification design to preserve action-class recall.
- Annotators receive frames overlaid with detected person boxes and manually add boxes for people missed by the detector.
- The hybrid bounding-box process trades off annotation accuracy against annotation cost.
- More than half of action-class labels derive their majority labels from the verification stage.
9. Additional details on the dataset
The AVA dataset exhibits realistic long-tailed class imbalance, uses sufficiently represented classes for benchmarking, and includes rare classes and consecutive atomic-action examples.
- The AVA trainval dataset has significant class imbalance consistent with Zipf’s law.
- Figure 11 reports that verification-stage labels improve recall for more than half of the action classes.
- The benchmark selects classes without asterisks to ensure sufficient numbers of test examples.
- Rare classes provide an opportunity to learn from few training examples.
- AVA includes examples of common consecutive atomic actions.
10. Examples of our action detection
The paper illustrates AVA action detections with temporally changing atomic actions, top true positives and false alarms, and label-frequency statistics.
- The best Faster-RCNN with I3D model produces the top true-positive and false-alarm detections shown in Figures 13 and 14.
- Pose: Pose labels are dominated by standing, sitting, and walking instances in the AVA trainval dataset.
- Person-Object Interaction: Person-object interaction labels include actions such as shooting, hitting an object, taking a photo, cutting, and turning a screwdriver.
- Figure 12 shows pairs of atomic actions changing over time for people marked by red bounding boxes.
- Figures 13 and 14 distinguish true positives in green from false alarms in red.