Source-linked AI summary
Video Action Transformer Network
Rohit Girdhar, João Carreira, Carl Doersch, Andrew Zisserman
TL;DR
Video action recognition and localization require contextual reasoning about surrounding people and objects, which person boxes alone may not provide. The paper introduces Action Transformer, using person-specific RPN queries with a modified Transformer to aggregate spatiotemporal context. On AVA, it achieves 25.0% mAP versus 17.4% for prior work using only raw RGB frames, while the authors note that the problem remains far from solved.
Problem
Recognizing and localizing human actions is difficult because many actions require reasoning about surrounding people, objects, and context beyond the person’s bounding box.
Method
Action Transformer combines I3D base features and RPN person proposals with a modified Transformer head that uses person boxes as queries to aggregate contextual video features.
Results
25.0% mAP versus 17.4% mAP on AVA establishes a new record, using only raw RGB frames and outperforming prior work with optical flow and sound.
Takeaways & Limitations
The network learns meaningful spatiotemporal context, tracking people and attending to hands and faces during action classification without explicit supervision.
Takeaways & Limitations
At 25% mAP, the action-recognition problem and the AVA dataset remain far from solved.
Abstract
from arXiv · showhide
We introduce the Action Transformer model for recognizing and localizing human actions in video clips. We repurpose a Transformer-style architecture to aggregate features from the spatiotemporal context around the person whose actions we are trying to classify. We show that by using high-resolution, person-specific, class-agnostic queries, the model spontaneously learns to track individual people and to pick up on semantic context from the actions of others. Additionally its attention mechanism learns to emphasize hands and faces, which are often crucial to discriminate an action - all without explicit supervision other than boxes and class labels. We train and test our Action Transformer network on the Atomic Visual Actions (AVA) dataset, outperforming the state-of-the-art by a significant margin using only raw RGB frames as input.
1. Introduction
The paper targets video action localization and recognition when interpreting a person’s action requires contextual information from surrounding people, objects, and earlier frames. It introduces Action Transformer, which aggregates such context and achieves state-of-the-art AVA performance using raw RGB frames.
- Motivation: Human actions are difficult to recognize because they often require jointly reasoning about the person and surrounding people or objects.Examples include listening, pointing, holding, and shaking hands; some actions also depend on context from earlier frames.
- Method: Action Transformer repurposes a Transformer as a head that uses I3D features and RPN proposals to aggregate spatiotemporal context for classifying a person’s action.The I3D model provides base features, while the RPN supplies person-localizing queries for the Transformer head.
- Learned behavior: The network learns to track individual people, contextualize their actions through other people’s actions, and attend to hand and face regions without explicit supervision.These behaviors emerge during supervised action classification rather than from separate tracking, attention, or object-detection supervision.
- Results: 25.0% mAP improves on 17.4% mAP on AVA, while raw RGB input outperforms prior approaches using additional optical flow and sound.The paper reports that this result set a new AVA record and was the top-performing approach on the ActivityNet leaderboard at submission.
- Limitations: At 25% mAP, the AVA action-localization and recognition problem remains far from solved.The paper analyzes failure modes and notes that some classes with relatively large training sets remain difficult to recognize.
2. Related Work
Prior video-understanding work largely focused on trimmed clips, small fine-grained datasets, or temporal detection without person-action attribution. The paper situates Action Transformer among person-centric relation and attention methods as a human-region-query approach to contextual action recognition.
- Video Understanding: Traditional video activity recognition often classifies trimmed clips, offering a limited view when multiple people perform different actions simultaneously.The related-work discussion contrasts this focus with fine-grained video understanding involving multiple people and actions.
- Video Understanding: Fine-grained video understanding methods had largely been evaluated on small datasets such as UCF-24 and JHMDB.Temporal action detection addressed temporal localization but not person detection or person-action attribution.
- AVA dataset and methods: AVA introduced long clips labeled with all people and their actions, creating a testbed for person-centric action understanding.The passage identifies AVA as addressing limitations of earlier datasets and methods.
- AVA dataset and methods: Compared with person-centric relation networks, Action Transformer uses person detections as queries to select regions for aggregation and reports a large-margin improvement over prior work.Its attention design reduces one side of the comparison to human regions and can sit atop varied base architectures.
- Attention for action recognition: Video attention research has used gating, pooling, pose guidance, region graphs, recurrent models, and self-attention, whereas this model applies self-attention around human regions.The paper presents its approach as complementary to these existing attention mechanisms.
3. Action Transformer Network
The Action Transformer detects people in a keyframe and classifies their actions using a spatiotemporal video trunk, region proposals, and a Transformer-based head that aggregates contextual information. Its architecture combines action-agnostic localization with person-specific queries and contextual attention, alongside an alternative I3D head for comparison.
- Overall design: The model ingests a short keyframe-centered clip and outputs bounding boxes for people in the central frame, each labeled with predicted actions.The network is designed to detect and classify all people at a given time point.
- Base network architecture: A 3D convolutional trunk generates spatiotemporal features, while an RPN proposes person boxes that the head classifies and refines.The RPN and box regression are action agnostic; the head predicts C action classes or background and regresses four box offsets.
- Action Transformer head: The Transformer head uses each person proposal as a query and the surrounding video clip as memory, updating the query with contextual information from other people and objects.Attention aggregates projected key and value features from the clip before classification.
- Action Transformer head: HighRes query preprocessing preserves the person’s spatial layout by converting a 7 × 7 RoIPool feature into a 128D query, unlike spatial averaging.Spatial averaging would lose the person’s spatial layout.
- Alternative head: An alternative I3D head applies the remaining I3D blocks to spatiotemporal region features without extracting Transformer context.This simpler head is used in the ablation study to measure the importance of contextual aggregation.
- Implementation: The implementation uses 64-frame, 400 × 400px inputs, a 7 × 7 query feature, and a full 16 × 25 × 25 trunk feature as Action Transformer context.The I3D head instead receives a 16 × 7 × 7 input; I3D layers are initialized from Kinetics-400 pretraining and fine-tuned end-to-end.
4. Experiments
Experiments on AVA evaluate action classification, localization, overall detection and classification, design choices, and comparison with prior work. The Action Transformer improves overall performance, especially when paired with I3D regression, while localization remains weaker than the I3D head.
- Evaluation setup: The AVA evaluation separates action classification and localization, including tests with ground-truth person boxes and classification-agnostic localization.Performance is reported using frame-level mean average precision at an IoU threshold of 0.5, with 60 action classes evaluated in the standard subset.
- Classification: The Action Transformer head gives a significant 5% classification boost over the I3D head with ground-truth boxes.The comparison uses R = 64 proposals; the Action Transformer head also has 2.3M fewer parameters than the I3D head in the LowRes QPr case.
- Localization: 92.9% to 87.7%: HighRes query preprocessing substantially narrows the Action Transformer’s localization gap relative to the I3D head at IoU 0.5.The I3D head with RPN boxes achieves almost 93% mAP, whereas the low-resolution transformer reaches 77.5%.
- Overall performance: 24.9 versus 20.5: combining I3D for regression with Action Transformer classification produces the best reported overall performance.Using the Action Transformer head alone reaches 24.4; the hybrid adds only 0.1 GFlops of computational overhead.
- Ablations: Class-agnostic regression, data augmentation, Kinetics pre-training, and 300 RPN proposals are important design choices for strong performance.Removing the first three choices causes significant drops, while reducing proposals from 300 to 64 significantly harms the Action Transformer model.
- Comparison with prior work: >7.5% on validation and >3.5% on test: the model outperforms the previous state of the art and the CVPR 2018 challenge winner using raw RGB input.The best model uses a single model rather than an ensemble, and adding temporal context at test time further improves performance.
5. Analysis
The Action Transformer learns interpretable spatiotemporal representations and attention patterns that track people and focus on relevant body parts. Performance varies with class, person-box size, and scene density, while qualitative analyses reveal contextual successes and recurring failures.
- Learned embeddings and attention: Learned embeddings track people across clips, with separate heads showing semantic and instance-level representations.Attention maps also emphasize faces, hands, and other parts of the person of interest.
- Performance analysis: Performance generally improves with larger person boxes and decreases as more people appear in a scene.Larger boxes make the person’s action easier to see, while additional people increase scene complexity.
- Performance analysis: Some actions remain difficult despite substantial training data, including smoking and eating.The analysis identifies these as anomalous tail cases rather than simply data-scarce classes.
- Qualitative results: The model uses contextual information to recognize actions such as watching a person and holding an object.Other people and interacted-with objects can provide cues unavailable from the target actor alone.
- Failure modes: Common failure modes include similar actions or interactions, incorrect identity assignment, and temporal-position mismatches.These errors are illustrated in misclassified videos, including difficult smoking examples.
6. Conclusion
The Action Transformer learns spatiotemporal context from people and objects to localize and classify human actions. It achieves a significant margin over prior state of the art on AVA, while remaining imperfect and potentially benefiting from motion input.
- Conclusion: The network learns spatiotemporal context from other human actions and objects for human-action localization and classification.Its embeddings and attention maps acquire semantic meaning through supervised action training.
- Conclusion: The Action Transformer exceeds the state of the art on the AVA dataset by a significant margin.The conclusion contrasts this result with prior systems that used motion or flow streams in addition to RGB.
- Conclusion: The model’s performance is far from perfect, motivating further improvement and investigation.The paper identifies adding flow as a likely way to boost performance and proposes additional avenues for future work.