Source-linked AI summary
Actor-Context-Actor Relation Network for Spatio-Temporal Action Localization
Junting Pan, Siyu Chen, Mike Zheng Shou, Yu Liu, Jing Shao, Hongsheng Li
TL;DR
Spatio-temporal action localization must recognize persons’ actions while accounting for contextual and higher-order interactions that pairwise relations may miss. ACAR-Net explicitly models actor-context-actor relations using a High-order Relation Reasoning Operator and Actor-Context Feature Bank, achieving state-of-the-art results and first place in the AVA-Kinetics ActivityNet Challenge 2020 task.
Problem
Pairwise relation modeling may miss indirect higher-order relations among actors and context that provide crucial clues for action localization.
Method
ACAR-Net models actor-context-actor relations with a High-order Relation Reasoning Operator and an Actor-Context Feature Bank using actor and grid-based context features.
Results
ACAR-Net achieves state-of-the-art performance on AVA and UCF101-24 and ranks first in the AVA-Kinetics action localization task of ActivityNet Challenge 2020.
Takeaways & Limitations
Modeling higher-order actor-context relations supports action localization and helps identify contextual regions relevant to the action.
Abstract
from arXiv · showhide
Localizing persons and recognizing their actions from videos is a challenging task towards high-level video understanding. Recent advances have been achieved by modeling direct pairwise relations between entities. In this paper, we take one step further, not only model direct relations between pairs but also take into account indirect higher-order relations established upon multiple elements. We propose to explicitly model the Actor-Context-Actor Relation, which is the relation between two actors based on their interactions with the context. To this end, we design an Actor-Context-Actor Relation Network (ACAR-Net) which builds upon a novel High-order Relation Reasoning Operator and an Actor-Context Feature Bank to enable indirect relation reasoning for spatio-temporal action localization. Experiments on AVA and UCF101-24 datasets show the advantages of modeling actor-context-actor relations, and visualization of attention maps further verifies that our model is capable of finding relevant higher-order relations to support action detection. Notably, our method ranks first in the AVA-Kineticsaction localization task of ActivityNet Challenge 2020, out-performing other entries by a significant margin (+6.71mAP). Training code and models will be available at https://github.com/Siyu-C/ACAR-Net.
1. Introduction
Spatio-temporal action localization requires recognizing actors through interactions with surrounding context, but pairwise relations can miss higher-order clues. ACAR-Net addresses this by explicitly reasoning over actor-context-actor relations and reports strong benchmark performance.
- Spatio-temporal action localization jointly localizes persons and recognizes their actions in videos.
- Recognizing actions often requires contextual interactions involving environments, other people, and objects, beyond visual appearance alone.
- Higher-order relations can reveal an actor’s action when relations with another actor or scene context alone are insufficient.
- ACAR-Net uses actor and grid-based context features, avoiding an extra predefined-category object detector while representing background, objects, and object parts.
- ACAR-Net explicitly models actor-context-actor relations through a High-Order Relation Reasoning Operator and an Actor-Context Feature Bank.
- ACAR-Net achieves state-of-the-art performance with significant margins on AVA and UCF101-24 and ranks first on the ActivityNet leaderboard at submission.
2. Related Work
Prior work on video understanding spans action classification, temporal localization, and spatio-temporal localization, with increasingly fine-grained spatial and temporal annotations. Relational reasoning methods model dependencies among frames, actors, and contextual entities for video recognition and localization.
- Action Recognition: Action-recognition research includes action classification, temporal localization, and spatio-temporal localization.
- Spatio-Temporal Action Localization: Spatio-temporal localization requires action instances to be localized in both space and time, with datasets such as AVA annotating actors’ atomic actions.
- Relational Reasoning for Video Understanding: Video-understanding methods use relational reasoning because recognizing an actor’s action depends on relationships with other actors and objects.
- Relational Reasoning for Video Understanding: Prior approaches capture temporal dependencies, long-range interactions, space-time region relations, or aggregated actor and scene features.
3. Method
ACAR-Net models higher-order actor-context-actor relations by combining actor and spatio-temporal context features, then extends this reasoning across time with an Actor-Context Feature Bank.
- Overall Framework: ACAR-Net processes person detections and video-backbone features, extracting actor features with RoIAlign before relation reasoning.The framework uses an off-the-shelf person detector and video backbone, with actor and context features passed to the ACAR module.
- First-order Relation Encoding: The module first encodes actor-context relations between each actor and spatial context locations.Actor features are replicated across context locations and concatenated with context features for convolutional relation encoding.
- High-order Relation Reasoning: HR2O models second-order relations by associating pairs of actor-context relations at the same spatial location, linking two actors through shared context.This restricts reasoning to actor-context-actor relations while avoiding unrestricted combinations of all relation features.
- High-order Relation Reasoning: A relation-network instantiation directly computes actor-context-actor features from two actor features and a context feature, then averages relations associated with each actor.The relation feature uses concatenation followed by two convolutional layers; the resulting feature is fused through residual addition.
- Action Prediction: After relation reasoning, a fully connected action classifier outputs confidence scores for each actor across action classes.The classifier operates on the resulting actor-context-actor feature maps.
- Actor-Context Feature Bank: The Actor-Context Feature Bank stores contextual relation features from past and future clips, enabling current actors to reason with long-term actor-context relations.Unlike the earlier Long-term Feature Bank, which stores actor features for first-order actor-actor interaction recognition, ACFB supports long-term higher-order reasoning.
4. Experiments on AVA
Experiments on AVA evaluate ACAR-Net’s components, relation-ordering choices, temporal feature support, and comparisons with prior methods. The results show consistent gains from higher-order relation reasoning, especially for interaction categories, with state-of-the-art performance across AVA variants and AVA-Kinetics.
- Experimental setup: AVA evaluates poses, human-object interactions, and human-human interactions using mAP, with softmax for poses and sigmoid outputs for interactions.The framework uses detected person boxes for inference and trains with both ground-truth and predicted boxes.
- Relation analysis: ACAR gains more mAP on human-object and human-human interactions than on poses, consistent with its actor-context-actor design.ACFB further improves the pose category, while attention visualizations show focused free-form context regions without object proposals.
- Ablation study: Both HR2O and ACFB provide significant gains over the baseline in the component ablation study.The baseline contains the video backbone, actor detector, and one-layer action classifier.
- Ablation study: Context-first relation ordering performs better than aggregating actor-actor relations before actor-context relations.ACAR-Net therefore encodes actor-context relations first for higher-order reasoning.
- Ablation study: Two modified non-local blocks outperform one, while adding another block worsens performance, possibly because of overfitting.The model adopts two non-local blocks as its default HR2O depth.
- Ablation study: 28.84 mAP with HR2O + ACFB improves over 27.83 mAP with HR2O alone, whereas replacing ACFB with LFB fails to match the baseline.The ACFB experiment uses a 21-second temporal window because of memory limitations.
- State-of-the-art comparison: 30.0 mAP on AVA v2.1 and 33.3 mAP on AVA v2.2 establish new state-of-the-art results under the reported settings.On AVA v2.2, the method uses single-scale testing and surpasses AIA with one-third of its temporal support.
- AVA-Kinetics: +6.71 mAP over other entries and 39.62 mAP on the test set place ACAR-Net first in the AVA-Kinetics ActivityNet Challenge 2020 task.Integrating ACAR into a competitive baseline also provides a +2.86 mAP gain.
5. Experiments on UCF101-24
Experiments on UCF101-24 evaluate spatio-temporal localization on 24 action classes using frame-mAP at IoU 0.5. ACAR-Net surpasses the strong baseline by a considerable margin, supporting the value of higher-order relation reasoning.
- Experimental setup: UCF101-24 contains 3,207 videos with spatio-temporal annotations for 24 action classes and is evaluated using frame-mAP at IoU 0.5.The experiments use the first split and a SlowFast R-50 backbone pretrained on Kinetics-400.
- Results: ACAR surpasses the strong UCF101-24 baseline by a considerable margin.The result is reported in Table 5 and is interpreted as evidence for the importance of high-order relation reasoning.
6. Conclusion
The conclusion identifies a need for more sophisticated relation reasoning in complex scenes and presents ACAR-Net as a higher-order solution. Experiments report state-of-the-art results on several challenging spatio-temporal action-localization benchmarks.
- Motivation: Complex scenes with multiple actors and varied contextual objects require more sophisticated relation reasoning than existing approaches provide.Current methods can miss important hints for recognizing actions.
- Contribution: ACAR-Net explicitly models higher-order relations between actors through their interactions with context.The conclusion frames this design as the paper’s response to limitations in current relation reasoning.
- Conclusion: Extensive experiments show that ACAR-Net outperforms existing relation-reasoning methods and achieves state-of-the-art results on several challenging benchmarks.The cited benchmark comparison is presented for spatio-temporal action localization.