Source-linked AI summary
Actor-Centric Relation Network
Chen Sun, Abhinav Shrivastava, Carl Vondrick, Kevin Murphy, Rahul Sukthankar, Cordelia Schmid
TL;DR
Spatio-temporal action localization needs relations among actors, objects, and scene elements to distinguish similar actions, beyond frame-level detections and temporal context alone. ACRN weakly supervises this relation mining, jointly learning actor-centric pair-wise relations from actor and global scene features for action classification. The approach outperforms baseline and alternative relation methods on JHMDB and AVA, with visualizations showing attention to relevant relations, although it has difficulty attending to the right relations for some actions.
Problem
Action localization remains challenging because actor appearance and temporal modeling alone may not capture spatial interactions needed to distinguish similar actions.
Method
ACRN weakly supervises actor-centric relation learning by computing and accumulating pair-wise information from actor and global scene features, with temporal context from 3D ConvNets.
Results
ACRN consistently outperforms actor-focused baselines and alternative context approaches on JHMDB and AVA, including a 2.4 frame-AP improvement over the Base-Model on JHMDB.
Takeaways & Limitations
Learned relation heat maps focus on relevant elements such as the ball and its motion, supporting state-of-the-art action detection performance on JHMDB and AVA.
Takeaways & Limitations
ACRN has difficulty attending to the right relations for some actions.
Abstract
from arXiv · showhide
Current state-of-the-art approaches for spatio-temporal action localization rely on detections at the frame level and model temporal context with 3D ConvNets. Here, we go one step further and model spatio-temporal relations to capture the interactions between human actors, relevant objects and scene elements essential to differentiate similar human actions. Our approach is weakly supervised and mines the relevant elements automatically with an actor-centric relational network (ACRN). ACRN computes and accumulates pair-wise relation information from actor and global scene features, and generates relation features for action classification. It is implemented as neural networks and can be trained jointly with an existing action detection system. We show that ACRN outperforms alternative approaches which capture relation information, and that the proposed framework improves upon the state-of-the-art performance on JHMDB and AVA. A visualization of the learned relation features confirms that our approach is able to attend to the relevant relations for each action.
1 Introduction
Action localization requires reasoning about actor relationships with objects, other actors, and scene context across time, which motivates the weakly supervised ACRN. The model jointly learns these relations and improves action detection while attending to relevant elements.
- Motivation: Actor appearance alone often cannot distinguish actions; spatial and temporal relationships with objects and other actors are essential.Catching and shooting a ball can be differentiated through actor-object interactions and evolving relative positions.
- Motivation: Fully supervised actor-object relation learning would require expensive annotations of actors and relevant objects.The proposed system instead aims to infer actor-object spatiotemporal relations from action annotations.
- Approach: ACRN computes pair-wise relations between actor and scene features, accumulates neighboring spatial information, and incorporates temporal context from 3D ConvNets.The design uses 1 × 1 convolutions for efficient relation computation and 3 × 3 convolutions to aggregate nearby locations.
- Evaluation: The approach is jointly trainable with an action detector and is evaluated on JHMDB and AVA against actor-focused and context-based alternatives.Experimental results report consistent outperformance over the baseline and alternative approaches.
- Evaluation: ACRN’s learned relation heat maps focus on the ball and its motion over time, illustrating attention to action-relevant relations.The visualizations provide qualitative evidence that the mined relational elements are relevant to distinguishing similar actions.
- Contribution: The paper reports state-of-the-art action detection performance on two datasets through learned actor-centric spatiotemporal relationships.This is identified as the paper’s primary contribution and supported by experiments on JHMDB and AVA.
2 Related work
Spatio-temporal action localization has progressed from frame-level object-detector-based recognition and temporal modeling, but spatial context remains underused. This limitation motivates relation modeling that can distinguish similar actions while avoiding the costly full supervision used by some static-image methods.
- Action recognition: Traditional action recognition classifies short trimmed clips, whereas untrimmed videos additionally require temporal localization of action instances.Spatio-temporal localization further requires determining where actions occur, especially with multiple actors.
- Action recognition: Many localization methods use object detectors for frame-level action classification, with some incorporating temporal information from multiple frames.These approaches extend object-recognition architectures toward spatio-temporal action localization.
- Action recognition: Existing methods that model temporal information often ignore spatial context involving humans, objects, and the surrounding scene.This omission can confuse similar actions such as jumping and shooting a basketball.
- Context in vision: Adding spatial context to a state-of-the-art action localization approach generates a significant performance improvement.The reported improvement motivates explicit modeling of relations beyond the actor appearance.
- Context in vision: Learned context has been studied for scene classification, object detection, and action recognition, extending earlier hand-designed context models.Deep convolutional networks provide a framework for learning contextual information rather than specifying it manually.
- Context in vision: Human-object relation models achieve strong static-image recognition but rely on full supervision of actors, actions, and objects, unlike the proposed approach.The cited methods target V-COCO and HICO-DET, whereas the paper addresses spatio-temporal video localization.
3 Action detection with actor-centric relation network
The framework augments actor-based action detection with an actor-centric relation network that learns relations between each actor and scene feature locations using weak actor-level supervision. These relation features are jointly integrated into action classification while preserving temporal context from 3D ConvNets.
- Framework motivation: The framework explicitly models relations between each actor proposal and other actors or objects, which are typically ignored by recent action detection methods.It uses weak actor-level supervision because detailed annotations of relevant objects and actors can be expensive and are often unavailable.
- Actor-centric relations: ACRN computes pairwise relation features from cropped actor features, a global feature map, and location information, then feeds aggregated relation features to action classification.Each feature-map cell is treated as an object, avoiding the need to generate or annotate explicit object proposals.
- Framework integration: The base model jointly trains actor localization and action classification end to end, and ACRN can be incorporated into this action detection pipeline.The same process is repeated for all proposed actor regions, producing representations for action classification and bounding-box regression.
- Framework architecture: 3D ConvNets provide temporal context, whose feature maps are temporally flattened before ROIPooling produces cropped actor features.The flattened representation is used to preserve compatibility with standard differentiable ROIPooling and reuse pretrained 3D ConvNet weights.
- Actor-centric relations: The relation network accumulates neighboring relation information with convolutional layers instead of relying only on average pooling, which could dilute sparse relevant relations.Actor features are duplicated across the feature map, combined with global features and embeddings, and processed using 1×1 and subsequent convolutional operations.
4 Experiments
Experiments evaluate ACRN through ablations, feature choices, state-of-the-art comparisons, and qualitative visualizations on JHMDB and AVA. ACRN improves detection performance and captures relations beyond actor boxes, although it struggles with some actions.
- Experimental setup: The experiments use JHMDB and AVA action-detection benchmarks, with frame-AP and video-AP evaluated at an intersection-over-union threshold of 0.5.JHMDB results are averaged over three splits, and video detections are linked into tubes.
- Relation reasoning ablations: ACRN improves JHMDB frame-AP by 2.4 over the Base-Model, while Resize+Concat and Santoro et al. do not outperform the baseline.This comparison indicates that the relation reasoning module design matters, not merely access to global feature maps.
- Temporal context: ACRN consistently improves over the Base-Model across all tested temporal lengths, while increasing the number of input frames generally helps both models.
- Feature layers and scales: Conv2c is the best single global feature on JHMDB, followed by Mixed 4f, while combining multiple layers does not necessarily improve overall performance.The two feature layers are complementary for many AVA actions, and performance remains relatively stable across scales.
- State-of-the-art comparison: ACRN further improves a Base-Model that already outperforms previous methods, with the largest JHMDB gains for catch (12%), jump (6%), shoot gun (5%) and wave (10%).Gains are smaller for near-saturated categories, while the Base-Model is slightly better on pick, throw and run.
- State-of-the-art comparison: On AVA, the biggest reported gains include answer phone (11%), swim (10%), dance (10%), kiss (8%), touch (6%), fight (5%) and play musical instruments (5%).Most of these categories involve human-human or human-object interactions.
- Qualitative results: Relation heatmaps show that ACRN attends beyond actor bounding boxes and varies with the actor and action, while also removing false alarms and recovering missed detections.The visualizations use CAM-based per-category relation heatmaps on JHMDB and AVA.
5 Conclusion
The approach automatically identifies relevant spatiotemporal elements for human actions, achieving clear localization gains while visualizations indicate those elements are relevant.
- The method automatically determines relevant spatiotemporal elements characterizing human actions in video.
- Experimental results demonstrate a clear gain for spatiotemporal action localization.
- Visualizations show that the mined elements are relevant to the actions.