Source-linked AI summary
Actor-Transformers for Group Activity Recognition
Kirill Gavrilyuk, Ryan Sanford, Mehrsan Javan, Cees G. M. Snoek
TL;DR
Group activity recognition must capture relationships among actors without relying solely on explicit spatial and temporal structures. The paper introduces actor-transformers that use self-attention over static pose and dynamic video representations, and reports state-of-the-art results on two public benchmarks.
Problem
Existing group activity recognition methods explicitly model spatial and temporal actor relationships based on actor locations.
Method
Actor-transformers apply a transformer to static 2D pose and dynamic 3D CNN actor representations, learning actor relations and combining them through studied fusion strategies.
Results
Actor-transformers achieve state-of-the-art results on the Collective and Volleyball benchmarks, outperforming previous best published results by a considerable margin.
Takeaways & Limitations
Self-attention can refine and aggregate actor-level features for group activity recognition without explicit spatial and temporal modeling.
Abstract
from arXiv · showhide
This paper strives to recognize individual actions and group activities from videos. While existing solutions for this challenging problem explicitly model spatial and temporal relationships based on location of individual actors, we propose an actor-transformer model able to learn and selectively extract information relevant for group activity recognition. We feed the transformer with rich actor-specific static and dynamic representations expressed by features from a 2D pose network and 3D CNN, respectively. We empirically study different ways to combine these representations and show their complementary benefits. Experiments show what is important to transform and how it should be transformed. What is more, actor-transformers achieve state-of-the-art results on two publicly available benchmarks for group activity recognition, outperforming the previous best published results by a considerable margin.
1. Introduction
The paper replaces explicit location-based spatial and temporal modeling with an actor-transformer that learns actor relations through self-attention. It combines static pose and dynamic video representations to recognize group activities, achieving state-of-the-art results on two benchmarks.
- Motivation: Existing group activity recognition methods commonly model spatial and temporal relationships between actors using their locations.The paper targets individual actions and group activities in applications including sports, crowd monitoring, surveillance, and behavior analysis.
- Contribution: The proposed transformer learns actor interactions and selectively extracts information relevant to group activity recognition without requiring a priori graph or recurrent structures.This approach uses self-attention to model dependencies across actors and aggregate actor-level information.
- Actor representations: Static actor representations use single-frame 2D pose features, while dynamic representations use 3D CNN features from stacked RGB or optical-flow frames.The dynamic branch captures motion, while pose features help distinguish actions with subtle motion differences.
- Fusion: Actor-transformers combine actor-specific static and dynamic features through multiple fusion strategies and study their complementary benefits.The representations are passed to a transformer that learns relations between actors before inferring group activity.
- Results: Actor-transformers achieve state-of-the-art results on the Collective and Volleyball benchmarks, surpassing previous best published results by a considerable margin.The paper presents this as its third contribution alongside the transformer formulation and static-dynamic actor representation study.
2. Related Work
Prior video action recognition work developed 2D, two-stream, 3D CNN, recurrent, attention, and pose-based representations. Group activity recognition extended these ideas to multiple actors, while this paper fuses pose and dynamic representations for group-level inference.
- Video representations: Video action recognition progressed from frame-based 2D CNNs and two-stream RGB-optical-flow models to 3D CNNs that represent time directly.Other approaches used fusion methods or recurrent networks to incorporate temporal information across frames.
- Attention: Attention mechanisms for video recognition have been implemented through LSTMs, pooling, graphs, and pose- or motion-guided inputs.Transformer networks are presented as a more recent attention-based direction.
- Pose representations: Pose-based action recognition has used handcrafted pose features, skeleton data, body-joint representations, and pose-guided attention.These approaches primarily addressed recognizing actions performed by a single individual actor.
- Position of this work: This work combines pose features with dynamic representations using a multi-stream strategy to infer group activity rather than only an individual action.The paper applies the fusion of complementary representations to group-level recognition.
- Group activity recognition: Group activity recognition developed from handcrafted actor features processed by probabilistic graphical models toward deep-learning approaches including RNN-type networks.Public Collective and Volleyball datasets contributed to increased attention to the task.
3. Model
The model recognizes group activities by refining and aggregating actor-specific static and dynamic features with a transformer, then predicting individual actions and group activity. It uses pose and 3D CNN representations, positional encoding, and early or late fusion to combine complementary information.
- Actor feature extractor: The model extracts static pose features and dynamic RGB or optical-flow features for each actor before transformer-based aggregation.HRNet supplies pose representations, while I3D followed by RoIAlign extracts actor-level spatio-temporal features.
- Prediction and training: The aggregated actor features support separate linear classifiers for individual actions and group activity, trained jointly with a weighted sum of cross-entropy losses.The model uses equal loss weights, λg = λa = 1, in all experiments.
- Fusion: Early fusion combines static and dynamic actor features before the transformer, whereas late fusion combines their classifier prediction scores.The two strategies separately probe how information from the pose and video branches is combined.
- Transformer: The transformer receives actor features as an unordered set and refines them through self-attention without explicit spatial or temporal modeling.Actor features are embedded into vectors, processed by encoder layers, and aggregated into action-level features.
- Transformer: Positional encoding adds each actor bounding box’s center coordinates to its feature representation, allowing implicit spatial information.The x and y coordinates are encoded in separate halves of the feature dimensions.
4. Experiments
Experiments evaluate actor-transformers through ablations and comparisons on the Volleyball and Collective datasets. Results support positional encoding, late fusion of complementary actor representations, and state-of-the-art performance across both benchmarks.
- Experimental setup: Experiments use the Volleyball and Collective group activity datasets, with ten-frame inputs and pose, RGB, or optical-flow representations.The Volleyball clips contain individual and group annotations, while Collective annotations cover five individual actions and derive group activity from the majority action.
- Actor-transformer ablation: Positional encoding improves Volleyball accuracy by around 1.3%, while additional transformer heads and layers do not help on the small dataset.The authors therefore use one layer with one head in subsequent experiments.
- Actor aggregation: The actor-transformer outperforms graph aggregation across backbone networks and matches activity maps on optical flow without explicit relationship representations or segmentation masks.It is slightly worse than activity maps on RGB, while avoiding multiple refinement stages.
- Fusion: 93.9% versus 93.1% shows that late fusion with static Pose benefits Flow more than RGB, while early fusion is similar to or worse than single-branch models.Flow alone reaches 91.5% versus RGB at 91.4%, but Pose + Flow reaches 93.9% versus 93.1% for Pose + RGB.
- State-of-the-art comparison: Pose + RGB and Pose + Flow surpass existing Volleyball results by 0.5% and 1.4% for group activity, and by 2.7% and 2.9% for individual actions.Combining all three representations provides no improvement over Pose + Flow.
- State-of-the-art comparison: On Collective, RGB + Flow reaches 92.8% accuracy, while Pose + RGB and Pose + Flow achieve state-of-the-art results and outperform the same-backbone activity-maps result by 7.0%.The individual branches score Flow 83.8%, Pose 87.9%, and RGB 90.8%; confusion is greatest between crossing and walking.
5. Conclusion
The paper proposes transformer-based refinement and aggregation of actor-level features for group activity recognition, combining static and dynamic representations. It achieves state-of-the-art results on two public benchmarks.
- The transformer refines and aggregates actor-level features for group activity recognition.
- Static pose and dynamic RGB or optical-flow representations are studied, including multiple combination strategies with complementary benefits.
- The actor-transformer matches or outperforms related approaches without task-specific modifications optimized for group activity recognition.
- The approach achieves state-of-the-art results on two publicly available benchmarks, surpassing previously published results by a considerable margin.