Source-linked AI summary
Hierarchical Deep Temporal Models for Group Activity Recognition
Mostafa S. Ibrahim, Srikanth Muralidharan, Zhiwei Deng, Arash Vahdat, Greg Mori
TL;DR
Group activity recognition requires identifying a scene-level activity from individual actions and their relations over time. The paper proposes a two-stage hierarchical LSTM model that first represents person-level dynamics and then aggregates them for group recognition, evaluating it on volleyball and Collective Activity datasets. The authors report improved performance over baselines lacking hierarchical consideration of individual and group activities.
Problem
Group activity recognition must infer a group's activity from individual person actions, their relations, and temporal dynamics at both person and group levels.
Method
A two-stage hierarchical LSTM architecture models each person's temporal activity and then aggregates person-level representations into a group-level temporal model.
Results
The architecture improves upon baseline methods lacking hierarchical consideration of individual and group activities on the Volleyball and Collective Activity datasets.
Takeaways & Limitations
The results support using hierarchical temporal representations that recognize and integrate player actions for overall team-activity recognition.
Takeaways & Limitations
On the Collective Activity Dataset, insufficient spatial-relation modeling contributes to confusion among crossing, waiting, and walking.
Abstract
from arXiv · showhide
In this paper we present an approach for classifying the activity performed by a group of people in a video sequence. This problem of group activity recognition can be addressed by examining individual person actions and their relations. Temporal dynamics exist both at the level of individual person actions as well as at the level of group activity. Given a video sequence as input, methods can be developed to capture these dynamics at both person-level and group-level detail. We build a deep model to capture these dynamics based on LSTM (long short-term memory) models. In order to model both person-level and group-level dynamics, we present a 2-stage deep temporal model for the group activity recognition problem. In our approach, one LSTM model is designed to represent action dynamics of individual people in a video sequence and another LSTM model is designed to aggregate person-level information for group activity recognition. We collected a new dataset consisting of volleyball videos labeled with individual and group activities in order to evaluate our method. Experimental results on this new Volleyball Dataset and the standard benchmark Collective Activity Dataset demonstrate the efficacy of the proposed models.
1 INTRODUCTION
The paper addresses group activity recognition by modeling both individual actions and their temporal relations at the scene level. It proposes a hierarchical two-stage LSTM architecture and evaluates it on volleyball and surveillance-oriented datasets.
- Group activity recognition classifies the activity performed by a group as a whole rather than only by individual players.
- The task is challenging because group labels may depend on one key individual or on many inter-related people and their context.
- A hierarchical deep temporal model uses one layer for individual people and a higher layer to learn discriminative group-level information.
- The proposed two-stage architecture models individual person activities first, then combines person-level information to represent group activities using LSTMs.
- The method is evaluated on the Collective Activity Dataset and a new Volleyball Dataset containing person detections, individual action labels, and group activity labels.
- The paper extends an earlier version with alternative pooling structures, an enlarged Volleyball Dataset, and additional empirical analyses.
2 RELATED WORK
Prior group activity recognition methods commonly used hand-crafted features and structured models to represent relations among people. This paper situates its LSTM-based hierarchical model within the broader shift toward deep representations and sports-video analysis.
- Earlier group activity recognition research modeled spatial and temporal relations among individuals, often using hand-crafted features and structured models.
- Related approaches represented social roles, pairwise interactions, group structures, or collective activities with CRFs, hierarchical models, probabilistic grouping, and graphical formalisms.
- Sport Video Analysis: Sports-video research spans player detection, tracking, action recognition, player interactions, and team-level activity classification.
- Sport Video Analysis: Sports-analysis methods have used trajectories, role-ordered representations, occupancy maps, team centroids, heat maps, and classifiers for player or team activities.
- Deep Learning: Deep CNNs unify feature and classifier learning, while recurrent models such as LSTMs handle variable-length space-time inputs.
- Deep Learning: This work combines deep structured learning with LSTM temporal modeling to learn discriminative information from time-varying sports activity data.
3 PROPOSED APPROACH
The proposed approach uses hierarchical LSTMs to model individual actions and group-level temporal dynamics from tracked people. CNN features, person-level temporal representations, pooling, and a second LSTM are combined for group activity recognition, with subgroup pooling to preserve spatial arrangements.
- 3 PROPOSED APPROACH: The method takes tracklets of people as input and considers person actions, their temporal dynamics, and group-level temporal evolution as cues for recognizing group activities.The paper frames person-level recognition as a first step and treats temporal changes at both levels as informative.
- 3 PROPOSED APPROACH: The hierarchical deep model is intended to learn low-level image features, person-level actions, temporal relations, and group dynamics in one end-to-end framework.The approach is presented as a deep-learning alternative to classic structured prediction with hand-crafted features.
- 3.1 Temporal Model of Individual Action: A CNN extracts spatial features from each person’s bounding box, while a person-level LSTM represents that individual’s action over time.The LSTM hidden state can be used to predict person-level actions from temporally evolving information.
- 3.2 Hierarchical Model for Group Activity Recognition: For each person, the model concatenates CNN appearance features with first-stage LSTM outputs before pooling the resulting temporal features across the scene.The person representation combines AlexNet fc7 features and first-stage LSTM outputs; max pooling produces the frame-level representation.
- 3.2 Hierarchical Model for Group Activity Recognition: The first-stage person representations are aggregated across people, and a second LSTM models the temporal dynamics of the resulting group representation to predict group activity.The pooled frame-level representation is passed to the second LSTM and then to a softmax classifier for group activities.
- 3.3 Handling sub-groups: The model can split players into subgroups and concatenate subgroup representations to retain spatial arrangements that whole-scene max pooling may lose.The paper motivates subgroup pooling by noting that pooling all players together can confuse left-team and right-team activities.
4 EXPERIMENTS
Experiments evaluate the two-stage model through ablations and comparisons on the Collective Activity and Volleyball Datasets. Results support hierarchical temporal modeling, while revealing dataset- and representation-specific confusions and limitations.
- Experimental setup: The experiments compare the proposed architecture with ablation baselines and previously published methods on the Collective Activity Dataset.The evaluation includes image-, person-, and temporal-model baselines, with model details specified for the two LSTM stages.
- Collective Activity Dataset: The two-stage LSTM model significantly outperforms baseline models, and adding temporal information improves frame- and person-based counterparts.The comparisons include B1 versus B4, B2 versus B5, and B3 versus the two-stage model.
- Collective Activity Dataset: The two-stage model performs comparably to state-of-the-art methods on the Collective Activity Dataset.Figure 4 visualizes example predictions, including errors in ambiguous actions and under anomalous camera zoom.
- Collective Activity Dataset: On the Collective Activity Dataset, talking and queuing are classified almost perfectly, while crossing, waiting, and walking are confused.The authors associate these errors with limited modeling of spatial relations between people, especially for crossing sequences constrained to an orderly path.
- Volleyball Dataset: On the Volleyball Dataset, the two-stage LSTM model outperforms baselines, and explicit person modeling is useful under rapidly changing camera backgrounds.A temporal model without explicit people performs worse than a non-temporal image baseline, whereas a temporal person-aware baseline performs comparably to it.
- Ablation analysis: Across both datasets, the first person-level LSTM is relatively more critical to performance than the second group-level LSTM.Both LSTMs contribute overall classification performance, but removing the first layer is more damaging according to the ablations.
- Volleyball Dataset: Dense-trajectory baselines show close performance in one-group form, while the two-group variation performs better; the proposed model outperforms these baselines.The authors suggest that two-group trajectories reduce confusions between left- and right-team activities.
- Volleyball Dataset: Grouping left- and right-team players before pooling produces more accurate high-level labels and reduces left-versus-right winpoint confusion.The grouped representation is compared with a single representation using max pooling.
5 CONCLUSION
The paper presents a deep hierarchical architecture for group activity recognition that models person-level temporal actions before combining them to recognize group activity. It introduces a Volleyball Dataset and evaluates the model on both that dataset and the Collective Activity Dataset, improving over deep-learning baselines without hierarchical consideration.
- The proposed architecture uses a two-stage process to learn temporal representations of person-level actions and combine individual representations for group activity recognition.
- The authors created a new Volleyball Dataset for training and testing the model.
- Evaluation covers both the new Volleyball Dataset and the Collective Activity Dataset.
- The architecture improves upon baseline methods that lack hierarchical consideration of individual and group activities using deep learning.