Source-linked AI summary
GroupFormer: Group Activity Recognition with Clustered Spatial-Temporal Transformer
Shuaicheng Li, Qianggang Cao, Lingbo Liu, Kunlin Yang, Shinan Liu, Jun Hou, Shuai Yi
TL;DR
Group activity recognition needs representations that capture complex spatial-temporal interactions among individuals, beyond separate modeling or direct aggregation. GroupFormer jointly refines individual and group representations with a clustered spatial-temporal Transformer, and it outperforms state-of-the-art methods on the Volleyball and Collective datasets.
Problem
Group activity recognition lacks methods that jointly model spatial-temporal context and group individuals according to their interconnected relations.
Method
GroupFormer uses decoders to bridge spatial and temporal features and clustered attention to model intra-group and inter-group relations while refining individual and group representations.
Results
GroupFormer outperforms most state-of-the-art methods by a considerable margin on two group activity recognition benchmarks.
Takeaways & Limitations
Joint spatial-temporal modeling and clustered attention provide the framework's basis for learning more informative group activity representations.
Abstract
from arXiv · showhide
Group activity recognition is a crucial yet challenging problem, whose core lies in fully exploring spatial-temporal interactions among individuals and generating reasonable group representations. However, previous methods either model spatial and temporal information separately, or directly aggregate individual features to form group features. To address these issues, we propose a novel group activity recognition network termed GroupFormer. It captures spatial-temporal contextual information jointly to augment the individual and group representations effectively with a clustered spatial-temporal transformer. Specifically, our GroupFormer has three appealing advantages: (1) A tailor-modified Transformer, Clustered Spatial-Temporal Transformer, is proposed to enhance the individual representation and group representation. (2) It models the spatial and temporal dependencies integrally and utilizes decoders to build the bridge between the spatial and temporal information. (3) A clustered attention mechanism is utilized to dynamically divide individuals into multiple clusters for better learning activity-aware semantic representations. Moreover, experimental results show that the proposed framework outperforms state-of-the-art methods on the Volleyball dataset and Collective Activity dataset. Code is available at https://github.com/xueyee/GroupFormer.
1. Introduction
Group activity recognition requires reasoning over complicated spatial and temporal relations among individuals. GroupFormer addresses this challenge by jointly modeling these relations and clustering individuals to refine group and individual representations.
- Group activity recognition is challenging because spatial and temporal transitions vary substantially in untrimmed multi-person scenarios.
- Prior methods leave two gaps: jointly modeling spatial-temporal context and grouping individuals according to interconnected relations.
- GroupFormer generates an initial group representation from individual and scene context, then refines individual and group representations with stacked Spatial-Temporal Transformers.
- Its decoders connect spatial and temporal features in a cross manner, while a query-key mechanism models spatial-temporal context jointly.
- Clustered attention dynamically assigns individuals to groups and models inter-group and intra-group relations to enrich global activity context.
- Experiments on the Volleyball and Collective datasets show that GroupFormer outperforms state-of-the-art methods by a significant margin.
2. Related Works
Related work spans recurrent, graph, attention-based, and Transformer approaches for modeling group activity. GroupFormer builds on these directions while using encoder-decoder mechanisms to connect spatial and temporal relations.
- RNN-based methods model individual dynamics but require substantial computational cost.
- Graph and attention methods capture spatial or temporal relations, but prior designs may limit temporal coverage or omit temporal relations within their own modules.
- GroupFormer combines a CNN backbone, Group Representation Generator, and Clustered Spatial-Temporal Transformer for group activity recognition.
- Its encoder augments semantic representations, while a decoder builds a bridge between spatial and temporal relations.
- Transformer self-attention is suited to capturing long-term dependencies, but naive application incurs quadratic computational cost.
- Spatial-temporal attention and graph-based methods model dynamic correlations through message passing or attention mechanisms.
3. GroupFormer
GroupFormer initializes group representations from scene and individual features, then refines individual and group representations through spatial-temporal Transformer modules. Its clustered attention mechanism focuses relation modeling within and across adaptively formed individual clusters.
- Overview: GroupFormer processes RGB video clips with an I3D backbone, extracting scene and individual features from convolutional feature maps and actor regions.Individual features additionally incorporate pose information obtained with AlphaPose.
- Group Representation Generator: The Group Representation Generator converts scene and individual features into tokens, then fuses them into an initialized group representation.Scene features produce visual scene tokens through spatial attention, while a decoder generates an individual token before fusion.
- Spatial-Temporal Transformer: Spatial and temporal encoders operate in parallel to generate contextual features, with the temporal encoder emphasizing dynamical evolution along time for each individual.The spatial encoder treats time as the batch dimension, whereas the temporal encoder treats spatial positions as the batch dimension.
- Spatial-Temporal Transformer: Cross-scheme individual decoders use spatial features as queries against temporal features and vice versa, integrating spatial-temporal context into individual representations.The resulting representations can be repeatedly processed by stacked Spatial-Temporal Transformer blocks.
- Group Decoder: A Group Decoder uses the enhanced individual representation as key and the group representation as query to summarize overall context for group activity prediction.The decoder contains multi-head cross-attention and a feed-forward network.
- Clustered Attention Mechanism: Clustered Spatial-Temporal Transformer replaces fully connected attention by adaptively grouping individuals and modeling both intra-group and inter-group relations.Mini-batch k-means forms clusters; intra-group attention refines individuals using same-cluster information, while inter-group attention models relations among cluster centroids.
4. Experiments and Analysis
Experiments on Volleyball and Collective datasets compare GroupFormer with prior methods and analyze its architectural choices. Results show gains from joint spatial-temporal modeling, clustered attention, and suitable CSTT depth.
- Experimental setup: GroupFormer is evaluated on the Volleyball and Collective datasets against state-of-the-art methods using accuracy metrics.The experiments include comparisons and ablation studies targeting individual components.
- State-of-the-art comparison: 95.7% activity accuracy is achieved on Volleyball with I3D+Pose inputs, outperforming previous methods.The RGB-only model also outperforms several prior methods using additional modalities.
- State-of-the-art comparison: 96.3% accuracy is achieved on Collective with I3D+Pose, while RGB-only features reach 93.6%.The I3D+Pose result is reported as a considerable improvement over previous methods.
- Spatial-temporal relations: Joint spatial-temporal decoding improves group activity accuracy by 3.1% over the baseline, while stacked modeling slightly exceeds parallel modeling.The parallel variant improves performance from 91.0% to 92.2%.
- Clustered attention: Four clusters produce the best clustered-attention result, and combining intra-group with inter-group attention provides the strongest performance.The cluster-choice comparison uses the Volleyball dataset.
- CSTT depth: Three CSTT blocks achieve the best result, whereas performance slightly degrades with four blocks.A single CSTT block outperforms the baseline by 2.6%.
5. Conclusion
GroupFormer is a transformer-based architecture that jointly models spatial-temporal context and uses clustered attention to organize individuals into groups. Experiments on two benchmarks show considerable improvements over most state-of-the-art methods.
- Conclusion: GroupFormer jointly models spatial-temporal context and uses clustered attention to exploit intra-group and inter-group relations.The architecture is designed for inferring group activity from individual interactions.
- Conclusion: Experiments on two benchmarks show that GroupFormer outperforms most state-of-the-art methods by a considerable margin.The conclusion summarizes results across the evaluated benchmarks.