Source-linked AI summary
Learning Actor Relation Graphs for Group Activity Recognition
Jianchao Wu, Limin Wang, Li Wang, Jie Guo, Gangshan Wu
TL;DR
Group activity recognition needs effective actor-relation modeling despite limited interaction annotations. The paper learns flexible Actor Relation Graphs with GCNs, including spatial and temporal sparsification, and reports state-of-the-art performance on the Volleyball and Collective Activity datasets.
Problem
Group activity recognition requires capturing discriminative appearance and position relations between actors, although the underlying interactions lack direct annotations.
Method
The method builds Actor Relation Graphs whose connections are learned end to end with GCNs, using localized spatial and randomized temporal sparsification for efficient video modeling.
Results
The approach achieves state-of-the-art results on the Volleyball and Collective Activity datasets.
Takeaways & Limitations
Learned actor graphs capture relation information for group activity recognition, as supported by ablations and visualizations of relation graphs and features.
Abstract
from arXiv · showhide
Modeling relation between actors is important for recognizing group activity in a multi-person scene. This paper aims at learning discriminative relation between actors efficiently using deep models. To this end, we propose to build a flexible and efficient Actor Relation Graph (ARG) to simultaneously capture the appearance and position relation between actors. Thanks to the Graph Convolutional Network, the connections in ARG could be automatically learned from group activity videos in an end-to-end manner, and the inference on ARG could be efficiently performed with standard matrix operations. Furthermore, in practice, we come up with two variants to sparsify ARG for more effective modeling in videos: spatially localized ARG and temporal randomized ARG. We perform extensive experiments on two standard group activity recognition datasets: the Volleyball dataset and the Collective Activity dataset, where state-of-the-art performance is achieved on both datasets. We also visualize the learned actor graphs and relation features, which demonstrate that the proposed ARG is able to capture the discriminative relation information for group activity recognition.
1. Introduction
The paper targets flexible, efficient modeling of actor relations for group activity recognition, learning appearance and positional connections with Actor Relation Graphs and Graph Convolutional Networks. Spatial and temporal sparsification improve efficiency, while experiments report state-of-the-art results on two benchmarks.
- 1. Introduction: Group activity recognition requires modeling relevant relations between actors, but training provides only individual-action and collective-activity labels.The underlying interaction information is not directly annotated.
- 1. Introduction: Graph Convolutional Networks automatically optimize graph connections end to end and support efficient relational reasoning for group activity recognition.ARGs can be added on top of existing 2D CNNs and generate video-specific relation graphs after training.
- 1. Introduction: Localized spatial ARGs restrict connections to local neighborhoods, while randomized temporal ARGs retain only selected frames for long-range video modeling.These variants sparsify graph connections to improve efficiency.
- 1. Introduction: Actor Relation Graphs represent actors as nodes and their pairwise appearance and position relations as edges.The graphs are designed to capture both relation cues explicitly.
- 1. Introduction: The approach reports state-of-the-art results on the Volleyball and Collective Activity datasets.The paper also visualizes learned actor graphs and relation features to examine the captured relation information.
2. Related Work
Prior group activity methods combine person-level CNN features with scene-level aggregation, while visual-relation research models object or entity relationships across several tasks. This paper applies learned sparse graph reasoning to actor relations, addressing the inefficiency of fully connected video graphs.
- Group activity recognition: Earlier group activity recognition methods used handcrafted visual features with probabilistic graphical or AND-OR grammar models, while newer methods use CNNs.Deep CNNs have improved performance in this task.
- Visual relation: Visual-relation research studies human-object interactions, pairwise scene relationships, and relational reasoning between objects or entities.Several approaches require additional interaction annotations or target tasks beyond group activity recognition.
- Neural networks on graphs: The framework samples video frames, extracts actor features, builds multiple relation graphs, applies GCN reasoning, and fuses relational and original features for prediction.The fused representation supports individual-action and group-activity classifiers.
- Visual relation: Explicitly modeling relations improves group activity recognition accuracy, with dot-product and embedded dot-product methods reaching 91.3% in the reported ablation.Both methods outperform the relation network in that experiment.
- Neural networks on graphs: Fully connected graphs across all actor-frame pairs are inefficient, motivating sparse graphs based on relative location and sparse temporal sampling.The paper combines GCNs with these sparsification strategies for more efficient learning.
3. Approach
The approach represents actors as nodes in learned relation graphs whose edge weights combine appearance and position cues, then applies graph convolution for relational reasoning. It extends this design with multiple graphs and sparse spatial and temporal connections to capture diverse relations efficiently.
- 3.1. Group Activity Recognition Framework: The framework extracts actor features from sampled video frames, builds relation graphs, and uses graph convolution before classifying individual actions and group activity.Actor-level relational representations support individual-action classification, while pooled actor representations support scene-level group-activity classification.
- 3.2. Building Actor Relation Graphs: Each graph node represents an actor, while G_ij encodes the importance of actor j’s feature to actor i.The graph is represented as an N × N matrix over the N actors.
- 3.2. Building Actor Relation Graphs: ARG explicitly combines appearance and position relations, fusing them into a scalar edge weight that is normalized across each receiving actor’s connections.The two relation types are modeled separately because they have different semantic attributes; softmax normalization makes each actor’s relation values sum to 1.
- 3.2. Building Actor Relation Graphs: Appearance relations can use dot-product similarity, embedded dot-product similarity, or a Relation Network over concatenated actor features.The embedded dot-product learns similarity in a transformed subspace, while the Relation Network projects the concatenated vector to a scalar followed by ReLU.
- 3.2. Building Actor Relation Graphs: Position relations are modeled either by zeroing connections beyond a distance threshold or by embedding relative distance into a high-dimensional representation.The thresholded design is called localized ARG; distance encoding uses sine and cosine functions with different wavelengths before scalar projection and ReLU.
- 3.2. Building Actor Relation Graphs: Multiple graphs with unshared weights jointly attend to different relation types, while randomized ARG sparsely samples K = 3 frames to improve long-range temporal efficiency.The authors report that sparse frame sampling improves recognition accuracy, partly by increasing training diversity and reducing overfitting on limited datasets.
4. Experiments
Experiments evaluate ARG on two group-activity benchmarks, analyze its components, compare it with prior methods, and visualize learned relations and representations. Ablations show gains from appearance, position, multiple-graph, and temporal modeling choices.
- Datasets and Implementation Details: Experiments use the Volleyball and Collective Activity datasets, with ablation studies, state-of-the-art comparisons, and visual analyses.The implementation includes ground-truth and proposal-based bounding-box experiments, with inference taking approximately 0.2 seconds per video on one TITAN-XP GPU.
- Ablation Studies: 91.3% accuracy is achieved by both dot-product and embedded dot-product appearance relations, outperforming the relation network in the single-graph setting.All GCN-based models outperform the base model, and embedded dot-product is selected for later experiments.
- Ablation Studies: Modeling position relation improves performance over appearance-only graphs, with distance masks slightly outperforming distance encoding.The distance mask is used in subsequent experiments.
- Ablation Studies: 92.1% accuracy is obtained with multiple graphs, improving over 91.6% with one graph; late fusion by summation performs best.Using 16 graphs is selected, while early fusion before graph convolution causes a substantial performance drop.
- Ablation Studies: 92.5% accuracy is reached when temporal graphs are added after sparse temporal modeling, exceeding separate-frame score fusion.The study samples K = 3 frames and finds temporal reasoning helps differentiate group-activity categories.
- Comparison with the State of the Art: ARG establishes state-of-the-art results on both benchmarks, including 91.0% group-activity accuracy on Collective Activity.On Volleyball, the Inception-v3 model improves group-activity accuracy by about 2% over the compared feature-extraction strategy and also improves individual-action recognition.
- Visualization: Visualizations indicate that learned graphs can identify a key actor, while ARG variants produce better-separated video representations, especially with multiple graphs and temporal aggregation.Figure 3 displays input frames, labels, and relation matrices; Figure 4 colors t-SNE points by group activity.
5. Conclusion
The paper presents ARG as a flexible and efficient approach for relational reasoning in multi-person scenes and reports new state-of-the-art results on two datasets. Ablations and visualizations support its ability to learn relation information for group activity understanding.
- Conclusion: ARG performs relational reasoning over learned actor graphs and achieves new state-of-the-art results on two group-activity datasets.The paper identifies further understanding of ARG and incorporation of more global scene information as future directions.