Source-linked AI summary
Social Adaptive Module for Weakly-supervised Group Activity Recognition
Rui Yan, Lingxi Xie, Jinhui Tang, Xiangbo Shu, Qi Tian
TL;DR
Previous methods are sensitive to varying numbers of people per frame and require explicit person locations, limiting their practical application. SAM uses the assumption that key instances are highly related, builds dense relations to select features, then uses sparse relations for relational embedding. The approach achieves state-of-the-art results on two datasets and automatically attends to key proposals and frames.
Problem
Previous methods are sensitive to varying numbers of people per frame and require explicit person locations, limiting their practical application.
Source: §1 · p. 2Method
SAM uses the assumption that key instances are highly related, builds dense relations to select features, then uses sparse relations for relational embedding.
Source: §1 · p. 2, §4.2 · p. 8Results
The approach achieves state-of-the-art results on two datasets and automatically attends to key proposals and frames.
Source: §6 · p. 14Takeaways & Limitations
The video-level-only setting supports more practical GAR scenarios, and SAM can be plugged into existing frameworks and trained end to end.
Source: §1 · p. 2, §6 · p. 14The feature-selection approach relies on the assumption that key instances are highly related to each other.
Source: §4.1 · p. 7
Abstract
from arXiv · showhide
This paper presents a new task named weakly-supervised group activity recognition (GAR) which differs from conventional GAR tasks in that only video-level labels are available, yet the important persons within each frame are not provided even in the training data. This eases us to collect and annotate a large-scale NBA dataset and thus raise new challenges to GAR. To mine useful information from weak supervision, we present a key insight that key instances are likely to be related to each other, and thus design a social adaptive module (SAM) to reason about key persons and frames from noisy data. Experiments show significant improvement on the NBA dataset as well as the popular volleyball dataset. In particular, our model trained on video-level annotation achieves comparable accuracy to prior algorithms which required strong labels.
1 Introduction
Group activity recognition must capture both individual behavior and relationships among people, but fully supervised approaches require person-level annotations and explicit locations. The paper introduces video-level weak supervision and SAM to select useful people and frames despite noisy proposals and irrelevant frames.
1 Introduction
- GAR models must represent individual behaviors and the relationships among people cooperating in group activities.Prior fully supervised approaches use person-level annotations, including bounding boxes and individual action labels.Source: §1 · p. 1
- Video-level weak supervision leaves person locations and actions unannotated, making it a simpler, lower-cost alternative to person-level supervision.It also creates uncertain inputs: proposal counts vary, many proposals are outliers, and long videos contain irrelevant frames.Source: §1 · p. 2
- SAM selects discriminative proposals and frames by assuming key people and frames are highly related, using dense relations to select features and sparse relations to embed them.The approach trained without full supervision achieves comparable performance to prior methods on the volleyball dataset.Source: §1 · p. 2
- The paper introduces weakly supervised GAR and the NBA benchmark, and proposes SAM to find effective person- and frame-level representations.The benchmark is collected under video-level supervision, without person-level labels.Source: §1 · p. 2
2 Related Work
Prior GAR methods use person-level supervision, and existing datasets have limitations in scale, quality, or activity structure. This work instead studies video-level supervision and applies relational reasoning to select relevant people from proposals.
2 Related Work
- Earlier GAR methods include two-stage aggregation and end-to-end detection, action inference, and group recognition, but still require person-level action labels and bounding boxes.A prior event-and-key-actor method avoided individual action labels but still required bounding-box annotation for 9,000 frames.Source: §2 · p. 3
- Existing GAR datasets are limited by annotation burden, and the popular Volleyball Dataset contains wrong labels; the paper's NBA benchmark adds long-term and fast-moving activities.NCAA also has dead video links and some activities recognizable from one key frame.Source: §2 · p. 3
- Relational reasoning has been used across vision tasks, including GAR methods modeling spatial relationships; this work uses it to choose relevant people from proposals under weak supervision.Examples include relational networks for visual question answering and relation modules for object detection.Source: §2 · p. 4
3 Weakly-supervised Group Activity Recognition
Weakly supervised GAR uses only video-level group labels, omitting each person's location and action. The paper introduces the NBA dataset, annotated using website logs and containing long, fast-moving basketball activities.
3.1 Weakly-supervised Setting
- Weakly supervised GAR recognizes a group activity from a video using only its video-level label during training, without person locations or action labels.The paper motivates this setting as more practical for real-world applications and easier for benchmark annotation.Source: §3.1 · p. 4
3.2 The NBA Dataset for Weakly-supervised GAR
- The NBA dataset contains basketball clips labeled with group activity only and no person-level annotations.The paper describes it as the largest and most challenging GAR benchmark to its knowledge.Source: §3.2 · p. 4
3.1 Weakly-supervised Setting
- The dataset annotation requires one group-activity label per clip, rather than each person's boxes and actions, under the paper's rough equal-labor-per-annotation assumption.The paper estimates the fully supervised setting as K boxes, K actions, and one group label, versus one group label for weak supervision.Source: §3.2 · p. 4
3.2 The NBA Dataset for Weakly-supervised GAR
- The dataset draws on 181 NBA games from the 2019 season, with long-term activity structure and fast-moving players that make person-level labeling difficult.The videos were collected from the web and are available upon request due to copyright restrictions.Source: §3.2 · p. 5
- An automatic annotation pipeline filters NBA website logs, matches valid records to frame timers using Tesseract-OCR, and segments videos by the resulting time points.Segments are saved at a fixed length.Source: §3.2 · p. 5
- The benchmark contains 9,172 six-second clips from 181 videos, divided into 7,624 training and 1,548 testing clips across nine activities.Videos are 1920 × 1080 and subsampled to 12fps; dunk and turnover are excluded for limited sample size, and free-throw is omitted as easy to distinguish.Source: §3.2 · p. 5
4 Approach
The approach builds social representations from person and frame features, then selects effective instances using their relationships. SAM implements this selection by pruning a dense relation graph and embedding the selected features; implementation details specify the feature-extraction pipeline and optimization.
4.1 Mining Key Instances via Social Relationship
- SAM selects K_p discriminative person features and K_f effective frame representations to address noisy proposals and irrelevant frames.The selection operates in spatial and temporal domains, respectively.Source: §4.1 · p. 7, Figure 2 · p. 7
4.2 Social Adaptive Module (SAM)
- A learnable dense relation graph scores pairwise feature relationships, and nodes with the strongest incoming and outgoing connections rank as most related.The relation matrix is computed from two feature embeddings and a matrix multiplication; the top-K nodes are retained.Source: §4.2 · p. 8
- SAM embeds the selected features using their sparse relation matrix, combining relational features with each selected feature through a residual connection.The method prunes a dense graph to a sparse graph so selection remains tied to the relation structure.Source: §4.2 · pp. 8–9
- SAM’s social assumption is that key instances are closely related, contrasting with prior methods that either model player-scene pairs or lack a mechanism for uncertain inputs.The authors state SAM could also be used upon those graph-based methods.Source: §4.1 · p. 7, §4.2 · p. 9
4.3 Implementation details
- The implementation detects and tracks people, extracts ResNet-18 features with RoIAlign, and uses ADAM for 30 training epochs.The approach excluding detection is optimized end-to-end; varying proposal counts are handled with a batch-size of 1 for the module.Source: §4.3 · p. 9
5 Experiments
Experiments show that SAM improves weakly supervised group activity recognition on NBA and Volleyball, while analyses examine proposal selection, activity confusion, and changing key players over time.
5.1 Quantitative Analysis on the NBA Dataset
- On NBA, selecting Kp = 8 persons from Np = 14 proposals improves Acc by 5.7% and Mean Acc by 4.1% over B1.B2, which builds relational embeddings without pruning, improves Acc by 2.2% and Mean Acc by 1.8%; applying SAM temporally to B3 yields the best Mean Acc.Source: §5.1 · p. 10
- On NBA, Spatial-SAM plus Temporal-SAM achieves the best Mean Acc and outperforms all baselines by a good margin.The combined model performs poorly on “3p-succ.”, which lacks long-term temporal structure.Source: Table 4 · p. 11, §5.1 · p. 11
- On NBA, “Ours w/o SAM” is hardly improved or worse than frame-level TSN and TRN, while dense-frame I3D and I3D+NLN perform poorly.The paper attributes the no-SAM result to noisy, irrelevant pre-detected proposals.Source: Table 4 · p. 11, §5.1 · p. 11
5.2 Qualitative Analysis on the NBA dataset
- Spatial-SAM remains above baseline across tested proposal counts, while Temporal-SAM improves Mean Acc when 4 < Kf < 10.Spatial-SAM is described as insensitive to Np and Kp; Temporal-SAM performs less well when Kf is too small or large, amid varying activity durations.Source: §5.2 · pp. 11–12, Figure 3 · p. 12
- NBA defense and offense activities are easily confused amid class imbalance, whereas 2-point and 3-point activities are relatively easy to distinguish.The paper relates 3-point shots to jumping behind the line without blocking, and 2-point shots to being blocked.Source: §5.2 · p. 12, Figure 3 · p. 12
- SAM’s top-K proposals change over an activity’s phases, focusing on ball handlers during preparation and shooting and players turning back during defensive rebound.The visualization describes the selected players as controlling or close to the basketball and forming a semantic group.Source: §5.2 · p. 12, Figure 4 · p. 13
5.3 Quantitative Analysis on Volleyball Dataset
- On Volleyball, the weakly supervised ResNet-18 approach surpasses almost all prior fully supervised methods, and its Inception-v3 version achieves the best performance.The ResNet-18 approach is the exception to the comparison with ARG built on Inception-v3; the paper reports that its weakly supervised approach goes far beyond weakly supervised ARG.Source: Table 5 · p. 14, §5.3 · p. 14
- On Volleyball, selecting K = 12 proposals from N = 16 improves accuracy by 1.6% with either Quan-N or Prob-N.The variants that build relational embeddings without pruning improve accuracy by only 0.9% and 0.4%.Source: §5.3 · p. 13, Table 5 · p. 14
6 Conclusions
The paper introduces weakly supervised GAR and an NBA dataset, then proposes SAM to find discriminative person- and frame-level features. It reports state-of-the-art results on two datasets and highlights relational reasoning as useful for visual understanding.
6 Conclusions
- The approach achieves state-of-the-art results on two datasets while automatically attending to key proposals and frames.SAM uses the assumption that discriminative features are highly related; it can be plugged into existing frameworks and trained end to end.Source: §6 · p. 14
- The work introduces a more practical weakly supervised GAR setting and collects a larger, more challenging NBA dataset from high-resolution basketball videos.The setting uses video-level labels and is described as friendlier to real-world scenarios.Source: §6 · p. 14
- Social relationships among visual entities may help high-level semantic understanding, with future work targeting weakly annotated or unannotated visual data.Source: §6 · p. 14