Source-linked AI summary
Detecting events and key actors in multi-person videos
Vignesh Ramanathan, Jonathan Huang, Sami Abu-El-Haija, Alexander Gorban, Kevin Murphy, Li Fei-Fei
TL;DR
Multi-person event recognition must distinguish the few people driving an event from the many others in the scene, without relying on expensive key-actor annotations. The paper tracks people, represents their tracks with recurrent networks, and applies time-varying attention for event recognition and detection. On a 257-game, 11-class basketball dataset with 14K annotations, the method outperforms state-of-the-art methods and localizes relevant players.
Problem
Multi-person videos contain many active people while only a small subset contributes to the event, and identifying those key actors requires expensive annotations.
Method
The model tracks people, encodes each track with an RNN, learns time-varying attention weights, and feeds attended features to another RNN for event classification and detection.
Results
The method outperforms state-of-the-art methods for isolated-clip classification and temporal event localization on a basketball dataset with 257 videos, 11 classes, and 14K dense annotations.
Takeaways & Limitations
The attention mechanism can identify relevant players without explicit key-player annotations, while supporting event recognition in multi-person videos.
Takeaways & Limitations
Tracking-based attention can remain focused on a player visible throughout an event instead of switching to the shooter, and performance is poorer for classes with little data.
Abstract
from arXiv · showhide
Multi-person event recognition is a challenging task, often with many people active in the scene but only a small subset contributing to an actual event. In this paper, we propose a model which learns to detect events in such videos while automatically "attending" to the people responsible for the event. Our model does not use explicit annotations regarding who or where those people are during training and testing. In particular, we track people in videos and use a recurrent neural network (RNN) to represent the track features. We learn time-varying attention weights to combine these features at each time-instant. The attended features are then processed using another RNN for event detection/classification. Since most video datasets with multiple people are restricted to a small number of videos, we also collected a new basketball dataset comprising 257 basketball games with 14K event annotations corresponding to 11 event classes. Our model outperforms state-of-the-art methods for both event classification and detection on this new dataset. Additionally, we show that the attention mechanism is able to consistently localize the relevant players.
1. Introduction
The paper addresses event recognition in multi-person videos, where many people are present but only a few drive the event. It introduces attention-based modeling and a densely annotated basketball dataset for this setting.
- Problem: Multi-person videos contain many active people, but only a smaller subset typically determines the main event.Identifying these key actors differentiates multi-person event recognition from single-person action recognition.
- Motivation: Key-actor annotations are expensive, motivating weakly supervised models that identify relevant people without explicit training annotations.The proposed model attends to key actors without being told who or where they are.
- Method: The model combines tracking, recurrent representations, and time-varying attention to select the most relevant person or track during an event.Attention integrates spatial and temporal information and can change as the event progresses.
- Dataset: The authors collect a basketball dataset with timestamp annotations for 11 event types across 257 videos, addressing the limited scale of prior multiperson datasets.The videos are approximately 1.5 hours long and contain events involving multiple people.
- Contributions: The method outperforms state-of-the-art methods on isolated-clip classification and temporal event localization while learning to attend to relevant players.These results are reported despite the absence of explicit annotations identifying relevant players.
2. Related Work
Prior video-recognition work includes engineered features, deep models, recurrent pooling, and weakly supervised localization, but largely targets single-actor clips. This paper applies attention to identifying relevant participants in multi-person events and introduces a corresponding dataset.
- Action recognition: Traditional video recognition used engineered features, while later end-to-end deep models and recurrent networks achieved competitive or state-of-the-art results.The paper builds on recurrent approaches while adding attention to event participants.
- Action localization: Weakly supervised localization methods often target short, single-actor videos, whereas other localization methods require training annotations.The paper instead addresses multi-person event recognition without explicit key-actor annotations.
- Attention: Earlier attention-based RNNs align input features with output words; this work uses attention to identify the most relevant person during different event phases.The attention target is a participant rather than a word, caption region, or video frame.
- Datasets: The paper contributes a multi-player basketball dataset with dense temporal event annotations in long videos.This dataset responds to the limited number of videos in prior multi-person datasets.
3. NCAA Basketball Dataset
The authors construct a dense, long-video basketball benchmark covering 11 event types and develop player detections for model input. The dataset provides clips, event timing, ball positions, and player boxes for evaluation and analysis.
- Collection: The dataset uses 257 recent NCAA basketball games, typically 1.5 hours long, and defines 11 key event types including successful or failed shots and steals.The games come from a subset of 296 NCAA games available on YouTube.
- Annotation: Annotators marked event end-points, while each event was assigned a four-second duration to provide temporal context and localization.The fixed duration avoids disagreement about event start times.
- Splits and clips: The videos were split into 212 training, 12 validation, and 33 test videos, yielding 11,436 training, 856 validation, and 2,256 test clips.Clips were four seconds long and subsampled to 6fps after filtering non-profile shots.
- Additional annotations: The dataset includes event labels and temporal boundaries, plus ball-position annotations for 850 test clips and player-box annotations on 9,000 training frames.These annotations support event evaluation and player-related analysis.
- Detection: The player detector achieved 0.7 average overlap at 0.8 recall on validation ground-truth boxes.Detections above confidence 0.5 produced 6–8 person detections per clip.
4. Our Method
The method recognizes multi-person events by combining global frame context with time-varying attention over player representations, while tracking-free attention provides an alternative when tracking is difficult.
- Motivation: Events in team sports are distinguished by actions of a small subset of players, motivating models that focus on event participants.For example, a basketball steal is defined by the passer and the player stealing the ball.
- Attention-based model: The model combines spatial and temporal attention to select the most relevant player track at each time-step.Detections are associated across frames through tracking, and attention weights can evolve with the event.
- Feature extraction: Each frame uses a 1024-dimensional global feature, while each player uses appearance and spatial features totaling 2805 dimensions.The player representation includes 1365 appearance dimensions and 1440 spatial dimensions.
- Event classification: The event-recognition LSTM uses the attended player feature to predict one of 11 event classes.The model is trained with a squared-hinge loss using class-specific weight vectors.
- Attention-based model: Player tracks are represented with a separate BLSTM that incorporates context from adjacent frames before attention combines player representations.The resulting attended player representation is passed to a unidirectional event-recognition LSTM.
- Tracking-free attention: A tracking-free model treats detections in each frame independently, allowing attention to switch flexibly between players when tracking is difficult.This variant uses player detection features directly instead of track-based BLSTM representations.
5. Experimental evaluation
Experiments on the NCAA basketball dataset evaluate event classification, temporal detection, and whether learned attention identifies relevant players. Attention-based models outperform baselines, while performance varies by event class and tracking can reduce attention selectivity.
- Experimental setup: The experiments cover event classification, event detection, and attention evaluation on the NCAA basketball dataset.Event classification uses isolated clips; detection uses untrimmed videos; attention is evaluated for shooter identification.
- Event classification: Global information combined with local player information outperforms models using only local player information or only global information for isolated-clip classification.The comparison is reported using mean average precision in Table 2.
- Event classification: Attention-based weighted player aggregation outperforms uniform averaging, tracking-based attention is slightly better than track-free attention, and MIL is less effective than the authors’ modeling variants.These comparisons concern isolated-clip classification settings.
- Event classification: Performance is poorer for classes with little data, while shot-based events benefit from attending to the shooter or defenders.The paper specifically identifies “slam dunk fail” as data-limited and highlights free-throws, layups, and 3-pointers as stronger cases.
- Event detection: Attention models outperform previous state-of-the-art methods for temporal localization, although detection is slightly worse than isolated-clip classification.“Steal” detection is particularly challenging because negative passing instances resemble steals.
- Attention analysis: Attention localizes meaningful court regions for layups, free-throws, and 3-pointers, but tracking-based attention may remain on one persistent defender instead of switching to the shooter.The tracking model’s persistence can bias attention toward players visible throughout the event.
6. Conclusion
The paper introduces an attention-based model for event classification and detection in multi-person videos that can identify key people without explicit key-person annotations. It also contributes a densely annotated basketball dataset and evaluates attention through shooter recognition and spatial visualizations.
- Conclusion: The model recognizes events and identifies key people in multi-person videos without explicit training annotations for those people.The conclusion describes the approach as an attention-based model for event classification and detection.
- Conclusion: The paper introduces a basketball dataset with dense event annotations and compares its method with state-of-the-art methods.The dataset provides the evaluation setting for the paper’s classification and detection results.
- Conclusion: Attention is evaluated for shooter recognition and through visualizations of the spatial locations attended by the model.These analyses examine whether the learned attention corresponds to event-relevant players and locations.