Source-linked AI summary
BasketEvent: Understanding Who Did What and When in Basketball Videos
Yu Zhang, Jiayuan Rao, Haoning Wu, Weidi Xie
TL;DR
Basketball understanding requires identifying who is responsible, what event occurs, and when decisive evidence appears, while existing approaches often separate perception from recognition. BasketEvent supplies player-grounded labels and temporal annotations, and PlayNet models player-centric interactions to predict events with temporal evidence; it outperforms representative baselines and captures coarse temporal evidence.
Problem
Basketball understanding requires identifying the responsible player, event, and decisive evidence interval, but existing perception and recognition work typically treats these tasks separately.
Source: §1 · p. 2Method
BasketEvent grounds NBA event labels to players and annotates 1,000 intervals; PlayNet tracks and identifies entities, models player, ball, and court interactions, and predicts player-level events with temporal evidence.
Source: §1 · p. 2Results
PlayNet significantly outperforms representative video-level and crop-based baselines and captures meaningful coarse temporal evidence; against the strongest competitor, F1-score rises from 0.46 to 0.68.
Source: §5.4 · p. 12, §6 · p. 15Takeaways & Limitations
Grounding events to players and time yields more fine-grained, interpretable, and temporally precise basketball video understanding.
Source: §1 · p. 3
Abstract
from arXiv · showhide
Comprehensive basketball video understanding requires resolving not only what event occurs, but also who is responsible and when the key evidence appears. However, exist- ing methods typically treat spatial perception and semantic recognition as isolated tasks, failing to ground events to individual players or pinpoint their temporal boundaries within complex collective dynamics. To bridge this gap, we introduce BasketEvent, a player- centric basketball event understanding dataset curated from real NBA broadcasts. In BasketEvent, event labels are grounded to the responsible players, and a manually an- notated subset of 1,000 samples with precise event intervals is provided to evaluate tem- poral evidence localization. Based on this data, we propose PlayNet, a player-centric reasoning framework that maps basketball videos to player-level event predictions with temporal evidence. Concretely, PlayNet tracks key entities, associates player identities, and reasons about events by modeling player-player, player-ball, and global court inter- actions, while aggregating sparse temporal evidence via gated pooling. Extensive experi- ments demonstrate that PlayNet significantly outperforms representative video-level and crop-based baselines, proving the superiority of player-centric modeling for fine-grained sports video understanding. Our data, code, and models will be made publicly available.
1 Introduction
Basketball event understanding must identify the responsible player, event, and decisive evidence interval, but existing perception and recognition work typically addresses these separately. BasketEvent and PlayNet target this gap with player-grounded labels, temporal annotations, and player-centric reasoning.
1 Introduction
- Basketball events can depend on brief cues amid fast, multi-player interactions, so understanding requires identifying who is responsible, what happened, and when decisive evidence appears.Examples include a shot release, hand-ball interaction, transient contact, or possession change.Source: §1 · pp. 1–2
- Existing methods often separate spatial perception from semantic recognition, and benchmarks largely test event recognition rather than identifying responsible players and decisive moments.Perception-focused methods recover player locations, while recognition-focused methods often make clip-level predictions.Source: §1 · p. 2
- BasketEvent uses real NBA broadcasts to provide player-grounded event labels and manually annotates 1,000 samples with precise event intervals for temporal localization evaluation.The automatic pipeline associates play-by-play events with responsible athletes; temporal intervals are manually annotated.Source: §1 · p. 2
- PlayNet tracks players and the ball, associates player identities, and models player-player, player-ball, and court context to predict player-level events with supporting intervals.Its gated evidence aggregation pools sparse, discriminative temporal cues.Source: §1 · pp. 2–3
- PlayNet outperforms conventional video-level and crop-based event recognition, supporting more fine-grained, interpretable, and temporally precise basketball understanding.The paper attributes this to grounding events in both players and time.Source: §1 · p. 3
2 Related Work
Prior sports-video research covers event recognition, localization, tracking, and interaction modeling, but player-centric event understanding remains underexplored. BasketEvent and PlayNet address this gap by grounding basketball events to responsible players and modeling their interactions with others and the ball.
2 Related Work
- Existing sports and basketball benchmarks generally lack fine-grained player-level event annotations, motivating a benchmark that grounds complex events to responsible players.Prior basketball datasets focus on group activity, action localization, or identity-aware captioning.Source: §2 · p. 3
- Sports-video methods often identify what happened and when, while identifying the responsible player and reasoning about interactions remains underexplored, particularly in basketball.The paper frames its task as jointly recognizing event category, responsible player, and supporting temporal evidence.Source: §2 · p. 3
- Prior interaction models use actor relations and broader scene context, while object-centric models represent human-object interactions; PlayNet applies these ideas to player and ball tokens for basketball events.Its modeling sequence covers global-player interaction, player-ball interaction, and temporal evidence.Source: §2 · pp. 3–4
3 Dataset
BasketEvent combines player-grounded event annotations from NBA broadcasts with a manually annotated temporal-evaluation subset. Its dataset design supports evaluation of event recognition, responsible-player assignment, and temporal evidence localization.
3 Dataset
- BasketEvent is presented as a benchmark for fine-grained basketball understanding, grounding events to responsible players and including a subset for temporal evidence localization.The dataset section covers construction, statistics, and a temporally annotated subset.Source: §3 · p. 4
3.1 Dataset Construction
- The construction pipeline aligns NBA play-by-play records with broadcast clips, labels six event categories from structured fields, and parses descriptions for four additional categories.The additional categories are Assist, Block, Steal, and Jump Ball; overlapping segments are merged with all valid player-event pairs attached.Source: §3.1 · p. 5
3.2 Data Statistics
- 35k broadcast videos spanning 90.6 hours provide 51k player-level event samples, with player and ball trajectories alongside event semantics.Table 1 compares dataset scale and annotation granularity across representative sports datasets.Source: §3.2 · p. 5, Table 1 · p. 6
- The game-level split contains 189 training games and 37 test games, with 34,931 videos and 51,663 player-level samples overall.Splitting by game is intended to avoid near-duplicate clips across splits and evaluate generalization to unseen games.Source: §3.2 · p. 6
3.3 Temporal Evaluation Subset
- The temporal-evaluation subset contains 1,000 manually annotated player-level samples, 100 per event category, drawn exclusively from unseen test games.Start and end timestamps mark the interval containing the target event; the subset jointly evaluates player identification, event recognition, and temporal localization.Source: §3.3 · pp. 6–7
4 Method
PlayNet formulates basketball understanding as player-level what-who-when prediction, then uses identity-grounded trajectories, visual and interaction reasoning, and gated temporal aggregation to produce event predictions with coarse evidence intervals.
4.1 Problem Formulation
- Each event instance links a responsible player identity and event category to the interval containing its decisive visual evidence, allowing multiple events per clip.This makes the player-event instance, rather than the video alone, the prediction unit.Source: §4.1 · p. 7
4.2 Identity-grounded Entity Trajectories
- PlayNet tracks on-court players and the ball, then associates each player trajectory with a roster identity using sampled crops and roster context.The resulting player and ball trajectories provide identity-grounded entities for later reasoning.Source: §4.2 · pp. 7–8, Figure 3 · p. 8
4.3 Trajectory-guided Visual Encoding
- A TimeSformer encodes each clip into spatiotemporal features, and ROIAlign uses entity bounding boxes to extract player and ball features.Geometric and entity-type embeddings add box position and shape information and distinguish players from the ball.Source: §4.3 · p. 8
4.4 Interaction-aware Evidence Aggregation
- Entity tokens attend to the global feature map and then to other players and the ball, incorporating court context and within-frame relations.These stages let player tokens incorporate information from teammates, opponents, and the ball.Source: §4.4 · p. 9
- PlayNet pools player tokens within clips, aggregates player representations across clips, and uses learned gate weights to focus on evidence clips and provide coarse temporal grounding.The ball is an auxiliary interaction entity, not a directly classified event instance.Source: §4.4 · pp. 9–10
4.5 Training and Inference
- Training uses a player-level weighted cross-entropy loss with label smoothing and bootstrapped targets, while inference pairs roster identity and predicted event with the highest-weight clip.The background class is down-weighted; the selected clip supplies the temporal evidence interval.Source: §4.5 · pp. 10–11
5 Experiments
The experiments evaluate PlayNet on player-level event recognition and temporal evidence localization, comparing it with crop-based baselines and testing component and sampling choices. PlayNet leads the recognition metrics, while ablations examine how interactions, context, and temporal coverage relate to performance.
5.1 Implementation Details
- PlayNet uses 12 clips of 8 frames, a pretrained TimeSformer backbone, and ROIAlign player and ball tokens, and is trained for 10 epochs with AdamW.The reported setup uses learning rate 5 × 10−5 and batch size 8.Source: §5.1 · p. 11
5.2 Evaluation Metrics
- In the 1,000-sample temporal subset, localization uses the clip with the highest gate weight and counts a hit only when event classification is correct and overlap exceeds θ.Recognition requires matching both the event category and responsible player; the evaluation reports Rec@1/3/5 and macro F1, plus Hit@0.3/0.5.Source: §5.2 · pp. 11–12
5.3 Baselines
- R3D, SlowFast, and TimeSformer are compared using player-region crops, while temporal localization is reported only for PlayNet because crop baselines provide no clip-level evidence.All compared methods use the same player-level event labels.Source: §5.3 · p. 12
5.4 Quantitative Results
- PlayNet outperforms all baselines across recognition metrics, improving F1-score from 0.46 to 0.68 and Rec@1 from 0.45 to 0.70 over the strongest competitor.It also achieves 0.42 Hit@0.3 and 0.38 Hit@0.5 on the temporal subset.Source: §5.4 · p. 12, Table 2 · p. 13
5.5 Ablation Studies
- 0.68 F1-score, 0.70 Rec@1, 0.89 Rec@3, and 0.95 Rec@5 are achieved by the full model, which also exceeds ROI+Gate on F1-score and Rec@1.Removing player-player interactions, ball dynamics, or global context degrades recognition; w/o Clip SA has the highest Hit@0.3, unlike the full model’s recognition lead.Source: Table 3 · p. 13, §5.5 · p. 13
- 0.68 F1-score and 0.70 Rec@1 are achieved at FPS=4 and M=12, the best overall recognition setting in the sampling ablation.Increasing M from 4 to 12 boosts performance, while M=16 is slightly lower across metrics; FPS=8 offers marginal high-recall gains but slightly lower F1-score.Source: §5.5 · pp. 13–14, Table 4 · p. 14
5.6 Qualitative Results
- Qualitative examples include correct single- and multi-event predictions, as well as errors in identifying the responsible player or localizing brief evidence.The reported failure cases occur when players are spatially close or discriminative evidence is temporally brief.Source: §5.6 · p. 14
6 Conclusion
The paper frames basketball understanding as resolving who did what and when, beyond conventional video-level classification. It introduces BasketEvent and PlayNet, reporting stronger recognition than representative baselines and coarse temporal evidence from clip-level gates.
6 Conclusion
- Figure 4 pairs sampled frames with player-grounded who, what, and when predictions; ground truth is black, with correct and incorrect predictions marked green and red.Source: Figure 4 · p. 15
- PlayNet significantly outperforms representative video-level and crop-based baselines, while temporal evaluation finds its clip-level gates capture meaningful, coarse-grained event evidence.The paper presents the task, benchmark, and framework as a foundation for structured, fine-grained basketball video understanding.Source: §6 · p. 15
- BasketEvent grounds event labels to responsible players and includes a manually annotated subset for temporal evidence localization.The framework tracks players and the ball, associates trajectories with identities, and models court context, player-player and player-ball relations, and sparse temporal evidence.Source: §6 · pp. 14–15