Source-linked AI summary
SoccerNet: A Scalable Dataset for Action Spotting in Soccer Videos
Silvio Giancola, Mohieddine Amine, Tarek Dghaily, Bernard Ghanem
TL;DR
The paper addresses the lack of scalable benchmarks for locating sparse events in long soccer broadcasts. It introduces SoccerNet, a large soccer-focused dataset and spotting task with baselines, achieving 67.8% mAP for minute classification and 49.7% Average-mAP for spotting. The dataset is intended to support further algorithm development and standardized evaluation.
Problem
Large-scale reference datasets for evaluating and comparing soccer action-localization methods are lacking, while manual annotation is costly.
Method
The paper constructs SoccerNet from soccer broadcasts and parsed match reports, defines timestamp-anchored event spotting, and provides classification and spotting baselines.
Results
67.8% mAP is achieved for minute classification, and 49.7% Average-mAP for event spotting.
Takeaways & Limitations
SoccerNet provides a scalable environment for further spotting-method development and standardized evaluation.
Takeaways & Limitations
The spotting baseline's best performance is described as far from satisfactory, motivating further algorithm development.
Abstract
from arXiv · showhide
In this paper, we introduce SoccerNet, a benchmark for action spotting in soccer videos. The dataset is composed of 500 complete soccer games from six main European leagues, covering three seasons from 2014 to 2017 and a total duration of 764 hours. A total of 6,637 temporal annotations are automatically parsed from online match reports at a one minute resolution for three main classes of events (Goal, Yellow/Red Card, and Substitution). As such, the dataset is easily scalable. These annotations are manually refined to a one second resolution by anchoring them at a single timestamp following well-defined soccer rules. With an average of one event every 6.9 minutes, this dataset focuses on the problem of localizing very sparse events within long videos. We define the task of spotting as finding the anchors of soccer events in a video. Making use of recent developments in the realm of generic action recognition and detection in video, we provide strong baselines for detecting soccer events. We show that our best model for classifying temporal segments of length one minute reaches a mean Average Precision (mAP) of 67.8%. For the spotting task, our baseline reaches an Average-mAP of 49.7% for tolerances $δ$ ranging from 5 to 60 seconds. Our dataset and models are available at https://silviogiancola.github.io/SoccerNet.
1. Introduction
Soccer broadcast analysis has practical value, but manual event annotation is costly and soccer events are sparse in long videos. The paper addresses this gap by defining soccer event spotting and introducing a scalable benchmark with baselines.
- Motivation: Manual annotation of soccer broadcasts can take over 8 hours for up to 2,000 annotations per game.The scale of European soccer requires substantial annotator effort and makes dataset construction expensive.
- Motivation: Soccer video understanding targets sparse, soccer-specific events within long broadcasts.The paper frames soccer broadcast understanding as a sub-problem of video understanding focused on a specialized event vocabulary.
- Contributions: SoccerNet contains 764 hours of video and 6,637 instances across Goals, Yellow/Red Cards, and Substitutions.The dataset is presented as a large-scale resource for soccer video understanding.
- Contributions: The paper defines event spotting as locating soccer events anchored to single timestamps in video.The contribution establishes the task around timestamp-anchored event definitions.
- Contributions: The minute classifier reaches 67.8% mAP, while the event spotter reaches 49.7% Average-mAP.These baselines cover video chunk classification and event spotting.
2. Related Work
Prior video datasets and soccer analytics studies provide limited support for large-scale action localization and standardized comparison. SoccerNet responds with a scalable soccer-focused dataset intended to support development and evaluation of spotting methods.
- Sports Analytics: Earlier soccer analytics methods rely on small, custom-made datasets containing at most a few dozen games.This limits their use as large-scale reference benchmarks.
- SoccerNet: SoccerNet is designed to be expanded and enriched with varied annotations to match the scale of soccer analytics.The proposed dataset is explicitly soccer-centric and scalable.
- Implications: Large-scale soccer datasets are intended to encourage algorithmic development and standardized evaluation.The paper connects scalable resources with development of sports analytics solutions.
- Datasets: Existing video datasets largely support classification rather than activity localization.Examples include Hollywood2, HMDB, UCF101, Sports-1M, Youtube-8M, and Kinetics.
- Datasets: Action-localization benchmarks are fewer, with prior resources spanning general activities, dense activities, and temporal segmentation.THUMOS14, ActivityNet, Charades, and AVA illustrate the smaller and varied benchmark landscape.
- SoccerNet: With one event every 6.9 minutes, SoccerNet contains sparse events in long untrimmed videos, making localization more difficult.Annotations are parsed at one-minute resolution and refined manually to one-second resolution.
3. Spotting Sparse Events in Soccer
The paper defines soccer events as actions anchored to single timestamps and introduces spotting as locating those anchors rather than temporal boundaries. It also proposes tolerance-based evaluation for sparse soccer events.
- Event Definition: Soccer events are actions anchored to a single time instance within a specific context and rule set.The definition addresses ambiguity in assigning temporal boundaries to soccer actions.
- Motivation: Existing benchmarks mainly use activities with temporal boundaries, whereas spotting targets atomic events in video.Prior work addresses boundary ambiguity through annotator consensus or dense short-duration annotations.
- Event Definition: Soccer rules provide a structured vocabulary for anchoring events such as goals, fouls, cards, and penalty kicks.This exploits well-established rules to reduce ambiguity in live broadcast actions.
- Spotting: Spotting finds an event’s anchor time instead of identifying its temporal boundaries with tIoU.A candidate is considered a hit when it falls within tolerance δ of the target anchor.
- Evaluation: A candidate spot is positive when it falls within tolerance δ of an event anchor, enabling AP and mAP evaluation for each tolerance.The paper also defines Average-mAP across predefined tolerances below one minute.
4. Data Collection
SoccerNet combines online soccer videos, clock-based synchronization, and match-report parsing to build a large annotated benchmark. Its annotations are cheaply obtained at minute resolution and refined to one-second anchors when needed.
- Collection Pipeline: The dataset is built by collecting online videos, synchronizing game and video times, and parsing web match reports for aligned annotations.These three steps form the core data-collection pipeline.
- Collecting Videos: 500 games provide 764 hours of untrimmed video, split randomly into 300 training, 100 validation, and 100 testing games.The collection includes two videos per game, one for each half period.
- Game Synchronization with OCR: OCR of the game clock at multiple video instants estimates the linear mapping between game time and video time.RANSAC handles clock misreadings while enforcing a unitary gradient and checks temporal integrity.
- Collecting Event Annotations: 6,637 events are extracted from match reports for goals, cards, and substitutions, then manually refined from minute-level annotations to one-second anchors.Anchors are defined using soccer rules, such as the instant the ball crosses the goal line for a goal.
- Scalability: Minute-level web annotations are free, while producing complete one-second annotations is estimated to take less than 10 minutes per game.This supports the claim that the dataset can be scaled cheaply and easily.
5. Experiments
The experiments evaluate minute-level event classification and temporal event spotting using multiple video representations, pooling methods, imbalance strategies, and training resolutions. NetVLAD with ResNet-152 features performs strongly for classification, while spotting performance depends on chunk duration and annotation resolution.
- Video Chunk Classification: ResNet-152 outperforms C3D and I3D with the custom CNN pooling layer that gathers temporal context.The authors attribute this pattern to temporal information already captured by C3D and I3D, unlike the spatially focused ResNet representation.
- Video Chunk Classification: VLAD-based pooling methods rank highest, followed by deep Fisher Vector and Bag-of-Words methods.NetRVLAD performs similarly to, or better than, NetVLAD while using averages rather than residuals.
- Video Chunk Classification: Weighting slightly improves classification mAP, whereas downsampling produces the worst results because less data is used per training epoch.Augmenting training with one-second annotations yields slightly better classification results.
- Spotting: 40.6% Average-mAP is achieved by the watershed spotting baseline trained on one-minute chunks, with performance decreasing more gradually as tolerance falls below 60 seconds.The spotting baselines are applied using sliding one-second windows, and the best spotting performance is described as far from satisfactory.
- Spotting: 50% Average-mAP is obtained by baseline (i) on 20-second chunks, while five-second chunks produce a performance drop.Across chunk durations, spotting mAP plateaus above the training chunk length and decreases below that threshold.
6. Future Work
The paper proposes extending SoccerNet with richer temporal modeling, additional event classes, semantic relationships, and audio-based cues. These additions are intended to enrich the dataset and improve soccer video understanding.
- 6. Future Work: Future work proposes RNNs to model the temporal evolution of soccer games.
- 6. Future Work: Adding more soccer-event classes could enrich SoccerNet and support learning potential causal relationships between events.The paper gives cards resulting from fouls as an example of such a potential relationship.
- 6. Future Work: Player, ball, and field semantics could provide additional information for soccer video understanding.
- 6. Future Work: Audio tracks and visual-audio sentiment analysis are proposed for localizing salient moments in games.
- 6. Future Work: The released 506,137 one-second match commentaries could support soccer-video event captioning.
7. Conclusion
The conclusion presents SoccerNet as a large-scale benchmark for soccer broadcast understanding, combining minute-level classification with fine-grained event spotting. The reported results are 67.8% mAP for classification and 49.7% or 40.6% Average-mAP for spotting, depending on annotation resolution.
- 7. Conclusion: SoccerNet provides a benchmark for soccer analysis using large-scale annotated soccer-game broadcasts and a defined event-spotting task.
- 7. Conclusion: 67.8% mAP is obtained for minute-level classification using ResNet-152 features and NetVLAD pooling with a 512-long vocabulary and coarse annotations.
- 7. Conclusion: 49.7% Average-mAP is achieved for spotting with fine annotations, compared with 40.6% using only weak annotations.
- 7. Conclusion: The authors suggest that focusing research effort on spotting could improve the state of the art in detection tasks.
8. Supplementary Material
The supplementary material reports dataset splits, feature dimensionality findings, spotting metric definitions, and quantitative and qualitative results across event classes and training window sizes.
- Dataset split: Training, validation, and testing contain 300, 100, and 100 games, respectively, with events assessed as equally distributed across the sets.
- Feature dimensionality: Reducing frame-feature dimensionality to 512 minimizes variance loss for ResNet, C3D, and I3D while reducing the complexity of subsequent pooling layers.
- Spotting metric: A candidate spot is positive when it falls within tolerance δ of a ground-truth spot, with Average mAP estimated from the mAP–tolerance curve between 5 and 60 seconds.
- Spotting results: 73.0% AP is achieved for Goals, compared with 59.3% for Substitutions and 52.1% for Cards using the best classification model and center-segment spotting baseline.
- Window-size results: A 20-second training window gives the best classification result, while performance drops with 5-second windows and requires one-second annotations.
- Qualitative results: Qualitative results show activations around ground-truth events over roughly 60 seconds, while smaller training windows reduce activation width to the chunk size.