Source-linked AI summary
SoccerNet-v2: A Dataset and Benchmarks for Holistic Understanding of Broadcast Soccer Videos
Adrien Deliège, Anthony Cioppa, Silvio Giancola, Meisam J. Seikavandi, Jacob V. Dueholm, Kamal Nasrollahi, Bernard Ghanem, Thomas B. Moeslund, Marc Van Droogenbroeck
TL;DR
Broadcast soccer videos require higher-level understanding for automatic editing, beyond the framewise analysis emphasized by earlier computer-vision work. SoccerNet-v2 provides a large manual-annotation corpus and benchmarks for action spotting, camera-shot understanding, and replay grounding. Its scope is broad, though the authors note that label noise remains present while still allowing performance improvements.
Problem
Earlier soccer computer-vision work focused on framewise elements, leaving higher-level understanding for camera-shot, replay, and advertisement decisions limited.
Method
The paper extends 500 untrimmed SoccerNet games with approximately 300k annotations and benchmarks three tasks: action spotting, camera-shot segmentation and boundary detection, and replay grounding.
Results
SoccerNet-v2 provides benchmark results, reproducible code, and public leaderboards across its three tasks.
Takeaways & Limitations
The dataset supports holistic study of broadcast soccer content, editing, and production through expanded actions and a novel replay-grounding task.
Takeaways & Limitations
The authors note that the annotations retain a reasonable level of label noise, although performance improvements remain possible.
Abstract
from arXiv · showhide
Understanding broadcast videos is a challenging task in computer vision, as it requires generic reasoning capabilities to appreciate the content offered by the video editing. In this work, we propose SoccerNet-v2, a novel large-scale corpus of manual annotations for the SoccerNet video dataset, along with open challenges to encourage more research in soccer understanding and broadcast production. Specifically, we release around 300k annotations within SoccerNet's 500 untrimmed broadcast soccer videos. We extend current tasks in the realm of soccer to include action spotting, camera shot segmentation with boundary detection, and we define a novel replay grounding task. For each task, we provide and discuss benchmark results, reproducible with our open-source adapted implementations of the most relevant works in the field. SoccerNet-v2 is presented to the broader research community to help push computer vision closer to automatic solutions for more general video understanding and production purposes.
1. Introduction
Broadcast soccer understanding matters for editing and production, but prior computer-vision work largely addressed framewise information rather than higher-level game content. SoccerNet-v2 responds with a large annotated corpus, three complementary tasks, reproducible benchmarks, and public leaderboards.
- Motivation: Broadcasting and commercial activities are major revenue sources for soccer clubs, motivating analysis of how broadcasts select shots around actions and players.
- Motivation: Automating video editing could affect millions of games beyond the approximately 10,000 annual Big Five games, but requires understanding both the game and broadcast production.
- Research gap: Prior soccer computer-vision work mainly localized fields, detected players, motion, pose, teams, balls, or pass feasibility at frame level.
- Research gap: Framewise information supports viewer experience and player statistics but falls short of higher-level understanding needed for camera-shot selection, replay selection, and advertisement placement.
- Contributions: Action spotting covers 17 action classes, camera-shot understanding addresses segmentation and boundary detection, and replay grounding retrieves replayed game actions.
- Contributions: The project provides reproducible benchmark results, code, and public leaderboards to support research in soccer understanding and broadcast production.
2. Related Work
Earlier video datasets established temporal activity understanding, while soccer datasets remained limited in action coverage or broadcast-production annotations. SoccerNet-v2 addresses this gap by combining broader soccer actions with editing-oriented tasks, especially replay grounding.
- Video understanding datasets: General video datasets progressed from trimmed classification toward fine-grained temporal or spatio-temporal annotations for longer videos.
- Soccer-related datasets: SoccerNet contains 500 complete games and 6k annotations but only three action classes, limiting task richness and interest.
- Soccer-related datasets: SoccerDB adds seven classes and player boxes but lacks a complete action set and editing annotations for full broadcast-production understanding.
- Soccer-related datasets: Other soccer datasets provide actions, shot transitions, and player boxes, but one has few annotations and reports neither experiments nor a proposed task.
- Broadcast production: Prior camera-shot studies classified views for replay detection, exciting-moment detection, or highlight generation using visual, field, player, or audio features.
- Replay grounding: Replay research generally detects replay shots, whereas grounding a replay to its corresponding salient broadcast action has been mostly overlooked.
- Replay grounding: SoccerNet-v2 introduces replay grounding to connect replay shots with salient broadcast moments, with possible uses in action retrieval and highlight production.
3. SoccerNet-v2 Dataset
SoccerNet-v2 is a large, densely annotated corpus of broadcast soccer videos covering actions, camera shots, and replays. Its annotations capture realistic broadcast imbalance and temporal context across multiple production-relevant categories.
- Dataset scale: ~300k annotations are temporally anchored within 764 hours of video from 500 SoccerNet games.The annotations were produced by 33 frequent soccer observers over approximately 1600 annotation hours.
- Actions: 110,458 actions cover 17 classes, averaging 221 actions per game or one action every 25 seconds.This represents 16x more timestamps and 14 additional classes than SoccerNet.
- Actions: Each action timestamp includes a binary visibility tag indicating whether the action is shown or must be inferred from the broadcast.Unshown actions account for 18% of actions and require temporal context beyond frame-based analysis.
- Camera shots: Abrupt, fading, and logo transitions represent 71.4%, 14.2%, and 14.2% of camera changes, respectively.Logo transitions are common in sports broadcasts despite being less typical in general videos or movies.
- Replays: Replay annotations bound each replay shot with two timestamps and link consecutive views of the same action to that action.Retrieving replayed actions is difficult because one to three replays may show different viewpoints absent from the original live broadcast.
4. Broadcast Video Understanding Tasks
SoccerNet-v2 defines three broadcast-video understanding tasks: action spotting, camera-shot analysis, and replay grounding. Together, they target semantic moments, editing transitions, and retrieval of replayed actions within complete games.
- Task overview: Three tasks are defined: action spotting, camera shot segmentation with boundary detection, and replay grounding.They address broadcast soccer understanding and production while also generalizing to broader long-video applications.
- Camera shot understanding: Camera-shot segmentation classifies each frame into 13 camera types, while boundary detection localizes transitions between shots.Segmentation is evaluated with mIoU, and boundary detection uses spotting mAP at a 1-second tolerance.
- Replay grounding: Replay grounding retrieves the timestamp of the action shown in a replay shot within the whole game.Grounding the replay to its action provides an estimate of the action’s importance.
5. Benchmark Results
SoccerNet-v2 benchmarks action spotting, camera-shot segmentation and boundary detection, and replay grounding using adapted reproducible methods. Results show CALF's strength in action spotting and replay grounding, while camera-boundary performance varies by transition type.
- Action Spotting: The benchmark adapts publicly available methods, including MaxPool, NetVLAD, AudioVid, and CALF, using ResNet video features for action spotting.CALF processes 2-minute chunks and combines temporal segmentation with spotting predictions for 17 action classes.
- Action Spotting: CALF performs best globally, on unshown instances, and on most action classes, while AudioVid performs best on shown instances and five of 17 classes.Audio features are particularly helpful for actions preceded or followed by the referee's whistle, but less useful for unshown instances.
- Camera Segmentation and Boundary Detection: Histogram leads camera boundary detection overall, while CALF performs best on fading transitions and Histogram remains effective on logos.Intensity and Content are particularly weak on logos, whereas classical methods are intrinsically tailored for abrupt transitions.
- Camera Segmentation and Boundary Detection: The camera benchmark evaluates segmentation with mIoU and boundary detection with mAP using basic CNN, CALF, Content, Histogram, Intensity, and CALF detection baselines.The task includes both temporal camera-shot labeling and transition localization.
- Replay Grounding: CALF achieves the best replay-grounding performance with 60-second chunks and the S2 sampling strategy.All methods perform best with chunk sizes around 60 seconds, suggesting a balance between insufficient and excessive temporal context.
6. Conclusion
SoccerNet-v2 releases a large annotated corpus and benchmarks for holistic broadcast-soccer understanding. It combines expanded action and camera tasks with novel replay grounding, reproducible code, and public leaderboards.
- Conclusion: The work expands action spotting, camera-shot segmentation and boundary detection, and introduces replay grounding.These tasks address holistic understanding of broadcast content, editing, and production.
- Conclusion: The authors provide reproducible experiment code and public leaderboards to drive research toward automatic holistic broadcast-video understanding and production.The stated scope also includes more generic video-understanding and production tasks.
7. Supplementary Material
The supplementary material specifies annotation rules, camera categories, annotation interfaces, file formats, annotator procedures, time costs, and human-level uncertainty measurements.
- Annotation Guidelines: Actions are annotated with single timestamps defined by explicit soccer-event rules, such as the ball crossing the line for a goal.Examples include substitutions, kick-offs, cards, offsides, free-kicks, penalties, fouls, clearances, shots, and corners.
- Camera Shot Definitions: The camera taxonomy includes 13 shot types, covering main-field views, goal-line and behind-goal cameras, close-ups, spider cameras, public views, and other artistic shots.The main center camera provides a wide field of view for understanding the game, while specialized cameras serve specific viewpoints or effects.
- Annotation Tools and Format: Annotators use separate tools to label actions, camera transitions, replay status, and replayed actions, with annotations stored in JSON files.Camera-transition records include fields such as change type, game time, camera label, linked action, position, and replay status.
- Annotation Process: Thirty-three engineering students and soccer fans completed approximately 1,600 annotation hours under detailed guidelines with support for ambiguous cases.Per-game annotation takes about 105 minutes for actions, 140 minutes for camera changes, and 70 minutes for replay-action association.
- Human-Level Performance: Timestamp annotations retain uncertainty despite well-defined rules, so the authors estimate human-level performance by comparing each annotator against the others on a shared match.Prior datasets show that temporal-boundary agreement can be imperfect, while some label noise can still permit performance improvements.