Source-linked AI summary
A Context-Aware Loss Function for Action Spotting in Soccer Videos
Anthony Cioppa, Adrien Deliège, Silvio Giancola, Bernard Ghanem, Marc Van Droogenbroeck, Rikke Gade, Thomas B. Moeslund
TL;DR
Action spotting localizes human-induced events at single timestamps, despite sparse annotations and missing action boundaries. The paper uses a context-aware loss to incorporate surrounding frames, improving results on SoccerNet and ActivityNet. It also illustrates how action semantics can be applied to automatic highlight generation.
Problem
Action spotting must localize events from single timestamps despite sparse labels, absent start and end times, and temporal discontinuities between adjacent frames.
Source: §1 · p. 2Method
The method uses time-shift encoding and a context-aware segmentation loss to train on frames according to their relative location around actions.
Source: §3.1 · p. 3, §3.2 · p. 4Results
49.7% to 62.5% Average-mAP on SoccerNet, with improved activity proposal and detection results on ActivityNet.
Source: §1 · p. 2Takeaways & Limitations
The paper qualitatively illustrates how action semantics can support automatic highlights generation, including sequences related to unannotated actions.
Source: §6 · p. 8, §5 · p. 8Actions in the first five minutes of a half are harder to spot, and halftime substitutions annotated at the second-half opening are not spotted in 28% of test matches.
Source: §4.1 · p. 6
Abstract
from arXiv · showhide
In video understanding, action spotting consists in temporally localizing human-induced events annotated with single timestamps. In this paper, we propose a novel loss function that specifically considers the temporal context naturally present around each action, rather than focusing on the single annotated frame to spot. We benchmark our loss on a large dataset of soccer videos, SoccerNet, and achieve an improvement of 12.8% over the baseline. We show the generalization capability of our loss for generic activity proposals and detection on ActivityNet, by spotting the beginning and the end of each activity. Furthermore, we provide an extended ablation study and display challenging cases for action spotting in soccer videos. Finally, we qualitatively illustrate how our loss induces a precise temporal understanding of actions and show how such semantic knowledge can be used for automatic highlights generation.
1. Introduction
Soccer video systems can identify spatial details, but understanding and locating salient actions in time remains challenging. The paper introduces a context-aware loss for action spotting and reports gains on SoccerNet and ActivityNet.
1. Introduction
- Automating broadcast production tasks requires recognizing salient game actions, a higher-level understanding beyond spatial information such as player, ball, and camera positions.The introduction points to live production, commentary, and highlight generation as applications of action understanding.Source: §1 · p. 1
- 49.7% to 62.5% Average-mAP on SoccerNet, while ActivityNet proposal and detection results also improve with the context-aware loss.The introduction reports a 0.15% improvement in activity proposals and a 0.38% improvement in activity detection on ActivityNet.Source: §1 · p. 2
- The loss uses temporal context around an action spot, reflecting that action annotations are sparse, omit start and end times, and can differ between adjacent frames.Frames far from an action are penalized heavily, with the penalty decreasing closer to it; frames just before are not penalized, while those just after are heavily penalized.Source: Figure 1 · p. 1, §1 · p. 2
2. Related Work
Prior work addresses broadcast segmentation, salient-moment detection, sports analytics, and temporal activity localization. This paper positions action spotting as a distinct task and uses temporal context around action spots to improve it.
2. Related Work
- Broadcast-video research has moved from camera-shot segmentation and production-pattern analysis toward semantic and machine-learning methods for identifying salient game actions.Prior approaches include Bayesian frameworks, recurrent aggregation of hand-crafted or deep features, and attention using pixel or player-location information.Source: §2 · p. 2
- Sports datasets such as Sports-1M, MLB-Youtube, and GolfDB target specific sports tasks; SoccerNet provides the action-spotting benchmark used here.The related-work discussion describes SoccerNet as analyzing deep frame-feature extraction and aggregation for soccer action spotting.Source: §2 · p. 2
- Activity-localization methods predict proposals or temporal boundaries, while action spotting uses a single temporal anchor and differs in its annotation requirements.The discussion contrasts boundary-based methods including BMN with spotting approaches, noting that ActionSearch requires sequences of temporal annotations not readily available outside ActivityNet.Source: §2 · p. 2
- The proposed context slicing divides the vicinity of each action into six temporal segments, with different training behavior for distant, pre-action, post-action, and transition frames.Time-shift encoding represents each frame's relative location to its closest past or future action and contains richer temporal context than the original binary spotting annotation.Source: Figure 2 · p. 3, §2 · p. 3
3. Methodology
The method re-encodes sparse single-frame action annotations for temporal segmentation and spotting, then combines context-aware segmentation and YOLO-like spotting losses. Temporal context determines how segmentation scores are trained around each action, while the spotting module predicts action locations and classes.
3.1. Encoding
- Time-shift encoding assigns each frame a class-wise offset to its closest past or future action, selected according to which action’s temporal context dominates.Frames around an action are divided into temporal segments; the encoding distinguishes regions where an action is not foreseeable, uncertain, present, or no longer occurring.Source: §3.1 · p. 3
- YOLO-like encoding represents each ground-truth action with a presence indicator, normalized frame location, and one-hot class label.The action vectors are arranged as a matrix with one row per ground-truth action.Source: §3.1 · p. 4
3.2. Loss and Network Design
- The segmentation loss uses the time-shift encoding to train per-frame class scores, revising the per-score loss to zero out scores that satisfy the specified margins.The segmentation loss is then computed over the video’s frames and classes.Source: §3.2 · p. 4
- The network combines temporal segmentation and spotting losses, with the spotting loss matching ground-truth actions to predictions and regularizing unmatched prediction confidence.Its architecture uses frame features and a temporal CNN, followed by segmentation and spotting modules; spotting predictions include confidence, location, and class values.Source: §3.2 · pp. 4–5
4. Experiments
The experiments evaluate the context-aware loss for SoccerNet action spotting and ActivityNet activity proposals and detection. They report improved results on ActivityNet, strong SoccerNet performance, and analyses of how context, ablations, and event conditions affect spotting.
4.1. Experiments on SoccerNet
- SoccerNet evaluates three action classes using provided I3D, C3D, or ResNet frame features, and measures Average-mAP over localization tolerances from 5 to 60 seconds.Training uses 240-feature-frame chunks, corresponding to two minutes of raw video, with an action randomly located in action-centered chunks.Source: §4.1 · p. 5
- The optimized model establishes state-of-the-art SoccerNet action-spotting performance across all tested frame-feature sets, with ResNet performing best.The paper reports robust performance near optimal context-slicing parameters and a substantial decline when slicing is strongly reduced.Source: §4.1 · pp. 5–6, Table 1 · p. 5
- Ablations report Average-mAP increasing from 46.8% to 62.5% with prediction matching, from 57.8% to 62.5% with context slicing, and from 59.0% to 62.5% with segmentation margins.The auxiliary segmentation task improves performance from 58.9% to 62.5%; using segmentation without context slicing gives 57.8%, below omitting segmentation at 58.9%.Source: §4.1 · p. 6, Table 2 · p. 6
- Actions in the first five minutes of a half-time are harder to spot; removing break-time substitutions annotated at second-half starts raises Average-mAP from 62.5% to 63.2%.These substitutions occur in 28% of test matches and represent 5% of all substitutions; none are spotted by the model.Source: §4.1 · p. 6, Figure 4 · p. 6
- Actions close to another action are more difficult to spot, and performance eventually plateaus even for actions farther apart.The analysis groups ground-truth actions by distance to the closest other action and computes Average-mAP for each bin.Source: §4.1 · pp. 6–7, Figure 5 · p. 7
4.2. Experiments on ActivityNet
- ActivityNet experiments replace BMN’s temporal evaluation module loss with the context-aware segmentation loss and tune shared slicing parameters on validation AUC.The evaluation covers activity proposal and activity detection tasks using the dataset’s ResNet features at 5 fps.Source: §4.2 · p. 7
- 0.15% and 0.12% gains on ActivityNet activity proposal AR@100 and AUC, and a 0.38% gain on activity detection Average-mAP, are reported over BMN.The architecture and parameter count remain unchanged when the BMN temporal evaluation module loss is replaced.Source: §4.2 · p. 7
5. Automatic Highlights Generation for Soccer
Temporal segmentation surfaces unannotated soccer actions, and those proposals can be incorporated into automatically generated match highlights.
5. Automatic Highlights Generation for Soccer
- High segmentation scores also identify unannotated goal opportunities and unsanctioned fouls that can be useful highlight material.A goal opportunity appears around the 29th minute in the example match.Source: §5 · pp. 7–8, Figure 7 · p. 8
- 80% of sequences scoring above η = 0.5 were judged to be goal opportunities by both observers.Because these opportunities are unannotated, the authors report precision rather than recall.Source: §5 · p. 8, Figure 8 · p. 8
- The highlights generator selects clips around goal or card spots and high-scoring goal opportunities, then assembles them chronologically.The selected clips start 15 seconds before a spot and end 20 seconds after; substitutions are omitted.Source: §5 · p. 8
6. Conclusion
The paper introduces context-aware segmentation and spotting losses for soccer action spotting and tests the segmentation loss on ActivityNet. It reports improved benchmark results and uses segmentation to identify unannotated actions for highlights.
6. Conclusion
- Average-mAP rose from 49.7% to 62.5% on SoccerNet, with most actions spotted within 10 seconds of their ground truth.The authors report a new state of the art on SoccerNet.Source: §6 · p. 8
- ActivityNet results improved by 0.15% in AR@100, 0.12% in AUC, and 0.38% in Average-mAP without changing the network architecture.The context-aware loss was added to the existing method.Source: §6 · p. 8
- The approach combines a context-aware segmentation loss with a YOLO-like spotting loss and iterative matching that does not require ordered predictions.The segmentation loss uses frame time-shifts from ground-truth actions.Source: §6 · p. 8
- Segmentation results were used to identify unannotated actions, including goal opportunities, and generate highlights without specific supervision.The conclusion describes this as a qualitative application.Source: §6 · p. 8
7. Supplementary Material
The supplementary material expands the notation, network, matching and time-shift encoding details, then reports additional benchmark analyses and qualitative examples.
7.1. Notations
- The notation defines class and frame counts, ground-truth and predicted action vectors, feature counts, and the temporal receptive field.It also lists class-specific context slicing parameters and common network-layer abbreviations.Source: §7.1 · p. 12
7.2. Detailed Network Architecture for SoccerNet
- The SoccerNet pipeline applies a temporal CNN to frame features, then uses segmentation and spotting modules to produce per-frame class scores and spot predictions.The temporal CNN mixes information across frames; the spotting module takes its features and segmentation scores as input.Source: §7.2 · pp. 12–13, Figure 9 · p. 13
7.3. Iterative One-to-One Matching
- Iterative one-to-one matching repeatedly pairs mutually closest remaining ground-truth and predicted locations, removing each matched pair before continuing.The illustrated example matches all ground-truth locations in two iterations.Source: §7.3 · p. 13, Figure 10 · p. 13, §7.4 · p. 14
7.4. Details on the Time-Shift Encoding (TSE)
- Time-shift encoding records distances to neighboring class-specific actions and selects the shift associated with the dominant influence; its loss is temporally continuous except at annotated action frames.Special cases cover video boundaries and classes absent from the video.Source: §7.4 · p. 14
7.5. Extra Analyses
- For cards, the model reaches 66% precision within 15 seconds; for substitutions it reaches 73%, while the baseline performs similarly at that tolerance.The model generally outperforms the prior baseline except for substitution precision at tolerances above 20 seconds.Source: §7.5 · pp. 14–15, Figure 12 · p. 15, Figure 13 · p. 15
- Extreme λseg values hurt spotting and segmentation, while values from 1 to 10 retain strong spotting and informative scores for goal opportunities.The reported best Average-mAP for spotting is around λseg = 1.5, while the best value for spotting unannotated opportunities might be around 10.Source: §7.5 · p. 15, Figure 14 · p. 16
- The ActivityNet Average-mAP improvement over BMN was statistically significant, with a Student’s t-test p-value of 2.3 × 10^-18.The authors report similar analyses for AR@100 and AUC.Source: §7.5 · p. 16
7.6. Extra Actions and Highlights Generation
- Qualitative examples include a goal opportunity and an unsanctioned foul that the highlights generator could include; a visible LED substitution panel may explain a false positive.The authors suggest the network may associate that panel with substitutions.Source: §7.6 · p. 16, Figure 16 · p. 17