PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-Shot Multimodal Event Spotting in Soccer Broadcasts

Sadia Kamal, Mahmudul Hasan, Tony Braskich

arXiv:2610.37583v1cs.CVcs.LGcs.MM

TL;DR

Soccer spotting must find brief events in complete broadcasts, while existing approaches often rely on soccer-specific training. This paper uses general-purpose multimodal models with storyboards and aligned commentary for zero-shot spotting; across three annotated matches, it reports improved spotting from combined evidence and strong storyboard performance at lower inference cost.

  • Problem

    Spotting brief events in full soccer broadcasts is challenging, and existing approaches commonly rely on annotated datasets and soccer-specific training.

    Source: §I · p. 1

  • Method

    The framework jointly uses sparse visual storyboards and aligned commentary with general-purpose multimodal models to detect, classify, and temporally localize events without task-specific training.

    Source: §I · p. 1

  • Results

    Across three annotated matches, combining visual and commentary evidence improved spotting, while storyboard input provided strong performance at lower inference cost than individual-frame input.

    Source: §VI · p. 7

  • Takeaways & Limitations

    The results demonstrate the potential of general-purpose multimodal models for soccer video understanding, while motivating larger-scale evaluation and human validation of highlight relevance.

    Source: §VI · p. 8

    The three-match evaluation limits conclusions about generalization across leagues, broadcast styles, and commentary conditions.

    Source: §VI · p. 7

Abstract

from arXiv · show

Automatic soccer event spotting supports highlight generation, video indexing, and sports analytics, but existing methods depend on large annotated datasets and soccer-specific training. We present a zero-shot framework that detects, classifies, and temporally localizes events in full-length soccer broadcasts without task-specific parameter training. Each match is segmented into overlapping clips represented by a compact visual storyboard and the aligned commentary transcript, which a general-purpose multimodal model processes jointly. On three manually annotated matches containing 130 events, the best configuration achieves 72.0% F1@5s and outperforms SoccerNet-trained CALF and NetVLAD++ in a cross-dataset setting, while the storyboard representation reduces inference cost by 72% compared with individual frames. We further analyze the effects of model family, input representation, and prompt optimization, and introduce an evidence-grounded excitement-scoring stage that ranks detected events with natural-language explanations for highlight selection.

I. INTRODUCTION

Spotting events in complete soccer broadcasts is difficult because events are brief, visually similar, and mixed with replays, while existing approaches rely on soccer-specific data and training. This work studies zero-shot multimodal spotting and localization with general-purpose models, alongside input, prompt, benchmark, and excitement-scoring questions.

I. INTRODUCTION

  • Soccer event spotting must locate brief, visually similar events in untrimmed matches that also contain replays, celebrations, and recaps.The task supports highlight generation, video indexing, content retrieval, and sports analytics.Source: §I · p. 1
  • Existing soccer spotting systems rely on annotated, soccer-specific training data and computational resources, despite gains from temporal modeling and multimodal information,.MatchVision and SoccerMaster use large-scale domain-specific training.Source: §I · p. 1
  • The study investigates zero-shot detection and temporal localization on complete broadcasts, where general-purpose multimodal models jointly process sparse visual information and aligned commentary.The questions cover feasibility, representation, prompts, supervised-method comparisons, and excitement scoring.Source: §I · p. 1
  • The paper formulates complete-broadcast spotting as joint detection, classification, and temporal localization without task-specific training.It also evaluates model choice, input representation, prompt optimization, accuracy, localization, and inference cost.Source: §I · p. 1
  • The framework is compared with supervised spotting methods under a common six-class protocol, and adds excitement scoring for highlight ranking.The excitement stage is assessed with two LLM-based judging protocols, including FlowJudge and a GPT-based evaluator.Source: §I · p. 1

A. Soccer Event Spotting and Multimodal Understanding

Prior soccer spotting and temporal-reasoning work often uses domain-specific training or specialized methods; this work examines general-purpose models for zero-shot spotting in full broadcasts. Its evaluation uses three manually annotated broadcast matches.

A. Soccer Event Spotting and Multimodal Understanding

  • Earlier soccer spotting work developed temporal, feature-aggregation, transformer, and multimodal methods, while MatchVision and SoccerMaster use soccer-specific representation learning,,,,,,,,.The proposed study instead examines full-broadcast event spotting with general-purpose multimodal models and no task-specific parameter training.Source: A. Soccer Event Spotting and… · pp. 1–2

B. Multimodal Temporal Reasoning and Prompt Optimization

  • TimeChat, Grounded-VideoLLM [15], and TimeSuite [16] use specialized representations or training for temporal video understanding; this study also tests prompt optimization with MIPROv2 and GEPA [8].The prompt methods are examined for their effects on detection and temporal localization.Source: B. Multimodal Temporal Reasoning and… · p. 2

A. Dataset

  • The evaluation dataset comprises three professionally produced, multi-camera YouTube broadcasts, manually annotated with 130 events across six categories.The categories are goal, shot, miss, corner, kickoff, and yellow card; the source videos, annotations, and evaluation metadata are publicly available.Source: A. Dataset · p. 2
  • Each match is divided into 15-second clips with 5-second overlap, using 15 uniformly sampled frames and commentary transcribed with Whisper Medium and aligned to each clip.This default followed preliminary tests in which shorter, denser clips produced higher recall but lower precision.Source: A. Dataset · p. 2

B. Problem Formulation

The formulation represents each full match as overlapping clips and assigns each clip a multimodal input and structured event output. The pipeline then samples frames and processes clips through spotting and later stages.

B. Problem Formulation

  • A match is divided into overlapping clips, each associated with a storyboard, aligned commentary, and match-timeline start and end times.The formulation names the match V and the clip set C = {C1, . . . , CN}.Source: B. Problem Formulation · p. 2
  • For each clip, the model outputs a detection flag, event category, clip-relative timestamp, confidence score, and supporting evidence.The output tuple is specified as a detection flag, category, timestamp, confidence, and evidence.Source: B. Problem Formulation · p. 2

C. Pipeline Overview

B. Problem Formulation

  • Unlike event-centered classification, the task requires deciding jointly whether an event occurs, what it is, and when it occurs.Source: B. Problem Formulation · p. 2

C. Pipeline Overview

  • The pipeline processes overlapping clips using sparse frame sampling, storyboard construction, commentary alignment, multimodal detection, temporal localization, consolidation, and excitement scoring.Source: C. Pipeline Overview · pp. 2–3

1) Sparse Frame Sampling:

  • Frames are sampled uniformly across each clip to provide observations throughout the temporal window.The default configuration extracts 15 frames from each 15-second clip.Source: 1) Sparse Frame Sampling · p. 3

2) Storyboard Construction:

The method compresses sampled frames into a chronological storyboard and pairs it with aligned commentary for multimodal event prediction. It also maps, consolidates, and prompt-optimizes predictions.

2) Storyboard Construction:

  • The default storyboard arranges 15 timestamped frames in chronological order in a 5 × 3 grid, allowing temporal progression to appear in one visual input.Each frame is 384 × 216 pixels, producing a 1920 × 648 storyboard; the comparison uses the same sampled frames individually at the same per-frame resolution.Source: 2) Storyboard Construction · p. 3

3) Commentary Alignment:

  • Aligned commentary is provided alongside the visual input as complementary semantic evidence about whether an event occurred and its outcome.Source: 3) Commentary Alignment · p. 3

4) Multimodal Event Detection:

  • The multimodal model predicts an event category or no important event, a relative timestamp, confidence, and supporting visual and textual evidence.Predictions use a structured format for automatic parsing and evaluation.Source: 4) Multimodal Event Detection · p. 3

5) Absolute Temporal Localization:

6) Temporal Consolidation:

  • Overlapping same-class predictions are clustered within 20 seconds, or 30 seconds for goals and corners, retaining the highest-confidence prediction in each cluster.Predictions from different event classes are never merged.Source: 6) Temporal Consolidation · p. 3

D. Prompt Optimization

  • Prompt optimization changes textual instructions while keeping model parameters fixed; GEPA and MIPROv2 use labeled development examples and apply optimized prompts unchanged at inference.The event taxonomy, multimodal input representation, and structured output format are preserved.Source: D. Prompt Optimization · p. 3
  • The manual storyboard prompt specifies the event taxonomy, event-specific decision guidance, joint visual-commentary reasoning, and distinction between live events and replays or recaps.Source: D. Prompt Optimization · p. 3
  • A two-prompt configuration separates initial event recognition and coarse localization from verification and timestamp refinement.The second prompt re-examines the initial prediction and multimodal evidence.Source: D. Prompt Optimization · p. 3

E. Models

The evaluation compares multimodal model families and input modes for zero-shot event spotting, then uses model-assigned excitement scores to rank detected events with explanations.

E. Models

  • The generative models are evaluated with sampled frames or storyboards plus aligned commentary, producing event category, relative timestamp, confidence, and supporting evidence.GPT variants span capability–cost operating points; Qwen3.5-4B is a lightweight open-source generative baseline, and Qwen3-VL-8B Embedding provides a representation-similarity baseline.Source: E. Models · p. 3

F. Excitement Scoring

  • GPT-5.4 Mini scores each retained event from its predicted category, confidence, visual evidence, and aligned transcript, without reprocessing the original video.The evaluation includes 175 post-consolidation predictions from six benchmark categories, each scored once.Source: F. Excitement Scoring · p. 3
  • The excitement score combines event importance, visual intensity, and transcript intensity using weights of 0.40, 0.35, and 0.25, respectively.The weights are heuristic design choices, not learned parameters, and can be adjusted for highlight-selection preferences.Source: F. Excitement Scoring · p. 4
  • The model also produces a confidence score and concise natural-language explanation to make its excitement assessment interpretable for highlight ranking.Source: F. Excitement Scoring · p. 4

IV. EVALUATION

Detection is evaluated by matching timestamp predictions to annotated intervals under a fixed tolerance, while excitement-score consistency is assessed through LLM judges’ rankings and score differences.

A. Evaluation Protocol

  • A prediction is a true positive when it matches an event of the same category within ±5 seconds; each prediction and ground-truth event can match at most once.Unmatched predictions are false positives and unmatched annotated events are false negatives; the six-class ground-truth set contains 130 events.Source: A. Evaluation Protocol · p. 4
  • Detection metrics include Precision@5s, Recall@5s, F1@5s, mAP, and localization MAE, measured from predictions matched to annotated intervals.mAP averages category-level AP using the same ±5-second criterion.Source: A. Evaluation Protocol · p. 4

B. Excitement-Score Evaluation

  • Excitement-score consistency is assessed on 175 predictions by two LLM judges, which receive event category, visual evidence, and commentary but not the original score, confidence, or explanation.Both judges score on a 0–10 scale; Spearman’s rank correlation is primary for ranking agreement, with MAE describing absolute calibration differences.Source: B. Excitement-Score Evaluation · p. 4
  • GPT-5.5 and GPT-5.6 Sol achieve the highest average detection recall across three matches; NetVLAD++ reaches 65.96% and CALF 43.06%.Source: Figure 2 · p. 4

V. RESULTS

Across three matches, GPT-5.5 with storyboard input leads the evaluated multimodal configurations on overall detection metrics, while results vary by model and event category.

A. RQ1: Overall Event Detection Performance

  • GPT-5.6 Sol achieves 70.05% F1@5s and 82.45% Recall@5s, while GPT-5.6 Terra reaches 67.75% F1@5s at $7.20 per match versus $18.01 for GPT-5.5 and GPT-5.6 Sol.GPT-5.6 Luna costs $0.72 per match and reaches 77.19% recall, but its 45.83% precision yields 57.32% F1@5s.Source: A. RQ1: Overall Event Detection… · p. 4
  • Qwen3.5-4B combines 60.59% recall with 30.46% precision, while Qwen3-VL-8B reaches 19.38% recall.Source: A. RQ1: Overall Event Detection… · p. 4
  • GPT configurations recover 77–83% of annotated events, compared with 65.96% for NetVLAD++ and 43.06% for CALF.Source: A. RQ1: Overall Event Detection… · p. 4
  • Recall varies by event category: Qwen3.5-4B has 90.5% goal recall but detects only 34.1% of shots, while NetVLAD++ leads corner recall at 95.5%.Table I macro-averages metrics across matches, whereas Table II pools detections across matches; on these broadcasts, zero-shot spotting performance varies across model families.Source: A. RQ1: Overall Event Detection… · pp. 4–5, Table II · p. 5
  • 72.02% F1@5s, 82.61% Recall@5s, 76.24% mAP@5s, and 0.27-second localization error make GPT-5.5 with storyboard input the top multimodal configuration.The reported results are average event-detection performance across three matches.Source: A. RQ1: Overall Event Detection… · p. 4, Table I · p. 5

B. RQ2: Effect of Input Representation and Modality

The single-match ablation finds that multimodal input improves spotting, with storyboard input offering the strongest aggregate performance at lower cost. Across three matches, manual prompting leads overall recall and F1, while optimization shifts detection trade-offs and cannot recover missing multimodal evidence.

B. RQ2: Effect of Input Representation and Modality

C. RQ3: Prompt Structure and Optimization

  • Manual storyboard prompting achieves the highest Recall@5s (82.45%) and F1@5s (70.05%) among the evaluated prompt strategies.Storyboard-2 instead has the highest precision (62.24%) and lowest temporal MAE (0.14 s), but recall falls to 71.5%.Source: C. RQ3: Prompt Structure and… · p. 5, Table IV · p. 6
  • MIPROv2 reaches 80.37% recall but 63.89% F1@5s, while GEPA records 55.41% precision, 71.83% recall, and 62.17% F1@5s.Both optimized prompts improve temporal localization over the storyboard baseline, with MAEs of 0.23 s for GEPA and 0.27 s for MIPROv2 versus 0.47 s.Source: C. RQ3: Prompt Structure and… · pp. 5–6
  • Class-wise recall varies by prompt: the storyboard prompt leads shot recall (90.2%), while MIPROv2 leads miss recall (73.1%).GEPA matches storyboard recall for corners and goals but performs worse on remaining classes; Storyboard-2 generally has lower class-wise recall.Source: C. RQ3: Prompt Structure and… · p. 6
  • Prompt optimization changes precision, recall, localization, and class sensitivity but does not uniformly improve event spotting.Many errors arise from multimodal evidence limitations, including sparse sampling and ambiguous outcomes, which prompt changes cannot recover.Source: C. RQ3: Prompt Structure and… · p. 6

D. RQ4: Comparison with Soccer-Specific Baselines

In a common six-class cross-dataset evaluation, zero-shot multimodal configurations outperform the evaluated SoccerNet-pretrained baselines, though transfer differences may affect the comparison. The excitement-scoring stage produces event rankings that agree with two LLM judges, but human evaluation is still needed to establish alignment with perceived highlight relevance.

D. RQ4: Comparison with Soccer-Specific Baselines

  • The strongest zero-shot multimodal configurations exceed CALF and NetVLAD++ in precision, recall, F1@5s, and mAP@5s under the common six-class evaluation.NetVLAD++ achieves 65.96% Recall@5s and 38.49% F1@5s; CALF achieves 43.06% Recall@5s and 19.60% F1@5s.Source: D. RQ4: Comparison with… · p. 6
  • The baseline comparison is cross-dataset: CALF and NetVLAD++ are evaluated without retraining, and annotation or broadcast differences may affect results.Source: D. RQ4: Comparison with… · p. 6

E. RQ5: Excitement-Score Analysis

  • Goals receive the highest mean excitement score (7.87), followed by shots (5.81) and misses (5.03), while kickoffs, yellow cards, and corners score lower.Both LLM-based judges broadly preserve this ordering.Source: E. RQ5: Excitement-Score Analysis · p. 6
  • GPT-based judging agrees more strongly with GPT-5.4 Mini than FlowJudge, with Spearman ρ = 0.926 and MAE 0.64 versus ρ = 0.716 and MAE 1.37.Both judges show positive rank agreement across all three matches.Source: E. RQ5: Excitement-Score Analysis · p. 6
  • Human evaluation remains necessary to establish whether excitement rankings align with perceived highlight relevance.Source: E. RQ5: Excitement-Score Analysis · p. 6

F. Error Analysis

The error analysis identifies confusion among related attacking events and possible false detections from recap footage. Sparse storyboard sampling may omit evidence needed to distinguish event outcomes, and prompt-based broadcast-state reasoning is not fully reliable.

F. Error Analysis

  • Misses and shots are confused: four of seven misses in Match 1 are classified as shots.Corner errors can also correspond to the goal-directed action following a set piece; some shots and goals are rejected as No Important Event.Source: F. Error Analysis · pp. 6–7
  • Class ambiguity depends on clip context, and sparse storyboard sampling may show an attack without clearly revealing its final outcome.Distinguishing a shot from a miss requires evidence of the attempt’s final trajectory.Source: F. Error Analysis · p. 7
  • A recap sequence was interpreted as a new goal, but the one-case qualitative sample cannot establish a general recap-confusion rate.The analysis notes recap content can cause false positives despite prompt instructions to distinguish live events from retrospective footage.Source: F. Error Analysis · p. 7
  • Prompt-based broadcast-state reasoning handles some replay or recap sequences but is not fully reliable.Explicit broadcast-state modeling may provide a more robust way to distinguish new events from retrospective content.Source: F. Error Analysis · p. 7

VI. CONCLUSION

The paper concludes that zero-shot multimodal models can spot and temporally localize soccer events in complete broadcasts, with storyboard input offering lower inference cost than individual-frame input. Excitement scores agree positively with LLM judges, but broader generalization and human validation remain open concerns.

VI. CONCLUSION

  • Across three annotated matches with 130 events, the framework combines zero-shot event spotting, temporal localization, and excitement scoring; storyboard input costs less to infer than individual-frame input.Prompt optimization changed precision–recall and class-level behavior but did not consistently improve overall performance.Source: §VI · p. 7
  • Excitement scores showed positive rank agreement with two LLM-based evaluators, particularly the GPT-based judge, but human validation is still needed to establish alignment with perceived highlight relevance.Table VII covers agreement with two LLM-based judges; Table VIII gives representative examples.Source: §VI · p. 7, Table VII · p. 8, Table VIII · p. 8
  • The three-match evaluation limits conclusions about generalization across leagues, broadcast styles, and commentary conditions.Source: §VI · p. 7
  • Sparse frame sampling may omit short-duration visual cues, and replay or recap sequences can introduce false-positive detections.CALF and NetVLAD++ were evaluated cross-dataset without retraining, so annotation and broadcast differences may affect the comparison.Source: §VI · pp. 7–8
  • The authors characterize general-purpose multimodal models as promising for soccer video understanding while motivating larger-scale evaluation, improved replay handling, and human validation of highlight relevance.Source: §VI · p. 8
Loading 2610.37583v1…