PaperlayerView exact revision on arXiv

Source-linked AI summary

SoccerNet-FoulRet: Retrieving Semantically Similar Soccer Foul Videos

Jacobus Arthur, Ahmad Sait, Batool Hani, Merey Ramazanova, Jan Held, Marc Van Droogenbroeck, Bernard Ghanem, Anthony Cioppa, Silvio Giancola

arXiv:2610.09742v1cs.CVcs.IR

TL;DR

Referees cannot easily compare contentious fouls with similar historical cases, and prior retrieval benchmarks do not define relevance by refereeing interpretation. SoccerNet-FoulRet introduces a human-verified semantic precedent benchmark and evaluates retrieval methods. Category supervision improves category relevance but transfers only modestly to precedent retrieval, while the strongest zero-shot model remains below 5% HitRate@10.

  • Problem

    Referees cannot easily compare current incidents with similar historical cases, and foul similarity requires semantic interpretation without a standard benchmark or ground truth.

    Source: §1 · p. 2

  • Method

    SoccerNet-FoulRet builds a human-verified precedent benchmark and evaluates retrieval using referee judgments alongside category-relevance labels.

    Source: §1 · p. 2, §3.1 · p. 5, §3.3 · p. 7

  • Results

    Category-level supervision improves action, offence, and severity retrieval but transfers only modestly to precedent relevance; InternVideo2-1B achieves under 5% HitRate@10 on human-verified precedents.

    Source: §1 · p. 3, §5 · p. 14

  • Takeaways & Limitations

    Semantic foul retrieval remains challenging, and category similarity does not fully capture human judgments of precedent relevance.

    Source: §1 · p. 3, §5 · p. 14

    The verified precedent pool is incomplete, so unreviewed clips that may be valid precedents receive no credit under the evaluation metrics.

    Source: §4.1 · p. 8

Abstract

from arXiv · show

Refereeing decisions in professional soccer remain inconsistent because referees cannot easily compare a contentious foul against similar past cases. We cast this as a retrieval problem and introduce SoccerNet-FoulRet, the first benchmark for semantic foul retrieval. Given a query foul, the task is to retrieve past fouls judged to be relevant precedents, regardless of camera angle, teams, or appearance. This differs from prior video-to-video retrieval, which matches clips by visual similarity or a shared event. Here, relevance is defined by refereeing interpretation. We build the benchmark from the SoccerNet-MVFoul dataset and evaluate retrieval ability of zero-shot video and vision-language embedders together with a task-specific fine-tuned baseline on 693 human-verified queries and category-relevance labels. Semantic foul retrieval remains challenging. The strongest zero-shot model achieves under 5% HitRate@10 on human-verified precedents, while category-supervised fine-tuning improves category relevance but transfers only modestly to precedent retrieval. We release SoccerNet-FoulRet to establish semantic foul retrieval as an open problem: https://github.com/SoccerNet/sn-foulret.

1 Introduction

Referees can struggle to compare contentious incidents with historical cases, while foul similarity requires semantic interpretation and lacks a standard benchmark. SoccerNet-FoulRet introduces a human-verified precedent benchmark and evaluates retrieval approaches, finding that category supervision transfers only modestly to precedent relevance.

1 Introduction

  • Referees may judge the same contact differently, and they cannot easily compare current incidents with similar historical cases.The work frames retrieval as a way to help officials compare similar fouls without automating the final decision.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Foul retrieval must infer semantic concepts from video, while the absence of a standard benchmark or ground truth makes its effectiveness difficult to measure.Source: §1 · p. 2
  • SoccerNet-FoulRet constructs a human-verified precedent base and evaluation protocol because SoccerNet-MVFoul contains no similarity links between incidents.The benchmark is built from merged gallery annotations and query–candidate pairs with human-verified precedent judgments.Source: §1 · p. 2
  • The evaluation compares a text-mediated VARS+XVARS pipeline, zero-shot video and vision-language embedders, and a task-specific fine-tuned baseline.It tests precedent retrieval as well as whether category supervision transfers to precedent relevance.Source: §1 · p. 2
  • Category-level supervision improves action, offence, and severity retrieval but transfers only modestly to human-verified precedent relevance.The benchmark evaluates both category-level relevance and human-verified precedent retrieval.Source: §1 · p. 3

2 Related Work

Prior sports-video work has expanded from classification to richer event understanding, while referee assistance remains relatively under-explored. Existing video retrieval benchmarks generally define relevance through visual overlap or shared events, not referee judgments of precedent relevance.

2 Related Work

  • The related work spans sports video understanding and video-to-video retrieval.Source: §2 · p. 3

2.1 Sports Video Understanding

  • SoccerNet-MVFoul supports foul classification and SoccerNet-XFoul supports explanations and question-answer pairs; this work repurposes them for retrieval.Their structured annotations and referee-oriented explanations help generate candidate pairs for human validation, while MVFoul labels also support category-level evaluation.Source: §2.1 · p. 3

2.2 Video-to-Video Retrieval

  • Prior retrieval benchmarks target near-duplicate videos or shared events, whereas precedent pairs can span independent matches and visually dissimilar clips.Copy detection also requires shared source footage, and event retrieval requires a shared physical event.Source: §2.2 · pp. 3–4
  • Concept-based retrieval ranks shared semantic concepts, but prior benchmarks do not capture human judgments of whether two fouls are relevant precedents.Existing systems commonly encode videos as vectors and rank them by embedding distance.Source: §2.2 · p. 4

3 The SoccerNet-FoulRet Benchmark

SoccerNet-FoulRet repurposes SoccerNet-MVFoul and SoccerNet-XFoul into a benchmark for semantic foul retrieval. It evaluates referee-judged precedent relevance separately from category relevance, using a training-split gallery and human-verified query–candidate pairs.

3.1 Task Definition

  • Precedent retrieval ranks gallery clips by referee-judged similarity, while category relevance uses Action, Action+Offence, and Action+Offence+Severity labels.Identical mechanics can receive different sanctions depending on pitch location and context.Source: §3.1 · p. 5, Figure 2 · p. 5

3.2 Retrieval Gallery

  • The benchmark’s training-split gallery contains 2,916 actions with categorical attributes and SoccerNet-XFoul question-answer pairs.VARS+XVARS uses these signals for coarse filtering and dimension-wise semantic comparison; zero-shot video and vision-language retrievers use raw clips instead.Source: §3.2 · pp. 5–6

3.3 Human-Verified Precedent Pairs

  • The verified precedent set retains 693 validation and test queries with at least one approved candidate, after reviewing five candidates per query.Queries whose candidates were all rejected are excluded.Source: §3.3 · p. 6
  • Human validation labels each query–candidate pair Valid or Rejected; at most five reviewed candidates per query make the judgments high-precision but incomplete.The authors served as annotators, drawing on refereeing and soccer-video-understanding expertise.Source: Figure 3 · p. 6, §3.3 · p. 7
  • The benchmark comprises 693 query fouls, about four approved precedents per query on average, and a gallery of 2,916 actions.The attribute distributions are long-tailed, with standing tackles about 44% of the gallery and severities 4–5 under 3%.Source: §3.3 · p. 7, Table 2 · p. 7

4 Experiments

The experiments evaluate precedent retrieval and category relevance using embedding models and a text-mediated pipeline. InternVideo2-1B leads on human-verified precedents, while category-level gains transfer only modestly to that task.

4.1 Evaluation Protocol

  • The protocol scores sparse human-verified precedents with HitRate, Recall, nDCG, and MRR, while category relevance uses mAP.Only a small subset of gallery clips is judged per query, so unjudged valid precedents receive no credit.Source: §4.1 · pp. 7–8

4.2 Implementation Details

  • VARS filters candidates by predicted Action and Offence, then XVARS profiles clips with referee-question answers ranked by averaged BGE cosine similarity.The embedding models rank clips directly by video similarity; the study includes zero-shot models and a Qwen3-VL-Embedding-2B task-specific adaptation.Source: §4.2 · pp. 8–9

4.3 Results

  • The engineered VARS+XVARS pipeline leads on no metric, sits at the random floor for HR@1 (0.14%), and returns no candidates for 17% of Test queries, scored as misses.Source: §4.3 · pp. 9–10
  • InternVideo2-1B leads overall on human-verified precedents, with HR@5 2.60% and HR@10 4.62%.The strongest zero-shot model is evaluated without domain-specific training.Source: §4.3 · p. 9, Table 3 · p. 10
  • Fine-tuning improves Qwen3-VL-Emb-2B across all nine category metrics, yet these gains transfer only modestly to precedent retrieval; scaling does not improve zero-shot retrieval within either model family.InternVideo2-6B trails InternVideo2-1B, while Qwen3-VL-Emb-8B does not outperform Qwen3-VL-Emb-2B.Source: §4.3 · pp. 10–11
  • Category relevance is measured with mAP across increasingly strict Action, Action+Offence, and Action+Offence+Severity labels.The +Severity evaluation uses 2,561 gallery clips and 650 queries; Action and +Offence use 2,905 gallery clips and 709 queries.Source: §4.3 · p. 11, Table 4 · p. 12

4.4 Ablation

  • Generic query instructions are best or tied-best almost everywhere, while more specific prompts generally do not improve performance.The ablation compares Generic, Domain, and Detailed instructions; InternVideo2 is excluded because it takes no instruction.Source: §4.4 · p. 12, Table 5 · p. 13

4.5 Qualitative Analysis

  • A verified precedent can be retrieved despite substantial differences in camera angle, lighting, and team kits.In the illustrative failure, VARS misclassifies an elbowing as a challenge, so strict category filtering removes valid precedents before ranking.Source: §4.5 · pp. 13–14

5 Conclusion

SoccerNet-FoulRet frames refereeing consistency as semantic foul retrieval, with the final verdict left to the referee. Results show that precedent retrieval remains difficult, category supervision transfers only modestly to verified precedents, and the judged pool has important limits.

5 Conclusion

  • Under 5% HitRate@10, InternVideo2-1B is the strongest zero-shot retriever, while category-supervised fine-tuning transfers only modestly to human-verified precedent retrieval.Fine-tuning improves category relevance substantially but does not make action, offence, and severity sufficient to capture precedent relevance.Source: §5 · p. 14
  • SoccerNet-FoulRet is the first benchmark for semantic foul retrieval, evaluating human-verified precedent judgments and category-relevance labels.The task retrieves relevant historical precedents rather than automating the referee’s verdict.Source: §5 · p. 14
  • Performance degrades sharply as category relevance tightens from Action to Action + Offence + Severity, indicating stronger matching of broad categories than finer refereeing details.The conclusion identifies intent, point of contact, and severity as semantics retrieval models should better capture.Source: §5 · p. 14
  • The human-verified pool is incomplete, and its limited annotator pool does not establish broader agreement across referees.Unjudged retrieved clips may still be valid precedents.Source: §5 · p. 14

A Task-Specific Fine-Tuning

The task-specific Qwen3-VL-Embedding-2B baseline is fine-tuned with supervised contrastive learning across three levels of foul-label similarity. Its objective averages query-to-gallery and reverse-direction losses, with training details specifying the batch construction and optimization setup.

A Task-Specific Fine-Tuning

  • Fine-tuning uses supervised contrastive learning at Action, Action+Offence, and Action+Offence+Severity levels, treating matching-label clips as positives and non-matching clips as negatives.Self-pairs are excluded.Source: A Task-Specific Fine-Tuning · p. 18
  • Label-balanced batches contain 12 clips in six label groups, with two distinct clips per group; missing severity labels exclude clips only from the strictest loss.Each anchor has one guaranteed positive, while other groups supply negatives at the strictest level and can become positives at coarser levels.Source: A Task-Specific Fine-Tuning · p. 18
  • Training runs for 10 epochs using AdamW at learning rate 10^-5, with LoRA applied only to trainable parameters and a cosine schedule with 10% warmup.LoRA uses rank 16, scaling factor 16, and dropout 0.10; gradient accumulation is eight steps and clipping uses maximum norm 1.0.Source: A Task-Specific Fine-Tuning · p. 18
  • The query-to-gallery loss uses normalized query and gallery embeddings with τ = 0.05; the reverse-direction loss and three label-level losses are averaged equally.The compared embeddings are q_i and g_j; eligible clips exclude the anchor itself.Source: A Task-Specific Fine-Tuning · p. 18

B Oracle Candidate Generation

Oracle candidate generation narrows the annotation workload by filtering gallery clips with foul labels and ranking candidates using referee-written explanations. Three jurisprudential Q&A tracks are used to maintain diversity among candidates for human review.

B Oracle Candidate Generation

  • Candidates are filtered with SoccerNet-MVFoul labels and ranked by SBERT similarity over ground-truth referee-written XVARS explanations.The explanations were written by approximately 70 professional referees.Source: B Oracle Candidate Generation · p. 18
  • Candidate ranking averages cosine similarities across three jurisprudential Q&A tracks covering physical mechanics, rule interpretation, and disciplinary severity.The tracks are used to preserve diversity in the candidate set.Source: B Oracle Candidate Generation · p. 19
  • The filtering and ranking procedure reduces annotation overhead by 99.83% while concentrating review on plausible precedent candidates.Source: B Oracle Candidate Generation · pp. 18–19

C Relevance Structure

The benchmark represents relevance as a query-conditioned, many-to-many relation: queries may share precedents, and each query–candidate pair is judged independently.

C Relevance Structure

  • The benchmark uses query-conditioned, many-to-many relevance, allowing multiple queries to share precedents.Each query–candidate pair receives an independent judgment.Source: C Relevance Structure · p. 19
  • Across 693 evaluated queries, 2,751 approved pairs span 1,353 unique precedents.Source: C Relevance Structure · p. 19
  • Of 1,353 unique precedents, 691 match multiple queries, and 652 queries share at least one precedent.Source: C Relevance Structure · p. 19

D Annotation Protocol

Valid precedent pairs were judged on both physical mechanics and regulatory severity or tactical context, beyond the four-attribute coarse filter. One annotator with 18 years of soccer experience and IFAB knowledge reviewed 3,518 pairs in about 50 hours.

D Annotation Protocol

  • A pair was labeled Valid only when it matched both Form (physical mechanics) and Substance (regulatory severity or tactical context).Source: D Annotation Protocol · p. 19
  • Validation went beyond four coarse attributes—Action Class, Offence, Contact, and Body Part—because pitch location can change a foul’s context.The passage contrasts an identical tackle in midfield with one inside the box denying a clear goal-scoring opportunity.Source: D Annotation Protocol · p. 19
  • A single annotator with 18 years of soccer experience and IFAB knowledge reviewed 3,518 approved or rejected pairs in approximately 50 hours.Source: D Annotation Protocol · p. 19

E Inter-Annotator Agreement

A second annotator independently reviewed a subset of 101 queries, with raw agreement of 67.13% and Cohen’s κ of 0.238. Agreement was higher for accepted precedents than rejected candidates, and the authors cite this as evidence of subjective relevance and a reason for broader annotation.

E Inter-Annotator Agreement

  • 67.13% raw agreement and Cohen’s κ = 0.238 were observed across 502 independently reviewed query–candidate pairs.Source: E Inter-Annotator Agreement · p. 19
  • The second annotator reviewed 101 queries; 99 had five judged candidates, while two had pools of four and three candidates.Source: E Inter-Annotator Agreement · p. 19
  • 76.46% agreement on accepted precedents exceeded 45.54% on rejected candidates, indicating less consistent judgments for rejected pairs.The authors connect this difference to the subjective nature of precedent relevance and motivate broader annotation with multiple refereeing perspectives.Source: E Inter-Annotator Agreement · p. 19
Loading 2610.09742v1…