PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond the Single Camera: Agentic Multi-View Reasoning in Sports Video Understanding

Kerui Chen, Jinglu Wang, Xiaoyi Zhang, Yan Lu

arXiv:2607.11844v2cs.CV

TL;DR

Sports video understanding benchmarks typically provide a single view, although occlusion and complementary camera evidence can matter for interpreting events and rulings. This paper introduces SportMV-Bench and proposes SportMV-Agent, which iteratively selects views, invokes perception tools, and reasons from evidence. Current MLLMs struggle to exploit multi-view information, while SportMV-Agent improves overall performance by a relative 15.61% over GPT-4.1.

  • Problem

    Existing sports QA benchmarks typically use single-view inputs, leaving MLLMs’ ability to reason across multiple views of the same event unevaluated.

    Source: Introduction · p. 1

  • Method

    The paper builds SportMV-Bench from official match recordings and proposes SportMV-Agent, which iteratively selects informative views, uses perception tools, and grounds reasoning in collected evidence.

    Source: Introduction · p. 2

  • Results

    Current MLLMs fail to exploit multi-view information effectively, while SportMV-Agent improves overall performance from 59.12% to 68.35% versus GPT-4.1, a relative gain of 15.61%.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The reported bottleneck is fine-grained visual perception and view selection rather than logical reasoning or domain knowledge.

    Source: Introduction · p. 2

    MLLM-based QA generation can produce hallucinated answer choices and questions that reveal information intended to be visually perceived.

    Source: Discussion of MLLM-Based QA Generation · p. 10

Abstract

from arXiv · show

Recent Multimodal Large Language Models (MLLMs) achieve strong performance on single-view video understanding benchmarks. However, sports videos involve dense occlusion, rapid motion, and complex interactions that are difficult to resolve from a single viewpoint. In practice, sports events are recorded from multiple camera angles, providing complementary evidence used by referees. Yet, no existing benchmark evaluates MLLMs on multi-view sports video understanding. To address this gap, we introduce SportMV-Bench, a comprehensive benchmark built from official match recordings, through a dedicated pipeline combining LLM-based generation, MLLM-based verification, and human filtering to ensure quality and consistency. SportMV-Bench containing 1022 multi-view video bundles and 3015 question-answer pairs spanning 10 sports across three categories: Perception-Aware Recognition (PAR), Rule-aware Event Interpretation (REI), and Adjudicative Decision Reasoning (ADR). Our analysis shows that current MLLMs fail to effectively exploit multi-view information, with the bottlenecks lying in fine-grained visual perception and view selection rather than logical reasoning or domain knowledge. We propose SportMV-Agent, an agentic framework that orchestrates an iterative loop of active view selection, perception tool execution, and evidence-grounded reasoning, achieving a significant 15.61% relative improvement over the strongest MLLM baseline.

Introduction

Sports QA benchmarks typically use a single view, although occlusion and complementary evidence across cameras can make single-view judgments incomplete or misleading. SportMV-Bench and SportMV-Agent address multi-view evaluation and reasoning; experiments identify perception and view selection as key bottlenecks.

Introduction

  • Single-view sports QA can miss or misread decisive actions, while resolving some rulings requires combining complementary evidence across views.The introduction describes occlusion hiding a decisive touch and a ruling that depends on evidence from two different views.Source: Introduction · pp. 1–2
  • SportMV-Bench introduces multi-view sports evaluation across 10 sports, with 1,022 video bundles and 3,015 QA pairs spanning PAR, REI, and ADR.The QA pairs are generated and checked through a pipeline combining automatic generation, multimodal verification, and human review.Source: Introduction · p. 2
  • Providing all views jointly improves accuracy by only 0.9 points over a random single view, while an oracle view yields a 13.4% relative gain.Annotated perception labels improve accuracy by +21.72 points; the findings identify visual perception, rather than logical reasoning or domain knowledge, as the primary bottleneck.Source: Introduction · p. 2
  • SportMV-Agent raises overall performance from 59.12% to 68.35% versus GPT-4.1, a relative gain of 15.61%.Its orchestrator iteratively selects informative views, invokes perception tools, and grounds reasoning in collected evidence.Source: Introduction · p. 2
  • The paper’s contributions combine a multi-view benchmark, an empirical diagnosis of MLLM weaknesses, and an agentic framework motivated by that diagnosis.The framework coordinates active view selection, perception-tool execution, and evidence-grounded reasoning.Source: Introduction · p. 2

Related Work

Prior sports datasets and QA benchmarks broadened task and sport coverage, but typically evaluate single-view inputs. SportMV-Bench instead evaluates asynchronous multi-view evidence through a three-level question hierarchy and a staged construction pipeline.

Related Work

  • Earlier sports datasets expanded from action classification and quality assessment to temporal parsing and spatiotemporal localization, while later QA benchmarks covered single- and multi-sport tasks but generally remained single-view.This leaves MLLMs’ ability to reason across multiple views under occlusion largely untested.Source: Related Work · p. 2

Sport-MV Benchmark

  • SportMV-Bench frames the task as answering a natural-language question by aggregating evidence across camera views of the same event, which may be asynchronous.Its overview presents ten sports and three question types alongside a real-match multi-view example.Source: Sport-MV Benchmark · p. 2, Figure 2 · p. 3

Dataset Construction

  • The dataset pipeline collects videos with official referee decisions, proposes report-grounded QA, applies multimodal checks, and uses expert review to remove ambiguous or weakly grounded items.The MLLM audit checks report consistency, commonsense answerability, video alignment, and whether the answer can be inferred from the video; items must pass three shuffled-view audits.Source: Dataset Construction · p. 3, Figure 3 · p. 4

Dataset Statistics

  • SportMV-Bench’s 3,015 questions span PAR, REI, and ADR, testing action perception, rule-sensitive event interpretation, and sanction decisions, respectively.The hierarchy is intended to reflect officiating from perceiving actions through interpreting events under rules to rendering decisions.Source: Dataset Statistics · pp. 3–4

Benchmark Analysis and Findings

The section establishes a multi-view evaluation protocol and uses controlled experiments to diagnose MLLM performance bottlenecks in view exploitation and sports-specific reasoning.

Baseline Evaluation Protocol

  • Even the strongest evaluated models perform poorly under the baseline protocol, which delegates cross-view selection and fusion to the MLLM.The baseline packs the question and sampled frames from all views into one sequence without explicit cross-view reasoning structure.Source: Baseline Evaluation Protocol · p. 4

Bottleneck Analysis

  • The analysis treats effective exploitation of multi-view information and robust sports-specific reasoning as distinct challenges for controlled experiments.The experiments are intended to extract insights that guide method design.Source: Bottleneck Analysis · p. 4

Can MLLMs effectively leverage multi-view information?

  • M-View (All) yields only marginal gains over 1-View (Random) and remains substantially worse than 1-View (Best); M-View (Guided) performs best.Redundant, occluded, or misleading angles can offset the benefit of complementary perspectives.Source: Can MLLMs effectively leverage… · p. 4, Table 1 · p. 4

What can improve sports-specific reasoning?

  • Fine-grained perception is decisive: annotated action and contact improve accuracy, while chain-of-thought reasoning can reduce it and sports-specific knowledge yields negligible gains.The analysis identifies view selection and fine-grained action recognition and contact detection as key bottlenecks.Source: Table 1 · p. 4, What can improve sports-specific… · p. 5

SportMV-Agent

SportMV-Agent takes a question, multi-view videos, and perception tools, then iteratively gathers and evaluates evidence until it can make a confident prediction.

SportMV-Agent

  • SportMV-Agent actively selects views, invokes perception tools, and accumulates evidence in an iterative loop rather than passively processing every perspective at once.The loop continues until sufficient evidence supports a confident prediction.Source: SportMV-Agent · p. 5

Orchestration

  • At each iteration, the orchestrator uses the question, active views, and accumulated evidence to plan an operation or emit a final answer based on confidence.The operation either collects new evidence or returns the prediction when confidence is sufficient.Source: Orchestration · pp. 5–6

Evidence Collection

  • Evidence collection combines context-dependent view selection with tool execution targeted to the current state.The toolbox includes action recognition, contact detection, and contact-part detection; the latter is triggered when contact is detected.Source: Evidence Collection · p. 6

Experiment

SportMV-Agent outperforms the strongest baseline on SportMV-Bench, and ablations assess the roles of its orchestrator, view selection, and perception tools. The experiments also compare its reasoning with a chain-of-thought baseline and describe the evaluation setup.

Experiment

  • The evaluation covers eight state-of-the-art MLLMs and reports exact-match accuracy averaged over three runs, with videos processed at 2 fps.GPT-4.1 is the orchestrator and Qwen2.5-VL-72B implements the perception tools.Source: Experiment · p. 6

Main Result

  • 15.61% relative improvement over GPT-4.1, raising overall performance from 59.12% to 68.35%.Table 2 reports quantitative comparisons of state-of-the-art MLLMs on SportMV-Bench, with results in percent.Source: Table 2 · p. 6, Main Result · p. 6
  • In the qualitative example, COT misses the decisive View 3 detail and calls a 3-pointer, while SportMV-Agent selects it and identifies a 2-point field goal without foul.The selected view reveals the player's foot-on-line detail.Source: Main Result · p. 6, Figure 5 · p. 7

Ablation Study

  • Ablations show performance drops of 10.99% with a weaker orchestrator and 7.72% without active view selection.Replacing the Qwen perception tool with GPT-4.1 improves overall performance by 0.41% (PAR improves by 2.86%, while REI and ADR decline slightly); restricting retrieval to top-1 rather than top-3 results drops performance by 2.17%.Source: Ablation Study · p. 7, Table 3 · p. 7

Conclusion

The conclusion introduces SportMV-Bench and SportMV-Agent, reporting model difficulties with selecting viewpoints, perceiving details, and integrating evidence across sports videos.

Conclusion

  • SportMV-Bench contains 3015 QA pairs across 10 sports and three question types, built from real-world match videos and filtered by an MLLM and human annotators.SportMV-Agent uses active view selection and invokes perception tools to address the challenges identified in the experiments.Source: Conclusion · p. 7

Understanding

The supplementary material lists additional discussion, examples, implementation and data-construction details, and dataset QA examples.

Understanding

  • The supplement covers MLLM-based QA generation, qualitative examples, implementation and data-construction details, and dataset QA examples.Source: Understanding · p. 10

Discussion of MLLM-Based QA Generation

The discussion describes three failure modes in MLLM-generated sports QA: hallucinated answer choices, questions that reveal their own visual cues, and overly general questions.

Discussion of MLLM-Based QA Generation

More Qualitative Example

The qualitative example contrasts SportMV-Agent’s active viewpoint switching with CoT’s missed evidence in a complex adjudicative decision.

More Qualitative Example

  • SportMV-Agent selects View 3 to detect offensive contact with the defender’s neck, then combines it with other views’ evidence of legal verticality and positioning to rule an offensive foul.CoT overlooks the contact and incorrectly classifies the incident as a defensive shooting foul.Source: More Qualitative Example · p. 10

More Implementation Details

The implementation details describe constrained action recognition and hierarchical contact-part detection, alongside an example of active multi-view reasoning.

More Implementation Details

  • Action recognition is restricted to dataset taxonomies: SoccerNet classes for soccer and FineSports categories for basketball.The passage says these action sets prevent open-set hallucinations and support more objective pipeline evaluation.Source: More Implementation Details · p. 10
  • SportMV-Agent switches to View 3 and uses contact-detection tools to identify neck contact, supporting an offensive-foul ruling.The example contrasts this with CoT, which overlooks the critical contact and rules a defensive shooting foul.Source: Figure 3 · p. 11
  • The general video-question example has correct options but is unrelated to sports knowledge.The action-recognition and contact-part tools use Top-3 retrieval results with confidence scores.Source: More Implementation Details · p. 10, Figure 4 · p. 11
  • Contact-part detection first classifies contact as upper or lower body, then identifies the specific anatomical part.The coarse-to-fine strategy is described as reducing initial search complexity and fine-grained confusion.Source: More Implementation Details · p. 11

More Data Construction Details

SportMV-Bench is assembled from officiating-review footage, processed into distinct camera views, and refined through expert review. The construction uses official rulings as answer sources and retains only reviewed items meeting the stated criteria.

Video Sources Collection

  • Review broadcasts supply the same decisive moment from several cameras and an authoritative ruling used as the QA answer source.The benchmark targets review mechanisms across ten sports; basketball footage comes from the NBA archive, soccer from SoccerNet, and eight sports from YouTube.Source: Video Sources Collection · p. 11, Table 1 · p. 11
  • Crawled clips are selected with event-level queries and filtered by sport-specific duration windows and title keywords.A blocklist excludes explainers, compilations, press conferences, and video-game footage.Source: Video Sources Collection · p. 12
  • Downloaded videos are capped per sport, deduplicated, and paired with official decisions and metadata as textual ground truth.Downloads are at up to 720p, and the material is used strictly for non-commercial academic research.Source: Video Sources Collection · p. 12

Multi-View Segmentation and Filtering.

  • Shot segmentation, content filtering, and visual clustering produce distinct viewpoints, retaining only events with at least two views.The resulting views are not temporally synchronized, requiring correspondence across unaligned clips.Source: Multi-View Segmentation and Filtering · p. 12

Human Filtering and Refining

  • Expert annotators assess visual answerability, consistency with the official ruling, uniqueness of the correct option, and clarity.They may keep, edit, or discard items; edits can neutralize question wording and refine distractors.Source: Human Filtering and Refining · pp. 12–13
  • Human filtering removes a further 13.81% of candidate items and yields the final 3015 QA pairs after disagreements are resolved by a third annotator.Each item is reviewed by at least two annotators and retained only when they agree.Source: Human Filtering and Refining · p. 13

Dataset QA Examples

Representative examples show questions, answer options, and multi-view clips across several sports.

Dataset QA Examples

  • Each example pairs a question and four options with multi-view clips, and answering requires evidence aggregation across cameras.The figure examples span baseball, volleyball, cricket, and rugby.Source: Dataset QA Examples · p. 13

Related papers

Loading 2607.11844v2…