PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Comprehensive Basketball Understanding

Yirong Hu, Jiayuan Rao, Yu Zhang, Shangzhe Di, Weidi Xie

arXiv:2608.23435v1cs.CVcs.AI

TL;DR

Existing basketball benchmarks under-explore the integration of event recognition, player identification, action localization, and structured game knowledge. This paper introduces BasketballBench and BasketballSkills, a multimodal benchmark and a skill-composing agent, and reports stronger performance from BasketballSkills across most benchmark tasks while identifying remaining tool-reliability challenges.

  • Problem

    Existing benchmarks mainly evaluate basketball capabilities one at a time, leaving their interactions under-explored despite the need to integrate heterogeneous evidence.

  • Method

    The paper constructs BasketballBench and develops BasketballSkills, which composes eight basketball-specific tools through four reusable skills selected and coordinated by a controller.

  • Results

    BasketballSkills achieves the best result on eight of ten tasks, while general-purpose MLLMs perform worse on compositional and knowledge-grounded questions.

  • Takeaways & Limitations

    The results support explicitly composing specialized perception and retrieval capabilities for comprehensive basketball understanding.

  • Takeaways & Limitations

    Tool unreliability and execution-control errors remain challenges, including upstream event-detection failures that can propagate into incorrect answers.

Abstract

from arXiv · show

Understanding a basketball game requires recognizing events, localizing actions, identifying players, and relating these to structured game knowledge. Existing benchmarks primarily evaluate these abilities one at a time, leaving the interactions among these abilities under-explored. We introduce BasketballBench, a multimodal benchmark comprising 7,980 questions across ten tasks in text, image, and video. It is built from the 2025-2026 NBA season and includes official playby-play, rosters and profiles for 530 active players, and 2,501 possession-level broadcast clips. We further propose BasketballSkills, an agent that composes eight basketball-specific perception and retrieval tools under four reusable skills that specify tool order, evidence bindings, and stopping conditions. Experiments show that current MLLMs struggle particularly on questions requiring the integration of multiple capabilities, whereas BasketballSkills outperforms them, highlighting the effectiveness of explicitly composing domain-specific capabilities for comprehensive basketball understanding.

1 Introduction

Reliable basketball understanding requires integrating recognition, player identification, spatiotemporal localization, broadcast interpretation, and structured knowledge. BasketballBench evaluates these capabilities individually and compositionally, while BasketballSkills organizes them into executable workflows.

  • Professional basketball understanding requires integrating heterogeneous visual, temporal, spatial, player, broadcast, and structured-knowledge evidence.
  • Existing sports benchmarks often evaluate capabilities separately across different annotations, output formats, and sports.
  • BasketballBench contains 7,980 annotated instances across ten text, image, and video tasks spanning knowledge retrieval, perception, and spatiotemporal understanding.
  • BasketballSkills comprises eight atomic tools and four reusable procedural skills that dynamically organize multi-step basketball workflows.
  • BasketballSkills outperforms the best commercial MLLM on eight of the ten benchmark tasks.

2 Related Work

Prior sports research covers many perception, reasoning, and analysis capabilities, but modular systems have not explicitly organized professional-sport capabilities into domain-grounded, systematically evaluated skills. BasketballSkills addresses this gap with reusable skills having explicit interfaces and dependencies.

  • Sports-understanding research spans player and event perception, localization, tracking, jersey recognition, structured interpretation, and multimodal reasoning.
  • Existing modular and skill-based systems primarily target general-purpose environments or narrowly defined reasoning settings.
  • BasketballSkills represents professional-basketball capabilities as reusable skills with explicit inputs, outputs, and dependencies.

3 BasketballBench

BasketballBench is a ten-task multimodal benchmark built from complementary NBA data sources and organized by evidence type and reasoning complexity. Its tasks range from knowledge retrieval and broadcast perception to spatial, temporal, and structured event understanding.

  • BasketballBench contains 7,980 QA pairs across ten text, image, and video tasks organized around knowledge, perception, spatial-temporal understanding, and event reasoning.
  • The benchmark is built from three complementary data sources from the 2025–2026 NBA season.
  • It includes 2,501 possession-level broadcast clips aligned with source games and structured play-by-play records.
  • Structured records include official play-by-play, schedules, team metadata, uniform assignments, coaching staffs, rosters, and profiles for 530 active players.
  • The ten tasks cover basketball knowledge and retrieval, broadcast and player perception, and spatiotemporal and event understanding.
  • Task construction combines structured-record templates with player photographs or event-aligned clips, and model-assisted labels are manually verified.
  • Temporal-localization questions receive full manual timestamp reannotation because official play-by-play timestamps can lag visible events.

4 Methodology

BasketballSkills combines typed atomic tools, reusable composite skills, and a language-model controller to answer heterogeneous multimodal basketball questions. The controller dynamically selects skills, executes tools, validates calls, and stops when sufficient evidence is collected.

  • Problem Formulation: BasketballSkills consists of an atomic tool library, a reusable skill library, and a language-model controller coordinating executions.
  • Problem Formulation: Tools are typed operators that return structured results usable by the controller or subsequent tools.
  • Problem Formulation: Tool invocation is optional and dynamically determined by the query and intermediate results.
  • Tool and Skill Libraries: The architecture contains eight perception and knowledge tools, including face, jersey, scorebug, tracking, event, shot-zone, temporal-localization, and knowledge-retrieval tools.
  • Inference Workflow: The controller selectively loads relevant skills, invokes specialized tools, adapts calls to observations, validates each call, and generates an answer after sufficient evidence is collected.
  • Tool and Skill Libraries: Four reusable skills specify tool ordering, intermediate evidence flow, and task-specific output construction.

5 Experiments

Experiments show that general-purpose MLLMs handle direct visual reading better than player–event grounding, knowledge-intensive retrieval, and compositional basketball questions. BasketballSkills leads on most tasks and procedural skills reduce video tool calls while preserving performance, although upstream tool errors can propagate.

  • General-purpose MLLMs: General-purpose MLLMs perform best on direct OCR-style tasks but struggle to bind events to the relevant players and recover complete structured events.Event-type F1 is consistently higher than full-event F1, which also requires participant correctness.
  • General-purpose MLLMs: 53.1% on TextQA, 70.5% on ImageQA, and 65.1% on VideoQA are the best modality-level averages for general-purpose MLLMs.Direct tasks reach 94.0% on Q4 and 95.0% on Q9, whereas demanding Q8 and Q10 results reach only 29.8% and 53.0%.
  • BasketballSkills: BasketballSkills achieves the best result on eight of ten tasks, including a tie on Q9, and surpasses the strongest general-purpose baseline by 59.0 percentage points on Q2.It also exceeds that baseline by 39.5 percentage points on Q3 and 30.7 percentage points on Q10.
  • BasketballSkills: BasketballSkills improves knowledge-intensive and compositional tasks by composing complementary skills for event recognition, player grounding, OCR, spatial understanding, and retrieval.The reported advantages extend to Q1–Q3, Q5, Q6, Q8, and Q10.
  • Ablations: 16.74% fewer tool calls on VideoQA reduces the average from 4.36 to 3.63 per query while aggregate performance remains comparable at 72.34%.TextQA and ImageQA performance is likewise preserved.
  • Qualitative Results: An erroneous Track-Level Event Detection result can propagate through later grounding and answer generation, revealing remaining challenges in tool reliability and execution control.The workflow also includes an example of unnecessary Scorebug Reading, indicating an overly cautious tool-use strategy.

6 Conclusion

The paper introduces BasketballBench and BasketballSkills to address the difficulty of comprehensive basketball understanding, especially when questions require multiple capabilities. BasketballSkills outperforms general-purpose MLLMs, while procedural skills reduce tool use on complex video queries without reducing aggregate performance.

  • BasketballBench contains 7,980 multimodal questions across ten tasks covering individual capabilities and their composition.
  • Current MLLMs perform worse on compositional questions requiring event recognition, player identification, spatiotemporal localization, and basketball knowledge.
  • BasketballSkills organizes basketball-specific tools into reusable procedural skills and outperforms general-purpose MLLMs on comprehensive basketball understanding.
  • Figure 3 illustrates that upstream tool errors can propagate through evidence composition to produce an incorrect final answer.

Supplementary Material

The supplementary material provides additional paper context and describes the data sources, knowledge base, construction protocol, and task-specific generation procedures used for BasketballBench.

  • The paper is titled “Towards Comprehensive Basketball Understanding.”
  • The supplementary material describes the data sources used to construct BasketballBench.
  • It also covers the structured knowledge base, common construction protocol, and task-specific generation procedures.

A.1 Data Sources

BasketballBench combines structured NBA records, official player imagery, broadcast clips, play-by-play events, and uniform metadata. These sources are normalized and linked to preserve player, team, season, and game context for benchmark construction.

  • BasketballBench integrates structured records, official player images, possession-level broadcasts, play-by-play records, and game-specific uniform metadata.
  • Sportradar provides information on all 30 franchises, including teams, players, schedules, statistics, box scores, officials, injuries, drafts, and free agents.
  • The image gallery contains one official headshot for each of 530 registered players, alongside 400 distinct manually verified evaluation photographs.
  • The broadcast set contains 2,501 possession-level clips from 33 games, while NBA.com provides 43,558 chronological play-by-play events from 266 games.
  • NBA_DB is a normalized SQLite knowledge base whose modules separate team and player records, season records, schedules, and game details.
  • Links among profiles, rosters, affiliations, statistics, schedules, and game records preserve season-specific player, team, and game context.

A.2.4 Use in Benchmark Construction

BasketballBench uses typed templates and linked visual, game, and database evidence to generate unambiguous questions across text, image, and video tasks. The construction process validates records, annotations, metadata, and answer choices before retaining instances.

  • Templates specify required entities, temporal constraints, relations, and answer types for knowledge and match-situation questions.
  • Task construction removes missing fields, unresolved links, duplicate prompts, and cases admitting multiple answers before quota-based sampling.
  • Multimodal instances link evaluation images to official headshots and database profiles, while videos align play-by-play events with participants and game metadata.
  • Video alignment uses source-game and play-by-play intervals, 1-fps sampling, and scorebug reading to map clip time to game time.
  • The benchmark contains 7,980 instances: 2,400 text, 600 image, and 4,980 video questions.
  • Scorebug Reading QA recovers teams, scores, period, and game clock from broadcast graphics, retaining manually verified fields.

B BasketballSkills Architecture and Implementation

BasketballSkills implements domain-specific perception and retrieval through typed tools, validated execution, and reusable workflows. Its shot-zone classifier reaches 0.6935 macro F1 and 0.7100 weighted F1 on 2,219 test examples.

  • Agent Architecture: The controller interprets each query, optionally loads a skill, invokes tools, and iterates over structured outputs within bounded execution.Runs allow at most 12 controller turns and 10 tool calls, while a verifier checks names, arguments, media availability, and duplicate calls.
  • Tool Library: BasketballSkills exposes eight typed atomic tools covering player and ball perception, event understanding, shot zones, jersey and face recognition, and structured knowledge retrieval.The library contains seven perception tools and one knowledge tool, with structured outputs for downstream composition.
  • Tracking: Player-and-ball tracking combines SAM 3, RF-DETR, and BoT-SORT to produce normalized trajectories, stable identifiers, missing observations, and anomaly flags.The detector–tracker pipeline filters referees, bench players, and staff using trajectory overlap and player detections.
  • Shot-Zone Classification: Shot-Zone Classification predicts six court regions from a made- or missed-shot track identified by Track-Level Event Detection.It uses a new six-class head on PlayNet’s pretrained PlayerEventModel and excludes isolated free throws and independent shooter selection.
  • Shot-Zone Classification: 0.7071/0.9148/0.9739 top-1/2/3 accuracy and 0.6935 macro F1 are obtained on 2,219 shot-zone test examples.The same evaluation reports 0.6813 macro precision, 0.7610 macro recall, and 0.7100 weighted F1.
  • Identity Tools: Jersey Recognition returns color, number or null, and confidence, while Face Recognition returns a matched identity only when face distance is at most 0.6.Color-plus-number output supports disambiguation of opponents sharing jersey numbers; insufficient face evidence yields null.

C Additional Evaluation Results

The appendix defines conservative evaluation procedures for parsing, exact matching, temporal localization, and ordered event extraction. Additional results identify task-specific difficulty patterns, including weaker team identification and confusion among adjacent shot zones.

  • Metrics: BasketballBench reports option accuracy for multiple-choice tasks, percentages for other scores, and seconds for temporal errors unless noted otherwise.The evaluation explicitly distinguishes parsed accuracy from all-example accuracy and treats unparsable outputs as zero credit in all-example accuracy.
  • Scorebug Reading: Scorebug Full Exact Match requires every score or clock field to match, with unparseable or missing responses counted as incorrect.Clock correctness requires matching both period and normalized MM:SS; team and score measures are slot-level accuracies.
  • Temporal Localization: Temporal Acc@1s requires absolute error of at most one second and, in game-clock mode, a matching period.The primary Q7 score averages video-timestamp and game-clock all-example Acc@1s across 1,580 questions.
  • Play Event QA: Q8 uses longest-common-subsequence matching to score ordered event sequences under Type and Type+Participant rules.The primary metric is F1Type+Participant, while F1Type isolates event-type recognition; deterministic JSON recovery precedes scoring.
  • Play Event QA: Participant Accuracy evaluates participant correctness only among event pairs aligned by the Type-only LCS.It is diagnostic because it does not directly penalize missing, additional, or incorrectly typed events.
  • Scorebug Reading: Team identification is the weakest Scorebug Reading component because MLLMs struggle to associate logos with the correct teams.Parse rates are generally high, so parsed and all-example accuracies are close for most models.
  • Action Localization: 90.0% left and 83.0% right Corner Threes are most distinctive, while In The Paint (Non-RA) is hardest at 44.0%.The hardest region is chiefly confused with adjacent Restricted Area and Mid-Range zones.

C.2.3 Temporal Localization QA (Q7)

Temporal Localization QA evaluates whether models can report event timing in either video-time or game-clock coordinates, with format handling materially affecting measured performance.

  • Temporal Localization QA measures event timing under video-timestamp and game-clock settings.The video setting asks for the clip timestamp, while the game-clock setting asks for the scorebug time.
  • 84 of 790 Qwen2.5-VL-7B outputs were recovered through Period-Alias Normalization, raising Parse Rate from 89.2% to 99.9%.The repair uses fixed equivalences between period-label spellings and does not access ground-truth time.
  • Qwen2.5-VL-7B’s all-example Acc@1s increased from 39.1% to 44.9% after period-alias normalization.The adjustment is intended to measure temporal localization rather than format compliance.
  • Qwen3.5-9B produced 415 interval-valued predictions, leaving 47.5% of video-timestamp outputs parseable and yielding 22.9% all-example Acc@1s.Alternative interval conversions raised Acc@1s to 41.0%, 43.8%, or 44.9%, but no unique conversion was assumed for reported results.
  • VideoLLaMA3-7B had no parseable Q7 outputs and was assigned 0.0 all-example Acc@1s.Parsed-output metrics and additional format-error diagnostics were therefore unavailable.

C.2.4 Play Event QA (Q8)

Play Event QA tests recovery of ordered events and participants, revealing that jointly identifying event types and players is substantially harder for current MLLMs than predicting event labels alone.

  • BasketballSkills substantially outperformed MLLMs on all key Play Event QA metrics except event-type recall.
  • GPT achieved 98% of BasketballSkills’ F1 score when predicting event labels alone, but only 65% when identifying events and associated players.The comparison isolates event-to-player association as a major weakness of current MLLMs.
  • Conservative JSON Recovery retrieved 11, 189, and 366 outputs for Qwen2.5-VL-7B, Qwen3.5-4B, and Qwen3.5-9B, respectively.The corresponding Parse Rates increased to 100.0%, 99.9%, and 99.9%.
Loading 2608.23435v1…