PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Comprehensive Basketball Understanding

Yirong Hu, Jiayuan Rao, Yu Zhang, Shangzhe Di, Weidi Xie

arXiv:2608.23435v1cs.CVcs.AI

TL;DR

Reliable basketball understanding requires integrating heterogeneous evidence, yet existing benchmarks have left interactions among these abilities underexplored. This paper introduces a multimodal benchmark and a basketball-specific agent that composes reusable skills; the agent outperforms the best commercial MLLM on eight of ten tasks, including a tie on Q9.

  • Problem

    Understanding a basketball play can require integrating event recognition, player identification, spatiotemporal localization, broadcast context, and structured basketball knowledge.

    Source: §1 · p. 1

  • Method

    The paper builds BasketballBench, a multimodal benchmark spanning ten tasks and text, image, and video, and BasketballSkills, a framework organizing basketball-specific tools into reusable skills.

    Source: §1 · p. 1, §4.1 · p. 4

  • Results

    BasketballSkills outperforms the best commercial MLLM on eight of ten benchmark tasks, including a tie on Q9.

    Source: §1 · p. 1

  • Takeaways & Limitations

    BasketballBench and BasketballSkills provide a foundation for advancing fine-grained, compositional multimodal understanding in basketball and other complex sports domains.

    Source: §6 · p. 7

    The examples reveal remaining challenges in tool reliability and execution control, including erroneous upstream detections propagating into incorrect answers.

    Source: §5.4 · pp. 6–7

Abstract

from arXiv · show

Understanding a basketball game requires recognizing events, localizing actions, identifying players, and relating these to structured game knowledge. Existing benchmarks primarily evaluate these abilities one at a time, leaving the interactions among these abilities under-explored. We introduce BasketballBench, a multimodal benchmark comprising 7,980 questions across ten tasks in text, image, and video. It is built from the 2025-2026 NBA season and includes official playby-play, rosters and profiles for 530 active players, and 2,501 possession-level broadcast clips. We further propose BasketballSkills, an agent that composes eight basketball-specific perception and retrieval tools under four reusable skills that specify tool order, evidence bindings, and stopping conditions. Experiments show that current MLLMs struggle particularly on questions requiring the integration of multiple capabilities, whereas BasketballSkills outperforms them, highlighting the effectiveness of explicitly composing domain-specific capabilities for comprehensive basketball understanding.

1 Introduction

Comprehensive basketball understanding requires integrating visual events, players, spatiotemporal actions, broadcast context, and structured knowledge. The paper introduces a multimodal benchmark and reusable skills to evaluate and compose these capabilities.

Motivation

  • A single basketball play may require combining event recognition, player identification, action localization, broadcast interpretation, and structured basketball knowledge.The introduction terms this integration of heterogeneous evidence comprehensive basketball understanding.Source: §1 · p. 1

Contributions

  • BasketballBench contains 7,980 annotated instances across ten text, image, and video tasks, spanning knowledge retrieval, broadcast and player perception, and spatiotemporal and event understanding.It includes both single-capability tests and composite questions to support model comparison and fine-grained error diagnosis.Source: §1 · p. 1
  • BasketballSkills organizes eight basketball-specific perception and retrieval tools into four reusable skills, dynamically selected by a language-model controller and validated by a lightweight verifier.The framework supports all ten tasks without predefined task labels or fixed pipelines, while keeping intermediate evidence inspectable.Source: §1 · p. 1
  • Across all benchmark tasks, BasketballSkills outperforms the best commercial MLLM on eight of ten tasks.The experiments cover all benchmark tasks.Source: §1 · p. 1

2 Related Work

Prior work advances sports perception, multimodal question answering, and modular reasoning, but leaves the interaction of sports capabilities underexplored and lacks shared, domain-grounded executable skills. BasketballSkills addresses this gap with reusable basketball-specific skills.

Sports understanding and evaluation

  • Sports research covers perception, structured basketball analysis, and broader multimodal reasoning, but most capabilities are evaluated separately and benchmarks still provide limited evaluation of their interactions.Prior work includes event and actor localization, action detection, tracking, jersey recognition, and basketball-specific calibration, identification, description, and player evaluation; newer benchmarks broaden question answering, rule and strategy reasoning, temporal evidence, and camera-view coverage.Source: §2 · p. 2

Skills for multimodal reasoning

  • Modular multimodal systems use reasoning-action loops, executable visual or video programs, and reusable skills; SportMV-Agent applies iterative planning and evidence collection to multi-view sports reasoning.Examples include ReAct (Yao et al. 2023), VisProg and ProViQ (Gupta and Kembhavi 2023), reusable skills in Voyager and MMSkills (Wang et al. 2024; Zhang et al. 2026a), and SportMV-Agent (Chen et al. 2026).Source: §2 · p. 2
  • Existing systems do not organize professional-sport capabilities into shared, domain-grounded executable skills that can be systematically evaluated; BasketballSkills represents basketball capabilities as reusable skills with explicit inputs, outputs, and dependencies.This extends modular and skill-based reasoning beyond general-purpose environments or narrowly defined reasoning settings.Source: §2 · p. 2

3 BasketballBench

BasketballBench evaluates comprehensive basketball understanding through ten text, image, and video tasks, linking visual events to spatial and temporal information, participants, identities, and structured game knowledge. It is built from 2025–2026 NBA data using aligned broadcasts, structured records, and player images.

3.1 Overview

  • BasketballBench contains 7,980 QA pairs across ten tasks and links visual events to spatial locations, timestamps, participants, on-court identities, and structured basketball knowledge.Its capability taxonomy covers domain knowledge, visual perception, spatial-temporal understanding, and event-level reasoning.Source: §3.1 · p. 2

3.2 Data Sources

  • The benchmark draws on 2,501 possession-level broadcast clips, official records and metadata, profiles for 530 active players, and player-image collections.It includes official headshots and 400 additional manually checked photographs; the additional images differ from the headshots, so identification requires recognizing players across images.Source: §3.2 · p. 2

3.3 Benchmark Tasks

  • The ten tasks span basketball knowledge and retrieval, broadcast and player perception, and spatiotemporal and event understanding.They include knowledge QA, match-situation QA, player image and video knowledge QA, scorebug reading, jersey-number recognition, action identification, action and temporal localization, and play-event sequence recovery.Source: §3.3 · pp. 2–3

3.4 Benchmark Construction

  • Ten tasks are constructed from QA templates, structured records, photographs, and event-aligned clips, with task-specific targets derived from aligned events, participants, clocks, and shot locations.Human checks include reannotating timestamps for 790 events to create 1,580 questions, reviewing all 1,000 event-sequence examples and correcting 258, and retaining 200 correct, readable examples from 300 candidates for Q4.Source: §3.4 · pp. 3–4

4 Methodology

BasketballSkills answers multimodal basketball queries by combining typed atomic tools into reusable workflows selected and coordinated by a language-model controller. The controller adapts tool calls to their observations and produces an answer after sufficient evidence is collected.

4.1 Problem Formulation

  • A query x = (q, m) combines text with multimodal and game context; controller πθ selects reusable skills and dynamically invokes typed tools whose structured outputs can inform later calls.Tool invocation is optional and determined by the query and intermediate results.Source: §4.1 · p. 4

4.2 Skills Architecture

  • BasketballSkills pairs eight typed atomic tools—seven perception tools and one structured-knowledge retrieval tool—with four reusable skills specifying tool order, evidence flow, and output construction.The skills support face-conditioned profile retrieval, shot analysis, identity-grounded play-by-play generation, and event-conditioned player knowledge retrieval. Implementations include Geitgey (2017) face matching, Qwen3.5-9B (Qwen Team 2026a) prompted visual reading, SAM 3 (Carion et al. 2025) with RF-DETR (Robinson et al. 2026) tracking, and PlayNet (Zhang et al. 2026b) event and shot-zone models.Source: §4.2 · pp. 4–5

4.3 Inference Workflow

  • At inference, the DeepSeek-V4-Flash controller (DeepSeek-AI et al. 2026) loads a relevant skill, adapts subsequent tool calls to observations, validates each call, and answers when evidence is sufficient.The controller receives the multimodal query, atomic tool library, and a lightweight skill catalog.Source: §4.3 · pp. 5–6

5 Experiments

BasketballSkills leads general-purpose MLLMs on most benchmark tasks, while procedural skills reduce tool use on video queries with comparable aggregate performance.

5.1 Experimental Setup

  • Evaluation compares three commercial and six open-source MLLMs, sampling videos at 2 fps except for Jersey Recognition’s eight uniformly sampled player crops.Choice tasks use accuracy, Q4 exact match, Q7 macro-averaged Acc@1s, and Q8 full-event F1; unparseable Q7 outputs count as incorrect.Source: §5.1 · p. 6

5.2 Main Results

  • General-purpose MLLMs perform best on direct visual-reading tasks but struggle with player–event grounding and knowledge-intensive questions; their best modality averages are 53.1% on TextQA, 70.5% on ImageQA, and 65.1% on VideoQA.Direct-task results reach 94.0% on Q4 and 95.0% on Q9, compared with 29.8% on Q8 and 53.0% on Q10; Q8 full-event F1 requires correct participants as well as event type.Source: §5.2 · p. 6, Table 3 · p. 7
  • BasketballSkills achieves the best result on eight of ten tasks, including a Q9 tie, and beats the strongest general-purpose baseline by 59.0 percentage points on Q2, 39.5 on Q3, and 30.7 on Q10.Its advantages include knowledge-intensive Q1–Q3 and compositional Q5, Q6, Q8, and Q10 tasks.Source: §5.2 · p. 6, Table 2 · p. 7

5.3 Ablations

  • Procedural skills cut VideoQA tool calls from 4.36 to 3.63 per query, a 16.74% reduction, while aggregate performance remains comparable at 72.34%.TextQA and ImageQA performance is also preserved, suggesting skill-guided control reduces unnecessary exploration in multi-tool workflows.Source: §5.3 · p. 6

5.4 Qualitative Results

  • Representative traces expose an overly cautious tool-use strategy and an erroneous Track-Level Event Detection result that propagates into an incorrect answer.The failure illustrates remaining challenges in tool reliability and execution control.Source: §5.4 · pp. 6–7

6 Conclusion

The paper introduces BasketballBench and BasketballSkills to advance multimodal basketball understanding. Experiments show that current MLLMs struggle with fine-grained and compositional questions, while BasketballSkills outperforms general-purpose MLLMs.

6 Conclusion

  • BasketballBench comprises 7,980 questions across ten tasks and text, image, and video modalities, while experiments reveal MLLM limitations in fine-grained spatial and temporal perception.Performance degrades further on compositional questions requiring multiple capabilities, including event and player recognition, action localization, and basketball knowledge.Source: §6 · p. 7
  • BasketballSkills outperforms general-purpose MLLMs by composing basketball-specific perception and retrieval capabilities into reusable procedural skills.The agent organizes basketball-specific tools into reusable skills.Source: §6 · p. 7
  • Together, BasketballBench and BasketballSkills provide a foundation for advancing fine-grained, compositional multimodal understanding in basketball and other complex sports domains.Source: §6 · p. 7

Supplementary Material · A BasketballBench Construction and Specifications

This section outlines how BasketballBench is constructed, including its data sources, structured knowledge base, common protocol, and task-specific generation procedures.

A BasketballBench Construction and Specifications

A.1 Data Sources … A.2.3 Entity Relationships

BasketballBench combines season-anchored basketball records and visual data with a normalized knowledge base. Its source metadata and database structure preserve game, player, and season context for benchmark construction and knowledge tasks.

A.1 Data Sources

  • BasketballBench integrates structured records, official player images, broadcasts, play-by-play, and game-specific uniform metadata, anchors instances in 2025–2026, and freezes sources before generation.Historical statistics and draft records are retained where needed.Source: §A.1 · p. 12
  • NBA.com headshots for 530 registered players form the face-recognition gallery, alongside 400 manually verified evaluation photographs distinct from each player’s gallery headshot.The headshots are attributed to National Basketball Association 2026b.Source: §A.1 · p. 12
  • The video set contains 2,501 possession-level broadcasts from 33 games, with game dates, teams, and active rosters linked to support play-by-play alignment and identity labels.Clips are 1280 × 720 at 25 fps, last 3.48–18.96 seconds, and average 9.45 seconds; linked metadata are not model inputs unless a task prompt includes them. The clips come from the BasketEvent test split (Zhang et al. 2026b).Source: §A.1 · p. 12
  • NBA.com provides 43,558 chronological play-by-play events from 266 games, while LockerVision uniform references enable offline conversion of player identities into visible color–number labels.Play-by-play records specify clock, action, participants, and applicable shot outcomes and locations; clips may align with several events. Uniform colors are manually verified, and LockerVision data and derived colors are not exposed to tools or evaluated models. The sources are attributed to National Basketball Association 2026b and National Basketball Association 2026a, respectively.Source: §A.1 · p. 12

A.2 Structured Basketball Knowledge Base

  • NBA_DB is an 18-table normalized SQLite database derived from Sportradar records that grounds four knowledge-oriented tasks while keeping generated questions and visual annotations outside the database.Sportradar’s NBA API supplies franchise, player, season, schedule, game, and related records.Source: §A.2 · p. 12, §A.1 · p. 12

A.2.1 Temporal Snapshot and Data Coverage

  • The primary snapshot spans the 2025–2026 season through Game 5 of the NBA Finals on June 13, 2026, with 1,423 schedule records and detailed box scores for all 1,230 regular-season games.Historical player-season statistics cover 2012–2025, and draft records span 2003–2026.Source: §A.2.1 · p. 12

A.2.2 Logical Organization

  • The database organizes records into team-and-player, season, schedule, and game-detail modules, separating season totals from per-game averages so templates can specify statistical scope.Source: §A.2.2 · p. 12

A.2.3 Entity Relationships

  • Player profiles connect to rosters, drafts, affiliations, and statistics, while schedules link home and away teams to game records, preserving player, team, season, and game context.Separating current membership from historical affiliation prevents present-day team or jersey data from being applied to another season.Source: §A.2.3 · p. 12

A.2.4 Use in Benchmark Construction … A.4.10 Q10: Player Video Knowledge QA

BasketballBench construction combines structured-source retrieval with checks linking questions to records and visual evidence. Its ten tasks span text, image, and video questions, with task-specific rules for identities, events, and answer formats.

A.2.4 Use in Benchmark Construction

  • Templates define entities, temporal constraints, relations, and answer types; knowledge questions retrieve player or game facts, and questions, answers, and distractors remain separate from the reusable NBA_DB.The two visual knowledge tasks first resolve a player from visual evidence and then retrieve the requested fact.Source: §A.2.4 · p. 12

A.3.2 Video–Record Alignment and Temporal Annotation

  • Video alignment uses readable, consistent scorebug clocks for candidate matching, but manually corrected event annotations provide Q7’s final video-time and game-clock labels.From 4,244 clips, 790 target events were manually annotated; each produced two questions, yielding 1,580 total, with annotations recording visible-event time and displayed game clock.Source: §A.3.2 · p. 13, §A.4.7 · p. 14

A.3.3 Uniform-Color Normalization

  • Color normalization maps uniform colors to controlled English terms; Q4 asks models to read verified broadcast-frame scores, periods, and clocks in fixed output formats.Q4 starts with 200 sampled frames, removes incomplete parses or invalid team metadata, and manually verifies retained fields.Source: §A.3.3 · p. 13, §A.4.4 · p. 14

A.3.1 General Generation Procedure

  • The benchmark contains 7,980 instances: 2,400 text, 600 image, and 4,980 video questions; task-specific selection and answer construction are summarized below, with shared rules in Section A.3.Source: §A.4 · p. 13

A.4.1 Q1: Basketball Knowledge QA

  • Q1 and Q2 each contain 1,200 four-way text questions, testing player and league knowledge or specific-game facts across 47 and 30 question types, respectively.Q1 distinguishes season totals from per-game averages; Q2 requires date and matchup to identify a game with the necessary records.Source: §A.4.1 · p. 13, §A.4.2 · p. 14

A.4.3 Q3: Player Image Knowledge QA

  • Q3 uses 400 player photographs to require visual identification before knowledge retrieval, while Q10 uses 1,000 possession clips to link an event-role player to database facts.Q3 photos are human-verified frontal views distinct from official headshots; Q10 excludes ambiguous multi-team cases when current-team membership is queried.Source: §A.4.3 · p. 14, §A.4.10 · p. 15

A.4.5 Q5: Action Localization QA

  • Q5 tests shot-location classification, Q6 identifies the shooter, and Q8 extracts ordered possession events with participant identities from video.Q5 retains single-shot possessions with unambiguous locations; Q6 uses team colors and jersey numbers; Q8 samples clips to preserve coverage of rare assists, steals, and blocks.Source: §A.4.5 · p. 14, §A.4.6 · p. 14, §A.4.8 · p. 15

A.4.9 Q9: Jersey-Number Recognition QA

  • Q9 presents eight chronologically sampled player crops and asks for the tracked player’s jersey number in a four-way question.Its 300 examples use distinct possession–player pairs, with annotated jerseys as answers and NBA-profile jersey numbers as distractors.Source: §A.4.9 · p. 15

B BasketballSkills Architecture and Implementation … B.4 Complete Agent-Visible Prompts

This section details BasketballSkills’ implementation, including its bounded controller, typed tools, and four evidence-composition skills with specified tool order and stopping conditions. It also describes the agent-visible prompts for three ordered workflows, which constrain how tool evidence establishes identities and how results are returned.

B.1 Agent Architecture

  • A temperature-zero DeepSeek-V4-Flash controller interprets queries, optionally loads a matching skill, invokes tools, and iterates over structured outputs under limits of 12 turns and 10 tool calls.A fixed Python verifier checks tool names, arguments, types, media and path availability, and duplicate calls; traces record execution for reproducibility, while media paths are anonymized until execution.Source: §B.1 · p. 15, §B.3 · p. 18

B.2 Atomic Tool Library

  • The eight typed, task-bounded tools cover tracking, event detection, shot-zone classification, jersey and face recognition, NBA knowledge retrieval, scorebug reading, and event-time localization.Four trajectory-dependent video tools reuse frozen precomputed artifacts when available or run online; caching affects latency, not semantics. Other tools expose constrained structured outputs, and event-time localization handles an already identified event rather than detecting one.Source: §B.2 · p. 15, §B.2.2 · p. 16, §B.2.5 · p. 17

B.2.1 Player-and-Ball Tracking / B.2.2 Track-Level Event Detection

  • Player-and-Ball Tracking combines SAM 3 masks, fine-tuned RF-DETR detections, and BoT-SORT association, while Track-Level Event Detection assigns canonical event types to tracked players.Tracking filters player trajectories using at least 0.3 IoU with RF-DETR detections in at least half of observed frames, and records normalized boxes, missing observations, stable IDs, and anomaly flags. Event detection orders predictions by temporal midpoint and keeps the highest-confidence duplicate; exact event timing requires a separate localization tool.Source: §B.2.1 · pp. 15–16, §B.2.2 · p. 16

B.2.1 Player-and-Ball Tracking

  • The RF-DETR detector reached 0.6504 mAP@50:95 on the test set, compared with 0.3882 mAP@50:95 for the mapped official COCO model.The selected epoch-25 EMA checkpoint also scored 0.8709 mAP@50, 0.7303 mAP@75, and 0.7489 AR@100 on 200 test images.Source: §B.2.1 · p. 16

B.2.3 Shot-Zone Classification

  • On 2,219 test examples, Shot-Zone Classification achieved 0.7071/0.9148/0.9739 top-1/2/3 accuracy and 0.6935 macro F1.The epoch-13 checkpoint was selected by validation macro-F1; it also reached 0.6813 macro precision, 0.7610 macro recall, and 0.7100 weighted F1.Source: §B.2.3 · p. 17

B.3 Composite Skill Library

  • Four composite skills coordinate tools for shot analysis, event-conditioned and face-conditioned player knowledge retrieval, and identity-grounded play-by-play generation.For shot analysis, tracking and event detection identify the shooter before zone classification or jersey recognition; play-by-play orders events and resolves participant tracks with jersey recognition.Source: §B.3 · p. 18, §B.4.3 · p. 20

B.4 Complete Agent-Visible Prompts

  • Controller-visible prompts specify tool dependencies and skill-first workflows, require answers grounded in collected evidence when tools are used, and prohibit repeated identical tool calls.Skills are loaded when their recurring evidence pattern matches; their instructions prescribe tool order and conditions such as grounding a shooter role in event detection or resolving identity before querying player knowledge.Source: §B.4.1 · pp. 18–19, §B.4 · p. 18, §B.4.3 · pp. 20–21

B.4.3 Agent-Visible Skill Prompts

  • Event-conditioned retrieval links detected event roles to jersey-resolved identities, then queries the roster and player database for the requested fact.It uses the team and jersey number together, does not query player facts before resolving identity, and stops if the participant, team, number, or roster identity cannot be resolved.Source: §B.4.3 · p. 21
  • Face-conditioned retrieval uses recognize_face as the sole identity evidence, then queries the NBA database for the requested fact and stops if recognition fails.It excludes answer-choice or basketball-knowledge inference and does not call tracking, event, jersey, or shot-zone tools.Source: §B.4.3 · p. 22
  • Identity-grounded play-by-play sequences tracked entities, detected event roles, and jersey recognition, joining identities by track ID and using question-provided team mapping.Event types, roles, and order come from detect_events; jersey information comes from recognize_jersey, and uncertainty is preserved when a participant identity cannot be grounded.Source: §B.4.3 · pp. 22–23

C Additional Evaluation Results … C.2.2 Action Localization QA (Q5)

The additional evaluation results make scoring conventions explicit and provide fine-grained results for basketball QA tasks. They show that action localization is strongest on corner threes and weakest in non-restricted-area paint regions.

C Additional Evaluation Results

  • The additional evaluation section defines non-multiple-choice scoring, makes parsing-failure treatment explicit, and reports fine-grained results.Its fine-grained results are breakdowns of the experiments reported in the main paper’s Table 2.Source: C Additional Evaluation Results · p. 23, §C.2 · p. 24

C.1 Evaluation Metrics

  • For accuracy-based tasks, all-example accuracy equals parse rate multiplied by parsed accuracy, so missing, failed, or unparsable responses receive zero credit.Multiple-choice tasks use option accuracy; other scores are percentages, and temporal errors are measured in seconds unless noted otherwise.Source: §C.1 · p. 23, C Additional Evaluation Results · p. 23

C.1.1 Scorebug Reading (Q4)

  • Q4’s Full Exact Match requires every score field or both clock fields to match, and missing or unparseable responses count as incorrect.Scorebug Reading has score and clock subsets; clock matching requires both period and normalized MM:SS to match.Source: §C.1.1 · p. 23

C.1.2 Temporal Localization QA (Q7)

  • Q7’s primary score is the macro-average of all-example Acc@1s for video-timestamp and game-clock predictions, equivalent to accuracy over all 1,580 questions.Game-clock Acc@1s requires the period to match; its MAE excludes period mismatches, and deterministic normalization maps ordinal period labels to numeric labels before fine-grained evaluation.Source: §C.1.2 · pp. 23–24, Table C.3 · p. 25

C.1.3 Play Event QA (Q8)

  • Q8 uses LCS-based F1 with type-only and type-plus-participant matching; participant accuracy is diagnostic because it does not penalize missing, additional, or mistyped events.A deterministic recovery procedure repairs only unambiguous malformed JSON structure without changing event types, participants, or order; outputs still invalid afterward count as empty sequences.Source: §C.1.3 · p. 24

C.2 Fine-Grained Results

  • The fine-grained scorebug results show team identification has the lowest accuracy, largely because models struggle to map team logos to the correct teams.Parse rates are generally high, so parsed and all-example accuracies are close for most models; the reported field accuracies use the conservative all-example measure.Source: §C.2.1 · p. 24, Table C.1 · p. 25

C.2.1 Scorebug Reading (Q4)

  • For Q4, the detailed results report all-example slot accuracies for team and score recognition and exact-match accuracy for the game clock.An unparseable or missing output is incorrect for each corresponding field.Source: §C.2.1 · p. 24, Table C.1 · p. 25

C.2.2 Action Localization QA (Q5)

  • 90.0% left and 83.0% right Corner Threes were most distinctive, while In The Paint (Non-RA) was hardest at 44.0%, chiefly confused with Restricted Area and Mid-Range.Table C.2 summarizes the element-wise average of two BasketballSkills runs’ row-normalized confusion matrices.Source: Table C.2 · p. 25, §C.2.2 · p. 26

C.2.3 Temporal Localization QA (Q7)

Temporal Localization QA evaluates event timing using either video timestamps or the scorebug’s game clock. The reported results show that parsing conventions affect measured localization accuracy, while some video-timestamp outputs cannot be uniquely converted into point predictions.

C.2.3 Temporal Localization QA (Q7)

  • The task evaluates event timing in two settings: the video timestamp of the event or the time displayed on the scorebug’s game clock.The first asks when the event occurs in the clip; the second asks for the displayed game-clock time.Source: §C.2.3 · p. 26
  • 84 of 790 Qwen2.5-VL-7B outputs were recovered, raising Parse Rate from 89.2% to 99.9% and all-example Acc@1s from 39.1% to 44.9%.Normalization also recovered outputs for Claude Sonnet 5 (5), Qwen3.5-4B (53), Qwen3.5-9B (7), and InternVL3.5-8B (78). It uses a fixed equivalence between period-label spellings, without ground-truth times, to measure localization rather than format compliance.Source: §C.2.3 · p. 26
  • 415 of Qwen3.5-9B’s 790 video-timestamp predictions were intervals, leaving 47.5% parseable and 22.9% all-example Acc@1s.Among those intervals, choosing the start, midpoint, or end made 143, 165, or 174 correct; with 181 correct point-valued predictions, the resulting all-example Acc@1s values were 41.0%, 43.8%, and 44.9%. Counting intervals that overlap the reference within one second yielded 50.5%, close to Qwen3.5-4B’s 51.0%; intervals were not repaired because they imply no unique point-valued conversion.Source: §C.2.3 · p. 26
  • VideoLLaMA3-7B produced no parseable Q7 outputs, so its all-example Acc@1s was 0.0 and parsed-output metrics were unavailable.The passage also reports that further format-error diagnostics were unavailable.Source: §C.2.3 · p. 26

C.2.4 Play Event QA (Q8)

On Play Event QA, BasketballSkills outperforms MLLMs on key metrics except event-type recall, while event-to-player association exposes a major MLLM weakness.

C.2.4 Play Event QA (Q8)

  • Without player identification, GPT achieves 98% of BasketballSkills’ F1 score, but when both events and players must be identified, this falls to 65%.MLMs also achieve markedly higher precision, recall, and F1 when predicting only event labels than when jointly identifying events and players.Source: §C.2.4 · p. 26
  • Qwen2.5-VL-7B, Qwen3.5-4B, and Qwen3.5-9B recovered 11, 189, and 366 outputs, raising Parse Rates from 98.9%, 81.1%, and 63.3% to 100.0%, 100.0%, and 99.9%, respectively.Their repaired all-example F1 scores are 1.6%, 8.2%, and 10.4%, respectively. Recovery exposes predicted content rather than optimizing scores, since making invalid predictions visible can add both matched and unmatched events.Source: §C.2.4 · p. 26

Related papers

Loading 2608.23435v1…