PaperlayerView exact revision on arXiv

Source-linked AI summary

Stepping VLMs onto the Court: Benchmarking Spatial Intelligence in Sports

Yuchen Yang, Yuqing Shao, Duxiu Huang, Linfeng Dong, Yifei Liu, Suixin Tang, Xiang Zhou, Yuanyuan Gao, Wei Wang, Yue Zhou, Xue Yang, Yanfeng Wang, Xiao Sun, Zhihang Zhong

arXiv:2603.09896v1cs.CV

TL;DR

Existing spatial-intelligence benchmarks provide limited evidence about VLM reasoning in dynamic sports scenes involving human motion and object interactions. CourtSI addresses this gap with court-anchored 3D reconstruction, scalable QA generation, and CourtSI-Bench evaluation, finding persistent human–AI and generalization gaps alongside substantial fine-tuning gains. The results position CourtSI as a scalable pathway for advancing sports spatial intelligence.

  • Problem

    Existing spatial datasets focus mainly on static scenes and rigid objects, providing limited coverage of dynamic human motion and interactions that sports expose.

  • Method

    CourtSI reconstructs metric-scale 3D sports scenes using court geometry, converts them into over 1M categorized QA pairs, and evaluates models with CourtSI-Bench.

  • Results

    A 23.5-percentage-point accuracy improvement follows fine-tuning Qwen3-VL-8B on CourtSI, while evaluations reveal a human–AI gap and limited generalization from existing benchmarks.

  • Takeaways & Limitations

    Sports scenarios expose spatial-intelligence limitations not sufficiently captured by existing benchmarks, and CourtSI supports broader evaluation through fine-tuning, unseen-sport testing, and commentary generation.

  • Takeaways & Limitations

    CourtSI-Bench contains spatial ambiguities that challenge accurate 3D localization and fine-grained relational understanding in current VLMs.

Abstract

from arXiv · show

Sports have long attracted broad attention as they push the limits of human physical and cognitive capabilities. Amid growing interest in spatial intelligence for vision-language models (VLMs), sports provide a natural testbed for understanding high-intensity human motion and dynamic object interactions. To this end, we present CourtSI, the first large-scale spatial intelligence dataset tailored to sports scenarios. CourtSI contains over 1M QA pairs, organized under a holistic taxonomy that systematically covers spatial counting, distance measurement, localization, and relational reasoning, across representative net sports including badminton, tennis, and table tennis. Leveraging well-defined court geometry as metric anchors, we develop a semi-automatic data engine to reconstruct sports scenes, enabling scalable curation of CourtSI. In addition, we introduce CourtSI-Bench, a high-quality evaluation benchmark comprising 3,686 QA pairs with rigorous human verification. We evaluate 25 proprietary and open-source VLMs on CourtSI-Bench, revealing a remaining human-AI performance gap and limited generalization from existing spatial intelligence benchmarks. These findings indicate that sports scenarios expose limitations in spatial intelligence capabilities captured by existing benchmarks. Further, fine-tuning Qwen3-VL-8B on CourtSI improves accuracy on CourtSI-Bench by 23.5 percentage points. The adapted model also generalizes effectively to CourtSI-Ext, an evaluation set built on a similar but unseen sport, and demonstrates enhanced spatial-aware commentary generation. Together, these findings demonstrate that CourtSI provides a scalable pathway toward advancing spatial intelligence of VLMs in sports.

1. Introduction

CourtSI introduces sports as a challenging setting for fine-grained spatial intelligence, addressing limited coverage of dynamic human motion in existing benchmarks. Its reconstruction engine and QA generation pipeline support scalable dataset construction, while evaluations expose current VLM limitations and gains from fine-tuning.

  • Existing spatial datasets mainly cover static scenes and rigid objects, leaving dynamic human motion and articulated interactions underexplored.Sports provide a challenging testbed because they combine high-intensity human motion with dynamic object interactions.
  • CourtSI and CourtSI-Bench establish a sports-focused testbed for fine-grained, human-centric spatial reasoning beyond activity-centric understanding.The benchmark extends sports VLM evaluation toward spatial counting, distance measurement, localization, and relational reasoning.
  • The semi-automatic data engine reconstructs metric-scale 3D sports scenes by anchoring camera and object locations to court geometry.Court correspondences support camera calibration and a unified world coordinate system, while player meshes and annotated ball positions populate the reconstructed scenes.
  • CourtSI converts reconstructed badminton, tennis, and table-tennis states into over 1M QA pairs spanning four spatial reasoning categories.Answers are automatically derived from reconstructed 3D states, enabling scalable training-data curation.
  • Evaluations of 25 VLMs reveal a human-performance gap, weak transfer from existing spatial benchmarks, and a 23.5-percentage-point accuracy gain after fine-tuning Qwen3-VL-8B.The evaluation also examines cross-sport generalization and spatial-aware commentary generation.

2. Related Work

Spatial-intelligence research extends VLM evaluation beyond image-plane reasoning toward 3D spatial relationships. Related benchmarks probe varied spatial dimensions, while sports understanding has developed through action, analysis, captioning, commentary, and unified task benchmarks.

  • Spatial intelligence characterizes VLM capabilities for reasoning about relationships among perceived 3D objects.The research direction responds to concerns that web-scale training primarily emphasizes relationships on the image plane.
  • Existing spatial benchmarks evaluate capabilities such as indoor spatial-layout browsing, sparse-view reasoning, and other dimensions of spatial understanding.These efforts combine dedicated datasets with specialized approaches for characterizing spatial capabilities.
  • Sports understanding research spans action recognition, sports analysis, captioning, commentary generation, and unified multi-task benchmarks.Language models have accelerated progress particularly in captioning and commentary generation.

3. CourtSI Dataset

CourtSI uses a semi-automatic, court-grounded reconstruction pipeline to generate large-scale spatial QA data from broadcast net-sport scenes. The resulting dataset and benchmark cover diverse spatial reasoning categories with separate scenes and quality-control procedures.

  • 3.2. Dataset Curation: CourtSI converts reconstructed 3D states into QA pairs spanning spatial counting, distance measurement, localization, and relational reasoning.Questions target balls, players, courts, camera and world views, and include numerical and multiple-choice formats using 94 templates.
  • 3.1. Data Engine: The data engine reconstructs sports scenes by estimating metric-aware cameras, localizing balls, and recovering player meshes in a world-grounded coordinate system.It uses court geometry, human-in-the-loop supervision, ball projection annotation, and mesh recovery to address unreliable monocular metric depth.
  • 3.2. Dataset Curation: The data use broadcast-view badminton, tennis, and table-tennis images filtered for extreme viewpoints before 3D reconstruction and QA generation.CourtSI-Bench maintains a relatively balanced distribution across sports for evaluation.
  • 3.2. Dataset Curation: 1,008,941 QA pairs are generated from 52,481 images across 1,057 scenes, while CourtSI-Bench contains 3,686 QA pairs from 1,988 images across 382 scenes.The dataset and benchmark have no scene overlap, preventing potential information leakage.
  • 3.2. Dataset Curation: CourtSI-Bench undergoes human verification, with two annotators reviewing reconstructed-scene visualizations and removing any QA pair flagged as incorrect.A separate multi-view dataset with calibrated cameras and triangulated locations is used to assess the data engine because monocular broadcast videos lack ground-truth 3D annotations.

4. Experiment

CourtSI-Bench evaluates 25 VLMs, human performance, and CourtSI fine-tuning across sports spatial reasoning tasks. Results reveal persistent metric-level and generalization challenges, while fine-tuning improves benchmark and downstream performance.

  • Evaluation Setup: 25 VLMs are evaluated on CourtSI-Bench using exact-match accuracy and T-MRA for numerical distance-measurement and localization answers.T-MRA permits a bounded error, with confidence thresholds from 0.5 to 0.95 and a 15 cm distance threshold.
  • Human Level Performance: Human evaluators outperform existing models across metrics, but their performance drops on metric-sensitive distance-measurement and localization tasks.The evaluation uses two volunteers on a uniformly sampled 5% subset of CourtSI-Bench.
  • Proprietary Models: Gemini-Pro achieves the best overall proprietary-model performance across most metrics, while uncontrolled outputs from Gemini3-Pro and Claude-Sonnet-4.5 require answer parsing.Without the additional LLM post-processing step, performance drops significantly.
  • Open-source General Models: Most open-source models achieve below 40% overall accuracy, and some nearly fail distance measurement under the loose T-MRA metric.Qwen3-VL-235B-A22B is the strongest open-source general model, with a limited gap relative to the best proprietary models.
  • Open-source Spatial Intelligence Models: Spatial-intelligence fine-tuning does not consistently improve performance over corresponding base models, indicating that existing benchmarks miss challenges in dynamic sports scenarios.The finding applies to models fine-tuned for spatial relationships and metric distance measurement.
  • SFT on CourtSI: Qwen3-VL-8B gains 23.5 percentage points in accuracy after CourtSI supervised fine-tuning, including more than 25 percentage points on distance measurement.The model improves consistently across all evaluation metrics.
  • Error Analysis: Error cases show that VLMs can form structured reasoning chains but still struggle with accurate 3D localization from 2D imagery and fine-grained relational understanding.Failures involve incorrect relationships relative to court geometry and counterfactual player configurations that introduce spatial ambiguity.
  • CourtSI-Ext: CourtSI-Ext contains 215 QA pairs from 111 images across 35 pickleball scenes, providing an unseen-sport cross-sport evaluation.The sport has court geometry similar to tennis and badminton.

5. Conclusion

CourtSI introduces a large-scale sports spatial-intelligence dataset and benchmark, using court geometry to support scalable data production and evaluation. Results across 25 VLMs show persistent limitations, while fine-tuning and broader evaluation demonstrate the dataset’s utility.

  • CourtSI contains over 1M QA pairs, alongside CourtSI-Bench for high-quality evaluation.
  • Court geometry serves as a metric anchor for a semi-automatic data engine that produces accurate and scalable supporting data.
  • Evaluation across 25 state-of-the-art VLMs reveals a clear human–AI performance gap and limited generalization from existing spatial-intelligence benchmarks.
  • Fine-tuning, cross-sport evaluation, and commentary generation broaden CourtSI-Bench’s evaluation scope and support its use for advancing sports spatial intelligence.

A.1. Pipeline Details

The pipeline calibrates camera parameters from court geometry, annotates balls through ground-plane projection, and recovers player meshes for metric sports-scene reconstruction. It also applies trajectory modeling and depth correction to improve reconstructed scene states.

  • Court and camera reconstruction: Court keypoints are calibrated with a PnP solver, and the resulting world coordinate system is reused across static-camera frames or propagated to dynamic views.The coordinate system is anchored to standardized court geometry and verified through reprojection.
  • Ball annotation: Ball trajectories are modeled as constant-acceleration motion, using annotated start, midpoint, and endpoint positions to estimate airborne motion.This reduces annotation effort for airborne ball tracking.
  • Player mesh recovery: PromptHMR estimates player meshes in camera coordinates from player detections and camera parameters, with manual refinement for inaccurate detections.The recovered meshes use the SMPL-X representation.
  • Depth correction: Estimated human-mesh depths are corrected manually and re-aligned with a similarity transformation centered at the camera.The transformation preserves 3D scale while uniformly rescaling the mesh along camera rays.

A.2. Comparison with Monocular Scene Reconstruction Methods

The appendix compares CourtSI’s reconstruction components with monocular methods and specifies the benchmark’s query-generation and output conventions. Results indicate strong calibration and player localization but unreliable metric ball depth.

  • Camera calibration: Camera-intrinsic errors are reported as percentage averages over video sequences for each sport, and the proposed calibration performs best across all three sports.The method explicitly leverages court geometry.
  • Ball localization: DepthAnythingV3 fails to estimate accurate metric ball depth even with ground-truth 2D ball locations and camera parameters.The ball’s small image footprint is identified as a likely source of failure.
  • Player localization: The proposed player-localization method outperforms baseline approaches using 3D pelvis-position error as the metric.
  • Query construction: Benchmark queries combine a pre-prompt, question, and post-prompt, spanning 13 primary categories, 20 generated types, and 94 unique templates.Outputs are constrained to numerical formats or single-option multiple-choice responses.
  • Query construction: The templates cover distance, counting, visibility, coordinates, and relational questions involving court, camera, players, and balls.Camera-perspective left/right references are distinguished from anatomical body-part references.

B.4. Data Distribution

CourtSI-Bench has a relatively balanced distribution across badminton, tennis, and table tennis. A table-tennis-specific Player-Zone exclusion maintains annotation validity because players do not stand on the table surface.

  • CourtSI-Bench’s data distribution across badminton, tennis, and table tennis is relatively balanced.Table 7 reports sample counts and per-sport percentages for CourtSI-Bench and CourtSI.
  • Player-Zone instances are excluded for table tennis because players do not stand on the table surface.The exclusion preserves annotation validity and consistency.

C.1. Evaluation on CourtSI-Bench Details

CourtSI-Bench uses interactive human evaluation with court-size references and task-specific metrics for localization. Localization predictions are scored by whether their 3D error falls within a 30 cm threshold.

  • Human Evaluation: Human evaluators receive the image, question, and sport-specific court size through an interactive panel.The information shown matches what VLMs receive.
  • Localization Metric: Localization outputs are represented as 3D coordinates, so relative distance error is replaced by binary accuracy.The metric assigns accuracy based on a smooth 30 cm threshold.

C.2. In-depth Error Analysis Details

The analysis examines perspective ambiguity in object relations and evaluates whether fine-tuning improves spatially aware sports commentary. It also uses pickleball as an unseen-sport setting and assesses commentary quality through volunteer comparisons.

  • Perspective Ambiguity: Object-object and object-line subtasks are selected because perspective ambiguity particularly affects their target entities.The analysis focuses on cases where projected proximity may obscure true spatial separation.
  • Perspective Ambiguity: A higher 3D-to-2D distance ratio indicates stronger perspective distortion, because subjects far apart in 3D appear close in the image plane.The ratio is used as a proxy for perspective-induced ambiguity.
  • Unseen-Sport Evaluation: Pickleball videos include balanced men’s and women’s matches and singles and doubles matches at 1080p and 25 fps.The videos support evaluation on a similar but unseen sport.
  • Spatial-aware Commentary: 100 distance-measurement commentaries are compared by three volunteers for linguistic quality and spatial awareness.The comparison evaluates fluency, expressiveness, natural spatial integration, correctness, and numerical-value use.
Loading 2603.09896v1…