PaperlayerView exact revision on arXiv

Source-linked AI summary

Perceiving and Predicting Dynamic Sports: Spatiotemporal Intelligence in Vision Models for Egocentric Tennis Video

Tudor Braicu, Joseph DelPreto, Jared Boyer, Mónika Farsang, Maximilian Stölzle, Tim Niklas Seyde, Daniela Rus

arXiv:2610.37619v1cs.CVcs.RO

TL;DR

Agile robots need timely visual state estimates and anticipation, but foundation-model performance from moving egocentric views remains under study. The paper builds an annotated tennis dataset and evaluates vision models across seven tasks, finding broadly competitive image encoders and VLMs and improved ball localization with supplementary depth.

  • Problem

    The paper examines whether foundation vision models can provide sufficiently accurate, timely state estimates and anticipation for dynamic robotic tasks from moving egocentric views.

    Source: §I · pp. 1–2

  • Method

    The authors annotate 85 egocentric tennis rallies with five synchronized allocentric views and evaluate foundation vision models on seven egocentric spatiotemporal tasks.

    Source: §I · p. 2

  • Results

    Across seven tasks, image encoders and VLMs are broadly competitive, unlike the video encoder; supplementary depth reduces ball localization error by 50–60%, while longer histories yield little benefit.

    Source: §I · p. 2, §V · p. 7

  • Takeaways & Limitations

    The estimates could support future real-time robot-control pipelines, with generated motion adapting as visual contact estimates are refined.

    Source: §I · p. 2, §V · p. 7

Abstract

from arXiv · show

Agile robots require spatiotemporal intelligence to interpret scenes, reason about interactions, and anticipate future events. Yet it remains unclear how well foundation vision models support these capabilities, particularly from a moving egocentric camera, which is a natural view for robots. To investigate this, we collect an annotated tennis dataset and train a suite of vision models on spatiotemporal tasks. Our dataset contains egocentric video synchronized with five allocentric videos, estimated global positions, labeled contact times, and annotated stroke types. We then evaluate seven prediction tasks from egocentric images: 2D ego player and 3D ball localization, future player and ball trajectories, forehand or backhand stroke anticipation, time to the next ego player ball contact, and 3D ball position at that contact. We compare image encoders, video encoders, and vision-language models with jointly trained prediction heads. The best performing models achieve mean current-frame localization errors of 0.14 m for the ego player and 0.54 m for the ball, future forecasting errors of 0.34 m for the ego player and 0.91 m for the ball over 1.1 s, and a mean contact-time prediction error of 0.14 s. Auxiliary depth information substantially improves 3D ball localization, suggesting that inferring metric depth remains a challenge. Longer image histories do not improve ball forecasting, indicating that representations exploit scene context rather than rely solely on inter-frame motion. Finally, we propose a conditional diffusion model to generate whole-body kinematic tennis motion from vision encoder predictions on egocentric video at real-time rates, adapting movements as predictions are refined. These results move towards a deeper understanding of how foundation vision models can support real-time dynamic robots using egocentric video.

I. INTRODUCTION

The paper asks whether foundation vision models can estimate current states and forecast future events from demanding egocentric video, where camera motion and occlusion complicate perception. It builds an annotated tennis dataset, evaluates seven tasks, and connects predictions to whole-body motion generation.

I. INTRODUCTION

  • Tennis makes spatial accuracy and timing consequential because players and the ball continue moving during perception, planning, and execution.The paper uses tennis to study localization, trajectory anticipation, and body-motion coordination for imminent contacts.Source: §I · p. 2
  • The evaluation addresses a gap in understanding whether foundation visual representations support metric state estimation and forecasting from a moving camera with small, blurred, or occluded objects.Egocentric video relies on a body-mounted camera rather than external sensors, but body and head motion mix camera and object motion.Source: §I · p. 2
  • The seven tasks cover current player and ball localization, future player and ball trajectories, stroke anticipation, time to contact, and ball position at contact.Together these tasks probe spatial and temporal understanding of pretrained visual representations.Source: §I · p. 2
  • A conditional motion diffusion model generates whole-body kinematic tennis motion from vision-based contact position, time-to-contact, and ball approach-or-departure predictions, replanning as estimates update.Its planning pipeline uses a finetuned vision-only LFM2.5-VL-450M variant capable of 100 Hz inference on modern desktop GPUs.Source: §I · p. 2
  • The dataset contains 85 rallies with 53,155 egocentric frames, five synchronized allocentric views, 3D ball and 2D court-frame player trajectories, stroke labels, and contact times.The dataset is to be released upon acceptance.Source: §I · p. 2, Figure 2 · p. 3
  • Across seven egocentric tennis tasks, image encoders and VLMs are broadly competitive, while performance varies by task; depth improves ball localization and longer histories do not necessarily help.The study also reports that supplementary depth reduces ball localization error by 50–60%.Source: §I · p. 2

A. Synchronized Egocentric and Allocentric Videos

The dataset synchronizes head-mounted egocentric tennis video with five static allocentric cameras and derives spatial labels from calibrated views. Rally timing and camera calibration support the position and contact annotations used in the study.

A. Synchronized Egocentric and Allocentric Videos

  • The recording uses a head-mounted egocentric camera and five static allocentric cameras, all recording at 29.97 frames per second; egocentric video is 3840 × 2160 pixels.Allocentric video resolution is at least 5120 × 2880 pixels, at 29.97 or 59.94 frames per second.Source: A. Synchronized Egocentric and… · p. 3
  • The dataset retains active rally footage spanning approximately 29.6 minutes, or 53,155 egocentric frames, across 85 rallies.The rallies end at the final racket-ball contact, second bounce, or net contact.Source: A. Synchronized Egocentric and… · p. 3

B. Camera Synchronization and Calibration

  • A three-clap signal visible in all videos supports manual post-processing alignment, while camera intrinsics and extrinsics are recovered using ChArUco calibration and court-corner correspondences.Calibration uses a pinhole camera model and Perspective-n-Point with OpenCV.Source: B. Camera Synchronization and… · p. 3

C. State Estimation and Annotation

  • The annotation targets include ball and ego-player positions and whether the ego player will next make a forehand or backhand stroke.The paper also marks ego-player ball-contact times and records opponent contact times.Source: C. State Estimation and Annotation · p. 3

1) Ball and Player Position Estimation:

The authors derive global ball and player position labels from reviewed detections in three calibrated allocentric views, alongside egocentric ball annotations and stroke/contact labels. The study evaluates image, video, and vision-language models on egocentric tennis prediction.

1) Ball and Player Position Estimation:

  • Reviewed ball and player detections from three allocentric cameras are triangulated into 3D ball and ego-player position labels, with the player constrained to the court plane.Labels are available for 99.9% of frames for the ball and 100% for the player; triangulation requires valid detections in at least two views.Source: 1) Ball and Player Position Estimation · p. 3

2) Stroke Type Annotations:

A. Vision Models

  • The evaluation spans image encoders, video encoders, and vision-language models to assess pretrained representations on egocentric tennis prediction.The paper frames this as an assessment of support for egocentric tennis prediction.Source: A. Vision Models · p. 3

1) Image Encoders:

The study compares image, video, and vision-language encoders using jointly trained task-specific prediction heads. It fine-tunes visual backbones and uses parameter-efficient language-model adaptation for VLMs.

1) Image Encoders:

  • Four ViT-L-class image encoders span self-supervised, language-supervised, dense-depth, and dense-segmentation training paradigms, with 300–450M parameters.The models are DINOv3, SigLIP 2, Depth Anything 3, and SAM 3.Source: 1) Image Encoders · p. 4

2) Video Encoders:

  • V-JEPA 2 is a 300M-parameter video encoder self-supervised to predict masked latent representations, using 2×16×16 tubelet patches for multi-frame processing.Source: 2) Video Encoders · p. 4

3) Vision-Language Models:

  • Three 3–4B VLMs are tested with their vision towers alone and with full language stacks to isolate language-backbone and multimodal-training contributions.A smaller 450M LFM2.5-VL variant supports compute-constrained real-time planning experiments.Source: 3) Vision-Language Models · p. 4

1) Prediction Head:

  • Task-specific heads use learnable cross-attention queries over encoder tokens followed by a two-layer MLP, with all heads jointly trained on a shared encoder.Continuous targets use L2 loss and stroke type uses cross-entropy loss.Source: 1) Prediction Head · p. 4

2) Fine-tuning and Training Details:

  • Training uses 30 epochs, learning rates of 10^-4 for new heads and 10^-5 for pretrained vision backbones, and LoRA rather than full-parameter language-model fine-tuning for VLMs.Source: 2) Fine-tuning and Training Details · p. 4

3) Vision Tower and Vision-Language Settings:

The section defines how vision-language models are probed and evaluated, then reports performance across seven egocentric tennis prediction tasks. Results vary by task: PLM 3B leads future ball forecasting, while V-JEPA 2 leads future player forecasting.

3) Vision Tower and Vision-Language Settings:

  • VLMs are evaluated using either vision-tower tokens alone or vision and language components together, with a standardized task prompt in the latter setting.For the vision-language setting, representations are tapped halfway through the language tower and its second half is discarded.Source: 3) Vision Tower and Vision-Language… · p. 4

A. Experimental Setup

  • Rallies are split wholly across training, validation, and test sets, with stratification by drill subtype and free play.The realized split is 63.1% train, 17.7% validation, and 19.2% test, corresponding to 53, 16, and 16 rallies.Source: A. Experimental Setup · p. 4

B. Multi-Task Prediction

  • Each model receives the anchor frame and a frame 0.1 s earlier, and jointly predicts seven outputs covering five targets.Targets include current and future player and ball positions, upcoming stroke type, time to contact, and ball position at contact.Source: B. Multi-Task Prediction · p. 4
  • PLM 3B has the lowest future ball error (0.91 m), while V-JEPA 2 has the lowest future player error (0.34 m) and the highest future ball error (1.27 m).Image encoders and VLMs perform comparably overall; shot-type accuracy is 98.2–99.9%, and time-to-contact errors are 141–149 ms.Source: B. Multi-Task Prediction · p. 4
  • Current 2D player errors range from 0.14 to 0.18 m, while current 3D ball errors range from 0.54 m for PLM 3B to 0.90 m for V-JEPA 2.DA3 has the lowest current player error; V-JEPA 2 remains strongest on player forecasting but weakest on ball localization.Source: B. Multi-Task Prediction · p. 4

C. Injecting Information to Probe Spatial Representations

The ablation tests whether auxiliary annotations improve current 3D ball localization. Depth substantially reduces SigLIP 2 error, while ball and player inputs alone do not help.

C. Injecting Information to Probe Spatial Representations

  • The ablation tests ball image position, ego-player court position, and planar ego-ball distance, individually and in combinations, as auxiliary inputs to a SigLIP 2 prediction head.The inputs are embedded and injected through cross-attention using adaptive layer normalization; an image-free MLP serves as a control.Source: C. Injecting Information to Probe… · pp. 4–5
  • Depth reduces SigLIP 2 error from 0.51 m with images alone to 0.21 m, and depth plus player position yields the lowest error, 0.19 m.Ball and player inputs alone provide no benefit, and adding ball position slightly degrades combinations.Source: Table II · p. 5, C. Injecting Information to Probe… · p. 5
  • The image-free MLP benefits most from depth but performs best with all three auxiliary inputs at 1.07 m, and its errors remain substantially higher than image-based variants.This indicates that the auxiliary inputs alone do not replace visual data.Source: C. Injecting Information to Probe… · p. 5

D. Varying History to Probe Temporal Representations

The history ablation finds that longer image histories do not improve ball forecasting for SigLIP 2, unlike a ground-truth-position baseline that benefits from a second observation. V-JEPA 2 improves slightly with more frames but remains worse than single-frame SigLIP 2.

D. Varying History to Probe Temporal Representations

  • The ablation varies the number and spacing of input frames for future 3D ball prediction, using four SigLIP 2 settings and three V-JEPA 2 settings.Single-frame history is omitted for V-JEPA 2 because its tubelets must be at least two frames long.Source: D. Varying History to Probe Temporal… · p. 5, Table III · p. 5
  • SigLIP 2’s future ball error is 1.02 m with one frame and 1.02–1.08 m with additional frames; V-JEPA 2 improves from 1.22 m to 1.18 m with more frames.V-JEPA 2 remains worse than single-frame SigLIP 2; the reported pattern is consistent with image models relying on single-frame spatial cues rather than inter-frame ball displacement.Source: Table III · p. 5, D. Varying History to Probe Temporal… · p. 5
  • A ground-truth-position MLP cannot forecast from one ball position (2.92 m error), but a second position 100 ms earlier reduces error to 0.30 m.Longer or denser histories provide only modest further gains.Source: D. Varying History to Probe Temporal… · p. 5, Table III · p. 5

E. Effect of Backbone Adaptation

Freezing both towers degrades every evaluated task for LFM2.5-VL 3B, with larger increases in ball-position errors than in player-position errors.

E. Effect of Backbone Adaptation

  • Freezing both towers raises current ball error from 0.57 to 0.96 m and future ball error from 0.98 to 1.47 m, while current and future player errors rise from 0.16 to 0.18 m and 0.36 to 0.45 m.Shot-type accuracy falls from 99.4% to 98.3%, time-to-contact error rises from 146.7 to 162.0 ms, and ball-at-contact error rises from 1.003 to 1.225 m.Source: E. Effect of Backbone Adaptation · p. 6

F. Conditional Motion Diffusion for Kinematic Tennis Play

The section uses vision-based tennis predictions to condition a motion diffusion model that generates and adaptively replans whole-body kinematic motion. The demonstration shows well-timed movement toward predicted contact, while recording mismatch limits precise racket–ball alignment and physical-execution claims.

F. Conditional Motion Diffusion for Kinematic Tennis Play

  • The conditioning combines predicted contact position and time with previous-contact timing, masks, and global pelvis position and heading.Motion is trained to reconstruct recorded motion from noisy samples conditioned on annotated tennis states.Source: F. Conditional Motion Diffusion for… · p. 6
  • LFM2.5-VL-450M supplies real-time visual conditioning using two recorded images approximately 100 ms apart.Its 86M-parameter encoder runs at 100 Hz on an RTX 5090.Source: F. Conditional Motion Diffusion for… · p. 6
  • The motion diffusion model replans as visual estimates refine the predicted contact position and time remaining.Each rollout predicts 24 frames at 60 Hz; six frames are executed before replanning.Source: F. Conditional Motion Diffusion for… · pp. 6–7
  • The generated player approaches the predicted contact point with well-timed motion, while contact estimates become more accurate as the ball nears.The demonstration uses a finetuned VLM vision encoder to provide real-time conditioning for whole-body ego-player motion generation.Source: F. Conditional Motion Diffusion for… · p. 7
  • Recorded motion has a mean ball–racket distance of 0.89 m at annotated contact, limiting motion learning and evaluation; the demonstration remains kinematic without a physical-execution policy.Precise racket–ball alignment remains unresolved and requires improvements in recording quality and vision prediction accuracy.Source: F. Conditional Motion Diffusion for… · p. 7

V. CONCLUSION

The paper studies spatiotemporal understanding and forecasting from dynamic egocentric video and finds competitive performance across seven tennis tasks. It also reports benefits from depth information, limits to longer visual histories, and a conditional diffusion motion-generation demonstration.

V. CONCLUSION

  • Across seven tasks, image encoders and VLMs provide broadly competitive performance, while additional depth improves ball localization and longer visual histories do not necessarily improve performance.The paper also presents whole-body motion generation via conditional motion diffusion.Source: §V · p. 7
  • The study uses tennis to examine spatiotemporal understanding and forecasting from dynamic egocentric video.Future work will test whether identified representations and adaptation strategies transfer to other highly dynamic settings.Source: §V · p. 7

Related papers

Loading 2610.37619v1…