PaperlayerView exact revision on arXiv

Source-linked AI summary

TT4D: A Pipeline and Dataset for Table Tennis 4D Reconstruction From Monocular Videos

Nima Rahmanian, Daniel Kienzle, Thomas Gossard, Dvij Kalaria, Rainer Lienhart, Shankar Sastry

arXiv:2605.01234v1cs.CV

TL;DR

Existing table-tennis reconstruction pipelines segment 2D tracks before reconstruction, but occlusions and missing detections make this fragile. TT4D instead lifts entire unsegmented rallies to 3D with a Full-Sequence Lifting Network, producing a 140+ hour multimodal dataset validated through downstream applications.

  • Problem

    2D-first time segmentation is highly sensitive to occlusions and missing detections, limiting scalable table-tennis reconstruction from broadcast monocular videos.

  • Method

    TT4D uses a Lift-First Pipeline that lifts entire unsegmented 2D ball tracks to 3D before time segmentation, using a Full-Sequence Lifting Network trained on 3M synthetic rallies.

  • Results

    TT4D provides a 140+ hour high-fidelity multimodal dataset with 3D ball trajectories, human meshes, spin vectors, and precise 3D-derived time segmentations, demonstrated through downstream applications.

  • Takeaways & Limitations

    The dataset supports virtual replay, player analysis, robot learning, racket-contact estimation, and generative modeling of competitive rallies.

Abstract

from arXiv · show

We present TT4D, a large-scale, high-fidelity table tennis dataset. It provides $140+$ hours of reconstructed singles and doubles gameplay from monocular broadcast videos, featuring multimodal annotations like high-quality camera calibrations, precise 3D ball positions, ball spin, time segmentation, and 3D human meshes over time. This rich data provides a new foundation for virtual replay, in-depth player analysis, and robot learning. The dataset's combination of scale and precision is achieved through a novel reconstruction pipeline. Prior methods first partition a game sequence into individual shot segments based on the 2D ball track, and only then attempt reconstruction. However, 2D-based time segmentation collapses under occlusion and varied camera viewpoints, preventing reliable reconstruction. We invert this paradigm by first lifting the entire unsegmented 2D ball track to 3D through a learned lifting network. This 3D trajectory then allows us to reliably perform time segmentation. The learned lifting network also infers the ball's spin, handles unreliable ball detections, and successfully reconstructs the ball trajectory in cases of high occlusion. This lift-first design is necessary, as our pipeline is the only method capable of reconstructing table tennis gameplay from general-view broadcast monocular videos. We demonstrate the dataset's fidelity through two downstream tasks: estimating the racket's pose \& velocity at impact, and training a generative model of competitive rallies.

1 Introduction

TT4D addresses the difficulty of reconstructing complete table-tennis gameplay from occluded monocular broadcast video by lifting unsegmented sequences to 3D before temporal segmentation. The resulting dataset combines large scale with multimodal, high-fidelity annotations for downstream sports and robotics applications.

  • Motivation: Broadcast footage enables scalable 4D monocular reconstruction, but table tennis requires recovering ball spin alongside positions and human meshes.The ball is small, fast, and frequently occluded; spin affects flight through the Magnus effect and bounce behavior.
  • Approach: The Lift-First Pipeline first lifts an entire unsegmented sequence to 3D, making subsequent temporal segmentation and refinement possible in the 3D domain.This reverses the conventional order of segmenting video before reconstructing each shot.
  • Technical contribution: The Full-Sequence Lifting Network processes long, complex unsegmented sequences and is trained on a synthetic dataset of 3 million rallies.It enables the 3D-first approach by supporting continuous gameplay reconstruction.
  • Dataset and applications: TT4D provides 140+ hours of reconstructed gameplay with precise 3D trajectories, human meshes, dense spin, and 3D-derived time segmentations.The dataset supports racket pose and velocity estimation at impact and generative modeling of competitive gameplay.

2 Related Work

Prior table-tennis reconstruction methods rely on fragile 2D segmentation or limited-scale datasets, while TT4D targets full-rally 3D lifting and richer annotations. The related work spans 2D tracking, trajectory lifting, spin estimation, and specialized datasets.

  • 2D ball tracking: 2D ball tracking remains difficult because the ball is small, fast, frequently occluded, and blurred by motion.Deep detector-based methods, including MIMO formulations and attention-based variants, address this tracking problem.
  • 3D trajectory lifting: Physics-based trajectory lifting methods minimize reprojection error but do not scale to jointly reconstructing full rallies with unknown hits and bounces.Learned methods improve scalability, but prior lifting networks still operate on isolated, pre-segmented shots.
  • Spin estimation: Recent spin-estimation methods infer spin from trajectories, visual markers, event cameras, or player motion, while learned lifting transformers regress spin from ball tracks.TT4D adapts this direction to dense, per-frame spin prediction.
  • Datasets: TT4D surpasses LATTE-MV’s 26 hours by an order of magnitude and adds dense 3D spin and robust 3D-derived time segmentation at scale.It also uses improved camera calibration and realistic 3D trajectories rather than simplified parabolic fits.

3 Methodology: The Lift-First Pipeline

The Lift-First Pipeline reconstructs complete table-tennis points by lifting unsegmented 2D tracks into 3D before annotation and filtering. Its stages combine preprocessing, full-sequence learned lifting, 3D-domain event reconstruction, and visual-physical quality control.

  • 3.1 Terminology: 4D Reconstruction recovers camera parameters, 3D ball positions, ball spin, and 3D human meshes over time.A segment runs from one player’s hit to another player’s hit, while a point runs from serve until the ball leaves play.
  • 3.2 Pipeline Overview: The pipeline lifts an entire unsegmented point to 3D before performing time segmentation and annotation directly from the 3D trajectory.This replaces fragile 2D image-level segmentation with a 3D-domain signal that is described as unambiguous.
  • 3.4 Stage 2: A transformer-based Full-Sequence Lifting Network predicts dense 3D ball trajectories and per-frame spin from calibrated 2D detections.Rotary positional embeddings based on timestamps handle varying frame rates and missing detections.
  • 3.4 Stage 2: The network extends prior lifting by supporting arbitrary-length unsegmented points, dense spin, and reconstruction of occluded detections through an interpolation-token design.Its training uses 3 million simulated full points, generated with MuJoCo to model continuous play and pre-serve tosses.
  • 3.5 Annotation: 3D-domain annotation identifies hits from ball-trajectory extrema and estimates racket-contact parameters using an OCP that fits Magnus-effect ball flight to observed bounce locations.The procedure augments the dataset with physically consistent racket-stroke parameters.

4 The TT4D Dataset

TT4D scales monocular table-tennis reconstruction to 146 hours from 45,946 broadcast games, with multimodal 3D data and analyses of ball positions and spin.

  • Dataset scope: 146 hours and 211,534 reconstructed points are sourced from 45,946 broadcast games spanning 2021–2024.The dataset handles stationary general-view cameras, singles and doubles gameplay, and video speeds of at least 25 FPS.
  • Pipeline filtering: TT4D processing clips broadcasts into points, trims non-gameplay footage, calibrates cameras, and filters reconstructed trajectories for consistency.The filtering stages include scoreboard changes, gameplay trimming, camera calibration, reprojection consistency, physical plausibility, visibility, and human-mesh checks.
  • Pipeline filtering: The conservative clipping heuristic succeeds on 56.8% of available points, while approximately 5% are lost to camera-pose changes or calibration failure.Additional filtering rejects points for reprojection discrepancies, physically implausible trajectories, human-mesh issues, insufficient segments, or low visibility.
  • Gameplay statistics: Ball-position densities show that the ball typically crosses 5–15 cm above the net and that cross-court shots are common.Right-to-left cross-court bounces are less concentrated than left-to-right cross-court bounces.
  • Gameplay statistics: Spin analysis assigns segments to topspin, backspin, sidespin-left, sidespin-right, or no-spin categories and finds unimodal distributions across all types.Topspin and backspin have slightly heavier tails, indicating broader variability and more extreme spin magnitudes.

5 Evaluation & Applications

The evaluation tests the lifting network’s metrics, robustness, full-point context, physical consistency, and downstream use for generative gameplay and racket reconstruction.

  • Evaluation setup: The evaluation reports 3D trajectory error for datasets with 3D ground truth, 3D spin error on synthetic data, and 2D reprojection error plus Macro F1 on 2D benchmarks.The metrics quantify positional, spin, reprojection, and topspin/backspin-classification performance.
  • Lifting network evaluation: The lifting network remains strong under half-frame-rate input and randomly missing 10% of detections, verifying robustness to in-the-wild conditions.Interpolated predictions cause only minor performance degradations under missing detections.
  • Lifting network evaluation: Processing the full point reduces mean Δr_3D from 21.71 cm to 18.95 cm versus individual segments on TT4DBench.The full-point approach consistently outperforms the individual-segments baseline across camera views.
  • Lifting network evaluation: The network produces continuous 3D trajectories through missing detections and processes more than 500 points/s on a 10-year-old Titan X GPU.Its continuous predictions support physical-consistency checks and large-scale reconstruction without per-segment fitting during inference.
  • Generative gameplay: A conditional flow-matching model generates full 20-step future trajectories from 10 observations and evaluates 10,000 multi-segment rallies.Generated rallies are compared with real gameplay for physical plausibility and temporal realism.
  • Downstream applications: Racket stroke parameters are recovered by solving an inverse control problem using reconstructed ball states, whose motion is difficult to track directly from monocular video.The parameters are the racket’s velocity and orientation at impact.
  • Downstream applications: TT4D also supports motion-tracking policies and behavior-aware robotic models that reason about other players’ strategic decisions.The paper demonstrates a motion-tracking policy trained on retargeted motion from the dataset.

6 Conclusion

TT4D reconstructs unsegmented table-tennis rallies in 3D before time segmentation, avoiding fragility from occlusions and missing detections. The pipeline produces a 140+ hour multimodal dataset and supports downstream racket-contact estimation and rally generation.

  • The Lift-First Pipeline lifts entire unsegmented rallies to 3D before performing time segmentation.
  • 3M synthetic full points train the Full-Sequence Lifting Network to process unsegmented rallies and model pre-serve tosses.
  • The resulting TT4D dataset contains 140+ hours of 3D ball trajectories, human meshes, spin vectors, and 3D-derived time segmentations.
  • TT4D fidelity and utility are demonstrated through physics-based validation, racket-contact estimation, and generative modeling of competitive rallies.

From Monocular Videos

The supplementary material provides additional dataset-generation, evaluation, experimental, qualitative, and visualization details.

  • The supplement adds dataset-generation and evaluation details, further experiments, qualitative analyses, and example rallies.

A Data Preprocessing Details

The preprocessing pipeline converts long broadcast videos into reliable reconstructed gameplay clips by clipping sequences, removing duplicated frames, calibrating cameras, and filtering human tracks.

  • Broadcast videos are clipped first at scoreboard advances and then into gameplay intervals using detected visual ball-motion patterns.
  • Duplicated frames can create false stationary-ball observations and apparent deceleration, degrading trajectory estimation.
  • Periodic duplication is estimated from non-trivial inter-duplicate spacings and the most frequent residue modulo the estimated period.
  • Clips with insufficient evidence for periodicity are treated as aperiodic, and detected duplicates are removed before frame-rate correction and re-encoding.
  • Camera calibration uses the table geometry and four known corners as 3D reference points, with table segmentation and Hough line intersections recovering camera parameters.
  • Human players are reconstructed with 4DHumans and SMPL, then transformed into the world frame using camera rotation and ground-plane positioning.
  • Average mesh locations are checked against the 3D table to filter referees, crowd members, and other non-player detections, especially in doubles gameplay.

B.1 Network Architecture

The lifting architecture embeds ball detections and table keypoints into location tokens, predicts 3D trajectories and dense spin, and uses physically simulated full points for training.

  • Network Architecture: The network consumes sequences of 2D ball detections and 13 table keypoints, which provide camera-calibration context for trajectory lifting.
  • Network Architecture: Disentangled Context Embedding separately projects ball positions and table keypoints, preserving ball information apart from calibration information.
  • Network Architecture: Missing ball detections are replaced with learnable interpolation tokens, while Deferred Upsampling Token Attention limits early attention to these uncertain tokens.
  • Network Architecture: Removing the baseline’s single learnable spin token enables dense spin prediction across full points rather than one initial spin estimate.
  • Network Architecture: The Trajectory Head predicts per-frame 3D positions and the Spin Head predicts per-frame 3D spin vectors.
  • Synthetic Dataset Generation: A synthetic dataset of approximately 3 million physically correct points is generated in MuJoCo with per-frame position and spin ground truth.
  • Synthetic Dataset Generation: The stitching pipeline recursively joins validated toss, serve, and return segments by matching segment initial states and checking shot validity.
  • Synthetic Dataset Generation: Synthetic throw segments address pre-serve tosses, a major real-world generalization failure point for clipping before lifting.

E ODE Fit

The pipeline fits a physically consistent ball trajectory to lifting-network predictions, explicitly modeling aerodynamic flight, table bounces, and racket–ball impacts. It then formulates racket reconstruction as a nonlinear optimal-control problem constrained by landing, net-clearance, and orientation requirements.

  • ODE-constrained ball fitting: The method estimates the ball’s initial position, velocity, and spin, then simulates its trajectory with aerodynamic and table-bounce physics.The simulation uses fixed-step RK4 integration and explicitly detects table-plane crossings.
  • ODE-constrained ball fitting: A robust nonlinear least-squares objective aligns the simulated trajectory with valid lifting-network observations while downweighting outliers.Huber loss handles unreliable observations, and bounds keep the recovered initial state physically plausible.
  • ODE-constrained ball fitting: Solving the fit yields an optimal initial state and a fully simulated, physically consistent trajectory for downstream analysis.The method reports position RMSE and predicted bounce counts for each segment.
  • Racket strike reconstruction: Unlike a simplified flight model, the reconstruction uses full aerodynamic dynamics, including the Magnus effect and vertical drag, with RK4 integration.The outgoing trajectory combines the racket–ball interaction model with the aerodynamic ODEs.
  • Racket strike reconstruction: Racket reconstruction uses a nonlinear optimal-control problem with single shooting and a linear racket–ball impact model in the racket reference frame.The model represents racket orientation with a unit quaternion and includes incoming and outgoing ball velocities and spins.
  • Racket strike reconstruction: The optimization balances target spin and landing precision while enforcing net clearance, table-facing racket orientation, and physically constrained racket states.CasADi and IPOPT solve the fixed-node single-shooting formulation.

H Additional Dataset Statistics

Additional analyses characterize TT4D’s camera coverage and filtering criteria. The dataset contains diverse focal lengths and camera poses, with some pose parameters showing a bimodal distribution and two especially frequent views.

  • Filtering and camera statistics: Camera extrinsic parameters show wide pose variation, while some parameters exhibit a bimodal distribution with especially frequent views.Figure SM5 visualizes translation and Euler-angle rotation distributions, and Figure SM6 shows two dominant camera poses.
  • Filtering and camera statistics: The supplementary quality criteria table documents the reconstruction and filtering stages used to curate the dataset.Table SM1 lists the individual thresholds for each filtering step.
  • Filtering and camera statistics: TT4D contains a diverse distribution of camera focal lengths across the dataset.The focal-length distribution is shown in Figure SM4.

I.1 Data and Representation

The generative-model data representation treats each 30 Hz trajectory as synchronized ball and two-player skeleton observations. Evaluation covers physical plausibility, gameplay realism and diversity, and lifting accuracy for position, spin, and reprojection.

  • Data and representation: Each trajectory contains ball observations in R3 and two player skeletons with 21 three-dimensional joints per frame.All trajectories are resampled to 30 Hz; the dataset contains 237,054 reconstructed points, approximately 151 hours.
  • Generative trajectory modeling: The model predicts 20 future observations conditioned on a 10-observation history using a time-dependent vector field.The implementation uses a DiT-style architecture and numerically integrates the learned ODE from Gaussian noise during inference.
  • Generative trajectory evaluation: Generated rallies are evaluated for physical plausibility, gameplay realism, and diversity.Physical checks include trajectory-fit error, ball-acceleration smoothness, joint-velocity continuity, and kinematic-limit violations.
  • Lifting evaluation: The primary 3D position metric measures mean Euclidean prediction error in centimeters, while 3D spin error measures mean Euclidean spin-vector error in Hz.Spin evaluation is restricted to synthetic data because real-world datasets lack ground-truth 3D spin vectors.
  • Lifting evaluation: For real-world data without 3D ground truth, evaluation uses 2D reprojection error in pixels after projecting predicted trajectories through the camera matrix.Datasets with binary topspin/backspin labels additionally use Macro F1 after transforming predicted spin into a ball coordinate frame.

K.1 Lifting Network Training

The lifting network is trained on synthetic rallies and is designed for efficient full-sequence processing. Supplementary evaluations report high inference throughput, qualitative racket and sequence reconstructions, and robot-motion deployment procedures.

  • Lifting network training: The 1.6-million-parameter lifting network is trained on 2.6 million synthetic rallies for 17 epochs using Adam with a 10^-4 learning rate.Training uses a single NVIDIA H100 GPU, exponential moving averages, and TTST validation performance for model selection.
  • Efficiency: Learning-based lifting replaces per-segment optimization with a single efficient forward pass, though training costs approximately two days on one NVIDIA H100 GPU.The network processes entire points rather than isolated segments.
  • Efficiency: 25 points per second on a Titan X enables real-time inference and allowed hundreds of hours of gameplay to be lifted in minutes.This throughput shifts the computational bottleneck to preliminary processing steps.
  • Downstream reconstruction: Figure SM7 qualitatively visualizes estimated racket orientation and velocity for a specified ball flight, incoming velocity, spin, and bounce target.The reconstruction is reported as visually realistic.
  • Robot deployment: Player-reconstruction motion is retargeted and smoothed before training a motion-tracking policy for Unitree G1 deployment.The procedure adds half-second transitions into and out of the motion before policy training.
  • Generative evaluation: Figure SM10 provides a qualitative generated sequence, while Figure SM8 shows the spatial distribution of generated ball locations.The generated sequence is displayed at 30 FPS with every 16th frame shown.
Loading 2605.01234v1…