Source-linked AI summary
Uplifting Table Tennis: A Robust, Real-World Application for 3D Trajectory and Spin Estimation
Daniel Kienzle, Katja Ludwig, Julian Lorenz, Shin'ichi Satoh, Rainer Lienhart
TL;DR
Monocular video makes 3D table-tennis trajectory and spin estimation difficult, and real broadcast footage lacks 3D ground truth. The paper combines 2D detectors trained with TTHQ annotations and a synthetic-trained uplifting model adapted to real-world detection artifacts, reporting zero-shot generalization to real-world scenarios.
Problem
Real monocular footage lacks 3D trajectory and spin ground truth, while synthetic-trained methods must handle noisy and incomplete real-world detections.
Source: §1 · p. 1, §3.4 · p. 4Method
A two-stage pipeline trains ball and table-keypoint detectors with TTHQ annotations and trains an artifact-robust 2D-to-3D uplifting network on synthetic data.
Source: §1 · p. 2, §3.4 · p. 4, §4 · p. 5Results
The synthetic-trained uplifting model achieves zero-shot generalization to real-world scenarios, and Segformer++ B2 is selected for the final pipeline due to superior overall performance.
Source: §1 · p. 2, §5.1 · p. 7Takeaways & Limitations
Separating perception and uplifting allows the system to use available 2D annotations and synthetic 3D supervision without requiring a unified real-world 3D ground-truth dataset.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Obtaining the precise 3D motion of a table tennis ball from standard monocular videos is a challenging problem, as existing methods trained on synthetic data struggle to generalize to the noisy, imperfect ball and table detections of the real world. This is primarily due to the inherent lack of 3D ground truth trajectories and spin annotations for real-world video. To overcome this, we propose a novel two-stage pipeline that divides the problem into a front-end perception task and a back-end 2D-to-3D uplifting task. This separation allows us to train the front-end components with abundant 2D supervision from our newly created TTHQ dataset, while the back-end uplifting network is trained exclusively on physically-correct synthetic data. We specifically re-engineer the uplifting model to be robust to common real-world artifacts, such as missing detections and varying frame rates. By integrating a ball detector and a table keypoint detector, our approach transforms a proof-of-concept uplifting method into a practical, robust, and high-performing end-to-end application for 3D table tennis trajectory and spin analysis.
1. Introduction
Monocular footage makes precise 3D trajectory and spin analysis difficult, while synthetic-trained methods face a gap between clean inputs and noisy real detections. The paper proposes a two-stage pipeline and introduces TTHQ to support its perception components and integrated evaluation.
1. Introduction
- Noisy, sparse, and imperfect real-world detections remain a practical deployment gap for methods trained on synthetic data.The paper targets 3D trajectory and initial-spin inference directly from monocular footage.Source: §1 · p. 1
- A two-stage framework separates front-end perception from back-end uplifting, allowing each stage to use different supervision despite missing real-world 3D ground truth.The front end uses 2D annotations, while the back end is trained on physically correct synthetic data.Source: §1 · pp. 1–2
- TTHQ provides annotated 2D ball trajectories, table keypoints, and spin information from publicly available YouTube videos.The dataset supports training and evaluating the detectors and benchmarking the integrated pipeline.Source: §1 · p. 2
2. Related Work
Prior work covers synthetic-data reconstruction, monocular estimation, and sports-specific detection, but existing methods and datasets leave gaps for a robust integrated pipeline. The paper builds on synthetic uplifting and introduces TTHQ to address those gaps.
Ball and Table Keypoint Detection
- Existing table-tennis methods often lack the precision and comprehensive keypoint set needed for direct pipeline integration.The paper uses Segformer++ [20] for high-resolution ball and table-keypoint detection.Source: §2 · p. 2
3D Trajectory and Spin Estimation
- Monocular reconstruction has used physics fitting [10] [15] [24] and single-frame cues [2] [21], while multi-camera triangulation [27] [43] is impractical for broadcast footage.These approaches can be susceptible to event-identification, calibration, or video-quality issues.Source: §2 · p. 2
- Kienzle et al. [22] showed zero-shot generalization from synthetic training, but did not address imperfect detections, missing observations, or varying frame rates.This paper adapts that uplifting model and expands synthetic training to include more match situations.Source: §2 · p. 2
3. Method
The method separates 2D perception from synthetic-data-trained 3D uplifting. It combines filtered ball detections and stable table keypoints with an uplifting network adapted to missing observations and variable frame rates.
3.1. General Overview
- The front end detects ball positions and 13 table keypoints, filters detections, and aggregates table points into a trajectory-level set; the back end predicts metric 3D trajectory and initial spin.Front-end components use TTHQ 2D annotations, while the uplifting network uses synthetic data to avoid requiring real-video 3D ground truth.Source: §3.1 · p. 3
3.2. Ball Detection
- Segformer++ processes three consecutive frames as a 9-channel input and predicts a heatmap for the central-frame ball position, refined by fitting a 2D Gaussian.The model targets tiny, fast-moving balls and uses temporal context for motion blur and occlusions.Source: §3.2 · p. 3
- Ball detections are retained only when Segformer++ and an auxiliary HRNet model agree, removing false positives at the cost of potentially incomplete trajectories.The paper notes that false positives can persist across frames, making common temporal filters insufficient.Source: §3.2 · pp. 3–4
3.3. Table Keypoint Detection
- The table detector identifies 13 characteristic points, then combines cross-model agreement with DBSCAN clustering over time to produce a stable keypoint set.The approach assumes a static camera; clustering helps when points are occluded for more than half the trajectory.Source: §3.3 · p. 4
3.4. 2D-to-3D Uplifting
- Custom timestamp-based RoPE encodes elapsed time for missing detections and variable frame rates, with random detection and frame-rate dropping during training to improve robustness.The table-keypoint embedding also handles varying numbers of visible points, and keypoints are randomly dropped during training.Source: §3.4 · pp. 4–5
- The back end embeds ball positions and visible table points into location tokens, prepends a learnable spin token, and uses transformer stages and MLP heads to predict trajectory and initial spin.It builds on Kienzle et al. [22] but is designed for imperfect and incomplete front-end inputs.Source: §3.4 · pp. 4–5
4. Data and Metrics
The section describes the TTHQ real-video annotations and synthetic training data, then sets out metrics for evaluating detections, projected trajectories, and binary spin predictions.
TTHQ Dataset
- 14 match videos and 5 highlight videos comprise TTHQ, with 9,092 ball-annotated frames, 257 table-keypoint frames, and binary spin labels for 57 trajectories.The 1920 × 1080 broadcast videos span camera angles, lighting conditions, and player skill levels.Source: §4 · p. 5
Synthetic Data
- 140,000 synthetic trajectories provide over 4.7 million 2D ball positions, with varied trajectories, initial conditions, and camera parameters for uplifting-network training.The data are generated with the MuJoCo physics engine and include serves and fault types.Source: §4 · p. 6
Existing Datasets
- Blurball supplies 50,000 annotated frames for ball-detector pretraining, while 16 TTST trajectories validate the uplifting network and 34 test it in a controlled setting.TTST contains 50 trajectories in total.Source: §4 · p. 6
Metrics
- Because realistic broadcast videos lack 3D ground truth, trajectory evaluation uses mean 2D reprojection error (m2DRE), comparing projected predictions with ground-truth 2D positions.The score averages reprojection error over trajectories.Source: §4 · p. 6
- Front-end detections use ACC@Xpx, while real-world spin is classified as topspin or backspin by the predicted vector’s y-component sign and scored with ACC and macro F1.The available human spin annotations are binary rather than full 3D spin vectors or spin strength.Source: §4 · p. 6
5. Experiments
The experiments evaluate the front-end detectors, back-end robustness, and complete pipeline across TTHQ and TTST. They report that the full system maintains strong performance on both datasets.
5.1. Ball Detection Evaluation
- Segformer++ models outperform WASB and VitPose for ball detection, especially on ACC@2px; the authors select B2 for the final pipeline.B2 has superior overall performance, while B0 offers a balance of accuracy and speed.Source: §5.1 · pp. 6–7, Table 1 · p. 6
5.2. Table Keypoint Detection Evaluation
- Segformer++ models outperform the other architectures for table keypoint detection, with the largest performance gap appearing on this task.The authors select B2 for its performance on ACC@2px.Source: §5.2 · p. 7, Table 2 · p. 7
5.3. Back-End Evaluation
- Under half-frame-rate and missing-detection transformations, the full back-end architecture outperforms the alternatives on both spin and trajectory metrics.Time-proportional RoPE encodes exact frame timing for spin prediction, while the transformer embedding handles missing data; missing detections remove 10% of frames and give each table keypoint a 10% removal chance.Source: §5.3 · p. 7, Table 3 · p. 7
5.4. Full Pipeline: Combining Front- and Back-End
- The full pipeline achieves low table and ball reprojection errors and high spin-classification accuracy on TTHQ and TTST, with TTST spin performance similar to perfect-input results.The authors conclude that front-end detections do not significantly degrade back-end spin prediction.Source: Table 4 · p. 8, §5.4 · p. 8
6. Conclusion
The conclusion presents a modular pipeline that separates 2D perception from 3D uplifting, using different supervision for each stage without requiring a unified 3D ground-truth dataset. It identifies custom time-proportional positional encoding and robust embedding as essential for handling real-world artifacts.
6. Conclusion
- The two-stage framework trains perception with 2D human annotations and uplifting with synthetic data, avoiding the need for a unified 3D ground-truth dataset.The authors present this separation as a response to the lack of 3D ground truth data.Source: §6 · p. 8
- Custom time-proportional positional encoding and the robust embedding module are identified as essential for handling occlusions and varying frame rates.Source: §6 · p. 8
- The authors present the pipeline as a practical sports-analytics tool and its two-stage design as a template for other 3D reconstruction problems with limited ground truth.Source: §6 · p. 8
A. Training Details
The front end is trained on TTHQ with task-specific augmentations and pretrained backbones, while the back end is trained on an expanded synthetic dataset with missing detections and varying frame rates simulated. Model selection uses TTST validation, and evaluation uses a common image resolution.
A. Training Details
- Front-end training uses Adam at a 10^-3 learning rate, batch size 4, Gaussian heatmaps, and L2 loss, with augmentation choices tailored to each detection task.Table-keypoint training omits flipping because it changes keypoint semantics and adds random perspective transformations.Source: A. Training Details · p. 11
- Front-end backbones are initialized with ImageNet or MAE weights; ball detectors are also pretrained on Blurball before fine-tuning on TTHQ, unlike table-keypoint detectors.Blurball lacks sufficiently diverse table-keypoint annotations, so those models are fine-tuned only on TTHQ.Source: A. Training Details · p. 11
- The back end is trained on 140000 synthetic trajectories, with 5% of ball and table-keypoint detections randomly dropped and frame rates sampled from 20 FPS to 60 FPS.Source: A. Training Details · p. 11
- The back-end model is selected using TTST validation as a compromise between spin F1 and trajectory m2DRE, with tied maximum F1 epochs resolved by m2DRE.Source: A. Training Details · p. 11
- All evaluations use 1920 × 1080 resolution, and predicted global-coordinate spin is transformed to the local ball coordinate system for evaluation and spin-class extraction.Source: A. Training Details · p. 11
B. Architectures
The architectures are tuned for comparable inference speed and single-GPU training, while the back end uses a transformer with timestamp-based rotary positional encoding. This encoding incorporates detection times so attention can model temporal relationships despite irregular spacing.
B. Architectures
- Front-end models are tuned for similar inference speeds and to fit a single Nvidia V100 GPU with 32 GB of memory.Source: B. Architectures · pp. 11–12
- The back end uses 16 transformer layers with embedding size 128 and four attention heads, followed by three-layer Spin and Location heads.Source: B. Architectures · p. 12
- Timestamp-based RoPE replaces discrete token position with a timestamp discretized at 2 ms, allowing attention to learn temporal relationships despite varying detection spacing.The timestamp is incorporated into the rotation applied to feature pairs.Source: B. Architectures · p. 12
C. Ball & Table Keypoint Filtering
The filtering stage uses agreement between two different detector architectures to reject false positives, with separate procedures for ball detections and static table keypoints. The authors describe the ball procedure as robust to player movements that can fool conventional trajectory filters.
C. Ball & Table Keypoint Filtering
- Ball detections are retained only when Segformer++ (B2) and WASB predictions are within 20 px, using architectural differences to reject mismatched false positives.The authors report that this handles consecutive false detections of players’ feet or paddles better than Kalman filtering or polynomial fitting.Source: C. Ball & Table Keypoint Filtering · p. 12
- The filtering illustrations distinguish retained detections in green, removed false positives in crimson, and WASB predictions for filtered frames in orange.Source: Figure 8 · p. 12
- Table-keypoint detections must be within 10 px across Segformer++ (B2) and WASB, then DBSCAN selects the largest cluster’s center for each keypoint.The procedure uses the table keypoints’ static nature across a trajectory.Source: C. Ball & Table Keypoint Filtering · p. 12, Figure 8 · p. 12
D. Comparison of Datasets
The dataset comparison positions TTHQ as the strongest resource for training robust front-end models and evaluating the full pipeline, while TTST and Blurball serve narrower roles because of their limitations in scene diversity, annotation density, or resolution.
Dataset comparison
- Table 5 compares video resolution, ball-annotated frames, table-keypoint-annotated frames, and spin-labeled trajectories; an asterisk marks keypoints generated from homographies.The caption notes that homography-derived table annotations can be numerous despite extremely low frame variety when a dataset comes from few videos.Source: Table 5 · p. 13
TTHQ
- TTHQ combines high-resolution video, diverse scenes, and numerous ball, table-keypoint, and spin annotations, supporting robust front-end training, spin validation, and full-pipeline evaluation.Its annotations are not trajectory-dense like TTST's, but its variety makes it a benchmark for overall pipeline performance.Source: D. Comparison of Datasets · p. 13
TTST
- TTST offers high-resolution video and dense trajectory annotations for evaluating the back end independently, but its limited annotated frames and scene diversity make it unsuitable for robust front-end training.Its annotations also include balls, table keypoints, and spin.Source: D. Comparison of Datasets · p. 13
Blurball
- Blurball has the most ball annotations and homographies that can generate pseudo table keypoints, but low video resolution and limited video diversity restrict it to ball-detector pre-training.Its table-keypoint annotations lack diversity and are unsuitable for training a robust keypoint detector.Source: D. Comparison of Datasets · p. 13, Table 5 · p. 13