PaperlayerView exact revision on arXiv

Source-linked AI summary

TTNet: Real-time temporal and spatial video analysis of table tennis

Roman Voeikov, Nikolay Falaleev, Ruslan Baikulov

arXiv:2004.09927v1cs.CV

TL;DR

Real-time table-tennis analysis must handle fast, ambiguous ball motion while extracting multiple types of video information. TTNet jointly spots events, detects the ball, and predicts semantic masks, and the authors release the OpenTTGames dataset. The network processes 120-fps video with a single consumer-grade GPU, and the dataset supports development and assessment of table-tennis analysis systems.

  • Problem

    Real-time table-tennis analysis must recover ball position amid small, blurred lookalikes and fast motion while extracting multiple data types from high-frame-rate video.

    Source: §1 · p. 1, §5.1 · p. 3

  • Method

    TTNet jointly spots in-game events, detects the ball, and predicts semantic masks, alongside the release of a labeled multi-task table-tennis dataset.

    Source: §1 · p. 2, §8 · p. 8

  • Results

    TTNet processes 120-fps video with a single consumer-grade GPU while detecting the ball, spotting fast events, and predicting semantic masks.

    Source: §8 · p. 8

  • Takeaways & Limitations

    OpenTTGames provides labeled multi-task data for developing and assessing computer-vision systems for table-tennis analysis.

    Source: §8 · p. 8

    With nine frames spanning less than 0.1 s, TTNet targets fast rally events; serves are outside the paper’s scope.

    Source: §5.2 · p. 5

Abstract

from arXiv · show

We present a neural network TTNet aimed at real-time processing of high-resolution table tennis videos, providing both temporal (events spotting) and spatial (ball detection and semantic segmentation) data. This approach gives core information for reasoning score updates by an auto-referee system. We also publish a multi-task dataset OpenTTGames with videos of table tennis games in 120 fps labeled with events, semantic segmentation masks, and ball coordinates for evaluation of multi-task approaches, primarily oriented on spotting of quick events and small objects tracking. TTNet demonstrated 97.0% accuracy in game events spotting along with 2 pixels RMSE in ball detection with 97.5% accuracy on the test part of the presented dataset. The proposed network allows the processing of downscaled full HD videos with inference time below 6 ms per input tensor on a machine with a single consumer-grade GPU. Thus, we are contributing to the development of real-time multi-task deep learning applications and presenting approach, which is potentially capable of substituting manual data collection by sports scouts, providing support for referees' decision-making, and gathering extra information about the game process.

1. Introduction

Table tennis analysis must handle very fast, small-ball imagery while extracting several kinds of information quickly enough for real-time use. TTNet addresses these demands with a lightweight multi-task architecture and the OpenTTGames dataset.

1. Introduction

  • Computer vision tasks in sports analytics include tracking players and equipment, estimating poses, and detecting game-related actions.For many such tasks, real-time performance is important for online analytics.Source: §1 · p. 1
  • Balls can exceed 30 m/s and average about 15 pixels in full HD, while lookalike white objects and motion make detection difficult.High-frame-rate video is needed to track trajectories and spot bounces and net hits.Source: §1 · p. 1
  • Real-time analysis calls for low inference time on high-frame-rate video and parallel extraction of segmentation masks, ball coordinates, and game events.The introduction identifies shallow convolutional networks with relatively few operations as a response to the speed demand.Source: §1 · pp. 1–2
  • TTNet is a lightweight architecture for simultaneous task processing, with 5.9 GFLOPs and inference below 6 ms on one NVIDIA RTX 2080Ti GPU.Source: §1 · p. 2
  • The authors introduce OpenTTGames to address the lack of public multi-task datasets focused on spotting quick game events.They collected and labeled table-tennis videos to enable consistent assessment of computer-vision techniques for the sport.Source: §1 · p. 2

2. Related works

Earlier table-tennis systems focused mainly on ball detection, while methods from other sports and broader multi-task vision address different combinations of tasks. The paper presents TTNet against the gap in combining small-object tracking, semantic segmentation, and temporal event spotting in one architecture.

2. Related works

  • Earlier table-tennis approaches focused mainly on ball detection, using contour analysis or adaptive color thresholding and background subtraction.A later stereo-camera setup reconstructed the ball’s 3D trajectory accurately but required four interrelated high-speed cameras.Source: §2 · p. 2
  • Tennis patch classification was considered unsuitable for real-time use, while DeepBall ran at 190 fps on full-HD football images when the ball moved relatively slowly.The cited comparison notes table-tennis balls move much faster.Source: §2 · p. 2
  • Sports video can support analysis beyond ball detection, including player performance, auto-umpiring, and predictive models, motivating multi-task approaches.Source: §2 · p. 2
  • UberNet combined datasets to train across computer-vision tasks end-to-end, while Luvizon et al. combined pose estimation and visual features for action recognition [12].Source: §2 · p. 2
  • The authors identify the unaddressed combination as very-small-object tracking, multiclass semantic segmentation, and temporal event spotting in one CNN architecture.Source: §2 · p. 2

3. Table tennis analysis

Table-tennis analysis for score updates depends on combining event spotting with spatial information. The proposed architecture simultaneously handles event spotting, object detection, and semantic segmentation.

3. Table tennis analysis

  • Score updates during a rally rely on correctly treating bounces and net hits to determine the current game state.Source: §3 · p. 2
  • Ball coordinates and table position help determine the bounce side and filter false-positive bounces, while continuous table-position updates account for slight movement.Source: §3 · p. 2
  • Player, referee, and scoreboard information supports handling occlusion, detecting let-serve gestures, and retrieving the score through OCR; TTNet addresses the required data with three task types.Source: §3 · p. 2

4. OpenTTGames Dataset

OpenTTGames was introduced because multi-task datasets combining temporal and spatial table-tennis data were absent. It supplies videos and annotations for assessment of these analysis tasks.

4. OpenTTGames Dataset

  • OpenTTGames was introduced to cover table-tennis analysis aspects and support experiments in the paper, addressing the lack of datasets combining temporal and spatial data.Source: §4 · p. 2
  • The dataset includes full-HD table-tennis videos recorded at 120 fps with annotations for in-game events and/or ball coordinates and segmentation masks.Events include bounces, net hits, and empty event targets; ball coordinates and masks were labeled with deep-learning-aided annotation models.Source: §4 · pp. 2–3
  • OpenTTGames contains five videos from different matches for training and seven short videos from other matches for testing, recorded under similar conditions with slight camera-angle variation.The target construction yielded 38,752 training, 9,502 validation, and 7,328 testing samples.Source: §4 · p. 3

5. Proposed method

TTNet combines event spotting, ball localization, and semantic segmentation in a multi-task pipeline. Its method uses multiframe inputs, two-scale ball detection, and shared features, with task-specific targets and an adaptive multi-task loss.

5. Proposed method

  • The pipeline processes nine-frame stacks to predict events, semantic masks, and ball position, sharing backbone features between branches and refining ball localization on original-resolution crops.The global detector operates on downscaled frames; the local detector crops around its prediction, and their features also feed segmentation and event spotting.Source: §5 · p. 3

5.1. Ball detection

  • Ball localization uses consecutive frames and two stages: a downscaled global detector followed by a crop-based local detector, with x- and y-coordinate targets represented as Gaussian curves.The target distributions are fitted around the labeled ball center, with variances tied to the average ball radius at the input scale.Source: §5.1 · pp. 3–4

5.2. Event spotting

  • Event spotting uses concatenated global and local features to detect bounces and net hits, but its under-0.1-second input window limits the model to fast events during rallies.Smooth multilabel targets accommodate blurred or absent exact event frames; central-frame labeling uses past and future context and introduces a prediction delay under 0.05 seconds.Source: §5.2 · p. 5

5.3. Semantic segmentation

  • The segmentation branch predicts overlapping human, table, and scoreboard masks for the last frame using a shared encoder, an upsampling decoder, and skip connections.Its loss combines 1 minus the smoothed Sørensen-Dice coefficient with binary cross-entropy.Source: §5.3 · pp. 5–6

5.4. Multi-task loss function

  • Multi-task training adaptively weights the ball, event, and segmentation losses using learned homoscedastic uncertainty, with a regularizing term to eliminate trivial solutions.The approach addresses different learning paces and interdependent predictions.Source: §5.4 · p. 6

5.5. Data preparation

  • Training samples use manually labeled events at the middle frame of a nine-frame sequence, with ball positions and segmentation masks targeted at the last frame.Ball coordinates and masks were aided by task-specific annotation networks; the final ball auto-labeler achieved 96.3% presence detection with 2.5-pixel average error locally and 96.4% with 14.3-pixel average error globally.Source: §5.5 · pp. 6–7
  • The TTNet VGG-style encoder performs better in almost all metrics and has lower full-pipeline inference time than a ResNet-18 backbone.The comparison is reported in Table 6.Source: Table 6 · p. 7

6. Experiments

The experiments compare TTNet backbones, input sequence lengths, and loss-balancing strategies, alongside describing training and evaluation procedures. The reported comparisons favor the TTNet backbone and adaptive loss balancing, while the best sequence length depends on the task.

6.1. Implementation Details and Training

  • OpenTTGames is used to evaluate spatial and temporal multitasking because no open dataset combines those data.Experiments used PyTorch 1.2.0 and a single NVIDIA RTX 2080 Ti; training used Adam, a learning-rate schedule, and temporally consistent image augmentations.Source: §6.1 · p. 7

6.2. Evaluation metrics

  • Ball presence is evaluated by detection accuracy, ball localization by RMSE on true-positive detections, event spotting by PCE and SPCE, and segmentation by IoU.SPCE counts an event as correct when its prediction-target difference is below 0.25 in the experiments.Source: §6.2 · p. 7

6.3. Feature extractor

  • The TTNet backbone has lower inference time, fewer parameters, and better performance than ResNet-18 in almost every task.Both encoders were trained with optimal input-frame counts and adaptive loss balancing.Source: §6.3 · p. 7

6.4. Target sequence length

  • For event spotting, 9-frame sequences give the best results and are used thereafter, while ball detection favors 3–5 frames and segmentation favors one frame.Nine frames slightly reduce ball-detection metrics; longer sequences do not degrade segmentation quality, and the 9-frame choice keeps event predictions at a short time lag.Source: §6.4 · p. 8, Table 7 · p. 8, Table 8 · p. 8

6.5. Loss balancing

  • Adaptive loss balancing gives the best metric values in more tasks and, most importantly, in event spotting.The comparison covers summed loss components, manually tuned constant weights, and adaptive balancing.Source: §6.5 · p. 8, Table 9 · p. 8

7. Inference on a live video stream

The system supplements TTNet predictions with a separate network for serve spotting and applies game rules to raw predictions for automated table-tennis refereeing.

7. Inference on a live video stream

  • A separate convolutional neural network, using the event-spotting branch’s architecture and training method, handles serve spotting.The authors provide a supplementary system-demo video showing the network in the auto-referee setting.Source: §7 · p. 8
  • A game-rules module applies rules to raw predictions to enable completely automated table-tennis refereeing.The system demo video is provided in supplementary materials.Source: §7 · p. 8

8. Conclusion

The paper presents TTNet for real-time extraction of event, ball, and segmentation data from table-tennis video, and releases a labeled multi-task dataset. The authors report processing 120-fps video with a single consumer-grade GPU and demonstrate end-to-end training of all branches.

8. Conclusion

  • TTNet extracts events, ball positions, and semantic masks from downscaled full HD table-tennis video at 120 fps using a single consumer-grade GPU.The ball detector uses two cascaded detectors at different resolutions, and the authors report pixel-level ball accuracy.Source: §8 · p. 8
  • The authors demonstrate that all TTNet branches can be trained simultaneously end to end.Source: §8 · p. 8
  • The paper releases a labeled multi-task dataset for developing and assessing computer-vision systems for table-tennis analysis.Source: §8 · p. 8

Related papers

Loading 2004.09927v1…