PaperlayerView exact revision on arXiv

Source-linked AI summary

TrackNet: A Deep Learning Network for Tracking High-speed and Tiny Objects in Sports Applications

Yu-Chuan Huang, I-No Liao, Ching-Hsuan Chen, Tsì-Uí İk, Wen-Chih Peng

arXiv:1907.03698v1cs.LGcs.CVcs.MMstat.ML

TL;DR

Small, fast sports balls are difficult to locate in broadcast video when images are blurry, faint, or occluded. TrackNet combines appearance recognition with trajectory learning across consecutive frames to generate ball-position heatmaps. It reports 99.7% precision, 97.3% recall, and 98.5% F1 on a tennis test set, with 95.3% precision, 75.7% recall, and 84.3% F1 in 10-fold cross-validation.

  • Problem

    High-speed, tiny balls in sports video are difficult to recognize and position accurately, limiting reliable ball-trajectory data for game analysis.

    Source: §I · p. 1

  • Method

    TrackNet uses a CNN and DeconvNet to generate heatmaps from one or more consecutive frames, learning ball appearance and trajectory patterns.

    Source: §IV · pp. 6–7

  • Results

    99.7% precision, 97.3% recall, and 98.5% F1-measure were reported for TrackNet Model II’; 10-fold cross-validation achieved 95.3% precision, 75.7% recall, and 84.3% F1-measure.

    Source: §V · p. 10

  • Takeaways & Limitations

    TrackNet can position fast, tiny objects in broadcast sports video without high-resolution, high-frame-rate recording, reducing recording and processing costs.

    Source: §VI · p. 11

    For badminton, large speed differences between drop shots and smashes make it difficult for the model to fit both scenarios perfectly.

    Source: §V · p. 11

Abstract

from arXiv · show

Ball trajectory data are one of the most fundamental and useful information in the evaluation of players' performance and analysis of game strategies. Although vision-based object tracking techniques have been developed to analyze sport competition videos, it is still challenging to recognize and position a high-speed and tiny ball accurately. In this paper, we develop a deep learning network, called TrackNet, to track the tennis ball from broadcast videos in which the ball images are small, blurry, and sometimes with afterimage tracks or even invisible. The proposed heatmap-based deep learning network is trained to not only recognize the ball image from a single frame but also learn flying patterns from consecutive frames. TrackNet takes images with a size of $640\times360$ to generate a detection heatmap from either a single frame or several consecutive frames to position the ball and can achieve high precision even on public domain videos. The network is evaluated on the video of the men's singles final at the 2017 Summer Universiade, which is available on YouTube. The precision, recall, and F1-measure of TrackNet reach $99.7\%$, $97.3\%$, and $98.5\%$, respectively. To prevent overfitting, 9 additional videos are partially labeled together with a subset from the previous dataset to implement 10-fold cross-validation, and the precision, recall, and F1-measure are $95.3\%$, $75.7\%$, and $84.3\%$, respectively. A conventional image processing algorithm is also implemented to compare with TrackNet. Our experiments indicate that TrackNet outperforms conventional method by a big margin and achieves exceptional ball tracking performance. The dataset and demo video are available at https://nol.cs.nctu.edu.tw/ndo3je6av9/.

I. INTRODUCTION

TrackNet addresses the difficulty of locating small, fast, blurry sports balls in affordable broadcast or consumer video. It combines ball appearance with trajectory patterns across frames to produce heatmap-based positions.

I. INTRODUCTION

  • High-end cameras and high frame rates make professional sports data collection costly, while broadcast-video acquisition could offer a lower-cost alternative.Small, fast balls can appear tiny and blurry, making trajectory data difficult to collect reliably.Source: §I · p. 1
  • TrackNet uses consecutive frames to learn trajectory patterns alongside ball appearance, addressing blur, afterimages, and occlusion.The approach replaces rule-based techniques with a deep learning network applied to multiple frames.Source: §I · pp. 1–2
  • The network uses VGG-16 feature maps and FCN-style upsampling to generate a heatmap, then estimates the ball position from that heatmap.Its position calculation assumes at most one ball on the court.Source: §I · p. 2
  • On broadcast tennis videos, experiments found TrackNet outperformed the conventional image-recognition algorithm and effectively located a fast-moving tennis ball; both single-frame and multiple-frame versions were implemented.The tennis set contains 20,844 labeled frames, and the badminton set contains 18,242 labeled frames.Source: §I · p. 2
  • 95.3% precision, 75.7% recall, and 84.3% F1-measure were achieved in 10-fold cross-validation.The evaluation used additional tennis videos alongside a subset of the earlier dataset to address overfitting.Source: §I · p. 2

II. RELATED WORKS

Related work covers costly professional tracking systems, methods for ball detection and trajectory estimation, and the CNN operations underlying TrackNet’s heatmap approach.

II. RELATED WORKS

  • Professional systems such as Hawk-Eye use high-end cameras and dedicated operators, making them too expensive for non-professional teams.The passage also situates ball and player trajectory analysis in prior performance and tactics research.Source: §II · p. 3
  • Earlier ball-tracking work includes multi-camera 3D detection, particle filtering, and data association for trajectory recovery.Small balls can resemble other objects and fast motion can blur them, producing false positives and false negatives.Source: §II · p. 3
  • CNNs provide learned image features, while convolution, pooling, and upsampling support feature extraction and pixel-wise prediction.The passage describes convolutional layers, ReLU, maximum pooling, upsampling, batch normalization, and loss-based training.Source: §II · pp. 3–4
  • TrackNet combines a CNN and DeconvNet to generate a heatmap from consecutive frames, with additional frames intended to help learn trajectory patterns.The heatmap is ideally Gaussian-centered on the ball image.Source: §II · p. 4

III. DATASET

The dataset combines labeled tennis broadcast frames with additional tennis-court footage and a badminton trial. Labels record ball visibility and position, and tennis labels also encode trajectory patterns.

Tennis dataset

  • 20,844 frames from 81 clips of the 2017 Summer Universiade men's final form the first tennis dataset.The clips cover complete plays from serve to score; the source video is 1280 × 720 at 30 fps.Source: §III · p. 4
  • Visibility labels mark absent, easily identified, hard-to-identify, and occluded balls; the respective counts are 659, 18,035, 2,143, and 7 frames.Neighboring frames can help label a ball that is hard to identify or occluded.Source: §III · pp. 4–5
  • For blurry or prolonged tennis traces, the labeled X and Y coordinates mark the latest ball position in its direction of travel.Source: §III · p. 5
  • Tennis trajectory labels encode flying, hit, and bouncing as 0, 1, and 2, respectively.Source: §III · p. 5

Additional tennis data

  • An additional 16,118 frames from nine videos across different tennis courts were collected to diversify the training data.The authors expect this variety to improve robustness across court settings.Source: §III · pp. 5–6

Badminton dataset

  • The badminton trial dataset contains 18,242 frames from a 2018 Indonesia Open final; its visibility labels mark the ball as absent or present, including blurry cases.Badminton coordinates use the latest position of a prolonged trace, considering its direction of motion.Source: §III · p. 6

IV. TRACKNET

TrackNet combines convolutional and deconvolutional networks to generate an object-detection heatmap from one or more consecutive frames. It trains on Gaussian heatmaps and converts the output into a ball position.

IV. TRACKNET

  • TrackNet combines a CNN and DeconvNet to generate an object-position heatmap, with three consecutive input frames used to learn trajectory patterns.The paper also implements a single-frame version; the number of input frames is a network parameter.Source: §IV · p. 6, Figure 9 · p. 7
  • The training target is a probability-like heatmap at input resolution, formed from an amplified 2D Gaussian centered on the labeled ball.Its variance refers to the ball-image diameter; the implementation uses σ^2 = 10, corresponding roughly to the region where G(x, y) ≥ 128.Source: §IV · pp. 6–7, Figure 8 · p. 6
  • The first 13 layers follow VGG-16, while layers 14–24 follow DeconvNet; upsampling recovers information lost through pooling for pixel-wise prediction.The network uses symmetric numbers of upsampling and maximum-pooling layers.Source: §IV · p. 7, Table II · p. 7
  • For each pixel, softmax probabilities over grayscale depths determine the heatmap value by selecting the depth with the highest probability.The network’s continuous heatmap is thresholded at 128, then the Hough Gradient Method finds a circle; its centroid is returned only when exactly one circle is identified.Source: §IV · pp. 7–8
  • Training uses cross-entropy based on the predicted pixel values and a ground-truth function that marks the Gaussian heatmap’s grayscale value.The supplied passage states that Q(i, j, k) is 1 when G(i, j) = k and 0 otherwise.Source: §IV · pp. 7–8

V. EXPERIMENTS

TrackNet’s consecutive-frame model performs strongly on tennis, including under 10-fold cross-validation, while badminton tracking is more challenging. The experiments compare frame-input variants and conventional image processing, and test transfer to badminton.

Experimental setup

  • The tennis evaluation uses 20,844 frames split randomly into 70% training and 30% testing, with training run for 500 epochs.The paper uses the loss curve to select 500 epochs to avoid underfitting and overfitting.Source: §V · p. 8, Figure 10 · p. 8

Comparison with conventional image processing

  • Compared with Archana’s algorithm, both TrackNet models reduce false positives and false negatives, increasing true positives and true negatives.The paper also states that Model II performs better than Model I.Source: §V · p. 9, Table IV · p. 9

Consecutive-frame tracking

  • Model II correctly positions 2 of 7 occluded balls, and the paper reports improved tracking of blurry or occluded objects with consecutive frames.The authors attribute the added information to trajectory patterns learned from neighboring frames.Source: §V · p. 10

Tennis results

  • 99.8% precision, 96.6% recall, and 98.2% F1-measure are achieved by three-frame Model II, outperforming single-frame Model I and Archana’s algorithm.The comparison reports 95.7% precision, 89.6% recall, and 92.5% F1-measure for Model I, and 92.5% precision, 74.5% recall, and 82.5% F1-measure for Archana’s algorithm.Source: Table V · p. 9, §V · p. 10
  • 95.3% precision, 75.7% recall, and 84.3% F1-measure are achieved by TrackNet Model II’ with 10-fold cross-validation.On the same test set, Model II’ achieves 99.7% precision, 97.3% recall, and 98.5% F1-measure.Source: §V · p. 10

Badminton results

  • On badminton, TrackNet-Badminton reaches 85.0% precision, 57.7% recall, and 68.7% F1-measure, versus 75.8%, 22.9%, and 35.2% for TrackNet-Tennis.Badminton’s positioning-error cutoff is 7.5 pixels, compared with 5 pixels for tennis; the discussion identifies high speed and speed variation as tracking challenges.Source: §V · pp. 10–11, Table VI · p. 10

VI. CONCLUSION

TrackNet combines convolutional and deconvolutional networks to position fast, tiny objects in tennis and badminton videos. Its multiple-consecutive-frame input supports trajectory learning, and the experiments report strong tennis performance and promising badminton tracking.

Conclusion

  • TrackNet is a heatmap-based network combining convolutional and deconvolutional neural networks to position high-speed, tiny objects such as tennis balls and badminton shuttles.The paper reports accurate predictions from broadcast sports videos without high frame rate or high resolution, reducing recording and processing costs.Source: §VI · p. 11
  • A scalable input allows multiple consecutive frames, which the paper uses to enhance learning of trajectory patterns for fast-moving objects.The experiments compare consecutive-frame input with conventional image processing on tennis video.Source: §VI · p. 11
  • The authors report exceptional tennis prediction performance and decent badminton tracking, describing the framework as promising for related applications.The paper presents this as evidence of extensibility to related applications.Source: §VI · p. 11

Related papers

Loading 1907.03698v1…