Source-linked AI summary
SportsMOT: A Large Multi-Object Tracking Dataset in Multiple Sports Scenes
Yutao Cui, Chenkai Zeng, Xiaoyu Zhao, Yichun Yang, Gangshan Wu, Limin Wang
TL;DR
Sports MOT lacks benchmarks suited to its tracking demands, which matter for player statistics and tactical analysis. The paper introduces SportsMOT and MixSort, combining a sports-focused dataset with auxiliary appearance association; benchmarking identifies association as the main challenge and reports improved performance on SportsMOT and MOT17.
Problem
Sports-scene MOT is underrepresented in existing benchmarks despite its relevance to player statistics and automatic tactical analysis.
Method
The paper builds SportsMOT and proposes MixSort, which adds a MixFormer-like appearance association model to tracking-by-detection trackers.
Results
Benchmarking identifies object association rather than localization as SportsMOT’s main challenge, while MixSort boosts performance on SportsMOT and MOT17.
Takeaways & Limitations
SportsMOT provides a platform for advancing multi-object tracking and sports analysis through both motion-based and appearance-based association.
Abstract
from arXiv · showhide
Multi-object tracking in sports scenes plays a critical role in gathering players statistics, supporting further analysis, such as automatic tactical analysis. Yet existing MOT benchmarks cast little attention on the domain, limiting its development. In this work, we present a new large-scale multi-object tracking dataset in diverse sports scenes, coined as \emph{SportsMOT}, where all players on the court are supposed to be tracked. It consists of 240 video sequences, over 150K frames (almost 15\times MOT17) and over 1.6M bounding boxes (3\times MOT17) collected from 3 sports categories, including basketball, volleyball and football. Our dataset is characterized with two key properties: 1) fast and variable-speed motion and 2) similar yet distinguishable appearance. We expect SportsMOT to encourage the MOT trackers to promote in both motion-based association and appearance-based association. We benchmark several state-of-the-art trackers and reveal the key challenge of SportsMOT lies in object association. To alleviate the issue, we further propose a new multi-object tracking framework, termed as \emph{MixSort}, introducing a MixFormer-like structure as an auxiliary association model to prevailing tracking-by-detection trackers. By integrating the customized appearance-based association with the original motion-based association, MixSort achieves state-of-the-art performance on SportsMOT and MOT17. Based on MixSort, we give an in-depth analysis and provide some profound insights into SportsMOT. The dataset and code will be available at https://deeperaction.github.io/datasets/sportsmot.html.
1. Introduction
SportsMOT addresses the limited attention given to sports-scene MOT by providing a large dataset designed around fast, variable-speed motion and similar yet distinguishable player appearance. Benchmarks indicate that object association, rather than localization, is the central challenge, motivating MixSort.
- Dataset and motivation: SportsMOT targets sports analysis tasks such as player-distance and speed statistics and automatic tactical analysis.
- Dataset and motivation: SportsMOT contains 240 videos, over 150K frames, and over 1.6M bounding boxes across basketball, volleyball, and football.The dataset is split into 45 training, 45 validation, and 150 test sequences.
- Dataset and motivation: Its two defining properties are fast, variable-speed motion and similar yet distinguishable appearance, requiring stronger motion and appearance association.Players’ jersey numbers and postures provide appearance distinctions despite broadly similar visual context.
- Benchmarking and framework: Benchmarking shows lower IDF1 and AssA but high DetA than MOT17, indicating that SportsMOT’s main difficulty is object association rather than localization.
- Benchmarking and framework: MixSort adds a MixFormer-like appearance-based association model to tracking-by-detection trackers and integrates it with their original motion-based association.The framework is intended to improve association in sports scenes.
2. Related Work
Related MOT datasets span autonomous driving, pedestrian, dancing, and sports-relevant settings, but emphasize different scene properties and tracking demands. SportsMOT is distinguished by combining fast, variable-speed motion with similar yet distinguishable appearance.
- Multi-object tracking datasets: Existing MOT datasets cover autonomous driving, road pedestrians, static or moving-camera pedestrian scenes, crowded scenes, and dancing scenes.
- Multi-object tracking datasets: MOT20 emphasizes extreme crowding and occlusion, while DanceTrack emphasizes uniform appearance and diverse motion.
- Multi-object tracking datasets: SportsMOT differs from DanceTrack through players’ similar yet distinguishable appearance and fast, variable-speed motion.
- Object association in tracking: Tracking association commonly combines motion and appearance similarities into a matrix that is solved with the Hungarian algorithm.SORT uses predicted-box IoU, while ByteTrack adds low-confidence detections and OC-SORT addresses Kalman Filter limitations.
- Object association in tracking: Appearance-based methods generate re-identification features from detections or shared detection backbones and fuse them with motion cues for association.
3. SportsMOT Dataset
SportsMOT is a manually annotated dataset of overhead professional games covering three sports, designed around fast, variable-speed motion and similar yet distinguishable player appearance. Its statistics and visual analyses characterize these properties and motivate association-focused evaluation.
- Video Collection: 240 video sequences from football, basketball, and volleyball are collected from high-quality professional games, each at 720P and 25 FPS without abrupt viewpoint switches.Only overhead shots are used, limiting extreme viewpoint changes.
- Annotation Pipeline: Annotators label every visible player with full-body boxes, persistent unique IDs, and predicted boxes under partial occlusion, while skipping players whose torsos are substantially out of view.Boxes are refined after KCF-assisted propagation and reviewed before removing boxes with w < 5 or h < 5.
- Fast and Variable-Speed Motion: SportsMOT has faster motion than MOT17 and DanceTrack, with football showing the lowest adjacent-frame IoU among its categories.Kalman-Filter-based IoU is also lower than in MOT17 and DanceTrack, indicating more variable-speed motion.
- Overview: SportsMOT contains over 1.6M bounding boxes and 240 video clips, with 45 training, 45 validation, and 150 test sequences.The dataset has over 150K frames and dense annotations for players on the court.
- Similar yet Distinguishable Appearance: Players have similar but distinguishable appearances because jerseys carry different numbers and players exhibit distinct postures, placing SportsMOT between MOT17 and DanceTrack in re-ID separability.The dataset therefore targets more discriminative and extensive visual representations for association.
- Evaluation Metrics: SportsMOT recommends HOTA, AssA, and IDF1 alongside detection metrics because HOTA measures detection and association accuracy equally.These metrics emphasize object association more than MOTA, which focuses more on detection accuracy.
4. Multi-Object Tracking on SportsMOT
MixSort enhances tracking-by-detection trackers by adding MixFormer-based appearance association to motion-based association. It computes fused visual and IoU similarities for Hungarian matching, then updates or initializes tracks online.
- MixSort Framework: MixSort adds a MixFormer-like appearance association model to tracking-by-detection trackers such as ByteTrack and OC-SORT.The framework is designed to enhance appearance-based association and can be applied to trackers following the tracking-by-detection paradigm.
- MixFormer Appearance Association: MixFormer extracts target-specific discriminative features and communicates between each target template and the current search region to compute visual similarities.The corner-based localization head is replaced with a heatmap prediction head that produces a confidence heatmap for similarity computation.
- Similarity Computation: For each track, MixSort uses a Kalman Filter to predict a location, crops a centered search region, and obtains a template–region heatmap from MixFormer.Detection similarity is read from the heatmap at each detection center, while detections outside the search region receive similarity zero.
- Association Strategy: The fused similarity matrix combines visual similarity V with IoU using M = α · IoU + (1 − α) · V, then supports Hungarian association.The weight coefficient α controls the contribution of IoU relative to MixFormer visual similarity.
- Tracking Pipeline: After association, MixSort updates templates for matched tracks, retains unmatched tracks until a threshold, and initializes tracks from sufficiently confident unmatched detections.During inference, one template is maintained per track and replaced only when the new detection has sufficiently large uncovered area.
- Training: MixFormer training converts each player's ground-truth trajectory into TrackingNet format and uses Gaussian heatmaps with pixel-wise focal loss.The Gaussian spread σ adapts to bounding-box size, while focal-loss hyperparameters are set to γ = 2 and β = 4 following CornerNet.
5. Experiments and Analysis
Experiments identify object association, rather than localization, as SportsMOT’s central difficulty. MixSort improves association by combining appearance and motion cues, with effects varying across trackers and sports.
- Benchmark Results: MixSort-byte and MixSort-OC outperform state-of-the-art trackers in HOTA, IDF1 and AssA on MOT17 under the private detection protocol.
- Benchmark Results: MixSort improves OC-SORT by 0.4 HOTA and ByteTrack by 1.6 HOTA on the SportsMOT test set.
- Exploration Study: Fused association outperforms both pure motion-based and pure appearance-based association, while pure motion association exceeds pure appearance association across all sports categories.
- Exploration Study: Appearance cues improve football by +1.3, volleyball by +0.9 and basketball by +0.3, with faster-motion scenes more dependent on appearance.
- Exploration Study: Basketball is the most difficult category at 66.17 HOTA, followed by football at 73.19 and volleyball at 76.91.
- Exploration Study: Basketball’s smaller courts and greater physical contact can produce more occlusion and blur, making association harder than in football.
- Exploration Study: 73.8 HOTA from fused IoU and MixSort cues exceeds the 71.5 HOTA achieved by simple IoU in the ablation study.
6. Conclusion
The paper introduces SportsMOT for sports-scene tracking and MixSort for appearance-based association. The dataset targets fast, variable-speed motion and similar yet distinguishable appearance.
- SportsMOT is a large-scale sports tracking dataset characterized by fast, variable-speed motion and similar yet distinguishable appearance.
- MixSort introduces a MixFormer-like association module into multi-object tracking.