Source-linked AI summary
DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion
Peize Sun, Jinkun Cao, Yi Jiang, Zehuan Yuan, Song Bai, Kris Kitani, Ping Luo
TL;DR
Multi-object tracking has often relied on appearance matching, but this is inadequate when objects look alike and move diversely. The paper introduces DanceTrack, a large-scale multi-human tracking dataset designed for such conditions, and benchmarks existing trackers. State-of-the-art methods show a significant performance drop on DanceTrack, while the dataset’s detection scores indicate that association is the central challenge.
Problem
Existing tracking datasets and methods often emphasize distinguishable appearance and regular motion, leaving tracking under similar appearance and diverse motion insufficiently addressed.
Method
The paper builds DanceTrack, a large-scale multi-human tracking dataset, benchmarks baseline trackers, and analyzes cues and evaluation metrics for challenging association.
Results
Existing trackers show a significant performance drop on DanceTrack, while higher MOTA and DetA than on MOT17 indicate detection is not the bottleneck and association performance drops.
Takeaways & Limitations
DanceTrack provides a platform for developing trackers that rely less on visual discrimination and more on motion analysis for uniform-appearance, diverse-motion cases.
Takeaways & Limitations
The dataset provides only bounding-box annotations, although pose or segmentation masks would be important for finer-grained study.
Abstract
from arXiv · showhide
A typical pipeline for multi-object tracking (MOT) is to use a detector for object localization, and following re-identification (re-ID) for object association. This pipeline is partially motivated by recent progress in both object detection and re-ID, and partially motivated by biases in existing tracking datasets, where most objects tend to have distinguishing appearance and re-ID models are sufficient for establishing associations. In response to such bias, we would like to re-emphasize that methods for multi-object tracking should also work when object appearance is not sufficiently discriminative. To this end, we propose a large-scale dataset for multi-human tracking, where humans have similar appearance, diverse motion and extreme articulation. As the dataset contains mostly group dancing videos, we name it "DanceTrack". We expect DanceTrack to provide a better platform to develop more MOT algorithms that rely less on visual discrimination and depend more on motion analysis. We benchmark several state-of-the-art trackers on our dataset and observe a significant performance drop on DanceTrack when compared against existing benchmarks. The dataset, project code and competition server are released at: \url{https://github.com/DanceTrack}.
1. Introduction
DanceTrack addresses the failure of appearance-dominant tracking when people look alike and move diversely. It introduces a large-scale dataset and benchmarks showing that existing trackers struggle in these conditions.
- Motivation: Existing multi-object tracking commonly uses detection followed by re-ID, relying heavily on appearance cues for association.This paradigm can fail catastrophically when objects share similar appearance, such as dancers wearing uniform clothes.
- Motivation: Existing datasets often contain objects with distinct appearance and regular or linear motion, encouraging methods to rely on appearance matching.The dominant paradigm therefore fails outside that biased distribution.
- Dataset: DanceTrack provides over 100K image frames featuring uniform appearance, diverse motion, complex gestures, frequent occlusion and crossover.Its properties make visual re-ID less discriminative and impose higher demands on motion modeling.
- Benchmark and analysis: The authors benchmark baseline methods on DanceTrack and provide comprehensive analysis of cues for more robust tracking in complicated real-life situations.The benchmark examines existing algorithms while the analysis searches for alternatives to appearance matching.
- Contributions: The paper contributes a large-scale dataset covering low appearance distinguishability and diverse non-linear motion patterns.DanceTrack is intended as a platform for developing trackers that use cues beyond visual discrimination.
2. Related Works
Prior multi-object tracking research includes human, vehicle, pedestrian and multi-camera datasets, but common benchmarks often underrepresent indistinguishable appearance and diverse motion. Appearance matching dominates many trackers, while motion-based methods have generally handled only simple motion, motivating greater attention to motion analysis.
- Multi-object tracking datasets: Popular human-tracking datasets such as MOT15, MOT17 and MOT20 remain limited in undistinguished appearance and diverse motion.MOT17 has few videos and scenarios, while MOT20 emphasizes occlusion but retains regular movement and distinguishable appearances.
- Multi-object tracking datasets: Other datasets target multi-camera tracking, video instance segmentation, or autonomous-driving scenarios involving vehicles and pedestrians.Examples include WILDTRACK, YouTube-VIS, KITTI, BDD100K, Waymo and KITTI360.
- Tracking by matching appearance: Appearance similarity is a dominant association cue in methods including JDE, FairMOT and QDTrack, as well as newer transformer-based trackers.These methods learn or compare visual representations to match objects across frames.
- Tracking by matching appearance: Appearance matching still fails in many real-world situations, motivating the use of additional cues.The paper specifically frames motion as an important alternative when appearance is unreliable.
- Motion analysis in object tracking: Motion-based tracking has used displacement cues and sequential tools such as particle and Kalman filters, but existing methods generally handle only simple linear motion.SORT is given as an example of a tracker built on a Kalman-filter motion model.
3. DanceTrack
DanceTrack is designed to evaluate multi-object tracking when appearance is uniform and motion is diverse, using group-dancing videos with frequent crossover, occlusion, and deformation. Its analysis shows higher appearance similarity and more frequent relative-position switches than established datasets, while adjacent-frame IoU remains comparable; evaluation uses HOTA and related metrics.
- Dataset Design: DanceTrack targets tracking scenarios with similar appearance, diverse motion, frequent crossover, occlusion, and body deformation.The design specifically avoids relying on object density alone, emphasizing dynamically changing occlusions and relatively large motion ranges.
- Dataset Construction: The dataset contains 100 videos split into 40 training, 25 validation, and 35 test videos, with training and validation annotations public.The split preserves similar distributions of average length, box counts, scenes, and motion diversity across subsets.
- Dataset Statistics: DanceTrack’s co-existing objects have higher appearance similarity than MOT17, making pure appearance matching less effective for association.Lower cosine distance between re-ID features indicates that object features are more entangled in DanceTrack than in MOT17.
- Dataset Statistics: DanceTrack and MOT datasets have close average adjacent-frame IoU, indicating that DanceTrack does not involve unreasonably fast object movement.The dataset’s challenge comes from motion diversity rather than an implausibly low frame rate or extreme movement speed.
- Dataset Statistics: DanceTrack has much more frequent relative-position switches than KITTI, MOT17, and MOT20, reflecting nonlinear motion, crossover, and inter-object occlusion.The frequency-of-switch metric captures whether pairs exchange left-right or top-down relative positions on adjacent frames.
- Evaluation Metrics: The evaluation protocol uses HOTA as the main metric, with AssA and IDF1 for association and DetA and MOTA for detection quality.The tools also report false positives, false negatives, and identity switches for fine-grained analysis.
4. Experiments
Experiments show that DanceTrack’s main difficulty is object association rather than detection, especially under uniform appearance and diverse, non-linear motion. Motion-aware and richer-modal cues improve association, while appearance matching can hurt.
- Oracle Analysis: Oracle analysis uses ground-truth boxes to isolate association as the tracking bottleneck.The study compares IoU, appearance, and motion-based association on MOT17 and DanceTrack.
- Oracle Analysis: On MOT17, association performance is nearly perfect, and IoU matching performs best, consistent with simple, regular motion.The result indicates that association is usually not the bottleneck on MOT17.
- Oracle Analysis: On DanceTrack, association metrics drop dramatically despite near-100 DetA and MOTA, and adding linear motion or appearance does not improve IoU matching.Appearance similarity makes all metrics worse than using no appearance cue.
- Oracle Analysis: DanceTrack re-ID features are highly entangled across objects, unlike the distinguishable feature space observed on MOT17.The t-SNE visualization qualitatively supports DanceTrack’s uniform-appearance property.
- Benchmark Results: Across benchmarked trackers, HOTA, IDF1, and AssA fall substantially on DanceTrack while MOTA and DetA are higher than on MOT17.This separates tracking-quality degradation from detection quality and highlights association as the central challenge.
- Association Strategy: OC-SORT achieves the best association performance among compared DanceTrack association methods, while DeepSORT performs worse than SORT.OC-SORT improves robustness for non-linear motion without relying on appearance matching.
- Association Strategy: Kalman Filter and LSTM motion models outperform naive IoU association, showing the value of temporal dynamics when appearance cues are unreliable.The experiment motivates stronger motion modeling for DanceTrack.
- Analysis of More Modalities: Segmentation masks and especially human pose improve tracking, whereas KITTI-trained depth information does not improve DanceTrack performance under domain shift.Masks help in crowded or occluded cases, and pose remains more robust when bounding boxes are unreliable.
5. Conclusion
DanceTrack is a multi-object tracking dataset designed around uniform appearance and diverse motion, exposing limitations in appearance-based association and encouraging more comprehensive tracking.
- DanceTrack is a new multi-object tracking dataset featuring objects with uniform appearance and diverse motion patterns.
- Its design prevents Re-ID shortcuts and reveals how existing datasets emphasize detection quality and appearance matching while underrepresenting other association cues.
- The dataset provides a platform for developing trackers that analyze complex motion patterns for more comprehensive multi-object tracking.
A. Change Log
This update revises benchmark tables by adding or updating trackers and introduces a more consistent motion-model comparison. It also documents a revised QDTrack configuration while preserving its tracking and embedding components.
- Version history: The document includes an initial arXiv version in its change log.
- Benchmark updates: MOTR is updated in Table 3, with further details available in its project repository.The change log points readers to the MOTR GitHub repository for additional information.
- Benchmark updates: GTR and OC-SORT are added to Table 3, while Table 5 compares motion models using a YOLOX detector for consistency with Table 4.The new comparison is explicitly designed to align detector usage across the relevant tables.
- Implementation updates: QDTrack is updated with YOLOX, ByteTrack-aligned augmentation and hyperparameters, and 24 DanceTrack training epochs, without changing its tracking algorithm or embedding component.The revised configuration replaces Faster R-CNN and preserves the tracking and embedding-learning components.