Source-linked AI summary
SparseTrack: Multi-Object Tracking by Performing Scene Decomposition based on Pseudo-Depth
Zelin Liu, Xinggang Wang, Cheng Wang, Wenyu Liu, Xiang Bai
TL;DR
Crowded scenes and frequent occlusions make reliable MOT association difficult, especially when low-score detections have similar locations. SparseTrack estimates relative pseudo-depth, decomposes dense targets into sparse depth-ordered subsets, and applies depth cascade matching with IoU association. It improves over ByteTrack across MOT17, MOT20, and DanceTrack and is comparable with recent state-of-the-art MOT methods on MOT benchmarks.
Problem
Crowded scenes and frequent occlusions challenge MOT, while low-score occlusions can cause location-association errors when detections have similar positions.
Method
SparseTrack estimates relative pseudo-depth from 2D images, decomposes targets into depth-ordered sparse subsets, and applies depth cascade matching for hierarchical IoU association.
Results
+2.0 HOTA, +0.7 MOTA, and +2.8 IDF1 over ByteTrack on MOT17, with additional gains on MOT20 and DanceTrack and comparable performance to recent state-of-the-art MOT methods.
Takeaways & Limitations
Target-set decomposition based on pseudo-depth provides an effective perspective for associating occluded targets in dense scenes, and DCM can be integrated into different trackers.
Takeaways & Limitations
SparseTrack depends strongly on detectors identifying partially obscured and low-confidence targets and struggles with rapid motion and deformation scenarios.
Abstract
from arXiv · showhide
Exploring robust and efficient association methods has always been an important issue in multiple-object tracking (MOT). Although existing tracking methods have achieved impressive performance, congestion and frequent occlusions still pose challenging problems in multi-object tracking. We reveal that performing sparse decomposition on dense scenes is a crucial step to enhance the performance of associating occluded targets. To this end, we propose a pseudo-depth estimation method for obtaining the relative depth of targets from 2D images. Secondly, we design a depth cascading matching (DCM) algorithm, which can use the obtained depth information to convert a dense target set into multiple sparse target subsets and perform data association on these sparse target subsets in order from near to far. By integrating the pseudo-depth method and the DCM strategy into the data association process, we propose a new tracker, called SparseTrack. SparseTrack provides a new perspective for solving the challenging crowded scene MOT problem. Only using IoU matching, SparseTrack achieves comparable performance with the state-of-the-art (SOTA) methods on the MOT17 and MOT20 benchmarks. Code and models are publicly available at \url{https://github.com/hustvl/SparseTrack}.
1 INTRODUCTION
SparseTrack addresses crowded-scene occlusion association by estimating relative pseudo-depth, decomposing dense targets into sparse subsets, and matching them hierarchically. The resulting IoU-only tracker improves over ByteTrack and is competitive with recent MOT methods across multiple benchmarks.
- 1 INTRODUCTION: Crowded scenes and frequent occlusions make MOT association difficult, while simultaneous IoU matching of low-score detections can mistake targets with similar locations.ByteTrack separately associates low-score detections, but its location association can deteriorate under many low-score occlusions or frequent overcrowding.
- 1 INTRODUCTION: The pseudo-depth method estimates relative target depth from 2D images using scene priors about camera height and approximately flat ground.Pseudo-depth is a reference for relative depth ordering rather than ground-truth 3D depth.
- 1 INTRODUCTION: +2.0 HOTA, +0.7 MOTA, and +2.8 IDF1 over ByteTrack on MOT17; SparseTrack also gains on MOT20 and DanceTrack.On MOT17, SparseTrack reaches 65.1 HOTA, 81.0 MOTA, and 80.1 IDF1; the reported gains are +2.1 HOTA, +0.4 MOTA, and +2.1 IDF1 on MOT20, and +7.8 HOTA, +1.7 MOTA, and +4.4 IDF1 on DanceTrack.
- 1 INTRODUCTION: DCM decomposes dense target sets into sparse subsets ordered from near to far, then associates detection and trajectory subsets at corresponding depth levels.Unmatched trajectories and detections are passed to the next depth level, reducing interference between targets at different depths.
- 1 INTRODUCTION: SparseTrack significantly outperforms previous IoU-only trackers and achieves comparable results with recent state-of-the-art methods across MOT benchmarks.The tracker uses only IoU matching while incorporating pseudo-depth decomposition and DCM.
2 RELATED WORK
SparseTrack addresses crowded-scene occlusion association by decomposing targets using depth information rather than relying only on temporal, appearance, or motion cues.
- Existing tracking-by-detection methods use positional, appearance, motion, or staged association cues, but occlusions and motion blur can reduce association reliability.
- SparseTrack’s pseudo-depth method projects scene depth relationships into 2D to support decomposition of dense target sets.
- SparseTrack uses depth information to decompose target sets, introducing a distinct approach to associating occlusions in crowded scenes.The paper positions this as the first use of depth information for target-set decomposition.
3 METHOD
SparseTrack performs tracking-by-detection with pseudo-depth estimation and depth cascade matching to decompose crowded scenes into sparser subsets for association. Its pseudo-depth is a relative ordering cue under common ground-plane assumptions, while DCM performs staged matching across depth levels and can be integrated into other trackers.
- Overall framework: SparseTrack processes each frame with YOLOX, estimates target pseudo-depth from bounding boxes and image size, then performs frame-by-frame data association.The tracker follows a tracking-by-detection paradigm and uses detected boxes and confidence scores as inputs to pseudo-depth estimation.
- Pseudo-depth estimation: Pseudo-depth assumes the camera is above the ground and scene objects lie on the same plane, enabling relative depth ordering rather than true 3D depth.The method uses a geometric construction whose value is sufficient to measure relative depth among objects on the same ground.
- Depth decomposition: The method partitions detections between the minimum and maximum pseudo-depth values into k uniform depth intervals, producing sparser target subsets from dense crowds.These intervals define different depth levels used by the association procedure.
- Depth cascade matching: Depth cascade matching associates sparse track and detection subsets in order across depth levels, using IoU distance and Hungarian matching within the cascade.The procedure generates sparse detection and track subsets before performing depth cascade matching.
- Depth cascade matching: SparseTrack uses fewer pseudo-depth levels for high-score detections and more levels for low-score detections, with 1–2 versus 4–8 levels respectively.The setting reflects the stated relationship between confidence scores and target occlusion, using finer decomposition for dense occlusions.
- Depth cascade matching: DCM is plug-and-play and can be integrated into various trackers.The paper states that specific settings are provided separately.
4.1 Setting
The experiments evaluate SparseTrack on MOT17, MOT20, and DanceTrack using standard detection, association, and overall tracking metrics. Dataset-specific depth levels, lost-track limits, input resolutions, and detector settings are selected for evaluation and ablation comparisons.
- Evaluation datasets: SparseTrack is evaluated on MOT17, MOT20, and DanceTrack, with ablations conducted on validation splits from these datasets.Half of the MOT17 and MOT20 training sets are used for validation, alongside a DanceTrack validation set.
- Evaluation metrics: The evaluation reports MOTA, FP, FN, IDs, IDF1, HOTA, AssA, and DetA to measure detection, association, and overall tracking performance.MOTA emphasizes detection-related errors, IDF1 and AssA emphasize association, DetA measures detection accuracy, and HOTA combines detection and association.
- Dataset settings: The default pseudo-depth levels are 3 for MOT17, 8 for MOT20, and 12 for DanceTrack, with lost tracks retained for 30 or 60 frames depending on the dataset.MOT20 and DanceTrack use 60-frame lost-track limits, while MOT17 uses 30 frames.
- Implementation settings: Inference uses 800 × 1440 inputs for MOT17 and DanceTrack and 896 × 1600 for MOT20, while adopting ByteTrack’s pretrained YOLOX detector and matching NMS settings.The same detector weights and NMS threshold are used for fair comparison with the baseline.
4.2 Evaluation of Different Benchmark
SparseTrack improves over ByteTrack across MOT17, MOT20, and DanceTrack while using simple IoU association, with the largest gains on DanceTrack.
- MOT17: +2.0 HOTA, +0.7 MOTA, and +2.8 IDF1 over ByteTrack on the MOT17 test set.SparseTrack uses the same pre-trained detector as the baseline and achieves comparable performance with simple IoU association.
- Overall comparison: SparseTrack achieves comparable or better performance than methods using appearance features, attention, graphs, or other advanced association components.The reported comparisons cover MOT17, MOT20, and DanceTrack benchmark evaluations.
- MOT20: +2.1 HOTA, +0.4 MOTA, and +2.1 IDF1 over ByteTrack on MOT20, a denser and more occluded benchmark.The method also reports very low IDs and false positives together with high HOTA.
- DanceTrack: +7.8 HOTA, +1.7 MOTA, and +4.4 IDF1 over ByteTrack on DanceTrack, with +7.0 AssA and +7.9 DetA.The comparison uses the same pre-trained detector and reports gains across association and detection metrics.
4.3 Ablations
Ablations show that more pseudo-depth levels generally improve low-score association until excessive layering stops helping, while resolution and GMC effects depend on the setting.
- The impact of the number of pseudo-depth levels: Association performance improves as pseudo-depth levels increase, but gains stop when the number of levels exceeds 7.The authors attribute this saturation to excessive sparsity among low-scoring detections and participating tracks.
- The impact of the input with different resolutions: Tracking performance for SparseTrack and ByteTrack strengthens as input resolution increases, with larger association improvements for SparseTrack.Detection gains diminish beyond a resolution threshold, while tracking continues to benefit from recognizing more severely occluded targets.
- Global motion compensation: GMC substantially improves SparseTrack on MOT17 but provides little tracking gain on MOT20.The difference is attributed to more pronounced camera motion in MOT17.
4.4 Comparison with Other Simple Methods
Against location-dependent association methods under matched detector and NMS settings, SparseTrack achieves the best tracking performance on all three validation sets.
- Comparison with other simple methods: SparseTrack achieves the best tracking performance on the MOT17, MOT20, and DanceTrack validation sets among the compared location-dependent methods.The comparison removes video-specific hyperparameters and uses identical YOLOX-x detector weights and NMS settings.
4.5 Visualization
Visualizations show how pseudo-depth decomposes dense occlusions into separate depth levels and supports more stable association than ByteTrack.
- Target set decomposition: Pseudo-depth assigns overlapping targets in dense occlusions to different depth levels, decoupling targets that would otherwise be difficult to associate.The decomposition is presented as the key subprocess enabling DCM to handle dense occlusions.
- Occlusion association: SparseTrack demonstrates greater stability than ByteTrack in challenging occlusion scenarios under consistent detector settings.The visual comparison uses four MOT challenge videos with the same publicly available detector hyperparameters.
4.6 Discussion
SparseTrack improves on MOT and DanceTrack but remains sensitive to detection quality, motion modeling, and pseudo-depth accuracy in difficult scenarios.
- SparseTrack has poor performance on the MOT public detection benchmark despite improvements on MOT and DanceTrack.
- Its association performance depends substantially on accurately detecting partially obscured and low-confidence targets.
- Rapid motion and deformation can make pseudo-depth relationships inaccurate, reducing tracking performance.
- At low frame rates, a simple Kalman filter can produce inaccurate motion cues and less accurate pseudo-depth information.
5 CONCLUSION
SparseTrack decomposes dense target sets by pseudo-depth and associates corresponding sparse subsets with Depth Cascade Matching. It achieves competitive MOT17 and MOT20 performance using only simple IoU association, while DCM can be integrated into existing trackers.
- SparseTrack estimates relative target depth, partitions targets into sparse subsets, and associates matching detection and trajectory subsets with Depth Cascade Matching.
- SparseTrack achieves competitive MOT17 and MOT20 performance using only simple IoU distance association, without appearance embeddings or enhanced motion prediction.
- The tracker presents target-set decomposition by pseudo-depth as an alternative perspective for addressing occlusion in dense scenes.
- DCM is plug-and-play and can be integrated into existing trackers for consistent performance improvement.