Source-linked AI summary
Multiple Object Tracking with Correlation Learning
Qiang Wang, Yun Zheng, Pan Pan, Yinghui Xu
TL;DR
Multiple object tracking needs better handling of similar distractors and long-range spatial-temporal dependencies than convolutional and short-context approaches provide. CorrTracker learns dense local correlations across object context and frames, with self-supervised constraints and multiscale feature propagation. It achieves state-of-the-art performance on four MOT benchmarks, including 76.5% MOTA and 73.6% IDF1 on MOT17.
Problem
MOT methods struggle to distinguish similar objects and efficiently model long-range spatial-temporal dependencies because appearance matching and convolutional receptive fields are limited.
Method
CorrTracker uses local spatial-temporal correlation volumes, self-supervised correlation learning, and multiscale feature-pyramid matching in an end-to-end architecture.
Results
CorrTracker achieves state-of-the-art performance across four MOT benchmarks, including 76.5% MOTA and 73.6% IDF1 on MOT17.
Takeaways & Limitations
Correlation learning strengthens discriminative embeddings and temporal modeling while supporting efficient inference for crowded-scene tracking.
Abstract
from arXiv · showhide
Recent works have shown that convolutional networks have substantially improved the performance of multiple object tracking by simultaneously learning detection and appearance features. However, due to the local perception of the convolutional network structure itself, the long-range dependencies in both the spatial and temporal cannot be obtained efficiently. To incorporate the spatial layout, we propose to exploit the local correlation module to model the topological relationship between targets and their surrounding environment, which can enhance the discriminative power of our model in crowded scenes. Specifically, we establish dense correspondences of each spatial location and its context, and explicitly constrain the correlation volumes through self-supervised learning. To exploit the temporal context, existing approaches generally utilize two or more adjacent frames to construct an enhanced feature representation, but the dynamic motion scene is inherently difficult to depict via CNNs. Instead, our paper proposes a learnable correlation operator to establish frame-to-frame matches over convolutional feature maps in the different layers to align and propagate temporal context. With extensive experimental results on the MOT datasets, our approach demonstrates the effectiveness of correlation learning with the superior performance and obtains state-of-the-art MOTA of 76.5% and IDF1 of 73.6% on MOT17.
1. Introduction
MOT methods struggle with distractors, occlusions, and limited spatial-temporal context. CorrTracker addresses these issues with local correlation learning for object-context relations and temporal information, achieving strong benchmark results.
- MOT aims to continuously locate multiple target trajectories and supports applications including video surveillance and autonomous driving.
- Tracking-by-detection separates detection and data association, limiting joint optimization and leaving challenges with distractors and frequent occlusions.
- Appearance-based matching produces inaccurate, indistinctive confidences when similar objects act as distractors.
- CorrTracker models object-context topology with a local spatial correlation volume, avoiding the computational cost of full correlation.
- CNN-based detectors underuse temporal information, while methods using adjacent frames remain limited by short temporal context and local receptive fields.
- CorrTracker reports 76.5% MOTA and 73.6% IDF1 on MOT17 while improving results across four MOT benchmarks.
2. Related Work
Related MOT work spans fast motion- and appearance-based trackers, multi-frame prediction methods, and correlation-based approaches. CorrTracker extends correlation modeling to dense spatial-temporal context and feature propagation.
- IOUTracker and Kalman-filter trackers prioritize speed but struggle to maintain stable tracking during target interleaving.
- Re-identification features improve discriminability, but separate patch computation creates a computational bottleneck.
- Tracktor, CenterTrack, Chained-Tracker, and TubeTK use adjacent frames to form short associations, but often incur identity switches from limited long-term dependencies.
- CorrTracker enhances appearance features with local correlation volumes that encode pairwise object and spatial-temporal neighborhood relations.
- Correlation operations produce dense local matching confidences, while CorrTracker additionally propagates features across multiple frames.
3. Methodology
CorrTracker models spatial and temporal associations through local correlation volumes across feature-pyramid levels, using self-supervision to improve discrimination and memory features to propagate temporal context. The resulting end-to-end framework combines correlation learning with detection and data association for efficient tracking in complex scenes.
- Overall framework: CorrTracker jointly extracts features, learns spatial-temporal correlations, predicts detections, and associates detections with trajectories in an end-to-end architecture.Feature extraction and correlation learning with detection are differentiable, while association assigns detections to likely trajectories.
- Motivation: Appearance-only matching can produce inaccurate confidences for similar distractors, while local structure models target relationships with neighboring context.Patch-based features lose correlation information between cropped regions and retain adjacency mainly through coordinates, which can increase identity switches.
- Spatial correlation: Spatial local correlation evaluates feature similarity only within a neighborhood, encoding each target’s relative displacement to nearby locations.For displacement d constrained by ∥d∥∞≤R, the vectorized correlation volume has size H_l × W_l × (2R + 1)^2.
- Multi-scale correlation: Multi-scale pyramid correlation expands contextual coverage while controlling the computational and storage costs of enlarging a single local search region.Correlation spans spatial context up to [0, R × D × 2^l] and passes information from higher pyramid levels toward lower levels.
- Temporal correlation: Temporal correlation extends the spatial module across frames, using reference images as memory to enhance features and improve consistency during occlusion and motion blur.The approach establishes multi-scale correlations between different frames rather than relying only on short adjacent-frame inputs.
- Self-supervised learning: Self-supervised correlation learning combines tracking supervision from identity labels with correspondence-flow colorization as a proxy task.Positions without objects are ignored, and the identity-based objective uses a class-balanced cross-entropy loss.
4. Experiments
Experiments evaluate CorrTracker through implementation details, ablations, benchmark comparisons, and qualitative visualizations. The results show improved association accuracy, strong benchmark performance, and robustness in crowded and occlusion-heavy scenes.
- Evaluation Setup: CorrTracker is evaluated on 2DMOT2015, MOT16, MOT17, and MOT20 using standard accuracy, identity, error, and runtime metrics.MOT20 contains four extremely crowded test videos, providing a particularly challenging evaluation setting.
- Ablation Studies: Compared with the baseline, spatial correlation increases IDF1 by 2.9% and reduces identity switches by 15%.The spatial module models geometric information and relative positional relationships, improving discrimination between similar distractors.
- Ablation Studies: Larger correlation regions improve MOTA, but gains become marginal beyond R = 5 as feature-map resolution limits further improvement.The local region expands the neighborhood used for matching while avoiding the overhead associated with full non-local correlation.
- Evaluation Setup: The experiments compare CorrTracker with eight state-of-the-art trackers spanning joint detection and embedding, multi-frame prediction, graph-network, and offset-prediction approaches.The comparison includes JDE, FairMOT, TubeTK, CTracker, MPN, JDMOTGNN, CenterTrack, and Tracktor++v2.
- Experiments on MOT Challenges: 76.6% MOTA on MOT16 and 2.8% higher MOTA with 1.3% higher IDF1 than FairMOTv2 on MOT17 establish leading benchmark performance.On MOT17, false negatives decrease by 30% relative to the compared multi-frame methods while identity switches remain comparable or better.
5. Conclusion
CorrTracker is a correlation tracking framework that models spatial and temporal relational structure to distinguish similar objects and strengthen temporal modeling. Experiments on four MOT challenges report state-of-the-art performance and efficient inference.
- CorrTracker densely matches targets with local context to learn discriminative embeddings from correlation volumes.
- The framework extends correlation modeling from spatial layout to adjacent frames, strengthening temporal modeling.
- Self-supervised learning constrains correlation volumes by explicitly predicting instance flow.
- CorrTracker achieves state-of-the-art performance and efficient inference across four MOT challenges.