Source-linked AI summary
Track to Detect and Segment: An Online Multi-Object Tracker
Jialian Wu, Jiale Cao, Liangchen Song, Yu Wang, Ming Yang, Junsong Yuan
TL;DR
Online trackers often detect objects without tracking input, limiting their ability to use motion information during detection. TraDeS jointly performs detection and tracking, using cost-volume offsets to propagate previous features for current detection and segmentation. Across four datasets and three tracking tasks, it achieves reported state-of-the-art performance with efficient inference.
Problem
Most joint detection-and-tracking methods perform detection standalone without exploiting tracking cues, despite tracking cues potentially helping detection in occlusion and motion blur.
Method
TraDeS uses cost-volume association to infer tracking offsets and motion-guided feature warping to propagate previous object features into current detection or segmentation.
Results
TraDeS achieves state-of-the-art performance with efficient inference across MOT, nuScenes, MOTS, and Youtube-VIS, spanning 2D tracking, 3D tracking, and instance segmentation tracking.
Takeaways & Limitations
Tracking cues can be integrated end-to-end to assist detection and, in return, benefit tracking across multiple tracking tasks.
Abstract
from arXiv · showhide
Most online multi-object trackers perform object detection stand-alone in a neural net without any input from tracking. In this paper, we present a new online joint detection and tracking model, TraDeS (TRAck to DEtect and Segment), exploiting tracking clues to assist detection end-to-end. TraDeS infers object tracking offset by a cost volume, which is used to propagate previous object features for improving current object detection and segmentation. Effectiveness and superiority of TraDeS are shown on 4 datasets, including MOT (2D tracking), nuScenes (3D tracking), MOTS and Youtube-VIS (instance segmentation tracking). Project page: https://jialianwu.com/projects/TraDeS.html.
1. Introduction
TraDeS addresses limitations of joint detection-and-tracking systems by using tracking cues to enhance detection and segmentation within a unified end-to-end model. Its cost-volume association and motion-guided feature propagation target detection degradation and difficult motion conditions.
- Tracking-by-detection separates detection and association, requiring two-stage processing that is inefficient and not end-to-end optimized.
- Joint detection and tracking performs both tasks in one forward pass, but commonly detects objects without exploiting tracking cues.
- TraDeS uses a cost volume to infer tracking offsets and a motion-guided feature warper to propagate previous object features into current detection or segmentation.
- Its cost-volume association learns embeddings that account for inter-class and intra-class differences while remaining compatible with detection training.
- TraDeS is evaluated across MOT, nuScenes, MOTS, and Youtube-VIS for 2D tracking, 3D tracking, and instance segmentation tracking, achieving state-of-the-art performance with efficient inference.
2. Related Work
Related work contrasts separate tracking-by-detection pipelines with unified joint detection and tracking systems. TraDeS extends this direction by using cost volumes to learn embeddings and infer tracking offsets rather than helping detection only after standalone prediction.
- Tracking-by-Detection: Tracking-by-detection first generates boxes with an object detector, then extracts appearance features and performs separate data association.
- Joint Detection and Tracking: Joint detection and tracking builds tracking-related prediction branches on an object detector or uses transformers and paired boxes to form tracklets.
- Tracking-guided Video Object Detection: Tracking-guided video detection uses tracking results to reweight scores after standalone detection, leaving detection and tracking separately optimized.
- Cost Volume: TraDeS extends cost volumes from depth and optical-flow correspondence to multi-object tracking for learning re-ID embeddings and inferring tracking offsets.
3. Preliminaries
TraDeS builds on CenterNet's point-based representation, adding a tracking-offset branch and allowing feature propagation from previous frames. The resulting enhanced features support current-frame detection and related task predictions.
- CenterNet: CenterNet maps an input image to a backbone feature map and predicts class-wise center heatmaps plus task-specific object properties.
- CenterNet: Objects are detected from peaks in the center heatmap and the corresponding task-specific predictions at those peak positions.
- TraDeS Overview: TraDeS can propagate features from multiple previous frames, with T > 1, for object-feature enhancement.
- Baseline Tracker: The baseline tracker adds a branch predicting a two-dimensional tracking-offset map for data association.
- Baseline Tracker: Each tracking offset represents the spatio-temporal displacement from a point at time t to its corresponding point at an earlier time t − τ.
4. TraDeS Tracker
TraDeS integrates tracking cues into detection through cost-volume association and motion-guided feature warping. It propagates previous center-aware features to enhance current detection and segmentation, then associates outputs with tracklets.
- Cost Volume based Association: CVA builds a dense cost volume from re-ID embeddings of two frames, with each entry representing cross-frame point similarity.Embeddings are downsampled before matrix multiplication constructs the 4-dimensional cost volume.
- Cost Volume based Association: Tracking offsets are estimated from cost-volume likelihoods and predefined horizontal and vertical offset templates, producing motion cues for propagation and association.The likelihoods are obtained after directional max pooling and softmax normalization; the templates encode offsets using feature stride s=8.
- Cost Volume based Association: CVA supervises the cost volume with focal loss so embeddings distinguish matching objects from other objects and background regions while remaining compatible with detection loss.The softmax-based objective encourages the correct previous-frame location while repelling other locations.
- Motion-guided Feature Warper: MFW uses CVA motion cues to deformably warp previous center-attentive features and aggregate them with current features for enhancement.The propagated feature is produced by a deformable convolution, with adaptive weights used for aggregation.
- Motion-guided Feature Warper: Feature enhancement targets occlusion and motion blur, where propagated features may make objects visually legible and help recover missed detections.The enhanced feature is fed to detection and mask heads, potentially reducing false negatives and improving tracklet completeness.
- Tracklet Generation: TraDeS produces 2D boxes, 3D boxes, and instance masks from separate heads, then connects detections and masks to prior tracklets through data association.Its association procedure can match detections with unmatched or historical tracklet embeddings and start new tracklets when no match exceeds the threshold.
5. Experiments
TraDeS is evaluated through ablations and benchmarks spanning 2D tracking, 3D tracking, and instance segmentation tracking. Its CVA and MFW modules improve association, feature propagation, detection, and tracking across challenging motion settings and datasets.
- Experimental Setup: TraDeS is evaluated on MOT, nuScenes, MOTS, and YouTube-VIS across 2D tracking, 3D tracking, and instance segmentation tracking.MOT uses 25–30 FPS labels, nuScenes uses 2 FPS labels, MOTS is mask-based, and YouTube-VIS evaluates mask tracklets with AP.
- Ablation Studies: CVA reduces IDS by 60% and improves IDF1 by 7.2 over the baseline in ablation experiments.The comparison evaluates the cost-volume association module, tracking offset, re-ID embedding, and two-round data association.
- Ablation Studies: MFW reduces FN by 1.5% and also reduces IDS by 43% while improving IDF1 by 6.2 over the baseline.The module uses motion cues to predict the DCN offset; false positives increase slightly in the reported ablation.
- Ablation Studies: TraDeS reduces IDS from 606 to 285 and FN from 29.5% to 27.8%, achieving 68.2 MOTA and 71.7 IDF1.The CVA tracking offsets guide MFW feature propagation, and the combined improvements produce the reported tracking performance.
- Ablation Studies: The CVA predicts tracking offsets under low frame rate, large motion, and unseen large motion, including on nuScenes when trained only on MOT.The reported visualizations contrast TraDeS with baseline and CenterTrack, which fail on the unseen large-motion setting.
- Benchmark Evaluations: On benchmarks, TraDeS exceeds the second-best tracker by 2.5 MOTA on MOT16 and 1.8 MOTA on MOT17 at 15 FPS, and outperforms comparison trackers on nuScenes and MOTS.On nuScenes it leads monocular trackers by a large margin on all metrics; on MOTS it exceeds TrackR-CNN in accuracy and speed.
6. Conclusion
TraDeS integrates tracking cues into detection through CVA and MFW, improving detection, tracking, and segmentation tracking across multiple tasks.
- TraDeS uses CVA to learn re-ID embeddings and model object motions with a 4D cost volume.
- MFW uses CVA-derived motion cues to propagate previous object features and enhance current detection or segmentation.
- Experiments and ablations across 2D tracking, 3D tracking, and instance segmentation tracking validate TraDeS's effectiveness and superiority.