Source-linked AI summary
RetinaTrack: Online Single Stage Joint Detection and Tracking
Zhichao Lu, Vivek Rathod, Ronny Votel, Jonathan Huang
TL;DR
Detection and tracking are often separated even though both matter for autonomous driving and can benefit from joint modeling. RetinaTrack modifies RetinaNet to produce instance-level features for unified detection and tracking, outperforming a state-of-the-art tracker while using less computation.
Problem
Detection and tracking are commonly separated, while benchmarks using fixed detections can exclude jointly trained models despite the importance of both tasks.
Method
RetinaTrack modifies RetinaNet to predict per-anchor instance-level features that provide unique vectors for detected objects and tracking association.
Results
RetinaTrack achieves state-of-the-art Waymo performance, outperforming current multi-object tracking while being significantly faster.
Takeaways & Limitations
Joint training can provide an effective, efficient baseline for future research in detection and tracking.
Takeaways & Limitations
The study focuses on RGB inputs, which are important but not the only sensing modality in modern autonomous vehicles.
Abstract
from arXiv · showhide
Traditionally multi-object tracking and object detection are performed using separate systems with most prior works focusing exclusively on one of these aspects over the other. Tracking systems clearly benefit from having access to accurate detections, however and there is ample evidence in literature that detectors can benefit from tracking which, for example, can help to smooth predictions over time. In this paper we focus on the tracking-by-detection paradigm for autonomous driving where both tasks are mission critical. We propose a conceptually simple and efficient joint model of detection and tracking, called RetinaTrack, which modifies the popular single stage RetinaNet approach such that it is amenable to instance-level embedding training. We show, via evaluations on the Waymo Open Dataset, that we outperform a recent state of the art tracking algorithm while requiring significantly less computation. We believe that our simple yet effective approach can serve as a strong baseline for future work in this area.
1. Introduction
RetinaTrack addresses the common separation of detection and tracking with a jointly trained, efficient model for autonomous driving. It modifies RetinaNet to support instance-level features and establishes strong Waymo baselines.
- Detection and tracking are commonly separated despite their close relationship, producing more complex and less efficient approaches.
- Autonomous driving makes accurate and fast detection and tracking mission-critical for predicting vehicle and pedestrian behavior.
- RGB cameras are emphasized because they avoid LIDAR range constraints, cost less, and detect smaller objects important for highway driving.
- RetinaTrack jointly trains detection and tracking by modifying a real-time RetinaNet detector to extract instance-level features.
- The model establishes strong 2D detection and tracking baselines on the Waymo Open Dataset and achieves state-of-the-art performance.
2. Related Work
Related work largely advances detection, tracking, or temporal aggregation separately, while joint detection-tracking remains uncommon. RetinaTrack differs by learning detection-associated embeddings for direct association in a lightweight unified pipeline.
- Deep learning has advanced detection and tracking, but joint training remains the exception and few works evaluate both tasks together.
- Video detection methods use temporal features, flow, attention, or smoothing primarily to improve detection rather than tracking.
- Trackers can smooth detection predictions, provide real-time updates between detector runs, or supply trajectories for behavior prediction.
- RetinaTrack directly learns detection association while using a unified lightweight detection and re-identification pipeline.
- Strong detections strongly affect tracking metrics, motivating benchmarks that use fixed detections but potentially exclude jointly trained models.
- Prior joint approaches include correlation-based motion prediction, heavier tubelet models, and detector-embedding combinations with different architectures.
3. The RetinaTrack Architecture
RetinaTrack modifies RetinaNet so detections receive unique anchor-level features suitable for embedding-based tracking, while preserving shared structure across feature-pyramid levels and tasks. It trains detection and tracking jointly using standard detection losses plus a triplet embedding loss.
- RetinaNet: RetinaNet associates K anchor boxes of different shapes with each feature-pyramid grid point and predicts classification logits and box-regression offsets for each anchor shape.The FPN produces feature maps at multiple spatial resolutions, which feed post-FPN subnetworks.
- RetinaNet: Vanilla RetinaNet cannot provide distinct per-instance features for detections from different anchor shapes at the same location because its convolutional parameters remain shared until final predictions.This ambiguity is especially problematic for tracking through occlusions.
- Anchor-level features: RetinaTrack splits the computational pathways by anchor shape earlier in the post-FPN subnetworks, producing intermediate features Fi,k uniquely associated with each detection.Separate parameters are used for each anchor shape but shared across FPN levels and across classification and localization at this stage.
- Anchor-level features: After the anchor-specific shared layers, RetinaTrack applies parallel task-specific classification and localization subnetworks whose parameters are shared across anchor shapes and FPN levels.This preserves compatible feature spaces after the anchor-level split while producing class logits and box-regression offsets.
- Training: Training minimizes an unweighted sum of RetinaNet’s sigmoid focal classification loss, Huber box-regression loss, and a BatchHard triplet embedding loss.The triplet loss encourages detections from the same track to have similar embeddings; experiments use margin m = 0.1 and sample 64 triplets.
4. Experiments
Experiments evaluate RetinaTrack on Waymo for detection, tracking, efficiency, ablations, joint training, and comparison with established trackers. The results show strong performance from anchor-level embeddings and jointly trained detection and tracking, including faster execution than Tracktor++.
- Experimental setup: The Waymo experiments use 200K annotated frames from five camera positions, focusing on 2D vehicle detection and tracking.Evaluation reports COCO AP, CLEAR MOT metrics, and inference time on Nvidia V100 GPUs.
- Experimental setup: Detection mAP and tracking MOTA are evaluated jointly, requiring separate score-threshold choices because the metrics use different operating-point assumptions.Detection mAP averages precision and recall across IoU thresholds, whereas MOTA depends directly on false positives, false negatives, and identity switches.
- Architectural ablations: Using K = 6 anchor shapes improves both detection and tracking, while forcing colliding anchors to share embeddings reduces MOTA and mAP.These results support RetinaTrack’s per-anchor embedding design over adapting vanilla RetinaNet for tracking.
- Architectural ablations: RetinaTrack achieves its best Waymo performance with two embedding projection layers, reaching 39.12 MOTA and 38.24 mAP.The depth ablation compares m3 = 0, 2, and 4 while fixing m1 = 3 and m2 = 1.
- Efficiency: RetinaTrack runs at 70ms per frame, faster than the Waymo sensor framerate of 10 Hz.The Waymo setting is slightly faster than COCO because the detector’s class-dependent top layers are lighter.
- Joint vs Independent training: Jointly trained visual embeddings outperform the IOU-only baseline, and RetinaTrack with detection and metric-learning losses jointly outperforms the tested alternatives.The IOU-only tracker remains a reasonably strong Waymo baseline, likely because of the strong detector.
- Comparison against state of the art: RetinaTrack outperforms Tracktor and Tracktor++ on detection and tracking, reduces false positives and ID switches, and is significantly faster than Tracktor++.On Waymo v1.1, the same trends hold: RetinaTrack outperforms both an IOU tracker and Tracktor++ with significantly faster running time.
5. Conclusion
RetinaTrack jointly trains detection and tracking while extending single-stage detectors with instance-level attributes. It outperforms current state-of-the-art multi-object tracking, runs significantly faster, and can track through long object disappearances.
- RetinaTrack jointly trains detection and tracking and extends single-stage detectors to handle instance-level attributes.
- Joint training outperforms the prevailing approach of training independent detection and tracking models.
- RetinaTrack outperforms the current state of the art in multi-object tracking while being significantly faster.
- RetinaTrack can track through long periods of object disappearance.
- The authors position RetinaTrack as a strong baseline for future research in detection and tracking.