Source-linked AI summary
Quasi-Dense Similarity Learning for Multiple Object Tracking
Jiangmiao Pang, Linlu Qiu, Xia Li, Haofeng Chen, Qi Li, Trevor Darrell, Fisher Yu
TL;DR
Existing tracking methods often learn similarity from sparse ground-truth matches and rely on location or motion cues. This paper proposes quasi-dense contrastive similarity learning with hundreds of region proposals, enabling simple nearest-neighbor association. QDTrack achieves strong results across four tracking benchmarks, including 68.7 MOTA at 20.3 FPS on MOT17 without external training data.
Problem
Existing methods use sparse ground-truth matching for similarity learning and do not exploit most informative image regions.
Method
Quasi-dense similarity learning densely matches hundreds of region proposals across image pairs and trains embeddings with multiple-positive contrastive learning.
Results
QDTrack outperforms existing methods across MOT, BDD100K, Waymo, and TAO benchmarks, achieving 68.7 MOTA at 20.3 FPS on MOT17 without external training data.
Takeaways & Limitations
The learned feature space supports simple nearest-neighbor association, while the method couples with existing detectors and enables end-to-end tracking.
Abstract
from arXiv · showhide
Similarity learning has been recognized as a crucial step for object tracking. However, existing multiple object tracking methods only use sparse ground truth matching as the training objective, while ignoring the majority of the informative regions on the images. In this paper, we present Quasi-Dense Similarity Learning, which densely samples hundreds of region proposals on a pair of images for contrastive learning. We can directly combine this similarity learning with existing detection methods to build Quasi-Dense Tracking (QDTrack) without turning to displacement regression or motion priors. We also find that the resulting distinctive feature space admits a simple nearest neighbor search at the inference time. Despite its simplicity, QDTrack outperforms all existing methods on MOT, BDD100K, Waymo, and TAO tracking benchmarks. It achieves 68.7 MOTA at 20.3 FPS on MOT17 without using external training data. Compared to methods with similar detectors, it boosts almost 10 points of MOTA and significantly decreases the number of ID switches on BDD100K and Waymo datasets. Our code and trained models are available at http://vis.xyz/pub/qdtrack.
1. Introduction
The paper addresses weaknesses of location- and motion-based association by learning instance similarity from many informative regions, then using that representation in a simple tracking framework.
- Location heuristics work poorly under occlusion and crowding, motivating motion estimation or displacement regression in existing methods.
- Prior methods often treat appearance similarity as secondary and learn from post hoc detection stages or sparse ground-truth boxes.
- Quasi-dense similarity learning matches hundreds of regions across image pairs, supplying additional positive examples and hard negatives.
- Multiple-positive contrastive learning trains each sample to distinguish all proposals in the other image simultaneously.
- QDTrack combines the approach with existing detectors, trains end-to-end, and uses nearest-neighbor association without displacement regression or motion priors.
- 68.7 MOTA at 20.3 FPS is reported on MOT17 without external training data, while BDD100K and Waymo results improve substantially over comparable methods.
2. Related work
QDTrack differs from tracking-by-detection methods by learning similarity from dense-connected contrastive pairs and associating objects directly in feature space.
- Tracking-by-detection methods detect objects frame by frame and associate them through estimated instance similarity, often using spatial proximity or motion priors.
- QDTrack learns instance similarity from dense-connected contrastive pairs and uses simple nearest-neighbor search for association.
- The method applies multiple-positive contrastive learning to densely matched quasi-dense samples, unlike image-level methods limited to one positive target.
3. Methodology
QDTrack learns an embedding space through quasi-dense matching of region proposals and uses it for object association. The method combines detector-generated proposals, multi-positive contrastive learning, auxiliary supervision, and bi-softmax nearest-neighbor inference.
- Quasi-dense similarity learning: Quasi-dense matching learns embeddings by matching potential object candidates at informative regions rather than using only ground-truth labels.The approach targets embeddings that associate identical objects and distinguish different ones for online tracking.
- Quasi-dense similarity learning: Faster R-CNN with FPN generates region proposals, while RoI Align extracts proposal features from pyramid levels for paired-image training.Training samples a key image and a temporally nearby reference image, with neighbor distance k ∈[−3, 3].
- Object association: At inference, maintained candidates are scored with bi-softmax, and objects are associated by nearest-neighbor search in the embedding space.Unmatched objects and backdrops are included among candidates; absent targets tend to receive low bi-directional consistency scores.
- Quasi-dense similarity learning: Each key-frame sample is matched against all reference-frame samples, allowing multiple positive targets and denser contrastive supervision.The formulation addresses unequal treatment of positives and negatives by accumulating the positive term in the multi-positive scenario.
- Quasi-dense similarity learning: An auxiliary L2 loss constrains the similarity logit according to whether a sample pair is positive or negative.The match indicator c is 1 for positive pairs and 0 otherwise.
4. Experiments
Experiments evaluate QDTrack across MOT, BDD100K, Waymo, TAO, and segmentation tracking, with ablations of training, inference, and cue choices.
- Datasets: QDTrack is evaluated on MOT, BDD100K, Waymo, and TAO tracking benchmarks, plus BDD100K segmentation tracking.
- Main results: 68.7% MOTA and 66.3% IDF1 are achieved on MOT17, exceeding CenterTrack by 0.9 MOTA points and 1.6 IDF1 points.
- Main results: 49.40% MOTA/L1 and 43.88% MOTA/L2 are obtained on the Waymo test set, with the ResNet-101 DCN model on par with HorizonMOT.
- Main results: 16.1 AP50 on TAO validation and 12.4 AP50 on test exceed the baseline by 2.9 and 2.2 points, respectively.
- Ablation studies: Training and inference strategies improve BDD100K validation IDF1 by 8.5 points, from 63.0% to 71.5%, while reducing ID switches by 30%.
5. Conclusion
The conclusion presents QDTrack as a detector-compatible tracking method that learns instance similarity from quasi-dense proposal pairs and supports simple association.
- QDTrack learns instance similarity from hundreds of region proposals on image pairs using multiple-positive contrastive learning.
- The resulting feature space distinguishes instances with a simple nearest neighbor search without additional tracking machinery.
- The method couples with most existing detectors and supports end-to-end training for multiple object and segmentation tracking.
B. Hyper-parameters
The tracker configuration fixes association thresholds, memory durations, embedding momentum, duplicate-removal thresholds, and bi-softmax matching, with dataset-specific settings for TAO.
- QDTrack uses QuasiDenseEmbedTracker with thresholds of 0.8 for track initialization, 0.5 for continuation, and 0.5 for data association.
- Tracklets are retained for 10 frames, backdrops for 1 frame, and embeddings are updated with momentum 0.8.
- Duplicate removal uses IoU thresholds of 0.3 for backdrops and 0.7 for classes, with bi-softmax as the matching metric.
- The same tracking parameters are used across datasets except TAO, whose evaluation uses 3D mAP rather than CLEAR MOT metrics.
- TAO lowers initialization and object-score thresholds to 0.0001 and omits backdrops because of the resulting numerous tracks.
C. Supplementary experiments
With public MOT17 detections, QDTrack outperforms existing results and improves over CenterTrack on both MOTA and IDF1.
- On MOT17 with public detectors, QDTrack outperforms CenterTrack by 3.1 MOTA points and 5.5 IDF1 points.
D. Additional ablation studies
The ablations examine embedding momentum and the sensitivity of γ1 and γ2. Momentum contributes limited improvement, while γ1 materially affects performance above a threshold.
- Embedding momentum considers the history of track embeddings but does not improve results substantially.The updated embedding is m ∗E1 + (1 −m) ∗E0.
- γ2 does not change the final results, whereas γ1 affects performance.
- Performance drops when γ1 exceeds 0.5 but is insensitive to lower values.
E. Oracle analysis
Oracle experiments separate detection and association quality. They show strong identity association with accurate detections and indicate that QDTrack is more constrained by detection than tracking performance.
- Detection-oracle MOTA exceeds 94% across all eight classes and approaches 100% for some classes.Ground-truth boxes make false negatives and false positives close to zero.
- 88.8% average IDF1 under the detection oracle is 38 points higher than the reported result.The gaps are smaller for car and pedestrian than for the other classes.
- The tracking oracle boosts mIDF1 by only 4.3 points, indicating detection is the stronger performance constraint.
F. Failure case analysis
QDTrack’s failures primarily arise from detection and appearance ambiguities. Classification errors, truncation, occlusion, and limited visible regions can disrupt association.
- Object classification: False negatives on small or occluded objects and false positives with similar appearances disrupt one-to-one association.Examples include mirror reflections and advertising boards mistaken for annotated persons.
- Object classification: Category changes can cause association failure because matched objects are required to share the same category.The paper illustrates a rider becoming a pedestrian when a bicycle is occluded.
- Object truncation/occlusion: Truncation can produce competing boxes with similar embeddings, influencing association when one box covers only part of the object.
- Object truncation/occlusion: Occlusion can yield low similarity when the visible regions before and after occlusion differ substantially.Extreme truncation is especially difficult because only limited appearance information is available.
H. Qualitative results
The paper presents qualitative tracking results on BDD100K and MOT17. These examples are sampled at intervals for illustration.
- Qualitative results are shown for the BDD100K dataset.
- Qualitative results are also shown for the MOT17 dataset.
- The displayed results are sampled from a certain interval for illustrative purposes.