Source-linked AI summary
QDTrack: Quasi-Dense Similarity Learning for Appearance-Only Multiple Object Tracking
Tobias Fischer, Thomas E. Huang, Jiangmiao Pang, Linlu Qiu, Haofeng Chen, Trevor Darrell, Fisher Yu
TL;DR
Existing MOT methods learn appearance similarity from sparse matching regions, leaving much image information unused. QDTrack addresses this with quasi-dense multi-positive contrastive learning and simple feature-space association, achieving competitive performance across benchmarks, including a new BDD100K state of the art, while adding negligible detector overhead.
Problem
Existing multiple object tracking methods use sparse ground-truth matching for similarity learning and ignore many informative image regions.
Method
QDTrack densely matches hundreds of informative regions with multiple-positive contrastive learning and uses bi-softmax plus nearest-neighbor search for association.
Results
QDTrack rivals state-of-the-art methods across the reported benchmarks and sets a new state-of-the-art on BDD100K with negligible detector inference overhead.
Takeaways & Limitations
The learned embedding space supports simple tracking and effective instance similarity learning from static images without video input or tracking annotations.
Takeaways & Limitations
QDTrack struggles with rapid appearance changes, classification errors, and inaccurate localization that includes background or other objects in bounding boxes.
Abstract
from arXiv · showhide
Similarity learning has been recognized as a crucial step for object tracking. However, existing multiple object tracking methods only use sparse ground truth matching as the training objective, while ignoring the majority of the informative regions in images. In this paper, we present Quasi-Dense Similarity Learning, which densely samples hundreds of object regions on a pair of images for contrastive learning. We combine this similarity learning with multiple existing object detectors to build Quasi-Dense Tracking (QDTrack), which does not require displacement regression or motion priors. We find that the resulting distinctive feature space admits a simple nearest neighbor search at inference time for object association. In addition, we show that our similarity learning scheme is not limited to video data, but can learn effective instance similarity even from static input, enabling a competitive tracking performance without training on videos or using tracking supervision. We conduct extensive experiments on a wide variety of popular MOT benchmarks. We find that, despite its simplicity, QDTrack rivals the performance of state-of-the-art tracking methods on all benchmarks and sets a new state-of-the-art on the large-scale BDD100K MOT benchmark, while introducing negligible computational overhead to the detector.
1 INTRODUCTION
QDTrack addresses the limits of sparse appearance supervision and unreliable location priors by densely learning instance similarity, then combining it with detectors and a simple association pipeline. The resulting system supports effective similarity learning from static images and achieves strong benchmark performance with negligible detector overhead.
- Motivation: Existing MOT methods often rely on spatial proximity or sparse ground-truth regions, leaving many informative image regions unused for appearance learning.Location priors can fail under non-linear motion, varying frame rates, and complex camera motion.
- Method: Quasi-dense similarity learning matches hundreds of informative regions across image pairs using multiple-positive contrastive learning.The method supplies more positive examples and hard negatives than sparse supervision and trains each sample against possible object regions.
- System: QDTrack combines object detectors, quasi-dense similarity learning, and an inference pipeline that handles false positives, missed detections, newly appeared objects, and terminated tracks.Its bi-directional similarity metric and backdrop matching are designed to improve association under these cases.
- Results: Quasi-dense instance similarity learning can learn effective representations from static images alone, without video input or association annotations.The paper reports this capability as an extension of its initial findings.
- Results: QDTrack rivals state-of-the-art tracking methods across the reported benchmarks and sets a new state-of-the-art on BDD100K.The embedding extractor adds negligible inference overhead to the detector, and the system supports joint end-to-end detection and similarity training.
2 RELATED WORK
Related MOT work addresses association through network optimization, motion and location cues, or appearance similarity. QDTrack focuses on visual appearance similarity while using dense contrastive pairs and nearest-neighbor association to simplify tracking without compromising reported accuracy.
- Tracking-by-detection: Tracking-by-detection methods detect objects frame by frame and associate detections using mechanisms such as network flow, optimization, motion, or appearance similarity.The surveyed approaches include network flow, quadratic pseudo-Boolean optimization, conditional random fields, motion cues, and visual appearance.
- Location and motion: Spatial and motion cues are effective for association but become brittle with varying frame rates and complex camera motion.Prior methods use Kalman filtering, optical flow, or displacement regression to estimate object similarity.
- Appearance similarity: Appearance-based methods use independent models or detector embedding heads, but prior approaches may rely heavily on motion and displacement predictions.The related-work discussion identifies limited image-information use and sub-optimal instance similarity learning as contributing issues.
- QDTrack: QDTrack learns instance similarity from densely connected contrastive pairs and associates objects with nearest-neighbor search in feature space.The paper presents this as a simpler tracking framework without compromising accuracy.
3 METHOD
QDTrack learns instance embeddings from quasi-dense object-region matches and uses those embeddings for appearance-based association. Its training combines detector and embedding objectives, while inference uses bi-softmax matching, nearest-neighbor association, and track management.
- Method overview: QDTrack defines quasi-dense matching over potential object regions, between sparse ground-truth matching and dense matching at every pixel location.Its main components are object detection, instance similarity learning, and object association.
- Quasi-dense similarity learning: The training pipeline samples object regions from key and reference images, extracts RoI-aligned embeddings, and labels region pairs using IoU thresholds.A lightweight embedding head operates alongside the detector’s bounding-box head.
- Quasi-dense similarity learning: Each key-frame sample is matched to all reference-frame samples, allowing multiple positive targets and broader negative supervision than sparse crops.The multi-positive formulation extends contrastive learning to account for several positives while addressing repeated negative counting.
- Inference: During inference, detections and embeddings are compared with bi-softmax, then associated through nearest-neighbor search and track-history updates.The pipeline is summarized as detection, similarity scoring, association, and track management.
- Object association: Bi-softmax favors matches that are mutually nearest in feature space and gives low scores to objects lacking a reliable counterpart.This supports rejection of unmatched detections and candidates during association.
- Track management: Track management removes inactive tracks after a fixed period, applies detection-confidence thresholds, and retains unmatched detections as backdrops.Backdrops remain matching candidates to help filter recurring false positives.
4 EXPERIMENTS
QDTrack is evaluated across diverse multi-object tracking benchmarks, detector settings, training regimes, and ablations. Results show competitive or leading performance, effective learning from static images, and strong gains from quasi-dense matching, while rapid appearance changes and classification errors remain limitations.
- Experimental scope: Experiments cover MOT17, MOT20, DanceTrack, BDD100K, Waymo, and TAO, including comparisons, detector variations, static-image training, and ablations.The evaluation spans multiple benchmark types and tests the method's flexibility across detectors and training conditions.
- MOT benchmarks: 63.5 HOTA on MOT17 and 60.0 HOTA on MOT20 demonstrate competitive performance using only appearance cues for association.On MOT17, QDTrack maintains 58.6 MOTA even at 1 FPS, while ByteTrack fails and FairMOT falls to 44.3 MOTA.
- Benchmark comparisons: 42.4 mMOTA, 55.6 mIDF1, and 73.9 IDF1 on the BDD100K test set outperform all other methods, including ByteTrack by 2.3 mMOTA and 2.6 IDF1.QDTrack also reports 44.0 MOTA and 56.8 IDF1 on Waymo validation, and 16.1 AP50 and 12.4 AP50 on TAO validation and test sets.
- Static-image training: Static-image training nearly matches full tracking supervision, with a 0.1-point mMOTA gap when using the same amount of training data.Non-consistent augmentations and MixUp improve static-image instance representations, while augmentation also raises supervised performance by 1.2 mMOTA and 2.1 mIDF1.
- Ablation studies: Quasi-dense matching improves overall IDF1 from 63.0% to 67.8%, while combined training and inference strategies raise it to 71.5% and reduce ID switches by 30%.More negative samples, multiple positives, bi-softmax, and backdrop handling each contribute to the gains.
- Limitations: QDTrack struggles with rapid appearance changes such as partial occlusion, and classification errors can produce truncated object tracks.These challenges arise under the appearance-only paradigm and define practical limits of the approach.
5 CONCLUSION
QDTrack learns instance similarity from quasi-dense object regions, producing embeddings that support simple nearest-neighbor association. It can be coupled with existing detectors and trained without video input or tracking annotations.
- QDTrack uses all object regions for similarity learning instead of sparse ground-truth regions.
- Quasi-dense matching yields an embedding space suited to distinguishing instances through nearest-neighbor search.
- The method couples with existing object detectors and feature extractors for end-to-end training.
- QDTrack learns effective instance similarity without video input or tracking annotations.
APPENDIX A ADDITIONAL DETAILS
The appendix specifies benchmark training settings and augmentation schedules, including detector, image scale, optimization, and late-training augmentation choices.
- Training setting: MOT17 uses 1440 × 800 images, while MOT20 uses 1600 × 896 images with batch size 32.
- Training setting: Training uses learning rate 0.0005, cosine annealing to 0.05 times the original rate, and one epoch of exponential warm-up.
- Augmentations: MixUp and Mosaic are disabled during the final ten training epochs.
ADDITIONAL DETAILS FOR MOT CHALLENGE
For challenging MOT scenes, QDTrack adds association safeguards for occlusions and ambiguous matches. Ablations show that distance thresholding, tracklet merging, and interpolation improve association metrics.
- Association strategies: Near-online merging addresses unreliable appearance cues immediately after objects reappear from occlusion.
- Association strategies: Distance thresholding ignores candidates farther than d = 50, while linear interpolation recovers boxes for fully occluded objects.
- Ablation results: On MOT17 validation, distance thresholding raises IDF1 from 73.6 to 74.5, a +0.9 improvement.
- Ablation results: Tracklet merging further raises IDF1 from 74.5 to 76.0, a +1.5 improvement.
- Ablation results: Linear interpolation further improves all reported metrics.
APPENDIX C ADDITIONAL ABLATION STUDIES
The appendix examines embedding momentum and the sensitivity of two similarity-loss parameters. Momentum adds embedding history but has limited effect, while γ1 requires careful setting.
- Momentum of the embeddings: Track embeddings are updated as m ∗ E1 + (1 − m) ∗ E0, combining the current and existing embeddings.
- Momentum of the embeddings: Embedding momentum does not improve results substantially but incorporates historical embeddings.
- Sensitivity of γ1 and γ2: γ2 does not change final results, whereas γ1 affects performance and causes a drop when higher than 0.5.
APPENDIX D ORACLE ANALYSIS
Oracle analyses indicate that QDTrack’s identity association is effective when detections are accurate, while overall performance remains more constrained by detection quality than by tracking.
- Detection oracle: Detection-oracle MOTA exceeds 94% across all evaluated classes and approaches 100% for some classes.The oracle uses ground-truth boxes, reducing false negatives and false positives.
- Detection oracle: 88.8% average IDF1 across eight classes is 38 points higher than the reported result.The oracle-to-method gaps are smaller for cars and pedestrians than for other classes.
- Oracle comparison: The oracle analysis attributes the large identity-consistency gap primarily to detection performance rather than tracking performance.The tracking oracle boosts mIDF1 by only 4.3 points, whereas accurate detections substantially improve identity metrics.
- Detection oracle: Higher recalls in the detection-oracle setting can introduce more ID switches because more detected instances are associated.This explains why oracle ID switches can exceed those of the reported method despite stronger detection quality.
APPENDIX E FAILURE CASE ANALYSIS
The failure analysis identifies detection classification, localization, occlusion, and extreme truncation as the main conditions that disrupt QDTrack’s association process.
- Object classification: False negatives and false positives disrupt one-to-one matching, especially for small or occluded objects and visually similar distractors.The analysis identifies inaccurate classification confidence as the main classification-related distraction.
- Object classification: Category switches, such as “rider” becoming “pedestrian” under bicycle occlusion, prevent association because matched objects must share a category.The paper characterizes inaccurate category as a less frequent classification distraction.
- Object truncation/occlusion: Truncation can produce partial false-positive boxes and full-object boxes with similar embeddings, confusing association.The two boxes may differ in detection score while still influencing association through similar feature representations.
- Object truncation/occlusion: Boxes covering entirely different regions before and after occlusion receive low similarity scores and may fail to associate.Association remains possible when boxes cover the same discriminative regions, even if they are not identical regions.
- Object truncation/occlusion: Extreme truncation leaves too little appearance information to distinguish instances effectively as they enter or leave the camera view.This is identified as a separate corner case from ordinary truncation and occlusion.
APPENDIX F VISUALIZATIONS
Inference visualizations show how detected objects are matched against active tracks, vanished tracks, and backdrop representations using bi-directional softmax.
- Inference visualization: Each detected object is matched to prior objects from consecutive-frame tracks, vanished tracks, or backdrops via bi-directional softmax.The visualizations use colors to indicate the matched prior object for each detection.
- Inference visualization: Most current-frame false positives are matched to backdrops in the visualized examples.This matching pattern is presented as evidence of how the inference procedure handles false positives.
APPENDIX G QUALITATIVE RESULTS
The appendices provide qualitative examples on BDD100K and MOT17 alongside visualized failure cases involving detection confidence, category consistency, localization, occlusion, and severe truncation.
- Qualitative results: Figures 12 and 13 present qualitative results on BDD100K and MOT17, respectively.The examples are sampled at a fixed interval for illustration.
- Failure cases: Inaccurate detection confidence creates false negatives and false positives that disrupt association.The figure identifies false negatives with yellow rectangles and false positives with red rectangles.
- Failure cases: Category inconsistency can prevent association when an occluded bicycle causes an object label to change from “rider” to “pedestrian.”The failure follows from the method’s category-consistency requirement.
- Failure cases: Truncated detections may cover different portions of an object while retaining similar feature embeddings, influencing association.The figure contrasts a partial false-positive box with a full-object box.
- Failure cases: Occlusion across different object regions can lower appearance similarity when the visible regions do not overlap meaningfully.The example compares detections covering different regions of the same object.
- Failure cases: Highly truncated objects are difficult to distinguish because they provide limited appearance information.This occurs when objects only partly enter or leave the camera view.