Source-linked AI summary
MOTRv2: Bootstrapping End-to-End Multi-Object Tracking by Pretrained Object Detectors
Yuang Zhang, Tiancai Wang, Xiangyu Zhang
TL;DR
MOTRv2 addresses weak detection in end-to-end multi-object tracking by combining MOTR with a pretrained YOLOX detector. YOLOX proposals serve as anchors, simplifying detection and allowing MOTR to focus on association, while improving results across major benchmarks.
Problem
Existing end-to-end trackers such as MOTR have inferior detection performance because joint detection and association create a conflict in the shared transformer decoder.
Method
MOTRv2 uses YOLOX-generated object proposals as dynamically determined anchor queries for MOTR, which then propagates track queries across frames.
Results
MOTRv2 achieves large improvements over MOTR and state-of-the-art performance across DanceTrack, BDD100K, and MOT17, including 43.6% mMOTA on BDD100K, 2.4% above Unicorn.
Takeaways & Limitations
Using pretrained detector proposals reduces MOTR's detection complexity and conflict, enabling end-to-end tracking to achieve high-performance multi-object tracking.
Takeaways & Limitations
MOTRv2 remains data-hungry, performs less well on smaller datasets, can produce duplicated track queries, and runs at 6.9 FPS when YOLOX and MOTR are combined.
Abstract
from arXiv · showhide
In this paper, we propose MOTRv2, a simple yet effective pipeline to bootstrap end-to-end multi-object tracking with a pretrained object detector. Existing end-to-end methods, MOTR and TrackFormer are inferior to their tracking-by-detection counterparts mainly due to their poor detection performance. We aim to improve MOTR by elegantly incorporating an extra object detector. We first adopt the anchor formulation of queries and then use an extra object detector to generate proposals as anchors, providing detection prior to MOTR. The simple modification greatly eases the conflict between joint learning detection and association tasks in MOTR. MOTRv2 keeps the query propogation feature and scales well on large-scale benchmarks. MOTRv2 ranks the 1st place (73.4% HOTA on DanceTrack) in the 1st Multiple People Tracking in Group Dance Challenge. Moreover, MOTRv2 reaches state-of-the-art performance on the BDD100K dataset. We hope this simple and effective pipeline can provide some new insights to the end-to-end MOT community. Code is available at \url{https://github.com/megvii-research/MOTRv2}.
1. Introduction
MOTRv2 addresses MOTR’s weaker detection by using detector proposals as anchors, easing the conflict between detection and association. It improves performance across DanceTrack, BDD100K, and MOT17.
- Motivation: MOTR’s detection performance is inferior to tracking-by-detection methods, motivating integration with an extra object detector.The paper attributes this limitation to conflict between joint detection and association.
- Method: MOTRv2 uses an anchor-based query formulation so proposals from an extra detector provide local detection priors to MOTR.This replaces MOTR’s learnable detect-query positional embeddings with anchor positional encodings.
- Method: MOTRv2 replaces MOTR’s detect queries with detector-generated proposal queries while retaining track-query propagation across frames.The design lets MOTR learn association given detection results from the extra detector.
- Results: 43.6% mMOTA on BDD100K was 2.4% better than Unicorn, while MOTRv2 also achieved state-of-the-art performance on MOT17.The paper also reports large improvements over original MOTR on DanceTrack, BDD100K, and MOT17.
2. Related Works
Related work spans tracking-by-detection pipelines and query-propagation methods. MOTRv2 combines long-term query propagation with detector-provided location priors rather than explicit similarity-matrix matching.
- Tracking by Detection: Tracking-by-detection methods detect objects frame by frame and separately associate bounding boxes across adjacent frames.Their performance depends strongly on object-detection quality.
- Tracking by Detection: SORT uses Kalman-filter predictions and IoU matching, while Deep-SORT adds appearance features and cosine-distance matching.Other methods jointly train detection and appearance embeddings.
- Tracking by Query Propagation: Query-propagation methods update queries across frames to recall the same instances over time.They can interact with image features in parallel or serially.
- Tracking by Query Propagation: Parallel query methods process short videos jointly but are memory-consuming and limited to clips of a few dozen frames.Serial methods instead interact with image features frame by frame and iteratively update track queries.
- MOTRv2: MOTRv2 inherits long-term query propagation while using a powerful detector to provide object-location priors for complex-motion tracking.The paper states that it outperforms existing matching and query-based methods in tracking performance.
3. Method
MOTRv2 modifies MOTR with detector proposals as anchors and propagates track queries across frames. Proposal and track queries jointly enter the transformer decoder to produce frame-by-frame predictions.
- Method Overview: MOTRv2 is presented through proposal query generation and proposal propagation modules.The method builds on MOTR’s end-to-end tracking framework and query-based design.
- Method Overview: YOLOX proposals serve as object anchors, easing MOTR’s joint detection-association conflict and improving detection performance.The architecture combines a state-of-the-art detector with a modified anchor-based MOTR tracker.
- Proposal Query Generation: The detector generates proposals for each frame, and the modified MOTR concatenates proposal queries with track queries to learn association.Proposal-query counts are dynamically determined from selected YOLOX proposals rather than fixed learnable queries.
- Proposal Query Generation: For each frame, YOLOX proposals encode box geometry and confidence; a shared learnable query is combined with score embeddings to form proposal queries.The figure marks proposal generation in orange and propagation in blue; ten learnable anchors are also added to recover missed objects.
- Proposal Propagation: On later frames, previous track queries and predictions are concatenated with current proposal queries and YOLOX proposals before transformer decoding.Updated track queries are transferred to the next frame for tracked-object prediction.
- Proposal Propagation: On the first frame, YOLOX proposals are refined by predicted offsets to produce boxes and updated track queries.The predicted boxes are obtained by adding relative offsets to the YOLOX proposals.
- Query Interaction: Self-attention lets proposal queries focus on new or missing objects while track queries relocate tracked objects and exchange information for the same instance.The paper reports high similarity between corresponding proposal and track queries in the attention visualization.
4. Experiments
Experiments evaluate MOTRv2 on DanceTrack, MOT17, and multi-class BDD100K using HOTA, detection and association accuracy, MOTA, and IDF1. Results show consistent gains from YOLOX proposals and alignment strategies, with strongest performance across complex-motion and multi-class settings.
- Datasets and Metrics: MOTRv2 is evaluated on DanceTrack, MOT17, and BDD100K using HOTA, DetA, AssA, MOTA, and IDF1.BDD100K evaluation targets multi-class multi-object tracking.
- State-of-the-art Comparisons: 73.4% HOTA is achieved on DanceTrack with extra association, validation-set training, and a four-model ensemble.Without these additions, MOTRv2 reaches 69.9% HOTA and exceeds the previous best method by 14.8%.
- State-of-the-art Comparisons: 43.6% mMOTA on BDD100K is 2.4% better than Unicorn, while MOTRv2 also surpasses MOTR* by 8.1% mMOTA and 8.3% mIDF1.The gains indicate stronger averaged multi-class tracking performance and association.
- Ablation Study: Four-dimensional proposal propagation improves association more than center-point propagation, showing the value of previous-frame width and height information.Sine-cosine positional encoding provides little association benefit over learnable positional embeddings.
- Ablation Study: 8.4% higher MOTA and 3.9% higher IDF1 result from aligning anchors alone on MOT17 valhalf.Removing MOTR predictions unmatched to YOLOX boxes adds 2.0% MOTA beyond anchor alignment.
5. Discussion
MOTRv2 combines MOTR with YOLOX proposals to improve end-to-end tracking while easing detection–association optimization. The method remains data-hungry, underperforms on smaller datasets, and incurs an efficiency cost from the MOTR component.
- MOTRv2 combines the MOTR tracker with the YOLOX detector to generate high-quality proposals for new-object detection.YOLOX proposals help MOTR detect new objects more easily and allow it to focus on association.
- The proposal-based design reduces MOTR’s detection complexity and addresses the conflict between detection and association tasks.The extra detector provides detection priors, while MOTR concentrates on associating instances across frames.
- MOTRv2 remains data-hungry and does not perform well enough on smaller datasets.
- 6.9 FPS results when YOLOX and MOTR are combined, with MOTR running at 9.5 FPS and YOLOX at 25 FPS on a 2080Ti.The main efficiency bottleneck comes from the MOTR component.