Source-linked AI summary
MOTR: End-to-End Multiple-Object Tracking with Transformer
Fangao Zeng, Bin Dong, Yuang Zhang, Tiancai Wang, Xiangyu Zhang, Yichen Wei
TL;DR
MOTR addresses the bottleneck created by post-processing similarity matching in multiple-object tracking by providing a fully end-to-end framework for temporal modeling. It uses track queries and temporal-learning components, achieving promising results on MOT17 and DanceTrack, including a 6.5% HOTA improvement over ByteTrack on DanceTrack.
Problem
Existing MOT methods separate appearance and motion modeling and rely on post-processing similarity matching, limiting end-to-end temporal information flow across frames.
Method
MOTR extends DETR with iteratively updated track queries, tracklet-aware label assignment, an entrance and exit mechanism, TAN, and CAL for end-to-end temporal modeling.
Results
MOTR achieves promising performance on MOT17 and DanceTrack, outperforming ByteTrack by 6.5% on the HOTA metric on DanceTrack.
Takeaways & Limitations
MOTR removes explicit track associations during inference and eliminates post-processing heuristics such as track NMS and IoU matching.
Takeaways & Limitations
MOTR's newborn-object detection performance is unsatisfactory, and frame-by-frame query passing limits training efficiency.
Abstract
from arXiv · showhide
Temporal modeling of objects is a key challenge in multiple object tracking (MOT). Existing methods track by associating detections through motion-based and appearance-based similarity heuristics. The post-processing nature of association prevents end-to-end exploitation of temporal variations in video sequence. In this paper, we propose MOTR, which extends DETR and introduces track query to model the tracked instances in the entire video. Track query is transferred and updated frame-by-frame to perform iterative prediction over time. We propose tracklet-aware label assignment to train track queries and newborn object queries. We further propose temporal aggregation network and collective average loss to enhance temporal relation modeling. Experimental results on DanceTrack show that MOTR significantly outperforms state-of-the-art method, ByteTrack by 6.5% on HOTA metric. On MOT17, MOTR outperforms our concurrent works, TrackFormer and TransTrack, on association performance. MOTR can serve as a stronger baseline for future research on temporal modeling and Transformer-based trackers. Code is available at https://github.com/megvii-research/MOTR.
1 Introduction
MOTR frames multiple-object tracking as end-to-end set-of-sequence prediction, replacing post-processing association with iteratively updated track queries. It adds tracklet-aware assignment and temporal modules to model object trajectories across video frames.
- Motivation: Existing MOT methods separate appearance and motion modeling and rely on similarity-based matching as post-processing.Appearance is commonly measured with pair-wise Re-ID similarity, while motion uses IoU or Kalman Filtering heuristics.
- Motivation: DETR provides end-to-end one-to-one assignment for object detection, but extending this approach to MOT sequence prediction remains an open question.DETR updates object queries through interactions with image features and matches them to ground truths.
- MOTR Framework: MOTR extends DETR by treating MOT as set-of-sequence prediction, with each track query representing and iteratively predicting one object trajectory.Track queries are transferred and updated across frames as hidden states for object tracks.
- MOTR Framework: Tracklet-aware label assignment supervises each track query with same-identity bounding-box sequences, while entrance and exit mechanisms handle newborn and terminated objects.The track query set has variable length: newborn queries are added and terminated queries removed.
- Temporal Modeling: Collective average loss and temporal aggregation network enhance temporal modeling during training and inference.CAL uses whole video clips for loss computation, while TAN aggregates historical information from previous track-query states.
- Results: 6.5% on HOTA: MOTR outperforms ByteTrack on DanceTrack, while requiring no explicit track association during inference.The reported DanceTrack comparison also gives MOTR an 8.1% advantage on AssA.
2 Related Work
Transformer-based MOT methods build on attention architectures and iterative sequence prediction, adapting encoder-decoder ideas from sequence-to-sequence tasks to tracking.
- Transformer-based Architectures: Transformers aggregate information through self-attention and cross-attention and have been applied across speech processing and computer vision.DETR combines CNNs, Transformers, and bipartite matching for end-to-end object detection.
- Iterative Sequence Prediction: Iterative sequence prediction uses an encoder representation and a task-specific hidden state that repeatedly interacts with it to generate sequence elements.This encoder-decoder pattern is established in machine translation and text recognition.
3 Method
MOTR extends DETR for end-to-end multiple-object tracking by carrying track queries across frames and dynamically managing object entrances and exits. TALA, QIM, TAN, and video-level supervision support consistent identity assignment and temporal modeling.
- Track queries: DETR’s detect queries are extended into track queries that predict tracked objects across successive frames.Detect queries remain available for newborn-object detection, while track queries are transferred and updated frame by frame.
- Tracklet-aware label assignment: Tracklet-aware label assignment matches detect queries only to newborn objects and preserves track-query assignments across frames.Track queries assigned in the previous frame are merged with newborn assignments for subsequent tracking.
- Object entrance and exit: The variable-length track-query set adds newborn-object queries and removes queries associated with terminated objects.During inference, confidence thresholds retain newborns and remove tracks whose scores stay below the exit threshold for consecutive frames.
- Overall architecture: MOTR concatenates detect and track queries with frame features in the Deformable DETR decoder, then uses QIM to generate queries for the next frame.The architecture processes the first frame with detect queries alone and later frames with both query types.
- Training and inference: MOTR collects predictions across a video clip and applies collective average loss for training, while inference processes the video stream online.The loss supervises the overall prediction bank rather than treating frames as isolated training examples.
- Temporal aggregation: TAN aggregates filtered current hidden states with previous track queries to enhance long-range temporal relation modeling.The temporal aggregation uses a modified Transformer decoder layer and supplies contextual priors for tracked objects.
4 Experiments
Experiments evaluate MOTR across MOT17, DanceTrack, and BDD100k, including comparisons, ablations, and training-sampling studies. MOTR shows stronger association on DanceTrack and MOT17 Transformer comparisons, while performance depends on temporal components and sampling intervals.
- Datasets and Metrics: MOTR is evaluated on DanceTrack, MOT17, and BDD100k using HOTA, AssA, DetA, MOTA, IDS, and IDF1.The evaluation follows standard multiple-object tracking protocols.
- State-of-the-art Comparison on MOT17: 4.5% higher IDF1 and 3.1% higher HOTA than concurrent Transformer trackers are reported on MOT17.MOTR also achieves 71.9% MOTA versus TrackFormer’s 65.0%, while TransTrack performs better on MOTA.
- State-of-the-art Comparison on DanceTrack: 6.5% higher HOTA and 8.1% higher AssA than ByteTrack are reported on DanceTrack.MOTR is inferior to some state-of-the-art methods on DetA, while the paper attributes large HOTA improvements mainly to TAN and CAL.
- Generalization on Multi-Class Scene: MOTR performs well on BDD100k multi-class scenes and achieves promising performance with fewer ID switches.The method uses dynamic-length track-query sets and modifies the classification branch for multi-class prediction.
- Ablation Study: Adding track queries improves IDF1 from 1.2 to 49.8, while TAN and CAL provide further gains in MOTA and IDF1.TAN adds 7.8% MOTA and 13.6% IDF1; CAL adds 8.3% MOTA and 7.1% IDF1.
- Ablation Study: Increasing the training sampling interval from 2 to 10 reduces IDS from 209 to 155, whereas intervals above 10 weaken long-range dynamics.Small intervals can lead to local optima, while larger appropriate intervals better simulate real scenes.
5 Limitations
MOTR achieves end-to-end online tracking but has limitations in newborn-object detection and training efficiency. The paper identifies both as open problems for Transformer-based MOT.
- Detection: Newborn-object detection remains unsatisfactory because detect queries are suppressed when detecting tracked objects.The shared Transformer decoder limits detection performance on newborn objects and contributes to weak MOTA performance.
- Training Efficiency: Frame-by-frame query passing limits training efficiency, while parallel decoding does not handle complex MOT scenarios effectively.The paper presents solving these two issues as an important research topic for Transformer-based MOT frameworks.