Source-linked AI summary
MotionTrack: Learning Robust Short-term and Long-term Motions for Multi-Object Tracking
Zheng Qin, Sanping Zhou, Le Wang, Jinghai Duan, Gang Hua, Wei Tang
TL;DR
Multi-object tracking struggles to preserve identities in dense crowds and under extreme occlusion. MotionTrack addresses short- and long-range association with interaction-aware and history-based motion modules, achieving state-of-the-art performance on MOT benchmarks.
Problem
Dense crowds and extreme occlusions make motion prediction and appearance-based identity association unreliable in multi-object tracking.
Method
MotionTrack jointly learns interaction-aware short-term motions and history trajectory-based long-term motions within a tracking-by-detection framework.
Results
MotionTrack achieves state-of-the-art performance on MOT benchmark datasets, ranking first on several MOT17 metrics and second on MOTA.
Takeaways & Limitations
Interaction modeling improves location prediction and data association, while learned history motions help refind targets after prolonged occlusion without complex person Re-ID components.
Takeaways & Limitations
The method models pedestrian motion patterns and relationships but ignores drivable information in interactions, which may weaken motion prediction.
Abstract
from arXiv · showhide
The main challenge of Multi-Object Tracking~(MOT) lies in maintaining a continuous trajectory for each target. Existing methods often learn reliable motion patterns to match the same target between adjacent frames and discriminative appearance features to re-identify the lost targets after a long period. However, the reliability of motion prediction and the discriminability of appearances can be easily hurt by dense crowds and extreme occlusions in the tracking process. In this paper, we propose a simple yet effective multi-object tracker, i.e., MotionTrack, which learns robust short-term and long-term motions in a unified framework to associate trajectories from a short to long range. For dense crowds, we design a novel Interaction Module to learn interaction-aware motions from short-term trajectories, which can estimate the complex movement of each target. For extreme occlusions, we build a novel Refind Module to learn reliable long-term motions from the target's history trajectory, which can link the interrupted trajectory with its corresponding detection. Our Interaction Module and Refind Module are embedded in the well-known tracking-by-detection paradigm, which can work in tandem to maintain superior performance. Extensive experimental results on MOT17 and MOT20 datasets demonstrate the superiority of our approach in challenging scenarios, and it achieves state-of-the-art performances at various MOT metrics.
1. Introduction
MOT must preserve target identities across video frames despite dense crowds and extreme occlusions, which make short- and long-range association difficult. MotionTrack addresses both ranges by learning interaction-aware and history-based motion within tracking-by-detection.
- Motivation: Dense crowds complicate motion learning because pedestrians avoid collisions through interactions, while small detection boxes weaken appearance-based discrimination.These conditions make short-range association difficult.
- Motivation: Extreme occlusions create long gaps and large appearance variations, limiting appearance-based re-identification of lost targets.The challenge concerns long-range association after prolonged target disappearance.
- Approach: MotionTrack uses a tracking-by-detection framework that learns both interaction-aware and history trajectory-based motions for short- to long-range association.The approach targets the two association problems in a unified tracker.
- Approach: The Interaction Module models target interactions with an asymmetric adjacency matrix and graph convolution to predict complex motions in dense crowds.Its predictions support short-range association and can track short-term occlusions.
- Approach: The Refind Module re-identifies lost targets from history trajectories through correlation calculation and error compensation.It is designed for long-range association after interruption.
2. Related Work
Related MOT work commonly uses tracking-by-detection or motion-based tracking, but existing motion models often underrepresent interactions in dense crowds. Occlusion methods address short- and long-term failures through detection, appearance, memory, trajectory, or iterative-prediction cues.
- Tracking paradigms: Tracking-by-detection generates frame-wise detections and associates them across frames, while tracking-by-regression continuously predicts each tracklet’s position.Both paradigms must handle short-range and long-range association.
- Motion modeling: Motion-based methods use tools such as Kalman filtering, regression, optical flow, tracking branches, or pooled interaction features to estimate target displacement.These approaches provide data-driven motion predictions within tracking-by-detection or regression frameworks.
- Motion modeling: Most existing motion models do not explicitly model interactions between targets, limiting their ability to estimate complex movements in dense crowds.The related-work discussion identifies interaction modeling as the missing capability.
- Occlusion handling: Short-term occlusion prevents high-quality detection features, whereas long-term occlusion loses targets behind obstacles for extended periods.The literature distinguishes these occlusion settings because they create different association difficulties.
- Occlusion handling: Prior long-term occlusion methods use cascaded appearance matching, feature memory, bird’s-eye-view trajectories, or IoU between iterative predictions and detections.These methods re-identify lost tracklets using appearance or trajectory-derived cues.
3. Method
MotionTrack performs short-range and long-range association within a tracking-by-detection framework. Its Interaction Module models inter-tracklet interactions for motion prediction, while its Refind Module matches lost tracklets to unmatched detections using trajectory history.
- Overview: MotionTrack processes detections and maintains alive and lost tracklet sets throughout the video.Tracklets can be initialized, updated, interrupted, or removed as tracking proceeds.
- Short-range Association: The tracker performs short-range association by modeling inter-tracklet interactions to predict motions and associate detections.The resulting predictions update alive tracklets and record predictions for lost tracklets.
- Long-range Association: For long-range association, the Refind Module correlates lost tracklets with unmatched detections using trajectory spatial distributions and velocity-time relationships.It combines trajectory and detection features to produce association probabilities, then selects highly correlated pairs.
- Interaction Module: The Interaction Module extracts asymmetric interactions with self-attention and convolution, retains high-attention relationships, and uses graph convolution plus an MLP to predict offsets.The predicted offsets are converted into coordinates and used for IoU-based association.
- Refind Module: The Refind Module represents each lost tracklet with its recent trajectory and computes a correlation matrix before re-identifying matching detections.The trajectory input records the last thirty alive locations, while detection features include the difference from the last alive location.
4. Experiments
MotionTrack is evaluated on MOT17 and MOT20 under the private-detection protocol, with experiments covering overall performance, component ablations, motion prediction, and long-term occlusion handling. It achieves leading association and tracking results while its Interaction and Refind Modules improve complementary aspects of tracking.
- Experimental setting: The evaluation uses CLEAR metrics, IDF1, and HOTA to measure detection, association, and combined tracking performance.Training the Interaction and Refind Modules uses only half of the MOT17 and MOT20 training sets.
- MOT17 results: MotionTrack ranks first on IDF1, HOTA, AssA, DetA, IDs, and Frag, and second on MOTA on MOT17.It exceeds the second-best tracker by +2.0 IDF1 and +3.1 AssA.
- MOT20 results: MotionTrack ranks second among all methods and first among online methods on the official MOT Challenge evaluation server.
- Ablation study: The Interaction Module improves IDF1, HOTA, AssA, and IDs, while the Refind Module improves all evaluated metrics over the baseline.
- Motion-model analysis: Interaction-aware motion produces more accurate predictions than the traditional Kalman Filter, alongside steady improvements in IDF1 and AssA.Prediction accuracy is measured by average IoU across targets.
- Occlusion analysis: Under long-term occlusion evaluation, competing approaches show dramatic performance decreases, whereas MotionTrack achieves consistent improvement.The comparison raises the occlusion upper limit from 30 to 120 frames.
5. Conclusion
MotionTrack combines Interaction and Refind Modules for short- and long-term association in online MOT. It predicts through occlusions and reports state-of-the-art performance without complex components, while remaining limited by its omission of drivable information.
- MotionTrack combines Interaction and Refind Modules to address short-term and long-term association in online MOT.
- Figure 6 depicts directed target interactions, with arrows indicating the affecting and affected targets.
- Figure 7 shows occluded locations obtained from iterative Interaction Module prediction and Refind Module error compensation.
- Interaction-aware motion supports more accurate location prediction, while history-based motion helps refind targets after prolonged occlusion.
- The tracker achieves state-of-the-art performance without using complex components such as person Re-ID.
- A major limitation is ignoring drivable information in pedestrian interactions, which may weaken motion prediction.