Source-linked AI summary
BoT-SORT: Robust Associations Multi-Pedestrian Tracking
Nir Aharon, Roy Orfaig, Ben-Zion Bobrovsky
TL;DR
Multi-object tracking must detect objects and preserve their identities despite localization, camera-motion, and appearance-association challenges. The paper introduces BoT-SORT and BoT-SORT-ReID by combining motion and appearance information with camera-motion compensation and an improved Kalman filter state. The trackers rank first on MOT17 and MOT20 across MOTA, IDF1, and HOTA.
Problem
SORT-like trackers face inaccurate box-shape estimation, camera-motion-induced localization errors, and a trade-off between MOTA from IoU and IDF1 from Re-ID.
Method
BoT-SORT integrates camera-motion compensation, direct width-and-height Kalman state estimation, and IoU–ReID fusion into ByteTrack; BoT-SORT-ReID adds a re-identification module.
Results
BoT-SORT and BoT-SORT-ReID rank first on MOT17 and MOT20 across the main MOT metrics: MOTA, IDF1, and HOTA.
Takeaways & Limitations
The method and its components can be integrated into other tracking-by-detection trackers, while IoU–ReID fusion benefits from both motion and appearance information.
Takeaways & Limitations
Camera-motion estimation may fail in dense dynamic scenes lacking background keypoints, and global motion calculation can be time-consuming for large images.
Abstract
from arXiv · showhide
The goal of multi-object tracking (MOT) is detecting and tracking all the objects in a scene, while keeping a unique identifier for each object. In this paper, we present a new robust state-of-the-art tracker, which can combine the advantages of motion and appearance information, along with camera-motion compensation, and a more accurate Kalman filter state vector. Our new trackers BoT-SORT, and BoT-SORT-ReID rank first in the datasets of MOTChallenge [29, 11] on both MOT17 and MOT20 test sets, in terms of all the main MOT metrics: MOTA, IDF1, and HOTA. For MOT17: 80.5 MOTA, 80.2 IDF1, and 65.0 HOTA are achieved. The source code and the pre-trained models are available at https://github.com/NirAharon/BOT-SORT
1. Introduction
MOT detects and tracks multiple objects while preserving identities, but SORT-like trackers face localization, camera-motion, and MOTA–IDF1 trade-offs. BoT-SORT addresses these limitations with camera-motion compensation, an improved Kalman filter state, and IoU–ReID fusion.
- MOT estimates multiple objects’ spatial-temporal trajectories in video for applications including autonomous driving and surveillance.
- Tracking-by-detection combines object detection with motion-based prediction and association of new detections to existing tracks.Kalman filtering is commonly used for motion modeling, while localization and Re-ID distances support global assignment.
- SORT-like Kalman filter state estimation can produce sub-optimal box shapes because estimating aspect ratio instead of width yields inaccurate width estimates.
- Camera motion can reduce predicted-to-detected box overlap and tracking performance, so BoT-SORT estimates camera motion through image registration and corrects the Kalman filter.
- Localization and appearance cues create a MOTA–IDF1 trade-off: IoU usually improves MOTA, whereas Re-ID usually improves IDF1.
- BoT-SORT integrates these improvements into ByteTrack and proposes IoU–ReID cosine-distance fusion for more robust detection–tracklet associations.
2. Related Work
Tracking-by-detection research increasingly exploits stronger detectors and motion-based association, while appearance-based tracking offers identity cues at higher computational cost. BoT-SORT-ReID extends BoT-SORT with a re-identification module in its tracking pipeline.
- Improved object detectors have raised tracking performance and shifted tracking-by-detection research toward data association and deep-learning-based methods.
- Kalman-filter motion models, typically using a constant-velocity assumption, remain a common basis for predicting object motion.
- Appearance and Re-ID models help discriminate objects but can struggle with partial occlusions and require costly additional feature extraction.
- ByteTrack abandons appearance information and matches high-confidence detections before associating low-confidence detections to maintain high speed and performance.
- BoT-SORT-ReID adds a re-identification module to BoT-SORT, whose pipeline contains a main online-tracking region and an optional post-processing region.
3. Proposed Method
BoT-SORT integrates a revised Kalman filter, camera-motion compensation, and IoU–ReID association to improve tracking-by-detection robustness. Its BoT-SORT-ReID extension adds appearance features while using thresholds and matching rules designed for robust associations.
- Overview: BoT-SORT integrates three main improvements into ByteTrack: a revised Kalman filter, camera-motion compensation, and robust IoU–ReID fusion.BoT-SORT-ReID extends BoT-SORT with a re-identification module.
- 3.1. Kalman Filter: Directly estimating bounding-box width and height gives the Kalman filter a more accurate state representation than estimating aspect ratio.The proposed state vector changes the representation used by recent SORT-like trackers, with corresponding process and measurement noise matrices modified accordingly.
- 3.2. Camera Motion Compensation (CMC): Camera-motion compensation uses affine image registration and RANSAC-based sparse background motion estimation to correct predicted track states.The translation component updates box centers, while the scale and rotation component affects the state vector and noise matrix.
- 3.2. Camera Motion Compensation (CMC): Full state correction, including velocity terms, is essential at high camera velocities, while correction may be omitted when camera motion is slow relative to the frame rate.The method is intended to make the tracker robust to camera motion, under the stated assumption that object positions change only slightly between adjacent frames.
- 3.3. IoU - Re-ID Fusion: The appearance model updates matched tracklet descriptors with an exponential moving average using momentum α = 0.9 and only high-confidence detections.High-confidence detections are used because appearance features can be vulnerable to crowds, occlusion, and blur.
4. Experiments
Experiments evaluate BoT-SORT variants on MOT17 and MOT20 using standard tracking metrics, ablations, diagnostic visualizations, and implementation settings. The results show strong benchmark performance, while the authors note camera-motion and runtime limitations.
- Experimental settings: Experiments use MOT17 and MOT20 under the private-detection protocol, including moving-camera and crowded-scene settings.
- Evaluation metrics: MOTA, IDF1, HOTA, false positives, false negatives, and identity switches measure detection, association, and tracking performance.
- Ablation studies: The ablation studies isolate updated Kalman filtering, camera motion compensation, output-track prediction, and ReID using consistent tracking parameters.
- Ablation studies: The proposed IoU-ReID fusion with IoU masking achieves the highest MOTA, IDF1, and HOTA among the evaluated ReID association strategies.
- Failure analysis: Current-MOTA identifies failure intervals, while the rotating-camera example shows CMC preserving high MOTA with unchanged detections and tracking parameters.
- Benchmark evaluation: On MOT17 and MOT20, BoT-SORT variants outperform other state-of-the-art trackers across MOTA, IDF1, and HOTA, including crowded MOT20 scenes.
- Limitations: High-density dynamic scenes can defeat camera-motion estimation, while global-motion computation and separate appearance extraction introduce runtime concerns.
5. Conclusion
BoT-SORT is an enhanced multi-object tracker designed for robust association and ranks first on MOT17 and MOT20 across the main MOT metrics. Its components can be integrated into other tracking-by-detection trackers, and the paper introduces cMOTA for investigating tracker failures.
- BoT-SORT ranks first on MOT17 and MOT20 in MOTA, IDF1, and HOTA.
- The tracker combines MOT bag-of-tricks for robust association and can be integrated into other tracking-by-detection trackers.
- cMOTA is introduced as an MOT investigation tool for analyzing tracker failures.
- BoT-SORT ranks second and BoT-SORT-ReID ranks first among MOT17 leaderboard trackers under the private-detector protocol on MOT20.
Appendix A. Pseudo-code of BoT-SORT-ReID
The BoT-SORT-ReID pseudo-code processes video frames by handling detections, extracting appearance features, compensating camera motion, associating detections with tracks, and updating or managing tracklets.
- BoT-SORT-ReID takes a video sequence, an object detector, an appearance-feature extractor, and detection-score thresholds as inputs.
- For each frame, the algorithm handles new detections and stores high- and low-score detections separately.
- The tracker extracts appearance features and finds a warp matrix from the previous frame to the current frame.
- Hungarian assignment matches detections and tracks using high-score and then low-score association costs.
- Matched tracklets are updated with a Kalman filter and appearance features, while unmatched tracks are deleted and new tracks are initialized.
- Offline post-processing is optional, and tracks rebirth is omitted from the pseudo-code for simplicity.
Appendix B. Kalman Filter Model
The Kalman filter estimates object state from measurements using a linear stochastic model with prediction and update steps. Its state and covariance estimates are recursively refined using the observation model, noise covariances, and Kalman gain.
- The Kalman filter estimates a state x from measurements z given an initial state x0.
- The transition matrix Fk maps the state from time k−1 to k, while Hk is the observation matrix.
- Process and measurement noise are modeled as independent, identically distributed normal random variables.
- The covariance matrices Qk and Rk may change at each time step, and the filter alternates between prediction and update.
- At each step, the filter predicts prior state and covariance, then computes the posterior state from the observation and covariance using the Kalman gain.
- The tracker uses constant-velocity model matrices corresponding to the state and measurement vectors defined in the preceding equations.