PaperlayerView exact revision on arXiv

Source-linked AI summary

ByteTrack: Multi-Object Tracking by Associating Every Detection Box

Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Fucheng Weng, Zehuan Yuan, Ping Luo, Wenyu Liu, Xinggang Wang

arXiv:2110.06864v3cs.CV

TL;DR

MOT methods commonly discard low-score detections, even though they can represent occluded objects and contribute to missing or fragmented trajectories. The paper introduces BYTE, which associates nearly every detection using tracklet similarity to recover objects and reject background, and combines it with YOLOX in ByteTrack. ByteTrack reaches state-of-the-art benchmark results, including 80.3 MOTA, 77.3 IDF1, and 63.1 HOTA on MOT17 at 30 FPS.

  • Problem

    MOT methods often discard low-confidence detections, although such boxes can represent occluded objects and their removal causes missing detections and fragmented trajectories.

  • Method

    BYTE separates high- and low-score detections, matches both groups to tracklets using similarity, and filters unmatched low-score boxes as background.

  • Results

    80.3 MOTA, 77.3 IDF1, and 63.1 HOTA are achieved by ByteTrack on the MOT17 test set at 30 FPS, with state-of-the-art performance also reported on MOT20, HiEve, and BDD100K.

  • Takeaways & Limitations

    BYTE improves existing trackers across nearly all metrics and provides a flexible association method for enhancing multi-object tracking.

Abstract

from arXiv · show

Multi-object tracking (MOT) aims at estimating bounding boxes and identities of objects in videos. Most methods obtain identities by associating detection boxes whose scores are higher than a threshold. The objects with low detection scores, e.g. occluded objects, are simply thrown away, which brings non-negligible true object missing and fragmented trajectories. To solve this problem, we present a simple, effective and generic association method, tracking by associating almost every detection box instead of only the high score ones. For the low score detection boxes, we utilize their similarities with tracklets to recover true objects and filter out the background detections. When applied to 9 different state-of-the-art trackers, our method achieves consistent improvement on IDF1 score ranging from 1 to 10 points. To put forwards the state-of-the-art performance of MOT, we design a simple and strong tracker, named ByteTrack. For the first time, we achieve 80.3 MOTA, 77.3 IDF1 and 63.1 HOTA on the test set of MOT17 with 30 FPS running speed on a single V100 GPU. ByteTrack also achieves state-of-the-art performance on MOT20, HiEve and BDD100K tracking benchmarks. The source code, pre-trained models with deploy versions and tutorials of applying to other trackers are released at https://github.com/ifzhang/ByteTrack.

1. Introduction

MOT systems often discard low-confidence detections, but these can represent occluded objects and removing them causes missing detections and fragmented trajectories. BYTE addresses this dilemma by associating high- and low-score detections with tracklets, while ByteTrack combines BYTE with YOLOX for strong benchmark performance.

  • Low-confidence detections can indicate occluded objects, so thresholding them away causes missing detections and fragmented trajectories.
  • When occlusion lowers a detection score from 0.8 to 0.4 and then 0.1, thresholding makes the corresponding tracklet disappear.
  • Considering every detection introduces background false positives, creating a detection dilemma that few MOT methods handle.
  • BYTE first matches high-score detections, then matches low-score detections to unmatched tracklets using motion or appearance similarity.
  • Applied to nine state-of-the-art trackers, BYTE improves nearly all metrics; CenterTrack rises from 66.1 to 67.4 MOTA and from 64.2 to 74.0 IDF1, while IDs fall from 528 to 144.
  • ByteTrack achieves 80.3 MOTA, 77.3 IDF1, and 63.1 HOTA on MOT17 at 30 FPS, and also reaches state-of-the-art performance on MOT20, HiEve, and BDD100K.

2. Related Work

Related work establishes tracking-by-detection and data association as central MOT components, while highlighting weaknesses from imperfect or discarded detections. BYTE focuses on using detections across the full score range rather than only improving association after thresholding.

  • 2.1. Object Detection in MOT: Object detection supplies the detection results underlying many MOT methods, which then improve tracking using those fixed inputs.
  • 2.1. Object Detection in MOT: Occlusion and motion blur increase missing and very low-scoring detections, motivating the use of information from previous frames.
  • 2.1. Object Detection in MOT: Most MOT methods retain detections above a 0.5 threshold because low-score boxes contain background, although occluded objects may also receive low scores.
  • 2.2. Data Association: Data association computes similarities between tracklets and detection boxes, then matches them using strategies such as Hungarian assignment or greedy assignment.
  • 2.2. Data Association: Location and motion cues support short-range matching, while appearance similarity helps re-identify objects after long occlusions.
  • 2.2. Data Association: BYTE argues that detection-box utilization determines the upper bound of association and therefore uses boxes from high scores to low ones.

3. BYTE

BYTE separates detections by score, performs a high-score association followed by a low-score association for remaining tracks, and treats unmatched low-score boxes as background. The method is flexible across trackers and is used with YOLOX in ByteTrack.

  • BYTE association procedure: BYTE divides detections into high-score and low-score sets, first associating high-score boxes with tracks and then using low-score boxes to recover unmatched tracks.
  • BYTE association procedure: The method predicts track locations with a Kalman filter and computes association using IoU or Re-ID distances in the first matching stage.
  • Flexibility: BYTE is compatible with other association methods; combined with FairMOT, Re-ID features are added to the first association while the remaining procedure stays unchanged.
  • BYTE association procedure: The second association uses IoU between low-score detections and remaining tracks, deletes unmatched low-score boxes, and retains unmatched tracks as lost tracks.
  • Track management: BYTE keeps lost tracks for up to 30 frames before deletion and initializes new tracks from unmatched high-score detections.
  • ByteTrack: ByteTrack equips YOLOX with BYTE to form a simple, strong tracker.

4. Experiments

Experiments evaluate BYTE’s association choices, robustness, generalization across trackers, and benchmark performance across MOT17, MOT20, HiEve, and BDD100K. Results show that associating low-score detections improves tracking accuracy and supports strong performance across settings.

  • Similarity analysis: BYTE evaluates motion and appearance similarities for two association stages across MOT17 and BDD100K validation sets.IoU or Re-ID can be used in the first association, while IoU is important for the second association; Re-ID is especially useful on BDD100K under large camera motion.
  • Comparisons with other association methods: 76.6 MOTA, 79.3 IDF1, and 159 IDs are achieved by BYTE versus SORT’s 74.6 MOTA, 76.9 IDF1, and 291 IDs on MOT17 validation.The comparison uses SORT as a Kalman-filter-based baseline and attributes the gains to recovering objects from low-score detections.
  • Comparisons with other association methods: BYTE is more robust than SORT to detection score thresholds because its second association recovers objects below the high-score threshold.The threshold comparison is conducted on the MOT17 validation set.
  • Analysis on low score detection boxes: BYTE increases MOTA from 74.6 to 76.6 by selecting true positives from low-score detection boxes on MOT17 validation.The analysis compares all low-score detections with the low-score boxes retained in BYTE’s tracking results.
  • Benchmark Evaluation: ByteTrack ranks first on MOT17, MOT20, HiEve, and BDD100K, while achieving 80.3 MOTA, 77.3 IDF1, and 63.1 HOTA at 30 FPS on MOT17.Reported results include 77.8 MOTA and 75.2 IDF1 on MOT20, 61.3 MOTA and 62.9 IDF1 on HiEve, and 40.1 test mMOTA on BDD100K.

5. Conclusion

ByteTrack combines the BYTE association method with YOLOX to deliver fast, accurate, and robust multi-object tracking. It ranks first on BDD100K and MOT17 while remaining suitable for real applications.

  • ByteTrack achieves 80.3 MOTA, 77.3 IDF1, and 63.1 HOTA on MOT17 at 30 FPS, ranking first among leaderboard trackers.
  • ByteTrack is robust to occlusion because it combines accurate detection with association of low-score detection boxes.
  • The tracker’s high accuracy, fast speed, and simplicity support its use in real applications.
  • ByteTrack ranks first on BDD100K and outperforms QDTrack by a large margin on most metrics.

A. Bounding box annotations

The implementation modifies YOLOX preprocessing and label assignment for MOT17 so detections can cover occluded or partially out-of-image objects. Other datasets retain YOLOX’s original clipping behavior.

  • MOT17 requires bounding boxes covering the whole body even when objects are occluded or partly outside the image.
  • For MOT17, YOLOX is modified to avoid clipping bounding boxes inside the image during preprocessing and augmentation.
  • MOT20, HiEve, and BDD100K clip annotations inside the image, so ByteTrack uses YOLOX’s original setting for those datasets.

B. Tracking performance of light models

Using light detection models, BYTE provides stable MOTA and IDF1 improvements over DeepSORT on the MOT17 validation set, indicating robustness to detection performance.

  • BYTE brings stable improvements in MOTA and IDF1 compared with DeepSORT using light detection models.
  • The comparison uses YOLOX detectors with different backbones, trained on CrowdHuman and half of MOT17.

C. Ablation Studies on ByteTrack

The ablations examine speed, detection-model comparisons, training-data choices, and interpolation settings. Results show that ByteTrack maintains useful accuracy-speed trade-offs and strong performance with limited training data.

  • Light models: Table 8 compares BYTE and DeepSORT with light detection models on the MOT17 validation set.
  • Speed v.s. accuracy: Table 9 compares input sizes using total running time, combining detection and association time.
  • Speed v.s. accuracy: ByteTrack achieves 75.0 MOTA at 45.7 FPS and 76.6 MOTA at 29.6 FPS across inference input sizes.
  • Training data: Using only the MOT17 half training set, ByteTrack achieves 75.8 MOTA on the validation set.
  • Interpolation: Table 11 compares different interpolation intervals on the MOT17 validation set.
  • Training data: Adding CrowdHuman, Cityperson, and ETHZ raises performance to 76.7 MOTA and 79.7 IDF1.
  • Training data: Training-data experiments suggest ByteTrack is not data hungry compared with methods requiring more than seven data sources.

D. Tracklet interpolation

ByteTrack uses tracklet interpolation to recover boxes for fully occluded pedestrians when visual detection is impossible. Interpolation is bounded by a maximum gap and improves tracking metrics.

  • D. Tracklet interpolation: Tracklet interpolation recovers fully occluded pedestrians whose visible ratio is zero in MOT17 annotations.These objects are obtained from existing tracklets because visual cues cannot detect them.
  • D. Tracklet interpolation: Interpolation applies when the gap between lost and recovered tracklet boxes satisfies t2 − t1 ≤ σ.The hyper-parameter σ sets the maximum interpolation interval.
  • D. Tracklet interpolation: At σ = 20, tracklet interpolation improves MOTA from 76.6 to 78.3 and IDF1 from 79.3 to 80.2.The method is used on MOT17, MOT20, and HiEve test sets under the private detection protocol.

E. Public detection results on MOTChallenge

Under the public detection protocol, ByteTrack is evaluated on MOT17 and MOT20 using a strict trajectory-initialization rule. It outperforms existing methods by substantial margins on both benchmarks.

  • E. Public detection results on MOTChallenge: Tables 12 and 13 compare state-of-the-art methods under the public detector protocol on MOT17 and MOT20 test sets.The best results are shown in bold in both tables.
  • E. Public detection results on MOTChallenge: ByteTrack initializes a new trajectory only when its IoU with a public detection box exceeds 0.8.Tracklet interpolation is not used under the public detection protocol.
  • E. Public detection results on MOTChallenge: On MOT17, ByteTrack exceeds SiamMOT by 1.5 MOTA points and 6.7 IDF1 points.The comparison is reported under the public detector protocol.
  • E. Public detection results on MOTChallenge: On MOT20, ByteTrack exceeds TMOH by 6.9 MOTA points, 9.0 IDF1 points, and 7.5 HOTA points.It also reduces identity switches by three quarters.

F. Visualization results.

ByteTrack visualizations show its handling of occlusion, motion blur, and small objects. Low-score detections support recovery and long-range association while preserving identities in the shown cases.

  • F. Visualization results.: The visualized difficult cases include occlusion, motion blur, and small objects across selected MOT17 sequences.Figure 5 selects six validation sequences to illustrate these cases.
  • F. Visualization results.: Low-score boxes reduce missing detections and support long-range association in the visualized sequences.The middle-frame pedestrian marked by a red triangle is recovered through BYTE.
  • F. Visualization results.: ByteTrack preserves identities without identity switches in the shown difficult cases.Figure 5 uses matching box colors to represent the same identity.
Loading 2110.06864v3…