PaperlayerView exact revision on arXiv

Source-linked AI summary

ByteTrack: Multi-Object Tracking by Associating Every Detection Box

Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Fucheng Weng, Zehuan Yuan, Ping Luo, Wenyu Liu, Xinggang Wang

arXiv:2110.06864v3cs.CV

TL;DR

MOT methods commonly discard low-score detections, even though they can represent occluded objects and contribute to missing or fragmented trajectories. The paper introduces BYTE, which associates nearly every detection using tracklet similarity to recover objects and reject background, and combines it with YOLOX in ByteTrack. ByteTrack reaches state-of-the-art benchmark results, including 80.3 MOTA, 77.3 IDF1, and 63.1 HOTA on MOT17 at 30 FPS.

  • Problem

    MOT methods often discard low-confidence detections, although such boxes can represent occluded objects and their removal causes missing detections and fragmented trajectories.

    Source: §1 · p. 1

  • Method

    BYTE separates high- and low-score detections, matches both groups to tracklets using similarity, and filters unmatched low-score boxes as background.

    Source: §1 · p. 2

  • Results

    80.3 MOTA, 77.3 IDF1, and 63.1 HOTA are achieved by ByteTrack on the MOT17 test set at 30 FPS, with state-of-the-art performance also reported on MOT20, HiEve, and BDD100K.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    BYTE improves existing trackers across nearly all metrics and provides a flexible association method for enhancing multi-object tracking.

    Source: §1 · p. 2, §3 · p. 4

Abstract

from arXiv · show

Multi-object tracking (MOT) aims at estimating bounding boxes and identities of objects in videos. Most methods obtain identities by associating detection boxes whose scores are higher than a threshold. The objects with low detection scores, e.g. occluded objects, are simply thrown away, which brings non-negligible true object missing and fragmented trajectories. To solve this problem, we present a simple, effective and generic association method, tracking by associating almost every detection box instead of only the high score ones. For the low score detection boxes, we utilize their similarities with tracklets to recover true objects and filter out the background detections. When applied to 9 different state-of-the-art trackers, our method achieves consistent improvement on IDF1 score ranging from 1 to 10 points. To put forwards the state-of-the-art performance of MOT, we design a simple and strong tracker, named ByteTrack. For the first time, we achieve 80.3 MOTA, 77.3 IDF1 and 63.1 HOTA on the test set of MOT17 with 30 FPS running speed on a single V100 GPU. ByteTrack also achieves state-of-the-art performance on MOT20, HiEve and BDD100K tracking benchmarks. The source code, pre-trained models with deploy versions and tutorials of applying to other trackers are released at https://github.com/ifzhang/ByteTrack.

1. Introduction

MOT systems often discard low-confidence detections, but these can represent occluded objects and removing them causes missing detections and fragmented trajectories. BYTE addresses this dilemma by associating high- and low-score detections with tracklets, while ByteTrack combines BYTE with YOLOX for strong benchmark performance.

  • Low-confidence detections can indicate occluded objects, so thresholding them away causes missing detections and fragmented trajectories.Source: §1 · p. 1
  • When occlusion lowers a detection score from 0.8 to 0.4 and then 0.1, thresholding makes the corresponding tracklet disappear.Source: §1 · pp. 1–2
  • Considering every detection introduces background false positives, creating a detection dilemma that few MOT methods handle.Source: §1 · p. 2
  • BYTE first matches high-score detections, then matches low-score detections to unmatched tracklets using motion or appearance similarity.Source: §1 · p. 2
  • Applied to nine state-of-the-art trackers, BYTE improves nearly all metrics; CenterTrack rises from 66.1 to 67.4 MOTA and from 64.2 to 74.0 IDF1, while IDs fall from 528 to 144.Source: §1 · p. 2
  • ByteTrack achieves 80.3 MOTA, 77.3 IDF1, and 63.1 HOTA on MOT17 at 30 FPS, and also reaches state-of-the-art performance on MOT20, HiEve, and BDD100K.Source: Figure 1 · p. 1, §1 · p. 2

2. Related Work

Related work establishes tracking-by-detection and data association as central MOT components, while highlighting weaknesses from imperfect or discarded detections. BYTE focuses on using detections across the full score range rather than only improving association after thresholding.

2.1. Object Detection in MOT

  • Object detection supplies the detection results underlying many MOT methods, which then improve tracking using those fixed inputs.Source: §2.1 · p. 2
  • Occlusion and motion blur increase missing and very low-scoring detections, motivating the use of information from previous frames.Source: §2.1 · p. 3
  • Most MOT methods retain detections above a 0.5 threshold because low-score boxes contain background, although occluded objects may also receive low scores.Source: §2.1 · p. 3

2.2. Data Association

  • Data association computes similarities between tracklets and detection boxes, then matches them using strategies such as Hungarian assignment or greedy assignment.Source: §2.2 · p. 3
  • Location and motion cues support short-range matching, while appearance similarity helps re-identify objects after long occlusions.Source: §2.2 · p. 3
  • BYTE argues that detection-box utilization determines the upper bound of association and therefore uses boxes from high scores to low ones.Source: §2.2 · p. 3

3. BYTE

BYTE separates detections by score, performs a high-score association followed by a low-score association for remaining tracks, and treats unmatched low-score boxes as background. The method is flexible across trackers and is used with YOLOX in ByteTrack.

BYTE association procedure

  • BYTE divides detections into high-score and low-score sets, first associating high-score boxes with tracks and then using low-score boxes to recover unmatched tracks.Source: §3 · pp. 3–4
  • The method predicts track locations with a Kalman filter and computes association using IoU or Re-ID distances in the first matching stage.Source: §3 · p. 4

Flexibility

  • BYTE is compatible with other association methods; combined with FairMOT, Re-ID features are added to the first association while the remaining procedure stays unchanged.Source: §3 · p. 4

BYTE association procedure

  • The second association uses IoU between low-score detections and remaining tracks, deletes unmatched low-score boxes, and retains unmatched tracks as lost tracks.Source: §3 · p. 4

Track management

  • BYTE keeps lost tracks for up to 30 frames before deletion and initializes new tracks from unmatched high-score detections.Source: §3 · p. 4

ByteTrack

  • ByteTrack equips YOLOX with BYTE to form a simple, strong tracker.Source: §3 · p. 4

4. Experiments

Experiments evaluate BYTE’s association choices, robustness, generalization across trackers, and benchmark performance across MOT17, MOT20, HiEve, and BDD100K. Results show that associating low-score detections improves tracking accuracy and supports strong performance across settings.

Similarity analysis

  • BYTE evaluates motion and appearance similarities for two association stages across MOT17 and BDD100K validation sets.IoU or Re-ID can be used in the first association, while IoU is important for the second association; Re-ID is especially useful on BDD100K under large camera motion.Source: §4.2 · p. 5

Comparisons with other association methods

  • 76.6 MOTA, 79.3 IDF1, and 159 IDs are achieved by BYTE versus SORT’s 74.6 MOTA, 76.9 IDF1, and 291 IDs on MOT17 validation.The comparison uses SORT as a Kalman-filter-based baseline and attributes the gains to recovering objects from low-score detections.Source: §4.2 · p. 5
  • BYTE is more robust than SORT to detection score thresholds because its second association recovers objects below the high-score threshold.The threshold comparison is conducted on the MOT17 validation set.Source: §4.2 · p. 6

Analysis on low score detection boxes

  • BYTE increases MOTA from 74.6 to 76.6 by selecting true positives from low-score detection boxes on MOT17 validation.The analysis compares all low-score detections with the low-score boxes retained in BYTE’s tracking results.Source: §4.2 · p. 7

Benchmark Evaluation

  • ByteTrack ranks first on MOT17, MOT20, HiEve, and BDD100K, while achieving 80.3 MOTA, 77.3 IDF1, and 63.1 HOTA at 30 FPS on MOT17.Reported results include 77.8 MOTA and 75.2 IDF1 on MOT20, 61.3 MOTA and 62.9 IDF1 on HiEve, and 40.1 test mMOTA on BDD100K.Source: §4.3 · pp. 7–8

5. Conclusion

ByteTrack combines the BYTE association method with YOLOX to deliver fast, accurate, and robust multi-object tracking. It ranks first on BDD100K and MOT17 while remaining suitable for real applications.

  • ByteTrack achieves 80.3 MOTA, 77.3 IDF1, and 63.1 HOTA on MOT17 at 30 FPS, ranking first among leaderboard trackers.Source: §5 · p. 8
  • ByteTrack is robust to occlusion because it combines accurate detection with association of low-score detection boxes.Source: §5 · p. 8
  • The tracker’s high accuracy, fast speed, and simplicity support its use in real applications.Source: §5 · p. 8
  • ByteTrack ranks first on BDD100K and outperforms QDTrack by a large margin on most metrics.Source: Table 7 · p. 9

A. Bounding box annotations

The implementation modifies YOLOX preprocessing and label assignment for MOT17 so detections can cover occluded or partially out-of-image objects. Other datasets retain YOLOX’s original clipping behavior.

B. Tracking performance of light models

Using light detection models, BYTE provides stable MOTA and IDF1 improvements over DeepSORT on the MOT17 validation set, indicating robustness to detection performance.

C. Ablation Studies on ByteTrack

The ablations examine speed, detection-model comparisons, training-data choices, and interpolation settings. Results show that ByteTrack maintains useful accuracy-speed trade-offs and strong performance with limited training data.

Light models

  • Table 8 compares BYTE and DeepSORT with light detection models on the MOT17 validation set.Source: Table 8 · p. 9

Speed v.s. accuracy

Training data

Interpolation

  • Table 11 compares different interpolation intervals on the MOT17 validation set.Source: Table 11 · p. 10

Training data

D. Tracklet interpolation

ByteTrack uses tracklet interpolation to recover boxes for fully occluded pedestrians when visual detection is impossible. Interpolation is bounded by a maximum gap and improves tracking metrics.

D. Tracklet interpolation

  • Tracklet interpolation recovers fully occluded pedestrians whose visible ratio is zero in MOT17 annotations.These objects are obtained from existing tracklets because visual cues cannot detect them.Source: D. Tracklet interpolation · p. 10
  • Interpolation applies when the gap between lost and recovered tracklet boxes satisfies t2 − t1 ≤ σ.The hyper-parameter σ sets the maximum interpolation interval.Source: D. Tracklet interpolation · p. 10
  • At σ = 20, tracklet interpolation improves MOTA from 76.6 to 78.3 and IDF1 from 79.3 to 80.2.The method is used on MOT17, MOT20, and HiEve test sets under the private detection protocol.Source: D. Tracklet interpolation · p. 10

E. Public detection results on MOTChallenge

Under the public detection protocol, ByteTrack is evaluated on MOT17 and MOT20 using a strict trajectory-initialization rule. It outperforms existing methods by substantial margins on both benchmarks.

E. Public detection results on MOTChallenge

  • Tables 12 and 13 compare state-of-the-art methods under the public detector protocol on MOT17 and MOT20 test sets.The best results are shown in bold in both tables.Source: Table 12 · p. 10, Table 13 · p. 10
  • ByteTrack initializes a new trajectory only when its IoU with a public detection box exceeds 0.8.Tracklet interpolation is not used under the public detection protocol.Source: E. Public detection results on… · p. 10
  • On MOT17, ByteTrack exceeds SiamMOT by 1.5 MOTA points and 6.7 IDF1 points.The comparison is reported under the public detector protocol.Source: E. Public detection results on… · p. 10
  • On MOT20, ByteTrack exceeds TMOH by 6.9 MOTA points, 9.0 IDF1 points, and 7.5 HOTA points.It also reduces identity switches by three quarters.Source: E. Public detection results on… · p. 10

F. Visualization results.

ByteTrack visualizations show its handling of occlusion, motion blur, and small objects. Low-score detections support recovery and long-range association while preserving identities in the shown cases.

F. Visualization results.

  • The visualized difficult cases include occlusion, motion blur, and small objects across selected MOT17 sequences.Figure 5 selects six validation sequences to illustrate these cases.Source: F. Visualization results · p. 10, Figure 5 · p. 11
  • Low-score boxes reduce missing detections and support long-range association in the visualized sequences.The middle-frame pedestrian marked by a red triangle is recovered through BYTE.Source: F. Visualization results · p. 10
  • ByteTrack preserves identities without identity switches in the shown difficult cases.Figure 5 uses matching box colors to represent the same identity.Source: F. Visualization results · p. 10, Figure 5 · p. 11

Related papers

Loading 2110.06864v3…