PaperlayerView exact revision on arXiv

Source-linked AI summary

StrongSORT: Make DeepSORT Great Again

Yunhao Du, Zhicheng Zhao, Yang Song, Yanyun Zhao, Fei Su, Tao Gong, Hongying Meng

arXiv:2202.13514v2cs.CV

TL;DR

MOT comparisons are complicated by differing models and tricks, motivating a stronger fair baseline and solutions to missing association and missing detection. The paper upgrades DeepSORT into StrongSORT, adds AFLink and GSI, and reports state-of-the-art results for StrongSORT++ across multiple benchmarks, with lightweight plug-in costs.

  • Problem

    Different detectors, ReID models, external datasets, and training or inference tricks make fair MOT comparison difficult, while missing association and missing detection remain inherent problems.

  • Method

    The paper upgrades DeepSORT with advanced detection, embedding, and inference components, then adds appearance-free AFLink for global association and Gaussian-process-based GSI for missing detections.

  • Results

    StrongSORT++ achieves state-of-the-art results on MOT17, MOT20, DanceTrack, and KITTI, while AFLink and GSI run at 1.7 ms and 7.1 ms per image on MOT17.

  • Takeaways & Limitations

    StrongSORT provides a strong baseline, and AFLink and GSI can be plugged into various trackers with negligible extra computational cost.

  • Takeaways & Limitations

    StrongSORT is relatively slow compared with joint and some appearance-free trackers, and AFLink cannot split mixed-up ID trajectories into accurate tracklets.

Abstract

from arXiv · show

Recently, Multi-Object Tracking (MOT) has attracted rising attention, and accordingly, remarkable progresses have been achieved. However, the existing methods tend to use various basic models (e.g, detector and embedding model), and different training or inference tricks, etc. As a result, the construction of a good baseline for a fair comparison is essential. In this paper, a classic tracker, i.e., DeepSORT, is first revisited, and then is significantly improved from multiple perspectives such as object detection, feature embedding, and trajectory association. The proposed tracker, named StrongSORT, contributes a strong and fair baseline for the MOT community. Moreover, two lightweight and plug-and-play algorithms are proposed to address two inherent "missing" problems of MOT: missing association and missing detection. Specifically, unlike most methods, which associate short tracklets into complete trajectories at high computation complexity, we propose an appearance-free link model (AFLink) to perform global association without appearance information, and achieve a good balance between speed and accuracy. Furthermore, we propose a Gaussian-smoothed interpolation (GSI) based on Gaussian process regression to relieve the missing detection. AFLink and GSI can be easily plugged into various trackers with a negligible extra computational cost (1.7 ms and 7.1 ms per image, respectively, on MOT17). Finally, by fusing StrongSORT with AFLink and GSI, the final tracker (StrongSORT++) achieves state-of-the-art results on multiple public benchmarks, i.e., MOT17, MOT20, DanceTrack and KITTI. Codes are available at https://github.com/dyhBUPT/StrongSORT and https://github.com/open-mmlab/mmtracking.

I. INTRODUCTION

StrongSORT addresses fairness and two persistent MOT gaps by upgrading DeepSORT and adding lightweight modules for missing association and missing detection. Its StrongSORT++ combination achieves state-of-the-art results across multiple benchmarks.

  • MOT comparisons are difficult because trackers use different detectors, ReID models, external training datasets, and training or inference tricks.
  • StrongSORT upgrades DeepSORT with a stronger detector, embedding model, and inference tricks to provide a strong and fair MOT baseline.
  • Missing association splits one object across tracklets, while missing detection refers to false negatives commonly caused by occlusion and low resolution.
  • AFLink uses spatiotemporal information without appearance features to improve the speed–accuracy trade-off for global tracklet association.
  • GSI uses Gaussian process regression to refine interpolated bounding boxes and produce more accurate, stable localizations.
  • 1.7 ms and 7.1 ms per image are the reported MOT17 running speeds for AFLink and GSI, respectively, while StrongSORT++ achieves state-of-the-art results on MOT17, MOT20, DanceTrack, and KITTI.

II. RELATED WORK

Related MOT work contrasts separate tracking-by-detection systems with joint trackers and global-link approaches. StrongSORT retains the DeepSORT-like paradigm while using advanced techniques, and AFLink pursues a simpler motion-based alternative to complex global association models.

  • Separate trackers detect targets first and then associate them using appearance, motion, or related information, whereas joint trackers train detection and other components together.
  • Appearance-free trackers can achieve high speed and state-of-the-art MOTChallenge performance, but abandoning appearance features can reduce robustness in complex scenes.
  • Global-link methods generate accurate but incomplete tracklets and associate them offline using global information.
  • AFLink shares motion-based association motivations with LGMTracker but uses a simpler, more lightweight framework and training strategy.

C. Interpolation in MOT

MOT systems combine appearance and motion cues for association, while interpolation methods address gaps caused by missing detections. The paper positions GSI as a motion-aware refinement over linear interpolation within the broader DeepSORT framework.

  • C. Interpolation in MOT: Linear interpolation fills trajectory gaps but ignores motion information, limiting the accuracy of restored bounding boxes.
  • C. Interpolation in MOT: GPR-based smoothing has been used for velocity prediction, whereas GSI applies Gaussian process regression to refine interpolated localizations in MOT.
  • A. Review of DeepSORT: DeepSORT uses appearance features from a CNN and a feature bank, with cosine distance serving as an association cost.
  • A. Review of DeepSORT: Its motion branch uses a Kalman filter for state prediction and update, then Mahalanobis distance measures spatiotemporal dissimilarity between tracklets and detections.
  • B. StrongSORT: Figure 2 compares DeepSORT and StrongSORT on MOT17 validation detections predicted by YOLOX.
  • A. Review of DeepSORT: DeepSORT’s matching cascade solves association through prioritized subproblems, each using the Hungarian algorithm.

B. StrongSORT

StrongSORT strengthens DeepSORT with improved detection, embedding, association, camera compensation, and state estimation components. These changes target matching quality, robustness to detection noise, and motion-related errors.

  • Advanced modules: StrongSORT replaces DeepSORT’s detector with YOLOX-X and its feature bank with an EMA-based appearance-feature updating strategy.The EMA strategy uses inter-frame feature changes and can reduce detection noise while lowering time consumption.
  • Advanced modules: The EMA appearance update uses the current matched detection embedding with momentum term α = 0.9.
  • ECC: ECC estimates global rotation and translation between adjacent frames to compensate for camera-motion noise.It aligns a reference image and warped image through an image-alignment criterion.
  • NSA Kalman: NSA Kalman adapts measurement-noise covariance using detection confidence, assigning higher state-update weight to higher-confidence detections.Lower effective measurement noise increases a detection’s influence on the updated state.
  • Association: StrongSORT combines appearance and motion costs for assignment and replaces matching cascade with vanilla global linear matching.The stated motivation is that stronger trackers are more robust to confusing associations, making additional cascade constraints limiting.

IV. STRONGSORT++

StrongSORT++ integrates the StrongSORT baseline with two lightweight, plug-and-play, model-independent, appearance-free algorithms to address missing association and missing detection.

  • StrongSORT++ is formed by integrating StrongSORT with AFLink and GSI.AFLink and GSI are presented as lightweight, plug-and-play, model-independent, and appearance-free algorithms.

A. AFLink

AFLink addresses missing association by predicting whether two tracklets connect using spatiotemporal information rather than appearance features. It extracts recent tracklet features and solves global linking through linear assignment.

  • AFLink predicts connectivity between two tracklets using only spatiotemporal information.This design avoids the computationally expensive appearance components used by many global-link approaches.
  • The model uses the most recent N = 30 frames from each tracklet, with zero padding for shorter tracklets.
  • Temporal convolutions with 7 × 1 kernels and a 1 × 3 fusion convolution integrate frame, x-coordinate, and y-coordinate information.The resulting feature maps are pooled, squeezed, concatenated, and passed to an MLP classifier.
  • AFLink training formulates tracklet association as binary classification with binary cross-entropy loss.The predicted probability and ground-truth label are denoted x_n ∈ [0, 1] and y_n ∈ {0, 1}.
  • During association, unreasonable tracklet pairs are filtered by spatiotemporal constraints before linear assignment uses predicted connectivity scores.

B. GSI

GSI improves trajectory gap filling by modeling nonlinear motion with Gaussian process regression. It smooths noisy tracked results and linearly interpolated positions using an adaptive smoothness factor.

  • GSI uses Gaussian process regression to model nonlinear motion while remaining a lightweight interpolation algorithm.It is proposed as an alternative to additional time-consuming modules such as single-object tracking, Kalman filtering, and ECC.
  • The model represents each trajectory coordinate as a function of frame id with Gaussian noise and fits nonlinear motion from tracked and linearly interpolated observations.The position coordinates are x, y, w, and h.
  • For new frames, GSI predicts smoothed positions from the Gaussian-process covariance structure.The covariance function is derived from a radial basis function kernel.
  • The hyperparameter λ controls trajectory smoothness and is adapted to trajectory length l as λ = τ ∗ log(τ^3/l), with τ set to 10.
  • GSI smooths the entire trajectory with an adaptive smoothness factor, addressing both noisy jitter and linear interpolation’s lack of motion information.In the illustrated comparison, raw tracked results are orange, linear interpolation is blue, and GSI is red.

V. EXPERIMENTS

Experiments evaluate StrongSORT on MOT17 and MOT20, with additional testing on KITTI and DanceTrack. The study uses standard MOT metrics and reports implementation settings including a single-V100 evaluation environment.

  • Datasets: Experiments use MOT17 and MOT20 under the private-detection protocol, with KITTI and DanceTrack included for additional StrongSORT++ evaluation.MOT17 and MOT20 provide the primary experimental datasets; KITTI and DanceTrack extend evaluation to autonomous-driving and multi-human tracking scenarios.
  • Datasets: MOT17 contains 7 training sequences with 5,316 frames and 7 test sequences with 5,919 frames, while MOT20 contains 4 training sequences with 8,931 frames and 4 test sequences with 4,479 frames.MOT20 emphasizes highly crowded scenes, whereas MOT17 is a standard MOT benchmark.
  • Experimental protocol: Ablations use the first half of each MOT17 training sequence for training and the last half for validation, with DukeMTMC, CrowdHuman, Cityperson, and ETHZ used in training or testing.The appearance extractor is pretrained on DukeMTMC, while the detector uses CrowdHuman and MOT17 training data.
  • Metrics: Evaluation reports MOTA, IDs, IDF1, HOTA, AssA, DetA, and FPS to measure detection, identity consistency, association, and speed.MOTA emphasizes detection-related errors, IDF1 measures ID-matching consistency, and HOTA combines detection and association scores.
  • Implementation: All experiments run on a server with a single V100 GPU, and AFLink training takes just over 10 seconds.Inference uses temporal and spatial distance thresholds of 30 frames and 75 pixels, respectively.

B. Ablation Studies

Ablations show that StrongSORT gains from stronger appearance, motion, filtering, and matching components. AFLink and GSI provide complementary improvements, while vanilla matching benefits stronger baselines and GSI supports more stable interpolation.

  • StrongSORT components: BoT improves IDF1 by +2.2, showing that more discriminative appearance features benefit association quality.BoT replaces the original feature extractor.
  • StrongSORT components: ECC increases IDF1 by +0.2 and MOTA by +0.3, while NSA improves HOTA by +0.4 without improving MOTA or IDF1.ECC supports more precise motion information, whereas NSA improves positioning accuracy.
  • StrongSORT components: EMA improves IDF1 by +0.4 and speed by +1.2 FPS, while matching appearance and motion costs improves IDF1 by +0.8.These components improve association, with EMA also improving runtime speed.
  • Matching strategy: Replacing matching cascade with vanilla matching improves StrongSORTv5 IDF1 by 1.4 because cascade constraints increasingly limit stronger trackers.The matching cascade benefits DeepSORT but becomes less advantageous as the baseline tracker improves.
  • AFLink and GSI: AFLink benefits poorer trackers more, including a 3.7 IDF1 improvement for CenterTrack, while GSI improves detection and association but can be confused by false associations in poorer trackers.GSI works better on stronger trackers, complementing AFLink’s larger gains on trackers with more missing associations.
  • GSI analysis: Figure 5 compares normalized velocity across frame IDs after linear interpolation and Gaussian-smoothed interpolation for six DeepSORT trajectories.The figure uses frame ID on the x-axis and normalized velocity on the y-axis.

C. Main Results

StrongSORT++ is evaluated against state-of-the-art trackers on MOT17, MOT20, DanceTrack, and KITTI. It ranks highly across benchmarks, with particularly strong association results, while detection performance is weaker on MOT20 under the shared threshold setting.

  • MOT17: StrongSORT++ ranks first on MOT17 for HOTA, IDF1, AssA, and DetA, and second for MOTA and IDs.It exceeds the second-best tracker by +2.1 IDF1 and +2.1 AssA.
  • Overall comparison: The reproduced DeepSORT baseline also performs well when equipped with YOLOX and tuned hyperparameters, supporting the effectiveness of the DeepSORT-like tracking paradigm.The benchmark comparisons cover MOT17, MOT20, DanceTrack, and KITTI.
  • MOT20: StrongSORT++ ranks first on MOT20 for HOTA, IDF1, and AssA and achieves significantly fewer IDs than other trackers.Its MOTA and DetA are slightly weaker, with 117,920 false negatives versus 87,594 for ByteTrack.
  • DanceTrack: StrongSORT++ achieves the best results on DanceTrack for most metrics using the same detections as ByteTrack.Appearance-related BoT and EMA optimizations are omitted because DanceTrack emphasizes motion analysis over visual discrimination.
  • KITTI: On KITTI, StrongSORT++ achieves comparable car results and superior pedestrian performance relative to PermaTrack.The KITTI configuration uses ECC, NSA Kalman, AFLink, and GSI with shared detection results from PermaTrack and OC-SORT.

D. Qualitative Results.

StrongSORT++ demonstrates robust tracking across normal, occluded, and moving-camera scenarios on MOT17, MOT20, DanceTrack, and KITTI. However, the tracker remains relatively slow and cannot correct false associations caused by mixed-up ID trajectories.

  • StrongSORT++ handles normal scenes, post-occlusion association, and moving-camera sequences in the MOT17 qualitative results.MOT17-01 represents normal scenarios, MOT17-08 shows correct associations after occlusion, and MOT17-14 uses a moving camera.
  • StrongSORT++ performs well under severe occlusion in the MOT20-04 qualitative results.
  • The DeepSORT-like design has relatively low running speed because it requires an extra detector and appearance model.
  • The visualization covers test sets from MOT17, MOT20, DanceTrack, and KITTI, with box colors representing object IDs.
  • AFLink restores missing associations but cannot split mixed-up ID trajectories into accurate tracklets.
  • StrongSORT++ achieves state-of-the-art results on MOT17, MOT20, DanceTrack, and KITTI after integrating StrongSORT with AFLink and GSI.
Loading 2202.13514v2…