PaperlayerView exact revision on arXiv

Source-linked AI summary

Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking

Jinkun Cao, Jiangmiao Pang, Xinshuo Weng, Rawal Khirodkar, Kris Kitani

arXiv:2203.14360v3cs.CV

TL;DR

SORT-based tracking is vulnerable when observations disappear during occlusion or when object motion is non-linear. OC-SORT addresses this by using observations to correct accumulated Kalman-filter errors, while retaining simple, online, real-time tracking and improving robustness across diverse datasets.

  • Problem

    SORT and related motion-based trackers suffer from state-estimation noise, error accumulation without observations, and an estimation-centric design under occlusion and non-linear motion.

  • Method

    OC-SORT uses Observation-centric Re-Update with virtual observations and adds Observation-Centric Momentum to association costs.

  • Results

    OC-SORT achieves state-of-the-art performance on multiple datasets while remaining online and real-time.

  • Takeaways & Limitations

    Observation-centric correction improves tracking robustness, especially for occlusion and non-linear object motion, without abandoning a simple tracker design.

  • Takeaways & Limitations

    The analysis and method remain tied to the limitations and assumptions of SORT’s motion-model framework, whose foundation is examined through three connected limitations.

Abstract

from arXiv · show

Kalman filter (KF) based methods for multi-object tracking (MOT) make an assumption that objects move linearly. While this assumption is acceptable for very short periods of occlusion, linear estimates of motion for prolonged time can be highly inaccurate. Moreover, when there is no measurement available to update Kalman filter parameters, the standard convention is to trust the priori state estimations for posteriori update. This leads to the accumulation of errors during a period of occlusion. The error causes significant motion direction variance in practice. In this work, we show that a basic Kalman filter can still obtain state-of-the-art tracking performance if proper care is taken to fix the noise accumulated during occlusion. Instead of relying only on the linear state estimate (i.e., estimation-centric approach), we use object observations (i.e., the measurements by object detector) to compute a virtual trajectory over the occlusion period to fix the error accumulation of filter parameters during the occlusion period. This allows more time steps to correct errors accumulated during occlusion. We name our method Observation-Centric SORT (OC-SORT). It remains Simple, Online, and Real-Time but improves robustness during occlusion and non-linear motion. Given off-the-shelf detections as input, OC-SORT runs at 700+ FPS on a single CPU. It achieves state-of-the-art on multiple datasets, including MOT17, MOT20, KITTI, head tracking, and especially DanceTrack where the object motion is highly non-linear. The code and models are available at \url{https://github.com/noahcao/OC_SORT}.

1. Introduction

The paper identifies limitations in SORT’s linear, estimation-centric motion handling and proposes OC-SORT, which uses observations to improve robustness under occlusion and non-linear motion.

  • SORT assumes approximately linear target motion, but this assumption breaks in practical scenarios involving occlusion and non-linear motion.
  • SORT accumulates state-estimation noise when observations are unavailable, causing tracking errors during occlusion.
  • OC-SORT uses virtual observations from historical and reactivating detections to correct accumulated errors through Observation-centric Re-Update.
  • Observation-Centric Momentum adds motion-direction consistency to the association cost.
  • OC-SORT remains simple, online, and real-time while improving robustness to occlusion and non-linear motion.

2. Related Works

Prior MOT work uses probabilistic association, detector-based tracking, and motion models, with Kalman-filter-based approaches remaining common despite limitations for non-linear motion.

  • Motion Models: Kalman filters and related motion models use Bayesian estimation to predict object states, but nonlinear variants still require motion-pattern assumptions.
  • Motion Models: Particle filters address non-linear motion through sampling-based posterior estimation but require exponential-order computation.
  • Multi-object Tracking: Modern multi-object tracking commonly combines object detectors with motion-based association, as exemplified by SORT and DeepSORT.

3. Rethink the Limitations of SORT

SORT’s predict-update design relies on Kalman-filter estimations when observations are missing, making its motion estimates increasingly sensitive to noise and temporal error accumulation.

  • Preliminaries: Kalman filtering maintains posterior state and covariance estimates through predict and update stages based on a transition model and observations.
  • Preliminaries: SORT uses detector measurements for updates, but when observations are absent it performs a dummy update by copying priori estimations into posterior parameters.
  • Estimation-centric Tracking: Estimation-centric tracking can become unreliable when nonlinear motion and occlusion occur together because it trusts state estimations without observations.
  • Sensitive to State Noise: δ ˙u and δ ˙v have variances proportional to 1/(∆t)^2, so small frame intervals amplify velocity-estimation noise.
  • Sensitive to State Noise: A one-pixel position shift can substantially vary estimated speed when consecutive-frame object displacement is only a few pixels.
  • Observation-Centric Correction: OC-SORT’s pipeline uses observation-centric re-update after reactivation to correct filter parameters over the previously untracked interval.
  • Temporal Error Magnification: During occlusion, propagated position noise grows with T^2; a 10-frame occlusion can shift the estimate by roughly a pedestrian’s size on MOT17.

4. Observation-Centric SORT

OC-SORT makes SORT observation-centric by correcting filter errors accumulated during untracked periods and incorporating motion direction into association. Its design includes ORU for re-activated tracks, OCM for association, and OCR for short-term recovery.

  • OC-SORT uses object momentum in association to improve robustness over occlusion and non-linear motion.
  • Observation-centric Re-Update (ORU): ORU backchecks an untracked period after re-activation and applies virtual observations to reduce accumulated filter error.The virtual trajectory is anchored by the last-seen and latest associated real observations, followed by predict and re-update operations.
  • Observation-centric Momentum (OCM): OCM adds a direction-consistency term to the association cost alongside negative pairwise IoU.The direction term compares the direction along an existing track with the direction toward a new observation, using an angle difference.
  • Observation-centric Momentum (OCM): Direction-estimation noise decreases as the time difference between the two observation points increases under the linear-motion model.The method uses this direction information while accounting for noise that is significant at small time differences.
  • Observation-centric Recovery (OCR): OCR makes a second association attempt from the last observations of unmatched tracks to unmatched detections.It is intended to handle objects that stop or are occluded for a short interval.

5. Experiments

OC-SORT is evaluated across diverse tracking benchmarks, including crowded, low-frame-rate, occluded, and highly nonlinear-motion settings. It achieves strong benchmark performance while retaining real-time efficiency, with ablations clarifying the roles and limits of its components.

  • Experimental Setup: OC-SORT is evaluated on MOT17, MOT20, KITTI, DanceTrack, and CroHD using shared or publicly available detections for fair comparisons.The datasets cover pedestrian, car, and head tracking, including crowded scenes, low frame rates, and highly nonlinear motion.
  • Benchmark Results: 62.1 HOTA on MOT20 is reported for OC-SORT under severe pedestrian occlusion, with performance comparable to other state-of-the-art methods on MOT17 and MOT20.The comparison uses the same detections as ByteTrack, while keeping the detection confidence threshold consistent across datasets.
  • Benchmark Results: OC-SORT sets a new state-of-the-art on DanceTrack, outperforming baselines by a great margin under highly nonlinear object motion.DanceTrack also includes close appearances, severe occlusion, and frequent crossovers, making it especially relevant to the method’s design goal.
  • Benchmark Results: OC-SORT improves pedestrian tracking on KITTI to a new state-of-the-art and achieves a large performance gap over PermaTr with 10x faster speed using the same detections.The default IoU association is a shortcoming for high-velocity or low-frame-rate car tracking, where consecutive bounding-box overlap can be very low or zero.
  • Efficiency: 793 FPS on an Intel i9-9980XE CPU is reported for OC-SORT with off-the-shelf detections, supporting online and real-time operation.The experiments use a shared parameter stack across datasets, although careful tuning could further improve performance.
  • Ablation Study: ORU provides significant gains on MOT17 and DanceTrack, while OCM helps especially on DanceTrack, where motion is more complicated and occlusion is heavier.The ablation also finds that increasing OCM’s ∆t from 1 can improve association, but values beyond the bottleneck hurt performance because linear motion becomes harder to maintain.

6. Conclusion

OC-SORT addresses SORT’s weaknesses under occlusion and nonlinear motion while retaining simple, online, real-time operation. Experiments across diverse datasets show substantial tracking gains, especially in these challenging conditions.

  • OC-SORT improves robustness to occlusion and nonlinear object motion while remaining simple, online, and real-time.
  • The method significantly outperforms state-of-the-art trackers across diverse datasets, with especially significant gains under occlusion and nonlinear motion.
  • SORT’s intrinsic Kalman-filter limitations hurt tracking accuracy when observations are unavailable for supervision.

A. Velocity Direction Variance in OCM

The analysis models velocity-direction estimation from two noisy observations under linear motion. It shows that larger temporal displacement reduces direction-estimation noise, while small displacement produces shifted and dispersed estimates.

  • Under linear motion with noisy states, velocity direction is estimated from two observations whose displacement differences form a jointly Gaussian pair.
  • Larger µw or µy concentrates the probability density near the true ratio µz = µy/µw and makes the tails decay faster.
  • Selecting observations with a larger temporal difference reduces the noise of arctan θ and improves direction estimation under linear motion.
  • With independent unit detection noise, the visualization shows that small µy or µw produces a less reliable distribution around the true value.

B. Interpolation by Gaussian Progress Regression

This section examines Gaussian Process Regression for generating virtual trajectories and interpolation in SORT. Although GPR offers a nonlinear trajectory-fitting option, the supplied discussion reports unsuccessful online gains and motivates careful kernel and interpolation design.

  • The section investigates whether Gaussian Process Regression can extend SORT beyond its linear-motion assumption for nonlinear trajectories.
  • GPR defines a Gaussian predictive function from observations and a kernel, producing a mean and variance for the queried output.
  • The authors report that GPR did not successfully improve tracking when used for online virtual-trajectory generation or offline interpolation.
  • B.1. Choice of Kernel Function in Gaussian Process: The RBF kernel uses a lengthscale l controlling the target function’s wiggles and an output variance σ^2 acting mainly as a scale factor.
  • B.1. Choice of Kernel Function in Gaussian Process: GPR is sensitive to kernel choice in some situations, so the authors evaluate different lengthscale settings for offline interpolation.

B.2. GPR for Offline Interpolation

The offline-interpolation study compares GPR configurations with linear interpolation on MOT17 and DanceTrack validation sets. Linear interpolation remains the stronger general baseline, while a two-stage LI-plus-GPR approach helps on DanceTrack but adds computation and offers only minor gains.

  • The study evaluates GPR interpolation on MOT17 and DanceTrack validation sets for trajectories longer than 30 frames and gaps shorter than 20 frames.
  • Linear interpolation stably improves tracking across multiple metrics, whereas directly using GPR to fill missing points hurts performance.
  • Interpolating with Linear Interpolation before GPR smoothing outperforms LI on DanceTrack but underperforms LI on MOT17.
  • The authors attribute GPR’s interpolation difficulty to limited trajectory data and discontinuities caused by missing intermediate points.
  • Because the improvement over linear interpolation is minor and GPR has heavier computational overhead, the authors do not recommend it for most MOT tasks.

C. Results on More Benchmarks

OC-SORT remains competitive across additional tracking benchmarks and public-detection settings, including crowded scenes with severe occlusion.

  • Results on HeadTrack: HeadTrack evaluates crowded head-tracking scenarios where focusing on partial objects can reduce occlusion compared with full-body tracking.Fine-grained part trajectories may support downstream action-recognition and forecasting tasks.
  • Public Tracking on MOT17 and MOT20: The public-detection results are reported in Table 12 for MOT17 and Table 13 for MOT20, with LI denoting Linear Interpolation.These tables provide the stated public-tracking evaluation context.
  • Public Tracking on MOT17 and MOT20: OC-SORT outperforms existing state-of-the-art methods on MOT17 and MOT20 using public detections.The improvement is reported as more significant on MOT20, which contains more severe occlusion scenes.

E. More Results on DanceTrack

On DanceTrack, OC-SORT is evaluated with detections trained only on MOT17 and already exceeds the listed baselines; visual comparisons show greater resilience to motion-related tracking failures.

  • More Results on DanceTrack: Visual comparisons show SORT suffering ID switches or fragmentation from nonlinear motion or occlusion while OC-SORT survives.Additional trajectory visualizations compare SORT and OC-SORT on the DanceTrack validation set.
  • More Results on DanceTrack: OC-SORT achieves higher tracking performance than the baselines on DanceTrack using a YOLOX detector trained only on MOT17.DanceTrack is intended to encourage improved association rather than detector-specific tuning.
  • More Results on DanceTrack: OC-SORT is motion-based but can incorporate appearance information, with reported improvements on MOT17, MOT20, and DanceTrack.The appearance-augmented variant is referred to as Deep OC-SORT.

G. More Discussion of State Noise Sensitivity

The analysis attributes SORT’s temporal noise sensitivity to noisy state and velocity estimates, while its process-noise convention suppresses velocity noise but limits rapid direction changes after occlusion.

  • State Noise Sensitivity: Velocity-estimation noise can accumulate into position error over time, with the general analysis accounting for transition noise in velocity terms.The paper notes that accumulation is faster than in its simplified analysis.
  • State Noise Sensitivity: SORT’s state-noise analysis considers noisy Kalman predictions whose process noise affects both state and velocity estimates.The state is represented as [u, v, s, r, u_dot, v_dot, s_dot].
  • State Noise Sensitivity: SORT commonly suppresses velocity noise through process-noise settings, treating velocity noise as secondary for typical deletion horizons below ten steps.This convention creates a side effect: SORT cannot quickly change a track’s velocity direction.
  • Intuition behind ORU: After occlusion, SORT may fail to steer toward the correct velocity direction because small velocity covariance makes re-associated observations insufficiently effective.ORU addresses this setting with historical virtual observations after re-association.
  • Intuition behind ORU: The constant-process-noise assumption conflicts with using small noise for consistent high-frame-rate observations when observations are missing.During missing observations, the direction difference between the true trajectory and the maintained linear direction grows.
  • Intuition behind ORU: The paper demonstrates the difference between SORT and OC-SORT empirically rather than providing a theoretical proof for re-association outcomes.A deeper theoretical comparison would require additional assumptions about true trajectories and observations.
Loading 2203.14360v3…