Source-linked AI summary
Simple Online and Realtime Tracking with a Deep Association Metric
Nicolai Wojke, Alex Bewley, Dietrich Paulus
TL;DR
Online tracking must preserve target identities despite occlusions, where SORT produces relatively many identity switches. This paper adds a pretrained deep appearance association metric to SORT, yielding fewer identity switches and competitive online-tracking performance while retaining real-time operation.
Problem
SORT produces relatively many identity switches and struggles to track targets through occlusions in online multiple-object tracking.
Method
The method extends SORT with a pretrained CNN appearance metric that combines visual appearance and motion for frame-by-frame measurement-to-track association.
Results
The tracker returns the fewest identity switches among online methods while maintaining competitive MOTA, fragmentation, and false-negative scores.
Takeaways & Limitations
Appearance information enables SORT to maintain identities through longer occlusions while remaining simple to implement and real time.
Takeaways & Limitations
The tracker assumes an uncalibrated camera with no ego-motion information and uses a constant-velocity Kalman-filter model.
Abstract
from arXiv · showhide
Simple Online and Realtime Tracking (SORT) is a pragmatic approach to multiple object tracking with a focus on simple, effective algorithms. In this paper, we integrate appearance information to improve the performance of SORT. Due to this extension we are able to track objects through longer periods of occlusions, effectively reducing the number of identity switches. In spirit of the original framework we place much of the computational complexity into an offline pre-training stage where we learn a deep association metric on a large-scale person re-identification dataset. During online application, we establish measurement-to-track associations using nearest neighbor queries in visual appearance space. Experimental evaluation shows that our extensions reduce the number of identity switches by 45%, achieving overall competitive performance at high frame rates.
1. INTRODUCTION
Recent progress in object detection has made tracking-by-detection the leading paradigm for multiple-object tracking.
- Recent progress in object detection has made tracking-by-detection the leading paradigm in multiple-object tracking.
tracking.
SORT is an online tracking framework that combines Kalman filtering with frame-by-frame Hungarian data association using bounding-box overlap. Although it performs favorably at high frame rates, its association metric causes relatively many identity switches and weak occlusion tracking, motivating a motion-and-appearance metric.
- Online tracking: Batch-optimized flow networks and graphical models cannot support online tracking when target identities must be available at each time step.These methods process entire video batches at once.
- SORT framework: SORT performs Kalman filtering in image space and frame-by-frame Hungarian association using bounding-box overlap.This simple approach achieves favorable performance at high frame rates.
- SORT performance: SORT with a state-of-the-art people detector ranks on average higher than MHT on standard MOT challenge detections.The result underscores the influence of object detector performance on overall tracking performance.
- SORT limitations: SORT returns a relatively high number of identity switches because its association metric is accurate only when state-estimation uncertainty is low.This limitation makes tracking through typical frontal-view camera occlusions difficult.
- Proposed improvement: The proposed method addresses SORT’s occlusion weakness by replacing bounding-box overlap with a more informed metric combining motion and appearance information.The supplied passage introduces this replacement as the solution to SORT’s deficiency, but does not provide further implementation details here.
2. SORT WITH DEEP ASSOCIATION METRIC
The method extends SORT with recursive Kalman filtering, frame-by-frame association, and a deep appearance metric. Motion and appearance cues are combined through gated matching and an age-prioritized cascade to improve association during occlusions.
- Tracking framework: SORT uses a conventional single-hypothesis tracker with recursive Kalman filtering and frame-by-frame data association.The framework largely follows the original SORT formulation and assumes an uncalibrated camera without ego-motion information.
- Association metrics: Motion association uses squared Mahalanobis distance, accounting for state uncertainty and rejecting unlikely matches at a 95% confidence interval.For the four-dimensional measurement space, the Mahalanobis threshold is t(1) = 9.4877.
- Association metrics: Appearance association uses the smallest cosine distance between CNN descriptors, complementing motion cues when camera motion or long-term occlusion makes location predictions unreliable.The CNN is pretrained on a person re-identification dataset containing over 1,100,000 images of 1,261 pedestrians.
- Association metrics: λ = 0 uses only appearance in the association cost under substantial camera motion, while retaining the Mahalanobis gate to reject infeasible assignments.The Mahalanobis gate still constrains assignments based on possible object locations inferred by the Kalman filter.
- Matching cascade: The matching cascade solves assignments by increasing track age, prioritizing tracks seen more recently and addressing uncertainty growth after prolonged occlusion.A final IoU stage handles unconfirmed and unmatched tracks of age n = 1, helping with sudden appearance changes and erroneous initialization.
3. EXPERIMENTS
Experiments on MOT16 show that integrating appearance information helps maintain identities through longer occlusions while preserving competitive tracking performance. The tracker remains computationally efficient, operating at approximately 20 Hz in real time.
- Evaluation setup: The tracker is evaluated on the MOT16 benchmark using detections from Yu et al., with SORT rerun on the same detections for fair comparison.MOT16 contains seven challenging test sequences, including moving-camera frontal views and top-down surveillance scenes.
- Evaluation setup: Evaluation uses λ = 0, Amax = 30 frames, and detections thresholded at a confidence score of 0.3.The remaining parameters are selected using separate benchmark training sequences.
- Tracking results: Appearance information maintains object identities through longer occlusions, increasing mostly tracked objects while decreasing mostly lost objects.Track fragmentation increases slightly because identities are maintained through occlusions and misses.
- Tracking results: The method achieves the fewest identity switches among online methods while maintaining competitive MOTA, track fragmentation, and false negatives.Tracking accuracy is mainly impaired by more false positives, which may be reduced by increasing the detection confidence threshold.
- Runtime: 20 Hz is the approximate runtime, with roughly half of processing time spent generating features.With a modern GPU, the system remains computationally efficient and operates in real time.
4. CONCLUSION
The paper extends SORT with appearance information via a pre-trained association metric, enabling longer-occlusion tracking while remaining simple to implement and real-time.
- The extension incorporates appearance information through a pre-trained association metric.This improves SORT’s ability to track through longer periods of occlusion.
- The resulting tracker remains simple to implement and runs in real time.The authors present it as a strong competitor to state-of-the-art online tracking algorithms.