Source-linked AI summary
Deep Learning in Video Multi-Object Tracking: A Survey
Gioele Ciaparrone, Francisco Luque Sánchez, Siham Tabik, Luigi Troiano, Roberto Tagliaferri, Francisco Herrera
TL;DR
MOT must track multiple unknown targets through video despite occlusions, interactions, and similar appearances. This survey reviews deep-learning methods for 2D single-camera MOT across four pipeline steps and compares them on MOTChallenge datasets, finding recurring patterns in top-performing approaches. It also identifies detection errors and robustness as continuing research boundaries.
Problem
MOT must distinguish and track multiple targets that can enter, leave, occlude one another, or share similar appearances, challenges that make direct single-object tracking unreliable.
Method
The paper surveys deep-learning MOT algorithms for 2D single-camera video, organizing methods by detection, feature extraction, affinity computation, and association.
Results
Top-performing methods commonly use high-quality detection, CNN appearance features with motion information, adapted SOT trackers online, or deep models within global graph optimization.
Takeaways & Limitations
Detection quality and appearance features are central to tracker performance, while online and batch methods benefit from different integrations of deep models.
Takeaways & Limitations
Modern detectors still produce substantial false negatives and false positives in complex dense-tracking scenarios, motivating strategies that reduce reliance on detections.
Abstract
from arXiv · showhide
The problem of Multiple Object Tracking (MOT) consists in following the trajectory of different objects in a sequence, usually a video. In recent years, with the rise of Deep Learning, the algorithms that provide a solution to this problem have benefited from the representational power of deep models. This paper provides a comprehensive survey on works that employ Deep Learning models to solve the task of MOT on single-camera videos. Four main steps in MOT algorithms are identified, and an in-depth review of how Deep Learning was employed in each one of these stages is presented. A complete experimental comparison of the presented works on the three MOTChallenge datasets is also provided, identifying a number of similarities among the top-performing methods and presenting some possible future research directions.
1 Introduction
MOT identifies and tracks multiple unknown targets in video while assigning persistent IDs, a task challenged by occlusions and similar appearances. This survey reviews deep-learning approaches for single-camera 2D MOT, organizes them across four pipeline steps, compares benchmark results, and discusses future directions.
- 1 Introduction: MOT tracks objects across video frames without prior knowledge of their appearance or number, assigning each detection a target ID to distinguish intra-class objects.Unlike detection alone, MOT associates bounding boxes with identities over time.
- 1 Introduction: Occlusions, object interactions, and similar appearances make direct application of single-object trackers unreliable, causing target drift and ID switches.These challenges motivated algorithms specifically designed for multi-target tracking.
- 1 Introduction: Deep learning contributes rich representations and complex feature extraction, with CNNs used for spatial patterns and LSTMs for sequential data.The survey frames these capabilities as the motivation for applying deep models to MOT.
- 1 Introduction: The survey comprehensively reviews deep-learning methods for 2D data from single-camera videos, including recent works absent from earlier reviews.Earlier surveys focused on MOT generally, RGB-D tracking, or formulations where deep learning was not the central topic.
- 1 Introduction: It identifies four common MOT steps, summarizes the deep-learning techniques used in each, compares results on commonly used datasets, and discusses future research directions.The technique tables also include available source-code links as a reference for future work.
2 MOT: algorithms, metrics and datasets
MOT commonly follows a tracking-by-detection workflow that detects objects, extracts features and motion predictions, computes affinities, and associates detections with target identities. Standard metrics quantify tracking errors and detection quality, while batch and online settings impose different information and speed constraints.
- 2.1 Introduction to MOT algorithms: Tracking-by-detection formulates MOT as associating frame-level bounding-box detections so boxes containing the same target receive the same identity.Modern detection frameworks provide the detections that guide this association process.
- 2.1 Introduction to MOT algorithms: Batch methods can use future frames and often achieve better tracking quality, whereas online methods use only present and past information and may be too slow for real-time deployment.Online operation is required in scenarios such as autonomous driving and robot navigation, but online does not necessarily mean real-time.
- 2.1 Introduction to MOT algorithms: MOT algorithms commonly detect objects, extract appearance or motion features, compute pairwise affinities, and associate detections by assigning consistent target IDs.Some methods merge or repeat stages, and others use detections to refine trajectories or manage track initialization and termination.
- 2.2 Metrics: CLEAR MOT and ID metrics provide standardized evaluation, covering errors such as false positives, false negatives, fragmentations, ID switches, trajectory coverage, and identity consistency.MOTA summarizes false positives, false negatives, and ID switches, while MOTP emphasizes bounding-box overlap and detection quality.
- 2.2 Metrics: MOTA can undervalue sustained identity tracking in applications that prioritize following an object continuously, motivating complementary global identity metrics such as IDP, IDR, and IDF1.The alternative metrics globally map computed and ground-truth trajectories to maximize correctly classified frames.
3 Deep learning in MOT
The survey organizes deep-learning applications in MOT around four common algorithmic stages and reviews how different models are used within each stage. It also summarizes techniques, operating modes, and available implementation resources for the reviewed works.
- 3 Deep learning in MOT: The survey reviews deep learning across each of the four previously identified MOT stages, including association and related track management.Track management includes operations such as track initialization and termination.
- 3 Deep learning in MOT: A summary table records the main techniques used in each MOT step, whether each method is batch or online, and available source code or other materials.The table is intended as a reference for the papers covered by the survey.
3.1 DL in detection step
Deep learning is used in MOT detection both to generate and refine object detections and, in some approaches, to provide alternatives to conventional rectangular bounding boxes. Faster R-CNN and SSD are especially common, alongside other detectors and CNN-based detection refinements.
- 3.1 DL in detection step: Custom detections often improve tracking performance by enhancing detection quality, with Faster R-CNN variants and SSD appearing frequently in MOT pipelines.Other detector families include YOLO, RRC CNN, SubCNN, and Tiny CNN.
- 3.1.1 Faster R-CNN: Replacing ACF detections with Faster R-CNN detections improved SORT’s MOTA by 18.9% on MOT15.SORT combined detection with Kalman-filter motion prediction and association.
- 3.1.1 Faster R-CNN: A modified Faster R-CNN with skip-pooling and multi-region features improved a proposed MOT algorithm’s MOTA by more than 30% on MOT16 and reached state-of-the-art performance.The detector was fine-tuned on multiple pedestrian-detection datasets, and its detections were later used by many MOT algorithms.
- 3.1.2 SSD: SSD has been compared with Faster R-CNN and R-FCN in pig tracking, where it performed better on that dataset.The tracking pipeline used DCF-based online tracking with HOG and Colour Names features.
- 3.1.3 Other uses of deep learning: CNNs can refine detection confidence, bounding-box localization, or false-positive filtering by incorporating affinities, correlation filters, or a secondary classifier.These uses include replacing SSD’s standard NMS, refining SSD regions of interest, and selectively applying a Faster-CNN network after SVM screening.
- 3.1.4 Other uses of deep learning: Some methods use deep networks to produce instance-aware semantic segmentation maps instead of conventional rectangular bounding boxes.The authors argue that instance shapes avoid including background structures or parts of other objects, potentially benefiting optical-flow tracking.
3.2 DL in feature extraction and motion prediction
Deep learning is used primarily to extract appearance features, with CNNs and Siamese CNNs providing representations for affinity computation. LSTMs additionally model nonlinear motion, while some approaches combine visual, pose, interaction, and motion information.
- CNN feature extraction: CNN-based feature extraction is the preferred use of deep learning because deep models learn meaningful high-level representations.Siamese CNNs are commonly trained with contrastive or related losses to learn discriminative features.
- CNN feature extraction: Autoencoder refinement, pretrained CNN features, and custom CNNs were integrated into classical trackers to improve visual representations.One early method refined natural-scene features with stacked autoencoders before SVM affinity computation and minimum-spanning-tree association.
- CNN feature extraction: DeepSORT adds a custom residual CNN whose normalized 128-feature output contributes cosine-distance affinity scores and reduces SORT’s ID-switch problem.The visual distance is combined with the original SORT affinity scores.
- Motion prediction: LSTMs are used for motion prediction and for combining appearance, pose, interaction, and trajectory information in more complex feature models.Typical motion-prediction systems feed sequences of bounding boxes into an LSTM and output the next-frame bounding box.
- Siamese networks: Siamese CNNs can serve either as feature extractors or direct affinity models, depending on whether inference discards the output probability.Feature-extraction variants use the final fully connected layer as a candidate vector, whereas affinity variants retain the full network.
3.3 DL in affinity computation
Deep learning models compute affinities either indirectly through learned features and explicit distances or directly through recurrent and convolutional networks. RNN-based approaches also address online association, track existence, and training objectives aligned with tracking metrics.
- Overview: Affinity computation may use explicit distances over CNN features or deep models that directly output affinity scores.The section covers recurrent networks, CNNs, and Siamese CNNs as direct affinity mechanisms.
- Recurrent neural networks and LSTMs: Milan et al. use an end-to-end RNN tracker with separate motion-prediction and association components, while another component predicts track birth and death.The association LSTM receives distances between predicted states and detections and outputs association probabilities.
- Recurrent neural networks and LSTMs: Multiple-LSTM systems combine learned appearance, motion, pose, or interaction signals into affinity scores for association.A pose-based triple-stream network separately models appearance, motion, and interaction similarity before combining the outputs.
- Uses of LSTMs in MHT frameworks: Maksai et al. train an RNN tracklet score as a proxy for IDF1 and expose the model to generated errors through hard examples and tracklet perturbations.The approach targets both loss-evaluation mismatch and exposure bias in recurrent MOT training.
- Other affinity models: Other methods use LSTMs to compute appearance affinity from ResNet features or combine learned affinity signals with spatial, temporal, and detection-confidence features.The resulting scores can be integrated with MHT or weighted feature-based association systems.
3.4 DL in Association/Tracking step
Deep learning supports the association and tracking stage by managing track status, splitting unreliable tracklets, computing confidence, and coordinating association decisions. These uses include RNNs, MLPs, and reinforcement-learning agents alongside classical tracking procedures.
- Overview: Deep learning models improve association with classical algorithms and manage track status, including decisions to start or terminate tracks.The survey groups these approaches into RNNs, deep MLPs, and deep reinforcement-learning agents.
- Recurrent neural networks: A recurrent model predicts each track’s existence probability to help determine when tracks should be initiated or terminated.This mechanism is part of Milan et al.’s broader RNN-based tracking system.
- Recurrent neural networks: A bidirectional GRU identifies where tracklets should be split to correct ID-switch errors introduced by occlusions.The method first generates high-confidence tracklets using NMS and Hungarian association with appearance and motion affinities.
- Deep multi-layer perceptrons: An MLP computes track-confidence scores from previous track scores and recent detection information, using them to manage track termination.Inputs include association scores and detection confidence.
- Deep reinforcement learning: Deep reinforcement-learning agents can influence Kalman-filter actions and make track-management decisions, while collaborative agents can also guide association.The surveyed systems include agents that decide whether to use predictions, measurements, or both.
3.5 Other uses of DL in MOT
Several deep-learning methods fall outside the survey’s four-step MOT scheme, including post-processing, likelihood estimation, auxiliary detections, adaptive models, and unified affinity–trajectory systems. These approaches attach deep models to broader probabilistic or tracking frameworks.
- Other uses: Deep reinforcement learning can perform bounding-box regression after tracking, independently of the underlying MOT algorithm.The procedure uses VGG-16 appearance features and the history of recent actions.
- Other uses: CNN detector ensembles and likelihood models can support probabilistic trajectory estimation, track birth and death, and particle-filter tracking.These systems use methods such as Markov Chain Monte Carlo or multi-Bernoulli filtering to construct track segments or weight particles.
- Auxiliary detections: Head detections can supplement body detections by helping distinguish true from false pedestrian boxes before graph-based association.Henschel et al. solve the resulting correlation-clustering problem with a modified Frank-Wolfe algorithm.
- Unified models: MetricNet unifies appearance affinity and Bayesian trajectory estimation using a person-reidentification CNN, an LSTM motion feature extractor, and fully connected layers.The model combines visual and motion components within one tracking framework.
- Adaptive tracking models: Adaptive CNN architectures can separate background and targets and decide when to refresh a tracking model, while target-specific layers capture appearance changes online.Chu et al. use PafNet, PartNet, and a refresh-decision CNN; Gan et al. use shared convolutional layers with target-specific fully connected layers.
4 Analysis and comparisons
The survey compares deep-learning MOT methods on MOTChallenge using whole test sets, separating public from private detections and online from batch methods. Results emphasize detection quality, false-negative reduction, appearance and motion features, and trade-offs in SOT-based and affinity-based approaches.
- Experimental setup: The comparison uses whole MOTChallenge test sets and separates algorithms by public versus private detections, with online and batch methods grouped separately.Results are organized across MOT15, MOT16, and MOT17 according to dataset and detection source.
- General observations: 56.5% MOTA vs. 42.3% for MOT15 and 71.0% vs. 49.3% for MOT16 shows that private detections achieve the best dataset-level performance.The survey attributes this pattern to detection quality dominating overall tracker performance.
- General observations: MOTA correlates strongly with false negatives, with Pearson coefficients of −0.95 on MOT15, −0.98 on MOT16, and −0.95 on MOT17.The analysis reports that false negatives typically outnumber false positives and ID switches substantially.
- Best approaches in the four MOT steps: Top-performing methods consistently use CNN appearance features, often alongside motion features from LSTMs, Kalman Filters, or Bayesian filters.Appearance features also support avoiding ID switches and re-identifying targets after long occlusions.
- Best approaches in the four MOT steps: Deep affinity networks can produce top-performing methods, but they are less ubiquitous and have not been shown essential for strong MOT performance.Reported approaches include Siamese CNNs, recurrent networks, and body-part attention for more accurate similarity measures.
- Other trends in top-performing algorithms: SOT-based online trackers can perform well after modification for occlusion recovery or model refreshing, but remain prone to drift and ID switches.The survey identifies applying SOT trackers with private detections as a possible research direction while noting their tracking-drift risk.
- Other trends in top-performing algorithms: Batch methods benefit from global graph optimization, with lifted multicut, heterogeneous association graph fusion, and correlation clustering appearing in top-performing methods.CNN-computed affinities support the minimum cost lifted multicut approach on MOT16.
- Other trends in top-performing algorithms: Bounding-box accuracy materially affects final performance, and the survey highlights deep reinforcement-learning-based box regression as a future research direction.A top MOT15 tracker achieved a relatively high MOTA score by regressing boxes from a previous state-of-the-art algorithm.
5 Conclusion and future directions
The survey identifies four MOT pipeline steps and finds that detection quality, CNN-based appearance features, and complementary motion modeling characterize strong approaches. It also highlights research directions spanning robustness, broader targets, association, error mitigation, post-processing, and reproducibility.
- The generic MOT pipeline comprises detection, feature extraction, affinity computation, and association.
- Detection quality strongly affects MOTA because false negatives remain a dominant error source.Higher-quality detections are described as the most effective way to reduce false negatives, while dense scenarios still produce false positives and false negatives.
- Strong trackers commonly combine CNN-based appearance features with motion features from LSTMs, Kalman filters, or other Bayesian filters.
- Deep learning has supported effective online SOT-based trackers and batch trackers integrated with global graph optimization.
- Future work includes robustness across camera conditions and target types, deeper use of learning for association, bounding-box regression, SOT trackers with private detections, and batch post-tracking correction.The survey also encourages public code to improve reproducibility.
A Appendix
The appendix summarizes the techniques used by the surveyed algorithms, organizing methods by MOT pipeline step and indicating implementation details such as operating mode and code availability.
- The appendix table summarizes the techniques used by each algorithm in the survey, following the papers’ order of presentation.
- The surveyed methods combine detection, feature extraction, motion or appearance modeling, association, and other tracking components.
- Examples include CNN-based features, Kalman filtering, LSTMs, correlation filters, Hungarian assignment, and custom or graph-based association procedures.
- The table records online or batch operation and indicates whether source code is available for the listed methods.
- The notation distinguishes appearance, motion, and feature components, while the final column provides clickable links to specified data.