PaperlayerView exact revision on arXiv

Source-linked AI summary

Multiple Object Tracking: A Literature Review

Wenhan Luo, Junliang Xing, Anton Milan, Xiaoqin Zhang, Wei Liu, Tae-Kyun Kim

arXiv:1409.7618v5cs.CV

TL;DR

Multiple Object Tracking involves challenges that complicate tracking objects across videos. This paper reviews various aspects of MOT, examines existing results, and discusses research issues and future directions, while benchmark analyses report that offline methods generally outperform online ones on most metrics.

  • Problem

    MOT remains complicated by key issues beyond common single-object-tracking challenges, including the non-rigid nature of pedestrians.

  • Method

    The paper provides a review discussing various aspects of multiple object tracking and examines publicly available algorithms for comparison.

  • Results

    Offline methods generally outperform online ones with respect to most metrics in benchmark comparisons.

  • Takeaways & Limitations

    Benchmark performance improves over the years, although this may reflect over-fitting to the dataset rather than general progress toward solving MOT.

  • Takeaways & Limitations

    Fair comparison is constrained when studies differ in their detection modules or do not describe those modules.

Abstract

from arXiv · show

Multiple Object Tracking (MOT) has gained increasing attention due to its academic and commercial potential. Although different approaches have been proposed to tackle this problem, it still remains challenging due to factors like abrupt appearance changes and severe object occlusions. In this work, we contribute the first comprehensive and most recent review on this problem. We inspect the recent advances in various aspects and propose some interesting directions for future research. To the best of our knowledge, there has not been any extensive review on this topic in the community. We endeavor to provide a thorough review on the development of this problem in recent decades. The main contributions of this review are fourfold: 1) Key aspects in an MOT system, including formulation, categorization, key principles, evaluation of MOT are discussed; 2) Instead of enumerating individual works, we discuss existing approaches according to various aspects, in each of which methods are divided into different groups and each group is discussed in detail for the principles, advances and drawbacks; 3) We examine experiments of existing publications and summarize results on popular datasets to provide quantitative and comprehensive comparisons. By analyzing the results from different perspectives, we have verified some basic agreements in the field; and 4) We provide a discussion about issues of MOT research, as well as some interesting directions which will become potential research effort in the future.

1. Introduction

MOT requires locating multiple objects, maintaining their identities, and producing trajectories, but remains difficult because of occlusions, changing object counts, similar appearances, and interactions. This review organizes MOT methods and evaluations comprehensively, compares reported results, and identifies open research directions.

  • MOT tracks multiple objects, maintains their identities, and yields individual trajectories from an input video.
  • Pedestrian tracking is the review’s main focus because pedestrians are non-rigid and account for at least 70% of current MOT research efforts.
  • MOT is complicated by frequent occlusions, track initialization and termination, similar appearance, and interactions among multiple objects.
  • The review provides a unified formulation, categorization schemes, and detailed discussions of MOT components including appearance, motion, interaction, exclusion, occlusion, and inference models.
  • It presents experimental results from existing publications on public datasets for quantitative comparisons and discusses current issues and future research directions.

2. MOT Problem

The review formulates MOT as estimating the sequential states of multiple objects from image-sequence observations, then organizes methods using complementary categorization criteria. It distinguishes approaches by initialization, processing mode, and output type, while highlighting practical differences between DBT and DFT and between online and offline tracking.

  • Problem Formulation: MOT is formulated as a multi-variable estimation problem that seeks the optimal sequential states of all objects given observations.Existing algorithms can be viewed as solving a MAP estimation problem, probabilistically or through deterministic optimization.
  • MOT Categorization: The review categorizes MOT methods by initialization method, processing mode, and type of output.These criteria follow how a task is initialized, processed, and produces results.
  • Initialization Method: Detection-Based Tracking detects objects in each frame and links detection hypotheses into trajectories, whereas Detection-Free Tracking manually initializes fixed objects and localizes them subsequently.DBT is also called tracking-by-detection; DFT does not accommodate newly appearing objects but does not require a pretrained object detector.
  • Initialization Method: The review notes that DBT performance depends strongly on the employed detector, which is trained in advance and may target only specific object classes.This dependence can make detection results for a particular video nonoptimal when the detector was not trained for that video.
  • Processing Mode: Online tracking uses only information available through the current frame, while offline tracking jointly uses observations from past and future frames.Offline processing may be split into shorter clips when handling all frames at once exceeds computational or memory limits.
  • Type of Output: Deterministic tracking produces constant outputs across repeated runs, whereas stochastic tracking can vary because inference involves randomness.Particle-filter inference and randomized particle generation are given as examples of stochastic behavior.

3. MOT Components

An MOT approach must discover objects in individual frames and recover their identities across continuous frames as trajectories. Its core design issues are measuring cross-frame similarity and inferring identity from those measurements.

  • The review presents the primary components of an MOT approach and organizes online-versus-offline tracking comparisons in Table 4.
  • MOT discovers multiple objects in individual frames and recovers their identity information across continuous frames as trajectories.
  • The two major design issues are measuring similarity between objects across frames and recovering identity from those similarity measurements.
  • Similarity modeling covers appearance, motion, interaction, exclusion, and occlusion, while identity recovery involves an inference problem.

3.1. Appearance Model

Appearance models compute affinity between observations from visual representations and statistical measurements. The review covers local, region, and other features, single- and multi-cue fusion, and the trade-offs among representations.

  • An appearance model has two components: visual representation describes object features, while statistical measuring computes similarity between observations.
  • Similarity between observations i and j is represented as S_ij = F(o_i, o_j), where F measures similarity between their visual representations.
  • Visual representations include local features, region features, and other cues such as depth, occupancy maps, gait, and deep descriptors.
  • Region features: Region representations range from zero-order color histograms and raw templates to first-order HOG and up-to-second-order covariance matrices.
  • Discussion: Feature choices involve trade-offs: local features are efficient but occlusion-sensitive, covariance features are more robust but computationally costly, and depth requires additional views or measurements.
  • Multiple cues: Multiple cues can improve robustness, but fusion is nontrivial; surveyed strategies include Boosting, Concatenating, Summation, Product, and Cascading.

3.2. Motion Model

Motion models estimate future object positions to reduce the search space, commonly assuming smooth motion. The review contrasts linear smoothness models with non-linear models for freely moving targets.

  • The motion model captures object dynamics, estimates potential future positions, and reduces the search space for tracking.
  • Most motion models assume objects move smoothly, except during abrupt motions, and commonly use linear dynamics with constant velocity.
  • Linear Motion Model: Linear motion affinity can use both forward and backward motion when estimating the relationship between tracklets.
  • Smoothness can be imposed on velocity, position, or acceleration when constructing motion affinities from tracklets.
  • Non-linear Motion Model: Linear motion cannot handle some cases, motivating non-linear models that produce more accurate motion affinity between tracklets.
  • Non-linear Motion Model: For freely moving targets, a support tracklet can explain the gap between two tracklets that a linear model would assign low linking probability.

3.3. Interaction Model

Interaction models represent how objects influence one another and the environment, especially in crowds. The review covers social-force and crowd-motion-pattern models that encode group behavior and collective motion.

  • Interaction models capture how one object influences other objects, including speed, direction, and destination adjustments to avoid collisions.
  • Social force models: Social-force models treat targets as agents whose behavior depends on individual forces, group forces, other objects, and the environment.
  • Individual force: Individual forces include fidelity to a desired destination and constancy of momentum.
  • Group force: Group forces include attraction, repulsion, and coherence, respectively keeping group members close, separated, and moving together.
  • Social force models: Social-force methods commonly minimize energy objectives containing individual- and group-force terms, or encode social force as a feature.
  • Crowd motion pattern: Crowd motion patterns are useful in highly crowded scenes because appearance and individual motion become ambiguous while collective motion remains comparatively reliable.

3.4. Exclusion Model

Exclusion models prevent physically impossible assignments in MOT by separating detections within a frame and trajectories across space-time. Detection-level exclusion is represented either softly through penalties and graphs or hard through explicit constraints, while trajectory-level exclusion penalizes excessive proximity or overlap.

  • Exclusion prevents two distinct objects from occupying the same physical space.
  • Detection-level exclusion: Detection-level exclusion forbids assigning two detections from one frame to the same target.
  • Trajectory-level exclusion: Trajectory-level exclusion prevents distinct trajectories from becoming infinitely close.
  • Detection-level exclusion: Soft models penalize assigning sufficiently distant simultaneous detections the same trajectory label.
  • Detection-level exclusion: Hard models impose cannot-link or non-overlap constraints so temporally overlapping tracklets cannot share a cluster.
  • Trajectory-level exclusion: Trajectory-level penalties increase as detections or trajectories become closer or overlap more in space-time.

3.5. Occlusion Handling

Occlusion is a major source of identity switches and trajectory fragmentation in MOT. Reviewed strategies use visible object parts, synthetic occlusion hypotheses, or buffered states to maintain or recover object trajectories.

  • Occlusion is a primary cause of identity switches and trajectory fragmentation.
  • Part-to-whole: Part-to-whole methods infer an object's state from visible parts while ignoring or downweighting occluded parts.
  • Part-to-whole: Part-wise appearance models update only unoccluded regions, while weighted affinities provide a softer alternative.
  • Part-to-whole: Human body-part association can recover part trajectories and assist recovery of the whole-object trajectory.
  • Hypothesize-and-test: Hypothesize-and-test methods generate occlusion proposals, then use them in cost-flow optimization or train a multi-person detector.
  • Buffer-and-recover: Buffer-and-recover methods retain observations and pre-occlusion states, then restore tracks when objects reappear.
  • Occlusion strategies do not form exhaustive or mutually exclusive categories, and multiple strategies may be combined.

3.6. Inference

MOT inference estimates object states and associations using probabilistic or deterministic models. Probabilistic methods provide a fuller formulation but are difficult to infer, whereas deterministic optimization often obtains good-enough solutions more practically.

  • Probabilistic inference: Probabilistic inference represents object states with dynamic and observation models and iteratively predicts and updates posterior distributions.
  • Probabilistic inference: Online tracking is especially suited to approaches that estimate states from existing past and present observations.
  • Probabilistic inference: Markov formulations assume the current state depends on previous states and observations are conditionally independent given states.
  • Probabilistic inference: Exact probabilistic integration requires simplifying assumptions because object-state distributions and multi-object state spaces are difficult to represent and integrate.
  • Probabilistic inference: Kalman, extended Kalman, and particle filters provide probabilistic inference alternatives, with particle filters using weighted samples without distributional assumptions.
  • Deterministic optimization: Deterministic approaches formulate association through graph matching, dynamic programming, network flow, conditional random fields, or MWIS.
  • Discussion: Deterministic optimization is used more often because probabilistic approaches are difficult to infer, while energy minimization yields good-enough solutions in reasonable time.

3.7. Summary

MOT systems combine components such as appearance, motion, interaction, exclusion, occlusion handling, and inference, but not every method includes them all. Appearance, motion, and inference are necessary in most methods, while other components depend on the setting.

  • MOT components are not orthogonal and can be combined or integrated for satisfactory performance.
  • Appearance, motion, and inference are mandatory in most methods, whereas interaction, exclusion, and occlusion handling may be omitted.
  • Crowd motion patterns are examples of components needed only in specific settings, such as extremely crowded scenarios.
  • Some systems combine appearance, motion, and exclusion terms in a deterministic objective or fuse features through neural networks.

4. MOT Evaluation

MOT evaluation requires metrics and datasets to measure components, compare methods, and assess detection and tracking performance. The review summarizes common metrics, datasets, code availability, and benchmark results while emphasizing that comparisons are often difficult to interpret fairly.

  • Evaluation metrics: MOT evaluation measures both detection and tracking performance, reflecting the widespread tracking-by-detection strategy.
  • Evaluation metrics: Tracking metrics cover accuracy, precision, completeness, and robustness, including MOTA, MOTP, ID switches, trajectory coverage, and recovery from occlusion.
  • Evaluation metrics: MOTA combines false positives, false negatives, and mismatches into one overall tracking measure, but it has acknowledged drawbacks and criticisms.
  • Datasets and implementations: Public datasets and source-code lists support quantitative comparison, but MOT datasets are smaller than SOT datasets and remain focused largely on pedestrians.
  • Benchmark results: Reported performance improves over publication years, although better models, improved detection, and possible dataset over-fitting may contribute to the observed progress.
  • Benchmark results: Offline methods generally outperform online methods on most metrics because offline tracking can use more information.

5. Summary

The review unifies MOT formulations, categorizes methods and components, and surveys evaluation resources and benchmark evidence. It identifies persistent reproducibility, generalization, detector-dependence, and scope issues while outlining directions including richer camera settings, 3D tracking, scene understanding, deep learning, and broader targets.

  • Review contributions: The review presents a unified MOT formulation, method categorizations, key system components, and an overview of metrics, datasets, code, and benchmark results.
  • Open issues: Current MOT research still contains unresolved issues and open problems despite substantial progress over recent decades.
  • Open issues: MOT performance depends heavily on object detectors, and missing detector descriptions can make comparisons between approaches infeasible.
  • Open issues: Complex algorithms are difficult to tune, implement, and reproduce, while performance may fail to generalize across camera views and video sequences.
  • Future directions: Future research includes multi-camera and 3D tracking, scene understanding, deep learning, joint vision tasks, and tracking non-human targets in varied scenarios.
  • Future directions: Deep learning can strengthen target observation and tracking, while neural modeling of target association remains in need of further research.
  • Future directions: Extending MOT beyond surveillance pedestrians to vehicles, animals, traffic scenes, and aerial photographs is identified as a promising direction.
Loading 1409.7618v5…