PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for Visual Tracking: A Comprehensive Survey

Seyed Mojtaba Marvasti-Zadeh, Li Cheng, Hossein Ghanei-Yakhdan, Shohreh Kasaei

arXiv:1912.00535v2cs.CVcs.LGeess.IV

TL;DR

Visual tracking is challenging, and exhaustive comparisons of deep-learning trackers across benchmarks had not previously been performed. This survey organizes methods, datasets, and metrics, then evaluates state-of-the-art trackers quantitatively and qualitatively to characterize their performance and limitations across scenarios.

  • Problem

    Exhaustive comparisons of deep-learning trackers across a wide variety of benchmarks had not previously been performed, despite visual tracking’s challenging and ill-posed nature.

  • Method

    The survey categorizes deep-learning trackers across nine aspects, investigates benchmark datasets and metrics, and compares state-of-the-art methods on seven visual tracking datasets.

  • Results

    Across benchmark comparisons, PrDiMP50, DiMP50, and ATOM frequently rank among the best trackers, while performance varies across datasets, metrics, and tracking scenarios.

  • Takeaways & Limitations

    The analyses provide practitioners with guidance for weighing which tracking methods to choose under different conditions and identify ongoing issues and future research directions.

  • Takeaways & Limitations

    Existing trackers have deficient scene understanding and may fail when multiple critical attributes occur simultaneously in a scene.

Abstract

from arXiv · show

Visual target tracking is one of the most sought-after yet challenging research topics in computer vision. Given the ill-posed nature of the problem and its popularity in a broad range of real-world scenarios, a number of large-scale benchmark datasets have been established, on which considerable methods have been developed and demonstrated with significant progress in recent years -- predominantly by recent deep learning (DL)-based methods. This survey aims to systematically investigate the current DL-based visual tracking methods, benchmark datasets, and evaluation metrics. It also extensively evaluates and analyzes the leading visual tracking methods. First, the fundamental characteristics, primary motivations, and contributions of DL-based methods are summarized from nine key aspects of: network architecture, network exploitation, network training for visual tracking, network objective, network output, exploitation of correlation filter advantages, aerial-view tracking, long-term tracking, and online tracking. Second, popular visual tracking benchmarks and their respective properties are compared, and their evaluation metrics are summarized. Third, the state-of-the-art DL-based methods are comprehensively examined on a set of well-established benchmarks of OTB2013, OTB2015, VOT2018, LaSOT, UAV123, UAVDT, and VisDrone2019. Finally, by conducting critical analyses of these state-of-the-art trackers quantitatively and qualitatively, their pros and cons under various common scenarios are investigated. It may serve as a gentle use guide for practitioners to weigh when and under what conditions to choose which method(s). It also facilitates a discussion on ongoing issues and sheds light on promising research directions.

I. INTRODUCTION

Visual tracking estimates an unknown target trajectory from an initial state, but its ill-posed, real-world setting creates substantial challenges. This survey addresses gaps in coverage, categorization, benchmark comparison, and comprehensive evaluation of DL-based trackers.

  • Visual tracking estimates an unknown target trajectory from only its initial state in a video frame.
  • Real-world tracking is challenging because targets, motion models, imaging characteristics, and environmental conditions vary substantially.
  • DL-based tracking has expanded beyond CNNs to include RNNs, autoencoders, GANs, Siamese networks, and custom architectures.
  • Earlier reviews omitted state-of-the-art Siamese and customized networks, long-term and aerial-view tracking, implementation details, and complete benchmark comparisons.
  • The survey categorizes extensive DL-based trackers across architecture, feature exploitation, training, objectives, correlation filters, aerial-view, long-term, and online tracking.

A. Contributions

The paper contributes a broad taxonomy, benchmark and metric comparison, and extensive experiments across short-term, long-term, and aerial-view tracking datasets. Its comparisons identify promising architectures, training strategies, feature combinations, and persistent tracking challenges.

  • A. Contributions: The taxonomy organizes DL-based trackers by architecture, feature exploitation, training, objectives, outputs, correlation filters, aerial-view, long-term, and online tracking.
  • A. Contributions: The survey presents issues and proposed solutions for DL-based tracking challenges to inform visual tracker design.
  • A. Contributions: Benchmark datasets are compared by videos, frames, classes or clusters, attributes, absent labels, and dataset overlap across short-term, long-term, and aerial-view tracking.
  • A. Contributions: Experiments evaluate state-of-the-art trackers on OTB2013, OTB2015, VOT2018, LaSOT, UAV123, UAVDT, and VisDrone2019, including challenging attributes.
  • A. Contributions: Siamese networks offer a promising balance between tracking performance and efficiency, while RL, GAN, and custom networks address decision-making, data, and scene-information issues.
  • A. Contributions: Offline end-to-end learning transfers generic features to tracking, while efficient online adaptation improves robustness but conventional online DNN training raises computational complexity.
  • A. Contributions: Deeper and wider backbones improve target-background discrimination, but pretrained networks are sub-optimal without tracking-specific training.
  • A. Contributions: Combined regression and classification objectives distinguish targets from backgrounds and support tight bounding-box localization.

II. DEEP VISUAL TRACKING TAXONOMY

The taxonomy organizes deep visual trackers by architecture, training, objectives, outputs, and deployment-oriented concerns. It highlights robust representation, data imbalance, and computational complexity as recurring design challenges.

  • DL-based trackers are categorized across nine aspects, including architecture, network exploitation, training, objectives, outputs, correlation filters, aerial-view, long-term, and online tracking.
  • The taxonomy includes CNN-, Siamese-, GAN-, RNN-, and custom-network trackers.CNNs provide hierarchical target representations, while Siamese networks compare target exemplars with search regions.
  • Robust target representation is pursued through large-scale offline training, feature fusion, target-specific models, and structural feature selection.
  • A single positive first-frame sample and noisy target-derived samples create over-fitting and training-data imbalance risks.Proposed responses include domain adaptation, update mechanisms, sample mining, and online ensemble learning.
  • CNN-based trackers remain computationally complex despite strong appearance representations.Reported remedies include lightweight architectures, feature selection, efficient optimization, attention, and correlation-filter computation.

2) Siamese Neural Network (SNN):

Siamese trackers compare target and search regions through shared two-stream processing. Their development focuses on discriminative target modeling, appearance adaptation, and balanced training samples.

  • Siamese networks compute a similarity map from paired target and search regions, supporting end-to-end tracking for real-time applications.
  • Discriminative target representation: SNN-based methods improve target discrimination through distractor-aware or target-aware features, multi-level fusion, specialized losses, and attention.
  • Adapting target appearance variation: Offline-only SNN training generalizes poorly to unseen targets, motivating online updates, background suppression, and one-shot local detection formulations.
  • Balancing training data: SNN trackers address sample imbalance using hard-negative stimulation, fixed foreground-to-background ratios, random sampling, and flow-guided sampling.

3) Recurrent Neural Network (RNN):

RNNs incorporate temporal information and memory into tracking, while custom networks combine complementary architectures and learning strategies. Their motivations include robustness, efficiency, adaptive decisions, and limited-data handling.

  • 3) Recurrent Neural Network (RNN):: RNN-based trackers capture spatio-temporal representations, contextual information, attention, background suppression, and previous target appearances.Convolutional LSTMs and multi-directional recurrent networks are used to model temporal dependencies.
  • 3) Recurrent Neural Network (RNN):: RNN-based methods are relatively limited because they are difficult to train and contain many parameters.
  • 5) Custom Networks:: Custom networks combine AE, CNN, RNN, SNN, detection, and deep reinforcement-learning components to compensate for ordinary trackers’ deficiencies.Meta-learning is also used to address few-shot learning problems.
  • 5) Custom Networks:: Custom trackers pursue robust localization through ensembles, target pose estimation, meta-learned models, class-agnostic detection, and segmentation masks.D3S uses segmentation masks with complementary geometric properties, while COMET targets small aerial objects with multi-scale features and attention.
  • 5) Custom Networks:: Custom-network designs also address complexity, model updating, limited data, search strategy, additional information, and online decision making.Examples include adaptive feature selection, parallel model updates, dense sampling, optical flow, and reinforcement-learning-based decisions.

B. Network Exploitation

Network exploitation ranges from reusing pretrained representations to training networks specifically for tracking. The survey distinguishes off-the-shelf features, end-to-end training, and dataset choices for specialization and efficiency.

  • DNNs are exploited either by reusing pretrained models or by training deep features for visual tracking tasks.
  • 1) Model Reuse or Deep Off-the-Shelf Features:: Off-the-shelf features transfer generic target representations into traditional trackers, commonly combining semantic and fine-grained features.These features are primarily pretrained on ImageNet object-recognition data.
  • Off-the-shelf features can limit performance because task objectives differ and offline representations may miss target variation or overfit initial templates.
  • State-of-the-art trackers mainly use end-to-end learning, with differences in offline training, online fine-tuning, complexity, data scarcity, overfitting, and unsupervised learning.
  • 1) Training Datasets:: Training uses general-purpose auxiliary datasets and tracking datasets such as LaSOT and TrackingNet to expose networks to task-specific challenging attributes.

3) Only Online Training:

Online-training trackers adapt whole or partial deep networks during tracking to discriminate unseen targets, but must control overfitting and divergence. Other approaches combine offline shared representations with online domain-specific adaptation, while augmentation and meta-learning address limited data and unseen scenarios.

  • 3) Only Online Training:: Online-training trackers update whole or partial DNNs during tracking to adapt network parameters to target appearance.Alternating direct training and inference can improve discrimination of particular targets, but requires strategies against overfitting and divergence.
  • 4) Both Offline and Online Training:: Offline and online training jointly provide shared and domain-specific representations for transferability and online domain adaptation.Shared features primarily discriminate targets from foreground information, while domain-specific features address intra-class distractors.
  • 5) Data Augmentation:: Geometric transformations and color-space augmentations are widely used to enlarge training data and reduce overfitting in the few-data tracking regime.GAN-based augmentation can additionally capture varied appearance changes and affect tracking performance.
  • 6) Meta-Learning:: Meta-learning transfers experience across tasks to construct more flexible target models for unseen targets and scenarios.Some trackers instead adjust model weights during tracking, which differs from the classic meta-learning definition.
  • 7) Network Objective:: Tracking networks are categorized by objective into classification-based, regression-based, or combined classification-and-regression methods.This objective-based categorization excludes trackers that only use pre-trained deep features without designing or training tracking networks.

1) Classification-based Objective Function:

Classification-based tracking generates candidate bounding boxes and selects the target by distinguishing target proposals from background. Regression-based and combined objectives instead directly localize or jointly refine targets using complementary objectives.

  • 1) Classification-based Objective Function:: Classification-based trackers generate hundreds of candidate bounding boxes from a search region and select the highest-scoring target proposal.They formulate proposal selection as binary classification between target and background across varied target classes, motion patterns, and sequences.
  • 2) Regression-based Objective Function:: Regression-based trackers directly localize targets in subsequent frames by minimizing regularized least-squares objectives.These methods generally require extensive training data and may refine L1 or L2-style losses for network optimization.
  • 3) Both Classification and Regression-based Objective Function:: Combined-objective trackers use classification to identify similar objects and ridge regression to support continuous target localization.Their stated goal is to bridge tracking-by-detection with continuous localization.
  • 4) Network Output:: Network outputs used for target estimation or refinement include confidence maps, bounding boxes, object scores, and additional output categories.The survey organizes these outputs into six main categories, including variants such as rotated boxes, voting maps, and feature maps.
  • 5) Exploitation of Correlation Filter Advantages:: Deep correlation-filter trackers seek robustness through deep features and efficiency through correlation layers, filters, or filter-specific objectives.DCF methods learn discriminative filters whose frequency-domain operations determine the target’s spatial location.

H. Long-term Tracking

Long-term tracking addresses videos in which targets disappear and reappear, motivating re-detection, memory, verification, and failure-recovery mechanisms. The survey also contrasts generic and aerial benchmarks by scenario properties, annotations, scale, and duration.

  • H. Long-term Tracking: Long-term tracking targets relatively long videos where objects may disappear and reappear, but comparatively few trackers address this setting.Short-term extensions use local-to-global search, detection modules, re-detection, memory management, negative-signal suppression, or online verification.
  • H. Long-term Tracking: Detection-inspired long-term trackers generate candidates, verify observations, mine hard distractors, and select targets across time.Examples combine query-guided proposal and classification networks, offline regression with online verification, or tracklet dynamic programming.
  • A. Tracking Benchmarks: Benchmarks are organized around generic or aerial applications and short- or long-term scenarios, with protocols measuring tracker accuracy and robustness.Dataset comparisons include applications, scenarios, characteristics, missing labels for unsupervised training, and dataset overlap.
  • A. Short-term Tracking Datasets: OTB2013 and OTB2015 provide fully annotated generic tracking sequences for more consistent and unbiased comparisons.OTB2015 extends OTB2013 specifically to support unbiased performance comparisons.
  • B. Aerial View Object Tracking: Aerial-view tracking introduces challenges including tiny objects, weather variation, dense environments, long occlusions, and major viewpoint changes.UAV123 provides sparse, low-altitude aerial-view sequences captured by professional, low-cost, and simulated flying-robot platforms.
  • C. Long-term Tracking Datasets: OxUvA and TLP support long-term evaluation through extended videos, with OxUvA additionally providing absent labels for frames where targets do not exist.OxUvA includes 14 hours of YouTube-BoundingBoxes videos, while TLP contains longer high-resolution sequences.
  • A. Short-term Tracking Datasets: LaSOT addresses small-scale, sparse-annotation, short-video, and category-bias problems with 1,120 training and 280 testing sequences.The training and testing subsets contain 2.3M and 690K frames, respectively, with equal numbers of videos per category.

C. Evaluation Metrics

Visual trackers are compared with complementary measures of accuracy, robustness, speed, and long-term presence handling. Short-term metrics include location error, overlap accuracy, failures, EAO, and AUC, while long-term metrics include precision, recall, F-score, and MaxGM.

  • 1) Performance Measures:: Evaluation measures and performance plots provide complementary views of tracker accuracy, robustness, and speed.Long-term measures additionally reflect re-detection and target-absence prediction capabilities.
  • A. Short-term Tracking Measures:: Center location error averages Euclidean distance between ground-truth and estimated target locations but ignores bounding-box overlap and tracking failures.Normalized precision reduces sensitivity to bounding-box size.
  • A. Short-term Tracking Measures:: Accuracy averages per-frame intersection-over-union overlap scores when tracker estimates overlap the ground-truth boxes.The overlap score is the intersection area divided by the union area of estimated and ground-truth bounding boxes.
  • A. Short-term Tracking Measures:: Robustness counts re-initializations after target loss, with failure detected when the overlap score reaches zero.This measure directly records tracking failures during the task.
  • A. Short-term Tracking Measures:: Expected average overlap combines accuracy and robustness by averaging per-frame overlaps through sequence ends, including zero overlaps after failure.The measure is defined over sequences of Ns frames using per-frame overlap averages Φ_i.
  • A. Short-term Tracking Measures:: AUC summarizes average success rates across predefined thresholds to rank overall tracking performance over a sequence.Success rates are normalized between 0 and 1.
  • B. Long-term Tracking Measures:: Long-term precision and recall integrate IoU scores over confidence thresholds, normalized respectively by frames with predictions and frames without absent targets.F-score combines precision and recall through their harmonic mean for ranking across thresholds.
  • B. Long-term Tracking Measures:: MaxGM uses true-positive and true-negative rates to evaluate correct localization and correct absent-target reporting across probabilistic thresholds.It avoids the zero value of the basic geometric mean for trackers that cannot predict absent targets.

2) Performance Plots:

The survey summarizes threshold-based performance plots for short-term and long-term visual tracking, including precision, success, expected average overlap, precision/recall, and F-score curves.

  • Short-term Tracking Plots: Precision plots report the percentage of frames whose estimated locations fall within each center-location-error threshold.
  • Short-term Tracking Plots: Success plots report the percentage of frames whose estimated and ground-truth regions exceed each overlap threshold.
  • Short-term Tracking Plots: Expected average overlap curves average overlap values across a specified interval for an individual video-sequence length.
  • Long-term Tracking Plots: Precision/recall plots compare long-term tracking performance and detection capability across thresholds.
  • Long-term Tracking Plots: F-score plots rank long-term tracking methods by their highest score on the curve.

IV. EXPERIMENTAL ANALYSES

The survey evaluates 48 DL-based trackers across seven benchmarks and compares overall performance, challenging attributes, and qualitative failure patterns. Results identify recurring top performers while showing that tracker strengths depend on dataset, metric, and visual challenge.

  • Quantitative Comparisons: 48 DL-based trackers are quantitatively compared on seven datasets: OTB2013, OTB2015, VOT2018, LaSOT, UAV123, UAVDT, and VisDrone2019-test-dev.
  • Quantitative Comparisons: VITAL, MDNet, and ASRCF lead combined precision-success results on OTB2013; SiamRPN++, ASRCF, and VITAL lead on OTB2015.
  • Quantitative Comparisons: PrDiMP50, DiMP50, and ATOM lead combined precision-success results on LaSOT, while PrDiMP50, DiMP50, and SiamRPN++ lead on aerial-view datasets.
  • Quantitative Comparisons: On VOT2018, DiMP50, SiamRPN++, and ATOM achieve the best EAO scores, while SiamRPN, SiamRPN++, and CFNet are the fastest trackers.
  • Most Challenging Attributes per Benchmark Dataset: The survey determines each benchmark’s most challenging attributes using median accuracy and robustness, or median precision and success, across attributes.
  • Most Challenging Attributes per Benchmark Dataset: SiamRPN++ handles several VOT2018 attributes accurately, while ATOM, DiMP50, and PrDiMP50 benefit from classification-regression networks and online optimization for fast adaptation.
  • Most Challenging Attributes per Benchmark Dataset: Discriminative target-background modeling addresses illumination variation, while IoU-based refinement in ATOM, DiMP, and PrDiMP handles aspect-ratio changes.

C. Discussion

The discussion links strong tracker performance to diverse architectures, specialized training, discriminative modeling, and online adaptation, while documenting important weaknesses and unresolved real-world failures.

  • Discussion: The strongest trackers span CNN, DCF, Siamese, custom, and other architectural families rather than one dominant design.
  • Discussion: Siamese and custom trackers seek a balance between performance and speed while integrating deep backbones, proposal networks, and tracking-specific learning schemes.
  • Discussion: Top-performing methods use data augmentation, adversarial learning, reformulated objectives, specified features, and efficient training procedures to address tracking challenges.
  • Discussion: The survey identifies similarity, meta-, multi-domain, adversarial, spatial-aware regression, and DCF learning as major training schemes among strong trackers.
  • Discussion: ATOM, DiMP, and PrDiMP are robust to CM, MC, SV, and ARC but can be strongly affected by SOB, LR, and OB attributes.
  • Discussion: MDNet-based trackers remain strong but incur high computational complexity, weak intra-class discrimination, and discrete scale estimation.
  • Discussion: DCF-based trackers remain competitive but suffer from limitations involving pre-trained models, aspect-ratio variation, model degradation, and appearance variation.
  • Discussion: Qualitative VOT2018 results favor DiMP50, ATOM, and SiamRPN++, yet simultaneous critical attributes still cause failures and limit real-world reliability.

V. CONCLUSION AND FUTURE DIRECTIONS

The survey organizes state-of-the-art deep learning trackers across nine methodological aspects, reviews benchmarks and metrics, and compares trackers on seven datasets. It identifies unresolved challenges involving online adaptation, aerial and long-term tracking, and scene understanding.

  • The survey develops a taxonomy of state-of-the-art trackers spanning nine aspects, including architecture, training, objectives, outputs, aerial-view, long-term, and online tracking.
  • It investigates benchmark datasets and evaluation metrics, then compares state-of-the-art trackers across seven visual tracking datasets.
  • Custom networks commonly combine efficient architectures with classification and regression or classification-plus-bounding-box-refinement branches, while often retaining generic backbones to avoid catastrophic forgetting.
  • Efficient online training remains important for rapid adaptation to unseen targets, with meta- and few-shot learning used to find favorable base-learner initializations despite overfitting concerns in shallow meta-networks.
  • Aerial tracking remains difficult because of tiny objects, viewpoint changes, dense multi-target scenes, out-of-view events, and large occlusions, motivating long-term approaches.
  • Long-term tracking and scene understanding remain underdeveloped: trackers need re-detection after failures and better interpretation of dynamic scenes, global structures, objects, and events.
Loading 1912.00535v2…