Source-linked AI summary
Visual Object Tracking with Discriminative Filters and Siamese Networks: A Survey and Outlook
Sajid Javed, Martin Danelljan, Fahad Shahbaz Khan, Muhammad Haris Khan, Michael Felsberg, Jiri Matas
TL;DR
Visual object tracking must estimate a target’s trajectory and state from an initial state despite changing appearances and distractors. This survey synthesizes DCF and Siamese tracking through theory, literature review, benchmark analysis, and open-challenge recommendations. It concludes that robust online target appearance modeling, accurate state estimation, segmentation, and suitable backbone architectures remain central directions.
Problem
Visual object tracking must learn an appearance model online from an initial target state despite occlusion, scale variation, deformation, environmental changes, and similar-looking distractors.
Method
The survey reviews DCF and Siamese formulations, more than 90 trackers, shared and specific challenges, and results across nine tracking benchmarks.
Results
The analysis identifies robust online target appearance modeling, accurate target-state estimation, segmentation, and backbone architectures as key areas shaping tracking performance and future development.
Takeaways & Limitations
The survey recommends integrating discriminative online modeling, accurate segmentation, and robust tracking frameworks to improve target representation, state estimation, and scale handling.
Takeaways & Limitations
Early Siamese trackers using relatively shallow AlexNet backbones remain limited in performance because their feature representations are not sufficiently strong.
Abstract
from arXiv · showhide
Accurate and robust visual object tracking is one of the most challenging and fundamental computer vision problems. It entails estimating the trajectory of the target in an image sequence, given only its initial location, and segmentation, or its rough approximation in the form of a bounding box. Discriminative Correlation Filters (DCFs) and deep Siamese Networks (SNs) have emerged as dominating tracking paradigms, which have led to significant progress. Following the rapid evolution of visual object tracking in the last decade, this survey presents a systematic and thorough review of more than 90 DCFs and Siamese trackers, based on results in nine tracking benchmarks. First, we present the background theory of both the DCF and Siamese tracking core formulations. Then, we distinguish and comprehensively review the shared as well as specific open research challenges in both these tracking paradigms. Furthermore, we thoroughly analyze the performance of DCF and Siamese trackers on nine benchmarks, covering different experimental aspects of visual tracking: datasets, evaluation metrics, performance, and speed comparisons. We finish the survey by presenting recommendations and suggestions for distinguished open challenges based on our analysis.
1 INTRODUCTION
Visual object tracking estimates a target’s trajectory and state from its initial state despite appearance changes and distractors. This survey reviews DCF and Siamese paradigms, their shared and specific challenges, benchmark performance, and future directions.
- Visual object tracking estimates a target’s trajectory and state from an initial state, supporting applications including autonomous driving, robotics, surveillance, sports analytics, and medical imaging.
- Generic tracking must learn an appearance model online despite occlusion, scale variation, deformation, illumination changes, motion blur, and similar-looking objects or backgrounds.
- DCF trackers learn online correlation filters and detect targets using FFT, whereas Siamese networks learn offline embeddings that compare template and search-region patches.
- Both paradigms share challenges in feature representation and full target-state estimation, while differing in issues such as DCF boundary artifacts and Siamese online adaptability.
- The survey reviews more than 90 trackers across nine benchmarks, distinguishes shared and paradigm-specific open challenges, and proposes recommendations for future research.
2 LITERATURE REVIEW
The survey positions itself against earlier VOT surveys by focusing on DCF and Siamese trackers and comparing them extensively across modern benchmarks.
- Earlier VOT surveys organized tracking methods by procedure, attributes, network structure, training, or datasets.
- This survey focuses on DCFs and Siamese networks, provides their background theory, and reviews their shared and specific open research challenges.
- It reviews more than 90 trackers, traces their evolution toward segmentation-based tracking, and compares them on nine popular visual tracking benchmarks.
3 DISCRIMINATIVE CORRELATION FILTERS
Discriminative correlation filters learn linear regressors online, using circularly shifted samples to approximate dense sampling and enable FFT-based computation.
- Discriminative correlation filters are supervised techniques for learning linear regressors that localize targets through correlation filtering.
- Circularly shifting training samples provides a computationally efficient approximation to dense sampling and enables the use of the fast Fourier transform.
3.1 Standard Single-Channel DCF Formulation
The standard single-channel DCF learns a filter whose circular convolution with training samples approximates desired response maps. Fourier-domain diagonalization reduces inference to efficient element-wise operations and FFTs.
- A single-channel DCF learns a filter w from training samples x_j and labels y_j so that x_j ∗ w approximates y_j.
- The formulation uses circular convolution, equivalent to cyclically shifting samples or periodically padding them before convolution.
- Because the convolution matrix is circulant, the Fourier transform diagonalizes it, allowing the inverse to be computed by element-wise division.
- The filter is learned by minimizing a regularized linear least-squares objective over concatenated samples and regression targets.
- The resulting Fourier-domain formula uses only element-wise operations and DFTs and runs in O(N log N) time with FFT, where N is the number of sample pixels.
3.2 Standard Multi-channel DCF Formulation
The multi-channel DCF formulation represents samples and filters across multiple feature channels, combines channel responses, and uses weighted least-squares learning with early or late fusion.
- Fusion strategies: Early fusion concatenates feature channels before training a single multi-channel classifier, whereas late fusion trains channel-specific filters and aggregates them with scaling factors β_d.
- Representation: Multi-channel DCFs represent each sample and filter with D feature channels, producing the total response by summing individual channel convolutions.
- Optimization: The generalized objective remains a linear least-squares problem with nonnegative per-sample importance weights α_j.
- Computational formulation: The general multi-channel objective can be solved as N D-dimensional linear systems using FFT-based computation, with complexity O(DN log N + ND^3).
3.3 Standard DCF Tracking Pipeline
The standard DCF tracking pipeline alternates between detecting the target with an online filter and recursively updating that filter using newly collected training samples.
- Pipeline: DCF tracking first learns a filter online, detects the target by tracking-by-detection, and then recursively updates the model after each frame.
- Target Detection: Target detection evaluates scores across all locations and selects the location n* that maximizes the target score function.
- Target Detection: When the estimate is sufficiently confident, the pipeline extracts a centered patch and constructs a new training sample for model learning.
- Model update: Online adaptation uses a learning rate γ ∈ [0, 1] to control how quickly the filter adapts to new data.
- Model update: The recursive update is obtained by assigning exponentially decaying sample weights α_j = γ(1 − γ)^(m−j) for j > 1 and α_1 = (1 − γ)^(m−1).
3.4 Open Issues in the Standard DCF Tracking Pipeline
Standard DCF tracking faces open issues in feature representation, optimization, boundary artifacts, inference efficiency, and target-state estimation, motivating spatial regularization, improved optimization, and scale-aware methods.
- Feature Representation: DCF trackers must find discriminative yet invariant features despite linear decision boundaries, with handcrafted, deep, hybrid, and end-to-end representations explored.
- Boundary Artifacts: Circular convolution creates boundary artifacts by replacing background content with repeated patches, reducing the diversity of background samples used for discrimination.
- Optimization: Spatial regularization penalizes filter coefficients in background regions more strongly, encouraging compact filters focused on the target.
- Optimization: Spatial-domain optimization in ATOM and DiMP bypasses boundary artifacts, while Fourier-domain methods remain computationally attractive for high-resolution feature maps.
- Optimization: Efficient and robust inference schemes remain a key problem because multi-channel loss minimization lacks a general efficient closed-form solution.
- Optimization: Conjugate-gradient computation exploits sparsity and blockwise operations to reduce feature-dimension complexity from quadratic O(D^2) to linear O(D).
- Target State Estimation: Fixed-size DCF bounding boxes cannot adequately handle scale variation, making accurate target-scale estimation an established challenge.
- Target State Estimation: Scale-filter methods reduce the search space and learn to discriminate target appearances across scales, but can degrade under sudden scale changes.
3.5 Evolution of DCFs to Segmentation-based Trackers
Segmentation-based extensions improve DCF tracking by providing more reliable object observations and spatially selective information for filter learning, especially with non-rectangular targets.
- Motivation: Object segmentation supplies reliable observations that address rotated boxes, occlusion, deformation, and scaling, potentially avoiding tracking failures.
- DCF extensions: Segmentation has been incorporated into DCF trackers to improve filter learning when targets are non-rectangular.
- DCF extensions: Color-histogram segmentation improves tracking under illumination changes, motion blur, and target deformation, while spatial reliability maps regularize filter learning.
4 SIAMESE TRACKERS
Siamese trackers learn offline similarity functions from target and search-image pairs, then localize targets by matching their learned representations. Their development addresses backbone limitations, distractors, model adaptation, scale variation, and loss design.
- Siamese Tracking Formulation: Siamese trackers learn a matching function offline from large collections of target-image pairs and evaluate it online during tracking.The template and detection branches share CNN parameters and process target patches from successive frames.
- Siamese Tracking Formulation: The SiamFC pipeline extracts feature maps from a target patch and a larger search region, then matches them using cross-correlation.The resulting response map represents local similarity between the template and search features.
- Performance: Siamese trackers are computationally efficient and have demonstrated state-of-the-art tracking performance; SiamFC tracks at 140FPS on a GPU.SiamFC compares each frame with the initial object appearance and predicts the location with the highest score.
- Backbone Architectures: Backbone architecture strongly affects feature quality: early trackers using modified AlexNet were limited because it produced relatively weak representations.ResNet-driven SiamRPN++ addressed this issue while also targeting spatial translational invariance.
- Offline Training: Standard Siamese trackers cannot exploit known distractor appearances during tracking, making semantically similar objects difficult cases.Hard negative mining adds semantic negative pairs and helped DaSiamRPN address data imbalance and drifting.
- Open Challenges and Design Choices: Siamese tracking research develops model updates, scale-handling modules, and alternative losses to address appearance change, scale variation, and representation learning.Examples include dynamic or gradient-guided template updates, RPN-based position and scale estimation, and contrastive or triplet losses.
- Online Model Update: Despite numerous update techniques, using no model update remains a robust and popular alternative, motivating simpler end-to-end trainable adaptation methods.The survey identifies further research on general techniques that improve Siamese tracking robustness.
5 EXPERIMENTAL COMPARISON
The comparison covers nine benchmarks spanning short- and long-term tracking, diverse object categories, resolutions, annotations, and challenge attributes. These datasets are designed to evaluate trackers under varied real-world conditions.
- Benchmark Datasets: OTB100 contains 100 videos from 22 object categories with 11 tracking attributes and an average resolution of 356 × 530.
- Benchmark Datasets: TC128 evaluates color effects using 128 annotated color sequences from 27 object categories and 11 tracking attributes.
- Benchmark Datasets: UAV123 and UAV20L provide short and long UAV sequences with diverse object categories, high-resolution imagery, and 12 tracking attributes.UAV123 contains 123 short sequences, while UAV20L contains 20 long simulator-generated videos.
- Large-Scale Benchmarks: TrackingNet, LaSOT, and GOT-10K provide large-scale evaluations with dense annotations, long sequences, broad object categories, and class-agnostic tracking settings.TrackingNet contains 60,643 sequences, LaSOT includes 1,400 train/test sequences, and GOT-10K contains 10,000 videos.
5.2 Performance Evaluation Measures
Tracker performance is evaluated with complementary measures of localization accuracy, overlap quality, scale-normalized precision, and reset-based robustness. The protocols differ between one-pass benchmarks and the VOT series.
- Precision: Distance precision measures the percentage of frames whose predicted target center lies within T pixels of ground truth, using T = 20 pixels for ranking.
- Overlap: Success plots evaluate bounding-box overlap using IoU across thresholds from 0 to 1, ranking trackers by area under the curve.
- Normalized Precision: Normalized precision evaluates relative localization error with respect to target size rather than using absolute pixel distance.
- Evaluation Protocols: OTB100, TC128, UAV123, and LaSOT use one-pass evaluation from first-frame initialization through sequence completion.
- Evaluation Protocols: VOT evaluation resets trackers after failure and reports Accuracy, Robustness, and Expected Average Overlap metrics.Accuracy measures overlap during successful periods, while Robustness counts target losses.
5.3 Quantitative Comparison
The survey compares representative DCF- and Siamese-based trackers across nine benchmarks, showing that recent methods generally outperform earlier approaches on challenging datasets. Performance on OTB100 has largely saturated, whereas LaSOT, GOT-10k, and TrackingNet show continued improvement.
- Tables 2–5 compare representative DCF-based and Siamese trackers across nine tracking benchmarks.
- DCF-based trackers: ECO achieves a 91.0% PR score on OTB100 but only 30.1% on LaSOT, illustrating weaker transfer to challenging large-scale datasets.
- DCF-based trackers: Recent end-to-end DCF frameworks, including DiMP and PrDiMP, achieve strong performance on both OTB100 and LaSOT.
- Siamese trackers: Recent Siamese trackers address online updating, re-detection, region refinement, box regression, and the connection between tracking and segmentation.
- Siamese trackers: Ocean reaches 48.9% EAO on VOT2018-ST, while D3S obtains 48.9% and 43.9% EAO on VOT2018-ST and VOT2020-ST, respectively.
- Benchmark trends: OTB100 performance has saturated, while LaSOT, GOT-10k, and TrackingNet show consistent improvements from recent trackers.
- Benchmark characteristics: Long sequences and multiple distractors in LaSOT and UAV123 emphasize robustness and re-detection capabilities.
5.4 Speed Comparison
Tracking speed is an important evaluation dimension, particularly for real-time requirements. Comparisons are complicated because speed depends on feature extraction, model updates, programming language, and hardware.
- Tracking speed is evaluated because practical systems may need to satisfy real-time requirements.
- Speed comparisons are confounded by feature extraction, model-update procedures, programming language, and implementation hardware.
- The VOT2014 committee introduced Equivalent Filter Operations to reduce hardware-related influence in speed evaluation.
6 DISCUSSION AND CONCLUSIONS
The survey identifies practical lessons and future directions for DCF and Siamese tracking, emphasizing end-to-end learning, robust target modeling, accurate state estimation, segmentation, efficient backbones, geometry, and connections to related areas.
- Recommendations: End-to-end offline learning, enabled by large-scale datasets, has become important for both Siamese and recent DCF trackers.The authors link optimal performance to jointly learning underlying features and prediction heads.
- Recommendations: Robust target modeling should integrate background appearance cues and support online updating during tracking.
- Recommendations: Siamese trackers have advanced bounding-box regression through object-detection techniques, including efficient anchor-free approaches such as Ocean.
- Future directions: Pixel-precise segmentation can improve target modeling, bounding-box regression, and scale estimation, motivating its integration into robust tracking frameworks.
- Future directions: ResNet remains popular for multiresolution feature extraction, but its computational cost motivates tracking-specific efficient backbones for constrained platforms.
- Future directions: Future tracking research may converge with video segmentation, SLAM, multi-target tracking, and open-world tracking involving unknown classes, segmentation, and shape modeling.