Source-linked AI summary
Unveiling the Power of Deep Tracking
Goutam Bhat, Joakim Johnander, Martin Danelljan, Fahad Shahbaz Khan, Michael Felsberg
TL;DR
Deep-feature trackers have not matched the performance of handcrafted-feature methods, motivating an investigation into the causes of this gap. The paper analyzes feature properties, tailors training for shallow and deep representations, and adaptively fuses them; on VOT2017, it reports a 17% relative EAO gain over the challenge winners.
Problem
Deep trackers have had limited success in generic object tracking, where handcrafted-feature trackers remain competitive despite deep learning’s broader computer-vision success.
Method
The paper systematically analyzes shallow and deep features, addresses limited data and low spatial resolution through feature-specific training, and adaptively fuses their predictions using a quality measure.
Results
17% relative EAO gain on VOT2017 gives the proposed approach an EAO of 0.378 versus 0.323 for the competition winners.
Takeaways & Limitations
Independently tailoring shallow and deep models and combining their complementary characteristics leads to state-of-the-art performance on the reported tracking datasets.
Takeaways & Limitations
The analysis assumes the detection score is twice continuously differentiable over a continuous domain, with discrete scores handled through interpolation.
Abstract
from arXiv · showhide
In the field of generic object tracking numerous attempts have been made to exploit deep features. Despite all expectations, deep trackers are yet to reach an outstanding level of performance compared to methods solely based on handcrafted features. In this paper, we investigate this key issue and propose an approach to unlock the true potential of deep features for tracking. We systematically study the characteristics of both deep and shallow features, and their relation to tracking accuracy and robustness. We identify the limited data and low spatial resolution as the main challenges, and propose strategies to counter these issues when integrating deep features for tracking. Furthermore, we propose a novel adaptive fusion approach that leverages the complementary properties of deep and shallow features to improve both robustness and accuracy. Extensive experiments are performed on four challenging datasets. On VOT2017, our approach significantly outperforms the top performing tracker from the challenge with a relative gain of 17% in EAO.
1 Introduction
Generic object tracking remains difficult for deep-feature methods because training data is scarce and precise localization is challenging. The paper studies these limitations and proposes independently trained shallow and deep models with adaptive fusion, reporting strong benchmark gains.
- Motivation: Deep trackers remain competitive with, and sometimes inferior to, handcrafted-feature trackers despite deep learning’s success elsewhere.They also do not consistently benefit from deeper or more sophisticated network architectures.
- Results: More powerful network representations consistently improve the proposed tracker on Need for Speed, whereas ECO does not benefit from them.The comparison keeps the shallow HOG and Color Names representation fixed while varying deep networks.
- Challenges: Limited training data is a central obstacle because the target model must learn from a single labeled frame while acquiring invariance to unseen appearance changes.Deep models are data-hungry, making online appearance-model learning especially difficult in tracking.
- Challenges: Low spatial resolution in deep convolutional features hampers accurate localization, where prediction errors can cause model drift and eventual tracking failure.Trackers therefore often complement deep representations with shallow activations or handcrafted features.
- Contributions: The proposed fusion strategy independently trains shallow and deep models, then combines their predictions using a quality measure that considers accuracy and robustness.This design targets the complementary characteristics of the two feature types.
- Results: 17% relative EAO gain on VOT2017 separates the proposed approach from the challenge winners, with scores of 0.378 versus 0.323.Experiments span Need for Speed, VOT2017, Temple128, UAV123 and OTB-2015, and the paper reports state-of-the-art performance on four datasets.
3 Analyzing Deep Features for Tracking
The analysis examines why deep features underperform in tracking and finds that scarce data, low spatial resolution, and feature-specific training choices matter. It motivates independent shallow/deep training and fusion to exploit their complementary strengths.
- 3.1 Motivation: Deep tracking is limited by the mismatch between large data requirements and the single labeled frame available at initialization.The analysis focuses on understanding deep features relative to shallow features under this scarce-data setting.
- 3.1 Motivation: Low spatial resolution makes deep features harder to use for accurate target localization, while their semantic invariance can support robustness.Shallow handcrafted features are primarily designed for precise localization, whereas deep features tolerate small translations and scale changes.
- 3.3 Data Augmentation: Data augmentation consistently improves deep-feature tracking but lowers performance for shallow features.On OTB-2015, every tested augmentation except shift improves deep features by more than 1%, while blur gives a 4% gain over the no-augmentation baseline.
- 3.4 Robustness/Accuracy Trade-off: Deep and shallow features require different accuracy-robustness trade-offs during training.A wide label function benefits deep features, while shallow features perform best with a low σ because their higher-resolution representations support accurate localization.
- 3.4 Robustness/Accuracy Trade-off: 5.3% AUC improvement comes from combining all evaluated augmentations for deep features, and a wider label function adds a further 0.5%.The combined configuration outperforms the baseline on OTB-2015, which uses no augmentation and the default σ.
- 3.5 Observations: These findings show that shallow and deep models should be trained independently before fusion because their complementary properties require different data and annotation strategies.The subsequent fusion problem is to combine the independently optimized models effectively.
4 Adaptive Fusion of Model Predictions
The approach adaptively fuses deep and shallow predictions because their complementary properties support robustness and accurate localization. It evaluates prediction quality through sharpness and confidence margins, then jointly selects model weights and target state.
- Complementary Model Predictions: Deep and shallow models have complementary characteristics: deep scores provide robust high-confidence modes but coarse localization, while shallow scores offer sharp localization with distractor peaks.The adaptive fusion combines these properties to produce a sharper and less ambiguous score function.
- Prediction Quality Measure: The prediction quality measure rewards both score sharpness around the predicted state and the margin over distractor peaks.A sharp peak supports accurate localization, while a large margin indicates a distinctive and robust prediction.
- Prediction Quality Measure: The measure balances accuracy and robustness through κ, which controls the rate at which the distance weighting approaches one as displacement increases.The formulation also extends to higher-dimensional state spaces, including translation and scale transformations.
- Prediction Quality Measure: A large quality measure indicates that distractor peaks do not make the prediction ambiguous, whereas a similar secondary peak yields quality approximately zero.This quality behavior explicitly captures robustness through separation from competing candidate locations.
- Adaptive Fusion Optimization: The fused score combines deep and shallow scores with weights β, and the method jointly estimates β and the target state by minimizing a constrained loss.The weights are non-negative, sum to one, and include a regularization term penalizing large deviations.
- Adaptive Fusion Optimization: For each candidate state sampled from local maxima, the method solves a three-variable quadratic program and selects the candidate with the lowest overall loss.This candidate-wise optimization adds little computational overhead.
5 Experiments
Experiments show that independently tailored deep and shallow models, combined through adaptive fusion, improve tracking accuracy and robustness across challenging benchmarks. The approach achieves state-of-the-art results, including a 17% relative EAO gain over LSART on VOT2017.
- Evaluation setup: The tracker is evaluated on NFS, VOT2017, UAV123, and Temple128, with OTB-2015 used for validation and hyperparameter tuning.Because dropout augmentation is stochastic, the tracker is run 10 times per sequence and average scores are reported.
- Ablative study: Fixed sum-fusion outperforms baseline ECO in robustness across a wide range of shallow-model weights on NFS and OTB-E.This supports using specifically tailored training procedures for deep and shallow features.
- Ablative study: Adaptive fusion addresses the trade-off between accuracy and robustness by dynamically computing model weights for each frame.Shallow features favor accuracy, whereas deep features are crucial for robustness; adaptive fusion improves both relative to fixed weights.
- Ablative study: Adaptive fusion emphasizes shallow features during scale changes and deep features under occlusion, clutter, and out-of-plane rotation.Early in the Soccer sequence, before major appearance changes, the shallow model is prominent.
- Network generalization: The proposed tracker provides consistent gains over ECO across VGG-M, GoogLeNet, and ResNet-50 architectures on NFS and OTB-E.Results are reported using AUC scores, while ECO fails to exploit more sophisticated deeper architectures.
- State-of-the-art comparison: 17% relative gain in EAO over LSART: the tracker achieves 0.378 versus 0.323 on VOT2017.The approach also obtains a 17% relative robustness gain and the best accuracy in the comparison.
- State-of-the-art comparison: 10% relative gain over CCOT: the approach significantly outperforms CCOT on NFS.On Temple128, it achieves an AUC of 62.2% versus ECO's 60.5%; on UAV123, it achieves 55.0% versus ECO's 53.7%.
6 Conclusions
The paper concludes that systematic analysis and complementary fusion of deep and shallow appearance models are central to improving visual tracking. Experiments report state-of-the-art performance across all four evaluated datasets.
- Individually tailoring training for shallow and deep features is crucial for achieving high robustness and accuracy.
- The proposed fusion strategy combines deep and shallow appearance models by leveraging their complementary characteristics.
- State-of-the-art performance is achieved on all four challenging datasets.
Supplementary Material
The supplementary material provides derivations, dataset details, and additional tracking results that extend the main paper's analysis and evaluation.
- Additional derivation: Section S1 derives properties of equation (4) from the main paper.
- Dataset details: Section S2 gives additional details about the OTB-H and OTB-E datasets introduced in the main paper.
- Additional results: Sections S3-S5 provide additional results on VOT2016, Need For Speed, UAV123, Temple128, and the full OTB-2015 dataset.
S1 Derivation of Quality Measure Properties
The quality measure rewards accurate, sharply localized predictions and robustness against distractor peaks. Its upper bound is tied to local curvature at the predicted optimum and can be tight.
- Assumptions: The derivation assumes a twice continuously differentiable detection score and a prediction at a local maximum.For discrete scores, suitable interpolation is used; the Hessian at the local maximum is negative semidefinite.
- Bound derivation: The upper-bound derivation uses the eigenvalue decomposition of the symmetric Hessian and directions associated with its eigenvalues.The relevant bound is obtained by considering eigenvectors corresponding to λ2 and λ1.
- Bound: The quality is bounded above by the minimum curvature at the local optimum.The bound is tight: some detection scores attain it, including a Gaussian centered at the optimum.
S2 Description of OTB-H and OTB-E
OTB-H and OTB-E are subsets of OTB-2015 constructed from per-video results of four state-of-the-art trackers, serving as validation and ablation-test sets.
- OTB-H contains 23 videos and is used as an explicit validation set for setting hyperparameters.
- OTB-E contains 73 videos and is used as the test set for the ablation study.
- Both subsets are constructed from OTB-2015 using per-video results from four state-of-the-art trackers.
- The listed OTB-H and OTB-E videos define the contents of the validation and ablation-test subsets.
S3 Results on VOT2016
On VOT2016, the method achieves an EAO score of 0.443 and outperforms the second-best method, ECO, by a relative gain of 18%.
- 0.443 EAO is achieved on VOT2016, significantly outperforming ECO with an 18% relative gain.
S4 Additional Results on NFS, UAV123 and Temple128
The method is evaluated with distance-precision plots on NFS, UAV123, and Temple128, where it significantly outperforms state-of-the-art trackers across all three datasets.
- Distance precision ranks trackers by the percentage of frames whose prediction-center distance from ground truth is below threshold 20.
- Precision plots are provided for NFS, UAV123, and Temple128, with the top 10 trackers' DP scores shown in the legend.
- 7.0% absolute gain is achieved on NFS over the state-of-the-art tracker.
- 4.4% absolute gain is achieved on Temple128 over the state-of-the-art tracker.
- 2.4% absolute gain is achieved on UAV123 over the state-of-the-art tracker.
S5 Results on OTB-2015
On the full 100-video OTB-2015 dataset, the method achieves the best AUC and DP scores, with absolute gains of 1.3% and 1.7%, respectively.
- 100 videos from OTB-2015 are evaluated using AUC and DP to rank trackers.
- The method achieves the best OTB-2015 scores in both AUC and DP.
- 1.3% absolute gain is achieved in AUC, while 1.7% absolute gain is achieved in DP.