Source-linked AI summary
Robust Visual Tracking via Hierarchical Convolutional Features
Chao Ma, Jia-Bin Huang, Xiaokang Yang, Ming-Hsuan Yang
TL;DR
Visual tracking must remain accurate under severe appearance variation, while existing approaches struggle to balance semantic robustness, spatial precision, and reliable recovery. The paper combines hierarchical CNN correlation filters with coarse-to-fine localization and a conservative long-term proposal-based classifier. Experiments on large-scale benchmarks show favorable accuracy and robustness against state-of-the-art trackers, with scope limitations related to initialization and fixed recovery parameters.
Problem
Existing deep trackers face challenges in exploiting CNN features effectively and extracting training samples, while deep semantic features can lack the spatial precision needed for tracking.
Method
The method learns adaptive correlation filters across hierarchical CNN layers for coarse-to-fine localization and uses a conservative long-term filter on local and whole-image proposals for scale estimation and re-detection.
Results
The proposed algorithm performs favorably against state-of-the-art methods in accuracy and robustness on OTB2013, OTB2015, VOT2014, and VOT2015.
Takeaways & Limitations
Combining semantic deep features with spatially precise hierarchical responses provides the tracker’s supported basis for handling appearance variation and localization.
Takeaways & Limitations
Performance is sensitive to first-frame center initialization and fixed re-detection parameters, and the approach performs less well on VOT2014 and VOT2015 than on OTB2013 and OTB2015.
Abstract
from arXiv · showhide
In this paper, we propose to exploit the rich hierarchical features of deep convolutional neural networks to improve the accuracy and robustness of visual tracking. Deep neural networks trained on object recognition datasets consist of multiple convolutional layers. These layers encode target appearance with different levels of abstraction. For example, the outputs of the last convolutional layers encode the semantic information of targets and such representations are invariant to significant appearance variations. However, their spatial resolutions are too coarse to precisely localize the target. In contrast, features from earlier convolutional layers provide more precise localization but are less invariant to appearance changes. We interpret the hierarchical features of convolutional layers as a nonlinear counterpart of an image pyramid representation and explicitly exploit these multiple levels of abstraction to represent target objects. Specifically, we learn adaptive correlation filters on the outputs from each convolutional layer to encode the target appearance. We infer the maximum response of each layer to locate targets in a coarse-to-fine manner. To further handle the issues with scale estimation and re-detecting target objects from tracking failures caused by heavy occlusion or out-of-the-view movement, we conservatively learn another correlation filter, that maintains a long-term memory of target appearance, as a discriminative classifier. We apply the classifier to two types of object proposals: (1) proposals with a small step size and tightly around the estimated location for scale estimation; and (2) proposals with large step size and across the whole image for target re-detection. Extensive experimental results on large-scale benchmark datasets show that the proposed algorithm performs favorably against state-of-the-art tracking methods.
1 INTRODUCTION
Visual tracking remains difficult under appearance changes, while existing deep trackers face challenges in exploiting CNN features and extracting training samples. The proposed approach combines hierarchical CNN features, adaptive correlation filters, coarse-to-fine inference, and long-term region-proposal-based recovery.
- Visual tracking must handle illumination variation, occlusion, background clutter, abrupt motion, and targets moving out of view.
- Existing deep trackers typically use CNNs as online classifiers, but their key challenges are exploiting CNN features and extracting effective training samples.
- The method combines semantic information from deeper CNN layers with precise localization from earlier layers to represent targets across appearance abstractions.
- Adaptive correlation filters are learned on hierarchical CNN layers, and multi-level response maps are searched in a coarse-to-fine manner.
- A conservatively updated long-term correlation filter supports scale estimation and target re-detection using region proposals.
- Experiments on OTB2013, OTB2015, VOT2014, and VOT2015 show favorable accuracy and robustness against state-of-the-art methods.
2 RELATED WORK
Related tracking methods use detection, correlation filters, deep features, or region proposals, each addressing different aspects of efficiency, representation, adaptation, or recovery. The proposed direction combines hierarchical CNN representations with multi-level localization and region proposals.
- Tracking by Detection: Tracking-by-detection methods update classifiers online from local positive and negative samples, but sampling inaccuracies can cause gradual drift.
- Tracking by Correlation Filters: Correlation-filter trackers are computationally efficient through fast Fourier transforms and typically regress circularly shifted features to soft labels.
- Tracking by Correlation Filters: This work differs from prior correlation-filter methods by learning filters on deep features and using multiple hierarchical convolutional layers instead of one filter.
- Tracking by Deep Neural Networks: Deep tracking methods use CNNs for target detection, but limited first-frame state annotations constrain available tracking training data.
- Tracking by Deep Neural Networks: Pre-trained CNN features can be regressed to binary labels, Gaussian soft labels, or bounding-box parameters to produce target response maps.
- CNN Feature Resolution: The conv5-4 layer is less effective for step-edge localization than conv3-4 because its spatial resolution is lower.
- Tracking by Region Proposals: Region proposals provide fewer candidate bounding boxes than exhaustive sliding windows, enabling CNN-based classification and supporting tracking applications.
3 OVERVIEW
HCFT* represents targets with convolutional features from multiple abstraction levels and estimates translation through hierarchical correlation responses. A long-term filter and EdgeBox proposals additionally address scale estimation and recovery after tracking failures.
- HCFT* uses deeper convolutional features for semantic robustness to appearance variation and earlier features for fine-grained spatial localization.
- The tracker learns an adaptive linear correlation filter for each convolutional layer and searches their response maps from coarse to fine for target localization.
- A long-term memory filter scores the patch at the estimated location and triggers whole-image proposal search when its confidence falls below T0.
- EdgeBox generates tightly spaced local scale proposals and widely spaced whole-image detection proposals for scale estimation and target re-detection.
4 PROPOSED ALGORITHM
The proposed tracker combines hierarchical CNN features with adaptive correlation filters, using coarse-to-fine responses for localization and a conservative long-term filter for scale estimation and recovery.
- Hierarchical Convolutional Features: Pre-trained CNN convolutional maps capture target appearance at multiple abstraction levels, balancing semantic discrimination against spatial resolution.Deeper features are more semantically discriminative, while earlier features retain finer localization detail; the method avoids pooling outputs and upsamples maps when needed.
- Correlation Filters: The tracker learns correlation filters from circularly shifted feature maps with Gaussian soft labels and locates targets at response-map maxima.Soft labels avoid hard-thresholded samples, and responses are computed efficiently in the Fourier domain before maximizing the map.
- Coarse-to-Fine Translation Estimation: Target translation is inferred coarse-to-fine by using deeper-layer maxima to constrain searches on earlier, higher-resolution response maps.Response values from later layers are weighted and propagated to earlier layers, with the final location selected on the finest-resolution layer.
- Coarse-to-Fine Translation Estimation: The soft weighting scheme compensates for inconsistent response ranges and remains robust to noisy maxima during heavy occlusion and abrupt motion.On KiteSurf, it tracked the target across the sequence, while alternatives and single-layer responses were unreliable; ablations reported around +1% accuracy on a 100-sequence benchmark.
- Region Proposals: EdgeBox generates tightly spaced local proposals for scale estimation and widely spaced whole-image proposals for target re-detection.A separately learned correlation filter uses gradients and intensities with conservative updates to preserve long-term target appearance.
5 IMPLEMENTATION DETAILS
The implementation uses hierarchical convolutional features, correlation filters, and a conservative long-term classifier within a coarse-to-fine tracking algorithm.
- The feature extractor uses VGGNet-19 convolutional layers conv3-4, conv4-4, and conv5-4, excluding pooling outputs to preserve spatial resolution.
- The algorithm initializes hierarchical correlation filters and a classifier, then iteratively crops search windows, estimates position and scale, and updates models until the sequence ends.
- The long-term classifier is updated only when confidence exceeds T0, while confidence below T0 triggers target re-detection.
- Scale estimation uses the highest-confidence proposal around the estimated position, whose width and height determine the updated target size.
- The implementation resizes feature channels to a fixed spatial size and uses shared layer parameters for translation estimation.
- Key settings include λ = 10^-4, Gaussian-label kernel width 0.1, update rate η = 0.01, cosine-window weighting, and layer weights 1, 0.5, and 0.25.
6 EXPERIMENTS
The method is evaluated on four visual-tracking benchmarks using standard protocols and compared with state-of-the-art trackers across accuracy and speed measures.
- The evaluation covers OTB2013, OTB2015, VOT2014, and VOT2015 using benchmark protocols and the same parameters for all sequences and sensitivity analyses.
- The reported measures include distance precision rate, overlap ratio, center location error, and tracking speed in comparison with state-of-the-art trackers.
6.1 OTB datasets
On OTB2013 and OTB2015, HCFT* combines hierarchical CNN features, coarse-to-fine localization, adaptive correlation filters, and proposal-based scale estimation and recovery, performing favorably against state-of-the-art trackers.
- Evaluation protocol: The tracker is evaluated on OTB2015 using OPE, SRE, and TRE, with distance precision at 20 pixels and overlap success measured by area under the curve.OTB2015 contains 100 sequences, while OTB2013 contains 50 and is a subset.
- Overall performance: HCFT* improves over HCFT by 3.3% in distance precision and 6.5% in overlap success on OTB2015.It also reduces center location error by 6.1 pixels on OTB2015.
- Overall performance: HCFT* gains 3.2% in distance precision and 5.3% in overlap success on OTB2013 compared with HCFT.The center location error is reduced by 6.3 pixels on OTB2013.
- Attribute-based evaluation: HCFT* performs well across illumination, rotation, scale variation, occlusion, deformation, motion blur, fast motion, out-of-view, clutter, and low-resolution attributes.With its recovery module, it gains 4.0% on occlusion and 6.1% on out-of-view scenes.
- Classifier analysis: The long-term-memory classifier remains stable under noisy updates and supports confidence scoring for detecting tracking failures.HCFT* performance is not sensitive to thresholds between 0.2 and 0.5; the experiments use T0 = 0.2.
- Qualitative evaluation: On the challenging MotorRolling sequence, HCFT* achieves a 94.5% distance precision rate when other methods fail to track targets well.The authors attribute performance to hierarchical CNN features, adaptive correlation filters, and region proposals for recovery and scale changes.
6.2 VOT2014 Dataset
On VOT2014, HCFT* is evaluated under baseline and region-noise initialization using accuracy and robustness ranks, and performs favorably among the compared trackers.
- Dataset and protocol: The VOT2014 experiments use 25 real-world sequences and include ground-truth initialization and randomly perturbed initial target locations.The challenge protocol re-initializes trackers with ground truth after failures.
- Evaluation results: HCFT* performs well in both accuracy and robustness under baseline and region-noise validations on VOT2014.The ranking uses overlap-based accuracy and the number of tracking failures for robustness.
- Evaluation results: HCFT* performs favorably among all other trackers in the VOT2014 accuracy-robustness plots.Trackers nearer the upper-right region represent stronger combined performance.
6.3 VOT2015 Dataset
On VOT2015, HCFT* is tested on 60 challenging sequences using accuracy and robustness metrics and achieves high overall rankings among evaluated trackers.
- Evaluation results: HCFT* ranks second in robustness and third in overall overlap on VOT2015.The results are reported as average ranks under baseline experiments.
- Dataset and protocol: The VOT2015 evaluation uses accuracy and robustness metrics across 60 challenging video sequences with substantial variation.Average accuracy and robustness ranks are compared with trackers submitted to VOT2015.
6.4 Failure Cases
Failure cases expose boundaries of HCFT*: re-detection can fail during similar-person occlusions, scale proposals can miss drastic changes, and unmodeled in-plane rotation can cause drift.
- Occlusion: Long-term occlusion can prevent re-detection when the target person resembles surrounding people.The re-detection scheme is not successfully activated in the Girl2 sequence.
- Scale variation: Small-step proposals tightly centered on the estimate do not handle drastic scale changes well.This limitation appears in the Jump sequence during scale estimation.
- Rotation: Because HCFT* does not model in-plane rotation, rotated targets can include background pixels as foreground and cause inaccurate updates and drift.This failure is illustrated by the Gymnastics sequence.
7 CONCLUSION
The proposed tracker combines hierarchical CNN features with correlation filters to improve visual tracking accuracy and robustness, including scale estimation and target re-detection. The approach performs favorably against state-of-the-art methods, but remains sensitive to initialization, empirical weighting, and fixed re-detection parameters.
- 7 CONCLUSION: The tracker uses semantic information from late convolutional layers and spatial details from early layers, combining hierarchical correlation maps for coarse-to-fine localization.A separate conservative correlation filter maintains long-term target appearance for scale estimation and re-detection using EdgeBox proposals.
- 7 CONCLUSION: The algorithm was evaluated on OTB2013, OTB2015, VOT2014, and VOT2015, with results favoring state-of-the-art methods in accuracy and robustness.
- 7 CONCLUSION: Performance is sensitive to the initial center location because correlation filters regress circularly shifted deep features to soft labels.This sensitivity is especially relevant when ground-truth rectangles contain substantial noisy background or their geometric centers differ from actual target centers.
- 7 CONCLUSION: The hierarchical location inference uses empirical weights without principled interpretations, motivating learning-based weighting in future work.
- 7 CONCLUSION: Fixed thresholds for activating re-detection and accepting or rejecting detections cannot be applied effectively to all sequences.The authors identify adaptive re-detection modules as a direction for recovering targets from tracking failures.