PaperlayerView exact revision on arXiv

Source-linked AI summary

Good Features to Correlate for Visual Tracking

Erhan Gundogdu, A. Aydin Alatan

arXiv:1704.06326v2cs.CV

TL;DR

The paper tackles the reliance of correlation-filter trackers on classification-pretrained features by learning deep fully convolutional features for tracking. It trains these features with an efficient backpropagation method based on the correlation-filter loss and reports improved benchmark tracking performance, including an 18% expected-average-overlap increase and 25% fewer failures.

  • Problem

    Correlation-filter trackers rely on features pretrained for object classification, whereas tracking requires robust features learned for correlation-filter localization.

  • Method

    The paper trains fully convolutional networks directly with a correlation-filter cost function using an efficient backpropagation formulation and stochastic gradient descent.

  • Results

    18% increase in expected average overlap and 25% fewer tracking failures are reported after fine-tuning a state-of-the-art network and integrating it into a top-performing VOT2016 correlation-filter tracker.

  • Takeaways & Limitations

    The learned features improve correlation-filter tracking performance across benchmark datasets while reducing dependence on classification-trained networks.

Abstract

from arXiv · show

During the recent years, correlation filters have shown dominant and spectacular results for visual object tracking. The types of the features that are employed in these family of trackers significantly affect the performance of visual tracking. The ultimate goal is to utilize robust features invariant to any kind of appearance change of the object, while predicting the object location as properly as in the case of no appearance change. As the deep learning based methods have emerged, the study of learning features for specific tasks has accelerated. For instance, discriminative visual tracking methods based on deep architectures have been studied with promising performance. Nevertheless, correlation filter based (CFB) trackers confine themselves to use the pre-trained networks which are trained for object classification problem. To this end, in this manuscript the problem of learning deep fully convolutional features for the CFB visual tracking is formulated. In order to learn the proposed model, a novel and efficient backpropagation algorithm is presented based on the loss function of the network. The proposed learning framework enables the network model to be flexible for a custom design. Moreover, it alleviates the dependency on the network trained for classification. Extensive performance analysis shows the efficacy of the proposed custom design in the CFB tracking framework. By fine-tuning the convolutional parts of a state-of-the-art network and integrating this model to a CFB tracker, which is the top performing one of VOT2016, 18% increase is achieved in terms of expected average overlap, and tracking failures are decreased by 25%, while maintaining the superiority over the state-of-the-art methods in OTB-2013 and OTB-2015 tracking datasets.

I. INTRODUCTION

The paper addresses the limitation of using classification-pretrained features in correlation-filter trackers by learning fully convolutional features directly for tracking. It proposes an efficient loss-specific training framework and integrates the learned networks into correlation-filter trackers.

  • Correlation-filter trackers commonly use deep features pretrained for large-scale object classification, motivating tracking-specific feature learning.
  • The proposed framework trains a fully convolutional network using paired patches and minimizes the difference between estimated and desired correlation responses.
  • The paper presents a framework for training fully convolutional deep networks with the correlation-filter cost function.
  • An efficient backpropagation formulation is provided for optimizing the network according to the correlation-filter-based loss.
  • The learned network is integrated into DSST, SAMF, and CCOT, boosting their performance on VOT2016, OTB-2013, and OTB-2015.

C. Correlation Filter Based Trackers

Correlation filter trackers efficiently learn filters from multi-channel features, with extensions for scale estimation, spatial regularization, deep features, and continuous resolutions. DSST and CCOT are summarized as the base trackers for integrating the proposed learned features.

  • Correlation filter foundations: Correlation filters minimize error between desired responses and circularly correlated object patches, enabling efficient frequency-domain optimization.Multi-channel extensions and HOG features broaden the original formulation.
  • Tracker extensions: Spatial regularization increases the search range, while background patches provide a trade-off between computational complexity and accuracy.These are presented as extensions addressing imperfect training examples and background modeling.
  • Deep and continuous features: Pre-trained deep CNN feature maps and continuous-domain filters address feature extraction and differing feature-map resolutions, respectively.The continuous formulation combines feature maps from different resolutions, including deep layers.
  • Evaluation context: The proposed framework is evaluated by integrating learned features into DSST and CCOT, while related work also includes customized deep architectures, recurrent models, and tracker combinations.The supplied related-work passages position these methods among several neural and ensemble tracking directions.
  • DSST: DSST estimates object location with multi-channel filters and selects scale factors using a separate scale correlation filter.The optimal scale is the index with the highest test correlation response.

B. Continuous Convolution Operators for Visual Tracking

Continuous convolution operators model correlation filtering over feature maps with different dimensions and resolutions. The formulation interpolates discrete signals, defines continuous target responses, and optimizes a spatially regularized filter cost.

  • Motivation: CCOT combines feature maps of different resolutions, especially deep feature maps from different network layers.This formulation differs from approaches assuming constant feature-map dimensions.
  • Continuous representation: Training samples may contain feature maps with different dimensions, represented as signals in a continuous-domain formulation.The support interval is assumed to be [0, T).
  • Continuous representation: An interpolation operator converts each discrete feature map into a continuous signal using an interpolation function.The interpolation function is denoted by b_d ∈ L2(T).
  • Filter formulation: The continuous correlation operator maps a sample to a target confidence response while estimating a multi-channel filter.Discrete inputs are first converted to the continuous domain before correlation.
  • Optimization objective: The continuous-domain cost function weights training samples by importance and uses a spatial penalty to suppress filter boundaries.The importance weight is α_j, and w regularizes the filter spatially.
  • Optimization objective: The filter cost is optimized through frequency-domain normal equations and Conjugate Gradient Descent, followed by multi-scale response evaluation.The best matching scale is selected from responses at S scales.

IV. PROPOSED FRAMEWORK FOR FEATURE LEARNING

The framework learns fully convolutional features directly for correlation-filter tracking by minimizing correlation-response error over triplet patches. Its loss optimizes network parameters around a correlation-filter solution, while convolutional feature maps provide shift-invariant representations.

  • Training formulation: Triplet samples pair a centered template, a shifted test patch, and a desired response peaked at the object’s correct displacement.The formulation extends directly from one-dimensional derivations to two-dimensional signals because the operations are separable across horizontal and vertical dimensions.
  • Feature design: The feature-generation function is designed to preserve shift invariance, motivating a fully convolutional CNN with convolutional, normalization, and pooling layers.The transformed test representation should correspond approximately to a shifted transformed template representation.
  • Objective: Stochastic gradient descent minimizes a cost over N triplets by reducing the difference between estimated and desired correlation responses.The network output and correlation-filter solution depend on the network parameters θ.
  • Objective: The proposed cost is minimized with respect to network parameters θ for a given correlation-filter solution, unlike the conventional cost that optimizes the filters themselves.The filter solution already penalizes filter norms, so the proposed cost removes the conventional regularization term.

C. Gradient of the Loss, L(θ)i

The gradient derivation applies the multivariable chain rule to the correlation-filter loss and expresses its required terms in the DFT domain. Standard backpropagation then computes parameter and activation gradients for randomly sampled batches.

  • Gradient derivation: The loss gradient with respect to network parameters is obtained through the multivariable chain rule.The derivation first identifies gradients with respect to network outputs and correlation-filter terms.
  • Gradient derivation: The error signal and intermediate variables provide the terms needed to express gradients with respect to feature maps and filter coefficients.These terms are substituted into the chain-rule expressions for the loss gradient.
  • Backpropagation: The Jacobians of the feature maps with respect to network parameters can be computed using standard backpropagation tools.The remaining Jacobian connecting filter coefficients and feature maps is converted to the DFT domain before evaluation.
  • Frequency-domain calculation: The derivative of the correlation-filter division is computed independently for each frequency component in the DFT domain.The formulation uses the derivative of the division rule after relating filter coefficients and feature maps frequency by frequency.
  • Frequency-domain calculation: The DFT-domain operations preserve real-valued signals through conjugation and circular time reversal operations.The matrix for circular time reversal fixes the first element while reversing the remaining signal positions.
  • Optimization: Gradients are computed for batches of triplets, followed by gradient-descent optimization over randomly sampled batches.The derivation is applied to the feature maps of each training example in a batch.
  • Limitation: Independent feature channels remain a future study because reducing denominator frequency components can cause gradient overflow despite learning-rate selection.The authors retain the coupled-channel formulation because it is optimal according to the referenced correlation-filter objective.

D. Computational Complexity and Its Reduction

Backpropagation is efficient in the DFT domain but scales with the number of feature maps, motivating an auxiliary low-channel layer to reduce training cost.

  • Complexity: O(dPlog(P)) is the complexity of backpropagating one triplet, where d is the number of feature maps and P is the signal length.The computational burden arises because the gradient operation is repeated for each of the d feature maps.
  • Complexity: 36, 16 and 6 frames per second are reported for d values 32, 64 and 128, respectively, in the Matlab implementation.Training speed decreases as the number of feature maps increases.
  • Complexity reduction: An auxiliary layer with fewer feature maps is added above VGG’s conv-5 layer to reduce computation time during convolutional-network training.The design retains the observation that localization robustness improves as the number of feature maps increases.

V. EXPERIMENTAL RESULTS

The experiments evaluate proposed tracker configurations on OTB and VOT benchmarks using dataset-specific tracking metrics. Training data are generated from video sequences with centered and shifted object patches, while dataset overlap and scale constrain evaluation and model training.

  • Evaluation datasets: The proposed configurations are evaluated on OTB-2013, OTB-2015 and VOT2016 datasets.OTB-2013 is a subset of OTB-2015, while VOT2016 is the 2016 challenge dataset of the VOT committee.
  • Evaluation metrics: OTB ranks trackers using success-curve AUC and also reports overlap precision based on predicted and ground-truth bounding-box overlap.The success curve counts successfully tracked frames across overlap-ratio thresholds.
  • Evaluation metrics: VOT2016 evaluates accuracy, failure, and expected average overlap as its three major tracking metrics.Accuracy is mean intersection over union, while failure is the mean number of failures per sequence.
  • Dataset generation: 200K training examples are generated from 60 VOT2015 sequences by cropping and resizing object-centered and shifted patches.The shifted patch center is sampled uniformly within ±0.3 times the object width and height.
  • Dataset limitations: The VOT2015-derived dataset overlaps with OTB-2013 because 11 sequences occur in both benchmarks, preventing evaluation on the full OTB-2013 set.The authors also describe VOT2015 as too small for conveniently training or fine-tuning state-of-the-art convolutional networks despite its generated samples.
  • Dataset limitations: Existing tracking benchmarks contain fewer than a thousand sequences and include overlapping datasets, limiting the data available for training large network models.The authors also note similarities in appearance across some benchmark sequences.
  • Custom architectures: Two custom architectures are trained on CFCF VOT2015, with one producing a single feature map and the other producing multiple feature maps.They are integrated into DSST as DSST CFCF and DSST MCFCF for single- and multiple-channel correlation filters.

C. CNN Architectures

The paper develops custom fully convolutional architectures for correlation-filter tracking and adapts VGG-M through fine-tuning and an auxiliary layer. These learned feature configurations are evaluated against alternative feature types and trackers.

  • Custom architectures: Two custom architectures produce either single or multiple feature maps for correlation filtering.The smaller architecture uses 101×101 RGB inputs, four convolutional layers, batch normalization, and ReLU activations.
  • VGG-M adaptation: VGG-M is truncated before fc6 because the framework accepts convolutional layers for shift-invariant tracking.The selected convolutional layers include conv-1 and conv-5 together with RGB channels.
  • VGG-M adaptation: A 32-feature-map auxiliary layer makes VGG-M training feasible despite its 512-channel final convolutional layer.The fine-tuned VGG-M convolutional layers are integrated into CCOT as CFCF.
  • Feature comparison: The learned multiple-feature configuration, DSST MCFCF, outperforms the compared feature combinations while using 11 feature maps instead of DSST’s 28 HOG maps.DSST CFCF uses 4 feature maps, whereas DSST MCFCF uses 11.

2) Comparison with respect to the state-of-the-art trackers:

The learned features perform competitively or favorably against state-of-the-art trackers across OTB-2013, OTB-2015, and VOT2016. The strongest reported VOT2016 comparison improves CCOT’s EAO while also reducing failures and computation.

  • OTB comparisons: With 27 learned feature maps, MCFCF CCOT approaches CCOT’s performance with 611 features and outperforms deepSRDCF with 96 features on OTB evaluation.It also performs favorably against deepSRDCF and SiamFC in center localization error.
  • OTB-2013: The proposed features perform close to CCOT across most OTB-2013 attributes, including scale variation, deformation, and background clutter.The attribute analysis reports AUC values for 11 attributes over 40 sequences.
  • OTB comparisons: On OTB-2013 and OTB-2015, the proposed configurations remain superior to baseline SAMF and CCOT while comparing favorably or comparably with deepSRDCF and HCF.The evaluation uses localization-error and overlap curves, with per-attribute AUC values reported for 100 OTB-2015 sequences.
  • OTB-2015: 67.8 AUC is achieved on OTB-2015, compared with 57.5 for DCFNet and 58.9 for CFNet.The paper attributes the difference to targeting accuracy rather than speed.
  • VOT2016: 18.7% EAO improvement over CCOT is reported on VOT2016, while the proposed tracker outperforms all existing participants shown.The comparison uses the VOT2016 ranking plot and table, where EAO combines accuracy and robustness.
  • VOT2016: 25% fewer failures and at least 3.5% higher raw accuracy are reported relative to CCOT, while optimization iterations decrease from 5 to 1.The paper also reports favorable per-attribute accuracy and failure values for most attribute types.

G. Ablation Study

The ablation study evaluates input color, convolutional-layer selection, training-set size, and failure cases for the proposed tracker on VOT2016. Fine-tuning and suitable layer combinations improve tracking performance, while failures remain under appearance and motion changes.

  • Utilization of chroma components: More than 11% EAO improvement over the pre-trained network accompanied fine-tuning, with 2% higher accuracy and 25% fewer failures.Grayscale input reduced performance but still reached an EAO of 0.26; fine-tuning improved EAO by 12% over ImageNet training.
  • Performance analyses of different layers: Conv-2 achieved the best individual EAO, accuracy, and robustness, while combining conv-2 with conv-5 performed favorably against either layer alone.The study compared individual lower- and higher-level convolutional layers and their combinations.
  • Training from scratch: Increasing training samples from 500 to 200K improved the average number of failures on VOT2016.The randomly initialized network outperformed the model trained with 500 samples, while very small training sets could cause overfitting.
  • Failure cases: Some CFCF and CCOT failure cases coincide, despite the proposed features improving tracking over features extracted from the pre-trained model.Reported possible challenges include train-test differences, abrupt aspect-ratio changes, and object deformation.
  • Overall outcome: The learned feature method improved the top-performing VOT2016 tracker by at least 18% in EAO and supported favorable benchmark performance.The framework was trained on VOT2015 and ILSVRC Video frames and integrated into state-of-the-art correlation-filter trackers.

APPENDIX A DERIVATION OF dL dhk AND dL

This appendix develops efficient derivatives of the correlation-filter loss with respect to filter components and network feature activations. The derivation uses the correlation theorem to express the gradients for efficient calculation.

  • The correlation theorem is used to rewrite the loss and derivative signals in forms suitable for efficient calculation.The derivation substitutes the error expression into the gradient and applies the theorem again.
  • The appendix first formulates the derivative of the loss with respect to a particular component of the correlation filter h_k[m].
  • The derivative of the loss with respect to x_l[m] is derived using the same correlation-theorem formulation.

OF THE PREVIOUS ONE

The analysis relates correlation errors across consecutive convolutional layers under an additive appearance-noise assumption. It argues that reducing correlation error in one layer is likely to reduce error in another when the two errors are nearly equal.

  • Appearance-noise model: The analysis models appearance differences between template and test patches as additive noise in the feature representation.The test feature in the DFT domain is represented as Z = X + µ, where µ is caused by appearance change.
  • Two-feature-map analysis: With two feature maps, the representation and noise are decomposed into X = X1 + X2 and µ = µ1 + µ2.The derivation assumes a fixed 1×1 convolutional kernel with values of 1 and ignores bias terms.
  • Correlation-error comparison: The resulting correlation response error is compared across single- and multiple-channel formulations.The derivation expands the correlation terms for the two feature maps and their individual noises.
  • Error relationship: Because both errors are proportional to µ1 and µ2, reducing their sum tends to reduce the individual noise terms.The argument extends to more than two feature maps under the same assumption.
Loading 1704.06326v2…