Source-linked AI summary
End-to-end representation learning for Correlation Filter based tracking
Jack Valmadre, Luca Bertinetto, João F. Henriques, Andrea Vedaldi, Philip H. S. Torr
TL;DR
Object tracking must learn representations for previously unseen targets from minimal supervision, while prior Correlation Filter methods used manually designed or task-mismatched features. The paper makes the Correlation Filter a differentiable layer with a closed-form derivative, enabling end-to-end feature learning; experiments show lightweight networks can reach state-of-the-art performance at high framerates.
Problem
Tracking must handle previously unseen object classes from a bounding box, while prior Correlation Filter methods used features that were manually designed or trained for another task.
Method
The paper interprets the Correlation Filter learner as a differentiable neural-network layer and propagates gradients through its closed-form solution to train CNN features end-to-end.
Results
Lightweight networks achieve state-of-the-art performance on multiple benchmarks at high framerates, whereas adding a Correlation Filter does not significantly improve sufficiently deep Siamese networks.
Takeaways & Limitations
The Correlation Filter can be integrated into training so shallow networks rival slower, deeper counterparts while retaining efficient online learning.
Takeaways & Limitations
Future work is needed to account for adaptation over time and to extend gradient back-propagation through learning problems to one-shot learning and domain adaptation.
Abstract
from arXiv · showhide
The Correlation Filter is an algorithm that trains a linear template to discriminate between images and their translations. It is well suited to object tracking because its formulation in the Fourier domain provides a fast solution, enabling the detector to be re-trained once per frame. Previous works that use the Correlation Filter, however, have adopted features that were either manually designed or trained for a different task. This work is the first to overcome this limitation by interpreting the Correlation Filter learner, which has a closed-form solution, as a differentiable layer in a deep neural network. This enables learning deep features that are tightly coupled to the Correlation Filter. Experiments illustrate that our method has the important practical benefit of allowing lightweight architectures to achieve state-of-the-art performance at high framerates.
1. Introduction
Object tracking requires adapting to previously unseen targets from minimal supervision, but online deep-network training is costly. This paper integrates a Correlation Filter into a Siamese CNN and finds that lightweight networks can achieve strong tracking performance at high framerates.
- Object tracking must re-detect an initially marked target of any class throughout a video with only a starting bounding box.
- Online CNN adaptation is difficult because tracking provides extremely limited training data while the networks contain many parameters.
- Correlation Filters efficiently adapt a target discriminator at every frame through a Fourier-domain solution to a ridge regression problem.
- Prior CNN-CF trackers applied the filter to pretrained CNN features without end-to-end integration, preventing component parameters from co-adapting to one objective.
- The proposed integration treats the Correlation Filter as a differentiable CNN layer, allowing errors to propagate back to the learned features through a closed-form derivative.
- The investigation finds that the Correlation Filter does not improve sufficiently deep networks, but enables ultra-lightweight networks to achieve state-of-the-art performance on multiple benchmarks at high framerates.
2. Related work
Related work has improved Correlation Filters and introduced efficient Siamese tracking networks, while also exploring differentiation through optimization procedures. This paper instead back-propagates through a conventional learning problem already used in tracking.
- Correlation Filter research has addressed periodic boundaries, multi-resolution feature maps, and more robust objectives, while this work adopts the basic formulation.
- Fully-convolutional Siamese networks impose the prior that the appearance-similarity function should commute with translation.
- The introduced Correlation Filter layer computes a regularized deconvolution solution rather than an upsampling convolution.
- Prior work differentiated SVD, Cholesky, and linear-system solutions so gradient descent could train networks containing these computations.
- Other approaches unroll iterative optimization or back-propagate through SGD, but these procedures can be computationally demanding.
- Unlike abstract meta-learning architectures, this paper propagates gradients through a conventional learning problem already widely used in tracking.
3. Method
The method integrates a Correlation Filter into a fully-convolutional Siamese network, combining offline CNN feature learning with efficient online template adaptation. The CF is evaluated in the forward pass and differentiated in closed form through Fourier-domain operations, enabling end-to-end training.
- Fully-convolutional Siamese networks: The framework processes a training image and search area with a shared CNN, then uses their feature maps for object localization.The target and search inputs produce feature maps that are cross-correlated, with the response maximum indicating the estimated target location.
- Fully-convolutional Siamese networks: Online tracking extracts a search window around the previous estimate and selects the location with the highest network score.The search window is four times the object area, and the target template is updated each frame using a moving average.
- Correlation Filter networks: The proposed architecture inserts a Correlation Filter block between the training feature map and the cross-correlation operator.The block computes a discriminative template by solving a regularized ridge-regression problem in the Fourier domain, with scale and bias parameters adapting scores for logistic regression.
- Correlation Filter: End-to-end training requires differentiating the CF template with respect to its input, which the method derives in closed form through Fourier-domain back-propagation.The derivation takes differentials of the defining system, rearranges them in the Fourier domain, and uses the resulting back-propagation map to train the CNN features.
- Correlation Filter: The CF template is robust against translations because the filter learns to discriminate the target patch from its circular shifts.The formulation minimizes discrepancy from a desired response while using quadratic regularization to prevent overfitting.
- Correlation Filter: The CF solution is computed efficiently in the Fourier domain, where convolutions and matrix operators become element-wise operations and diagonal system solving.The method represents the solution through signals k and α, and uses the discrete Fourier transform, complex conjugation, and element-wise multiplication.
4. Experiments
The experiments evaluate CFNet against Siamese and feature-transfer baselines, then assess adaptation, benchmark performance, and speed. CFNet is most beneficial with shallow networks, where it supports competitive accuracy with substantially lower computational and parameter costs.
- Comparison to Siamese baseline: 31% and 13% relative improvements over the baseline occur at network depths one and two, while differences become less meaningful at depths three through five.CFNet remains relatively unaffected by depth, whereas baseline performance increases with depth.
- Comparison to Siamese baseline: Updating the template with a running average is helpful for both Baseline and CFNet architectures at every tested depth.The comparison includes update rates of 0.01 and 0.
- Feature transfer experiment: CFNet’s training-aware CF integration is critical at depths one and two but becomes redundant at greater depths, where Baseline features combined with CF achieve similar performance.ImageNet features combined with CF perform significantly worse than CFNet and the Baseline, especially in deeper layers.
- Importance of adaptation: CFNet consistently outperforms its constant variant, showing that back-propagating through image-dependent Lagrange multipliers is important for improving over the Siamese baseline.The constant variant keeps the multipliers fixed as offline-learned network parameters.
- Speed and practical benefits: The two-layer CFNet runs at 75 frames per second with fewer than 4% of the five-layer baseline’s parameters and requires 600kB of storage.The one-layer CFNet remains competitive with fewer than 1% of the baseline’s parameters and under 100kB of storage; CFNet variants are modestly superior to state-of-the-art trackers.
5. Conclusion
The Correlation Filter network back-propagates gradients through an online learning algorithm to optimize feature representations. It benefits shallow networks, while adding a Correlation Filter does not significantly improve sufficiently deep Siamese networks; future work includes temporal adaptation and related learning problems.
- CFNet back-propagates gradients through an online learning algorithm to optimize the underlying feature representation.The method uses an efficient back-propagation map for the solution to a system of circulant equations.
- For sufficiently deep Siamese networks, adding a Correlation Filter layer does not significantly improve tracking accuracy.
- Integrating the Correlation Filter during training enables shallow networks to rival slower, deeper counterparts.
- Future research may extend the method to adaptation over time, one-shot learning, and domain adaptation.
A. Implementation details
Training uses ImageNet Video pairs sampled with temporal separation, while tracking applies spatial and scale heuristics to stabilize localization and size updates.
- Training uses all 3862 ImageNet Video training videos, with more than 1 million annotated frames and multiple objects per frame.Optimization runs for 100 epochs with mini-batches of size 8.
- Each training epoch samples approximately 12 frame pairs per video, with pairs at most 100 frames apart.
- A spatial cosine window penalizes large displacements in the tracking score map.
- Scale tracking evaluates the previous scale and adjacent scales using a geometric step of 1.04.
- Scale updates are discouraged with a 0.97 response factor and smoothed using a rolling average with learning rate 0.6.
B. Back-propagation for the Correlation Filter
The Correlation Filter’s differentiable block is obtained by expressing its differential as linear maps and using their adjoints as back-propagation maps, including Fourier-domain computation.
- The back-propagation map is the adjoint of the linear maps defining the Correlation Filter differential.
- The adjoint maps can be computed in the Fourier domain because Parseval’s theorem preserves inner products.
- The component back-propagation expressions combine into the map for the entire Correlation Filter block in equation 10.
C.1. Kernel linear regression
Kernel linear regression formulates regularized learning through a dual representation, while the Correlation Filter exploits circulant structure and Fourier computation; reverse-mode differentiation uses adjoint maps.
- Kernel linear regression: Kernel linear regression learns a weight vector from example inputs and targets using regularization.The examples form matrix X and the targets form vector y.
- Kernel linear regression: The constrained formulation introduces residuals and Lagrange multipliers, then obtains the final solution from the dual variable.
- Kernel linear regression: The dual solution is more efficient than the primal solution when the feature dimension d exceeds the number of examples n.
- Single-channel Correlation Filter: For the Correlation Filter, circular cross-correlation defines the template-learning operation, and circulant structure enables efficient Fourier-domain computation.
- Multi-channel Correlation Filter: The dual formulation is especially efficient for multi-channel Correlation Filters, whose computation scales linearly with the number of channels.
- Back-propagation: Reverse-mode differentiation obtains input gradients by applying the adjoint of the derivative’s linear map to output gradients.
E. Back-propagation for multi-channel case
The multi-channel back-propagation derivation rearranges Fourier-domain differentials and inner products to obtain adjoint maps. The section also describes random-search hyperparameter selection and its validation-based evaluation protocol.
- E. Back-propagation for multi-channel case: Fourier transforms are applied to the multi-channel differential equations to derive the back-propagation procedure.
- E. Back-propagation for multi-channel case: Rearranging inner products yields the adjoint back-propagation maps for the differential transformations.
- E. Back-propagation for multi-channel case: The multi-channel computation scales linearly with the number of channels.
- F. Hyperparameter optimization: 300 random hyperparameter vectors are evaluated on 129 validation videos because tracker evaluations are expensive and discontinuous.
- F. Hyperparameter optimization: The best training epoch is selected using validation tracking results with natural parameters, while test benchmarks are not used for hyperparameter optimization.
G. Detailed results on the OTB benchmarks
This section presents additional OTB benchmark curves and records the research support acknowledged by the authors.
- G. Detailed results on the OTB benchmarks: Figures 9 to 14 show OTB toolkit curves for OTB-2013, OTB-50, and OTB-100.
- G. Detailed results on the OTB benchmarks: The research was supported by Apical Ltd., EPSRC grant Seebibyte, and ERC grants HELIOS and IDIU.