Source-linked AI summary
Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Martin Danelljan, Andreas Robinson, Fahad Shahbaz Khan, Michael Felsberg
TL;DR
Conventional DCFs are restricted to single-resolution feature maps, limiting their use of multi-resolution deep features. The paper learns continuous convolution operators with implicit interpolation, improving tracking benchmarks while enabling sub-pixel localization for feature-point tracking.
Problem
Conventional DCF formulations are restricted to single-resolution feature maps, limiting joint fusion of multi-resolution features.
Method
The paper learns discriminative convolution operators in the continuous spatial domain using implicit interpolation and continuous confidence-function labels.
Results
The framework achieves superior performance across three object-tracking benchmarks and substantially improves accuracy and robustness for real-time feature-point tracking.
Takeaways & Limitations
Continuous-domain learning integrates multi-resolution feature maps and supports accurate sub-pixel localization for object and feature-point tracking.
Takeaways & Limitations
The framework does not use video data to learn an application-specific deep feature representation, which the authors expect could further improve object tracking.
Abstract
from arXiv · showhide
Discriminative Correlation Filters (DCF) have demonstrated excellent performance for visual object tracking. The key to their success is the ability to efficiently exploit available negative data by including all shifted versions of a training sample. However, the underlying DCF formulation is restricted to single-resolution feature maps, significantly limiting its potential. In this paper, we go beyond the conventional DCF framework and introduce a novel formulation for training continuous convolution filters. We employ an implicit interpolation model to pose the learning problem in the continuous spatial domain. Our proposed formulation enables efficient integration of multi-resolution deep feature maps, leading to superior results on three object tracking benchmarks: OTB-2015 (+5.1% in mean OP), Temple-Color (+4.6% in mean OP), and VOT2015 (20% relative reduction in failure rate). Additionally, our approach is capable of sub-pixel localization, crucial for the task of accurate feature point tracking. We also demonstrate the effectiveness of our learning formulation in extensive feature point tracking experiments. Code and supplementary material are available at http://www.cvl.isy.liu.se/research/objrec/visualtracking/conttrack/index.html.
1 Introduction
DCF-based tracking efficiently exploits shifted training samples, but conventional formulations cannot jointly use feature maps at different resolutions. This paper learns continuous convolution filters that fuse multi-resolution deep features and support sub-pixel localization.
- DCF methods efficiently exploit all spatial shifts of training samples through the discrete Fourier transform.
- Shallow CNN layers provide high spatial resolution for accurate localization, while deeper layers are discriminative; fusing them in DCF remains open.
- Conventional DCF requires all feature channels to share one spatial resolution, making direct multi-resolution fusion impossible without resampling artifacts or redundant data.
- The proposed formulation learns continuous-domain convolution filters using implicit interpolation, producing a continuous target confidence map while jointly integrating multi-resolution features.
- 82.4% mean overlap precision improves the OTB-2015 state of the art from 77.3% across 100 videos.
- Continuous confidence maps enable sub-pixel localization and support accurate feature point tracking, evaluated extensively on MPI Sintel.
2 Related Work
Prior work established DCF extensions for multi-channel and deep features, while this paper positions continuous convolution operators as a discriminative alternative for object and feature point tracking.
- Multi-channel DCF extensions incorporate high-dimensional descriptors such as HOG and Color Names to improve tracking.
- Deep CNN features offer high-level visual information, whereas shallow convolutional features are more suitable for visual tracking.
- The proposed discriminative framework targets feature-point motion and introduces continuous operators that integrate multi-resolution features and enable sub-pixel localization.
3 Learning Continuous Convolution Operators
The paper formulates continuous convolution operators by interpolating discrete feature samples into a continuous spatial domain and learning filters in the Fourier domain. This removes the equal-resolution restriction of conventional DCFs while supporting continuous confidence outputs and sub-pixel localization.
- Scope: The formulation is generic, extends to arbitrary spatial dimensions, and is applied to visual object and feature point tracking.The paper develops the theory first in one dimension and then generalizes it to images and higher-dimensional domains.
- Continuous formulation: Feature channels may have different spatial resolutions, eliminating a restriction of conventional DCF formulations.The number of spatial samples Nd can vary across channels extracted from the same image patch.
- Continuous formulation: The formulation learns a linear convolution operator that maps feature samples to continuous target-confidence functions over a spatial interval.The operator interpolates each feature channel, convolves it with a corresponding filter, and sums the responses.
- Continuous formulation: Training samples are labeled with continuous confidence functions, allowing sub-pixel information to enter the learning process.The learned operator consequently produces continuous confidence functions for accurate target localization.
- Optimization: The continuous learning problem is minimized in the Fourier domain using finite-dimensional filter representations and normal equations.The finite subspace limits each filter to coefficients indexed by |k| ≤ Kd; increasing Kd raises computation and memory costs.
- Optimization: The normal equations form a sparse linear system when the spatial penalty has few non-zero Fourier coefficients.The penalty function controls the filter’s spatial extent and assigns larger penalties to background regions.
4 Our Tracking Frameworks
The tracking frameworks localize targets by maximizing a continuous confidence function, using grid search followed by Newton optimization. Object tracking combines multi-resolution deep features, while feature point tracking uses a simplified single-channel setting.
- Localization: Both tracking frameworks localize targets by maximizing the learned continuous confidence function.A feature map is extracted from an image region, and the confidence score is optimized over the continuous interval.
- Localization: Localization first uses a discrete grid search and then refines the maximizer with Newton’s method using analytic gradients and Hessians.The grid search is implemented through a scaled inverse DFT of the non-zero Fourier coefficients.
- Object tracking: The object tracking framework extracts multi-resolution feature maps from a pre-trained deep network and updates weighted training samples over time.It stores at most 400 samples and uses a five-scale search with relative scale factor 1.02.
- Object tracking: Conjugate Gradient optimization gives linear O(D) scaling in the number of feature channels instead of the quadratic O(D^2) complexity of Gauss-Seidel.This makes the optimization suitable for high-dimensional deep features.
- Feature point tracking: Feature point tracking assumes a single-channel feature map and a constant penalty, reducing the normal equations to a diagonal system.Under these assumptions, the filter coefficients can be obtained directly.
- Object tracking: Combining all three convolutional layers produces the best mean OP and AUC results in the OTB-2015 baseline comparison.The table caption attributes the result to the importance of multi-resolution deep feature maps for object tracking.
5 Experiments
Experiments evaluate the continuous convolution operator on object and feature point tracking, comparing multi-resolution fusion, resampling, robustness, and state-of-the-art performance. Across these tests, the approach improves object-tracking benchmarks and supports accurate, robust feature point tracking.
- The framework is evaluated on OTB-2015, Temple-Color, VOT2015, and MPI Sintel for object and feature point tracking.
- Baseline Comparison: Combining RGB, shallow, and deep convolutional layers produces the best OTB-2015 result, whereas intermediate layers add no significant gain.The three-layer combination is used in subsequent object-tracking experiments.
- Baseline Comparison: 68.2% AUC is achieved by the continuous formulation on OTB-2015, compared with 47.7% for discrete DCF with common-resolution resampling and 60.8% with layer-1-resolution resampling.The latter discrete result requires a 5-fold increase in data size; the continuous formulation avoids explicit-resampling artifacts.
- Temple-Color Dataset: 5.0% is the reported mean OP improvement over DeepSRDCF across the OTB-2015 and Temple-Color comparison in Table 2.On Temple-Color specifically, C-COT reaches 70.4% mean OP and gains 3.8% absolute AUC over the previous best method.
- OTB-2015 Dataset: 5.1% is the OTB-2015 mean OP improvement over DeepSRDCF, with the approach reaching 82.4% versus 77.3%.The comparison covers 20 state-of-the-art methods.
- VOT2015 Dataset: 20% lower failure rate is obtained on VOT2015 without significant accuracy degradation.VOT2015 evaluates both overlap accuracy and robustness, restarting trackers after failures.
- Feature Point Tracking: 0.449 inlier EPE and 0.886 inlier ratio demonstrate accurate and robust feature point tracking on MPI Sintel.The method also improves precision over MOSSE at sub-pixel thresholds below 1 pixel.
6 Conclusions
The framework is validated for object and feature point tracking, integrating multi-resolution features and supporting sub-pixel localization. Its object-tracking performance is superior on three benchmarks, while feature point tracking gains accuracy and robustness; application-specific video training remains unused.
- The framework addresses both object tracking and feature point tracking using continuous-domain discriminative convolution operators.It integrates multi-resolution feature maps and supports accurate sub-pixel localization.
- Experiments on three object tracking benchmarks report superior performance compared to the state of the art.
- The method obtains substantially improved accuracy and robustness for real-time feature point tracking.
- The study does not use video data to learn an application-specific deep feature representation.The authors identify this as a direction expected to further improve object tracking performance.