Source-linked AI summary
ECO: Efficient Convolution Operators for Tracking
Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, Michael Felsberg
TL;DR
DCF trackers have gained accuracy and robustness while losing speed and incurring over-fitting risk from increasingly complex models. ECO revisits the DCF formulation with factorized convolution, compact sample modeling, and conservative updates. Experiments report improved performance and speed across four benchmarks, including a 60 FPS CPU variant.
Problem
DCF trackers' increasing accuracy has come with reduced speed and larger models that risk severe over-fitting under scarce tracking data.
Method
ECO combines a factorized convolution operator, compact generative sample model, and efficient model update strategy to reduce parameters, learning complexity, and over-fitting.
Results
ECO concurrently improves tracking performance and speed across VOT2016, UAV123, OTB-2015, and TempleColor, with a fast variant reaching 60 frames per second on a single CPU.
Takeaways & Limitations
The fast variant is especially suitable for computationally restricted robotics platforms while maintaining competitive performance.
Takeaways & Limitations
The complexity analysis excludes FFT computations because they account for roughly 1% of C-COT's total time.
Abstract
from arXiv · showhide
In recent years, Discriminative Correlation Filter (DCF) based methods have significantly advanced the state-of-the-art in tracking. However, in the pursuit of ever increasing tracking performance, their characteristic speed and real-time capability have gradually faded. Further, the increasingly complex models, with massive number of trainable parameters, have introduced the risk of severe over-fitting. In this work, we tackle the key causes behind the problems of computational complexity and over-fitting, with the aim of simultaneously improving both speed and performance. We revisit the core DCF formulation and introduce: (i) a factorized convolution operator, which drastically reduces the number of parameters in the model; (ii) a compact generative model of the training sample distribution, that significantly reduces memory and time complexity, while providing better diversity of samples; (iii) a conservative model update strategy with improved robustness and reduced complexity. We perform comprehensive experiments on four benchmarks: VOT2016, UAV123, OTB-2015, and TempleColor. When using expensive deep features, our tracker provides a 20-fold speedup and achieves a 13.0% relative gain in Expected Average Overlap compared to the top ranked method in the VOT2016 challenge. Moreover, our fast variant, using hand-crafted features, operates at 60 Hz on a single CPU, while obtaining 65.0% AUC on OTB-2015.
1. Introduction
DCF trackers have improved accuracy and robustness, but increasingly complex models reduce speed and increase over-fitting risk. ECO addresses these issues through a compact formulation and update strategy while targeting improved performance and real-time capability.
- Motivation: Online visual tracking must estimate target trajectories accurately and robustly under real-time computational constraints.It supports applications including surveillance, autonomous driving, UAV monitoring, traffic control, and human-computer interfaces.
- Motivation: DCF advances in accuracy and robustness have substantially reduced tracking speed compared with simpler trackers.MOSSE is about 1000× faster than C-COT but obtains only half the accuracy on VOT2016.
- Motivation: Large DCF models with hundreds of thousands of trainable parameters are prone to severe over-fitting because tracking provides scarce training data.C-COT continuously updates about 800,000 parameters during online learning.
- Contributions: ECO introduces a factorized convolution operator, a compact generative sample model, and an efficient update strategy to reduce complexity and over-fitting.The components reduce parameters, learning memory and time, and over-fitting to recent samples.
- Results: 80% fewer model parameters, 90% fewer training samples, and 80% fewer optimization iterations are reported relative to the baseline.The method is evaluated on VOT2016, UAV123, OTB-2015, and TempleColor, including a 60 FPS fast variant on a single CPU.
2. Baseline Approach: C-COT
C-COT learns continuous-domain multi-channel convolution filters from training samples to produce target detection scores. Its formulation integrates multi-resolution features and enables continuous localization, while Fourier-domain optimization exploits the problem's structure.
- C-COT formulation: Independent feature resolutions are integrated naturally through continuous-domain convolutions without explicit re-sampling.This allows each visual feature to use its own cell size or resolution.
- C-COT formulation: The predicted target detection scores are continuous functions, enabling accurate sub-grid localization.This is a second advantage of the generalized C-COT formulation.
- C-COT formulation: C-COT learns a continuous, multi-channel convolution filter from a collection of training samples to predict target detection scores.Feature maps are transferred into the continuous spatial domain using interpolation models.
- Objective: Training minimizes weighted classification error with L2 regularization and a spatial penalty that mitigates drawbacks of the periodic assumption.The labeled detection scores are periodically repeated Gaussian functions, and sample weights α_j determine their contributions.
- Optimization: Changing to the Fourier basis yields an equivalent loss whose convolution structure can be optimized efficiently.Fourier coefficients represent periodic functions, while feature coefficients are obtained through the DFT.
- Optimization: C-COT uses Conjugate Gradient iterations to solve the quadratic normal equations by exploiting the sparse structure of the optimization problem.The matrix contains diagonal blocks based on Fourier-domain feature terms.
3. Our Approach
The approach jointly targets over-fitting and computational bottlenecks in DCF tracking through factorized filters, compact sample modeling, and sparse model updates. These changes reduce model, sample, and optimization complexity while preserving or improving tracking robustness and performance.
- Overview: The method reduces DCF learning complexity by targeting filter dimensionality, sample count, and conjugate-gradient iterations separately.The complexity analysis motivates reducing D, M, and NCG through the three proposed components.
- 3.1. Factorized Convolution Operator: Factorized convolution replaces separate channel filters with a smaller set of jointly learned basis filters and coefficients.The factorized operator can be viewed as dimensionality reduction followed by convolution, with the projection and filters learned discriminatively.
- 3.1. Factorized Convolution Operator: The factorized representation stores projected feature maps and reduces subsequent learning complexity from O(NCGDM ¯K) to O(NCGCM ¯K).The projection matrix is learned from the first frame and remains fixed afterward.
- 3.2. Generative Sample Space Model: A generative sample-space model removes redundancy by representing training data with diverse mixture components instead of many consecutive samples.The model addresses repeated appearance samples and forgotten earlier appearance aspects in the baseline representation.
- 3.2. Generative Sample Space Model: The sample-space model reduces the number of learning samples from M to L, with L set to M/8 while improving tracking performance in experiments.The projected samples are combined with the factorized convolution because the projection matrix is constant after the first frame.
- 3.3. Model Update Strategy: Sparse model updates start optimization every NSth frame, reducing average conjugate-gradient iterations to NCG/NS; NS ≈5 generally improved tracking results.The authors mainly attribute the improvement to reduced over-fitting to recent training samples.
4. Experiments
Experiments on four benchmarks show that ECO systematically improves tracking performance and speed through progressively integrated contributions. The tracker achieves strong state-of-the-art results with deep features and a 60 FPS hand-crafted-feature variant on a single CPU.
- ECO was evaluated on VOT2016, UAV123, OTB-2015, and TempleColor.
- VOT2016: 13.0% relative EAO gain and 20-fold speedup were achieved over the baseline on VOT2016.The final EAO score was 0.374; speed was measured on a single CPU without feature extraction.
- VOT2016: 6× complexity reduction followed factorized convolution, while the sample space model added a 2.9% relative EAO gain and 8× lower learning complexity.
- VOT2016: ECO achieved the best VOT2016 failure rate of 0.72 and a 15.1% relative EAO improvement over TCNN with an almost 5-fold EFO speedup.
- OTB-2015: ECO-HC reached 65.0% AUC at 60 FPS on a CPU, outperforming SRDCFad’s 63.4% AUC on OTB-2015.
- OTB-2015 and TempleColor: ECO achieved 70.0% AUC on OTB-2015 and improved over C-COT by 0.8% AUC on TempleColor.
5. Conclusions
The conclusion presents ECO as a collective treatment of computational complexity and over-fitting in DCF tracking, combining parameter, sample, and update reductions. Experiments across four benchmarks report improved performance and frame rate.
- ECO revisits the DCF formulation with factorized convolution, a compact generative sample model, and an efficient model update strategy.
- The proposed components reduce model parameters, learning samples, and optimization complexity while maintaining sample diversity and improving robustness.
- Experiments on VOT2016, UAV123, OTB-2015, and TempleColor demonstrate state-of-the-art performance with improved frame rate.
Supplementary Material
The supplementary material details ECO’s optimization and complexity analysis, then reports attribute-level results on VOT2016 and OTB-2015. These evaluations show improvements over C-COT across multiple tracking attributes.
- Complexity analysis: ECO’s learning complexity is analyzed through the dominant Conjugate Gradient operations, while FFT preprocessing contributes roughly 1% of C-COT’s total time.The analysis identifies block-wise matrix-vector products and regularization operations as the principal computational terms.
- Optimization: ECO’s optimization uses a Gauss-Newton procedure for the nonlinear least-squares problem and Conjugate Gradient to solve the resulting quadratic subproblem.The equivalent loss is expressed using interpolated feature maps and point-wise matrix-vector products in the Fourier domain.
- VOT2016 results: On VOT2016, ECO achieves the best results on three attributes and improves over baseline C-COT on all five attributes.The five attributes are camera motion, illumination change, occlusion, size change, and motion change.
- OTB-2015 results: On OTB-2015, ECO achieves the best performance on 8 of 11 attributes and improves over baseline C-COT on 9 of 11 attributes.The comparison uses the same combination of deep and hand-crafted features for ECO and C-COT.