Source-linked AI summary
Learning Spatially Regularized Correlation Filters for Visual Tracking
Martin Danelljan, Gustav Häger, Fahad Shahbaz Khan, Michael Felsberg
TL;DR
Visual tracking needs robust appearance models despite limited training data, while standard DCFs are degraded by periodic-assumption boundary effects. The paper proposes SRDCF with spatial regularization and efficient Gauss-Seidel online optimization, achieving the best overall results on four benchmark datasets, including an 8.0% gain on OTB-2013 and 8.2% on OTB-2015 in mean overlap precision.
Problem
Visual tracking must learn robust target appearance models from a very limited set of training samples, but standard DCF periodic assumptions introduce boundary effects that degrade model quality.
Method
SRDCF spatially regularizes correlation-filter coefficients, supports larger negative-sample regions, and uses Gauss-Seidel optimization for efficient online learning.
Results
The approach achieves the best overall results on ALOV++ and VOT2014 and absolute mean overlap precision gains of 8.0% on OTB-2013 and 8.2% on OTB-2015.
Takeaways & Limitations
SRDCF outperforms existing trackers on all four evaluated benchmark datasets.
Takeaways & Limitations
A direct sparse-solver implementation for the resulting normal equations is computationally demanding, motivating a specialized optimization scheme for online learning.
Abstract
from arXiv · showhide
Robust and accurate visual tracking is one of the most challenging computer vision problems. Due to the inherent lack of training data, a robust approach for constructing a target appearance model is crucial. Recently, discriminatively learned correlation filters (DCF) have been successfully applied to address this problem for tracking. These methods utilize a periodic assumption of the training samples to efficiently learn a classifier on all patches in the target neighborhood. However, the periodic assumption also introduces unwanted boundary effects, which severely degrade the quality of the tracking model. We propose Spatially Regularized Discriminative Correlation Filters (SRDCF) for tracking. A spatial regularization component is introduced in the learning to penalize correlation filter coefficients depending on their spatial location. Our SRDCF formulation allows the correlation filters to be learned on a significantly larger set of negative training samples, without corrupting the positive samples. We further propose an optimization strategy, based on the iterative Gauss-Seidel method, for efficient online learning of our SRDCF. Experiments are performed on four benchmark datasets: OTB-2013, ALOV++, OTB-2015, and VOT2014. Our approach achieves state-of-the-art results on all four datasets. On OTB-2013 and OTB-2015, we obtain an absolute gain of 8.0% and 8.2% respectively, in mean overlap precision, compared to the best existing trackers.
1. Introduction
Visual tracking must learn robust target appearance models from very limited data, but standard DCF methods suffer boundary effects from their periodic assumption. SRDCF addresses this with spatial regularization, enabling larger negative-sample regions and strong benchmark performance.
- Visual tracking requires generalizing target appearance from limited samples despite occlusions, fast motion, and deformations.
- DCF methods use periodic samples to train and detect efficiently with the FFT, but this assumption creates unwanted boundary effects.
- Boundary effects restrict target search, reduce negative-sample quality, and cause over-fitting, harming tracking under fast motion, deformation, and occlusion.
- SRDCF spatially penalizes filter coefficients by location, allowing learning on larger image regions without corrupting positive samples.
- The method uses Gauss-Seidel optimization for efficient online learning and sub-grid interpolation for pixel-dense detection scores.
2. Discriminative Correlation Filters
Discriminative correlation filters learn multi-channel convolution filters by exploiting circular correlation and Fourier-domain computation. This makes training efficient, but the periodic assumption introduces boundary effects that impair the learned tracking model.
- DCF is a supervised technique for learning a linear classifier or regressor, using circular correlation for efficient training and detection.
- Circular correlation implicitly includes shifted training samples and enables Fourier-domain computation with the FFT.
- The DCF filter contains one M × N convolution filter per feature layer and is learned by minimizing L2 error between responses and labels.
- Fourier-domain normal equations have a block-diagonal structure, yielding MN linear systems of size d × d for the DFT filters.
- 26%?
- Periodic extension introduces boundary effects in both training and detection, producing inaccurate image descriptions and tracking models.
3. Spatially Regularized Correlation Filters
SRDCF adds spatially varying regularization to penalize filter coefficients outside the target, reducing background emphasis and addressing boundary effects. Its sparse Fourier formulation supports efficient iterative optimization, while the learned filters emphasize target-region appearance.
- Spatial regularization: Spatial regularization penalizes filter coefficients according to location, assigning higher weights outside the target region.The regularization replaces the standard term with a spatial weight function and suppresses coefficients outside the target.
- Fourier-domain formulation: The spatially regularized objective is reformulated in the Fourier domain using Parseval’s theorem and a convolution representation of the regularizer.The resulting real-valued system preserves Hermitian symmetry while representing spatial regularization through the matrix C(w_hat).
- Fourier-domain formulation: A sparse spectrum for the spatial weights yields a sparse real-valued linear system whose filter coefficients are recovered from its solution.The system’s sparsity depends on the number of non-zero Fourier coefficients in w_hat.
- Spatial regularization: Higher background weights reduce background emphasis and produce filters that focus visual information within the target region.Compared with standard DCF, the proposed filters assign less influence to background features and emphasize target appearance.
- Optimization: Direct sparse solution of the normal equations is computationally demanding, motivating the proposed iterative optimization for online learning.The computational difficulty remains even with standard regularization and few feature channels.
- Optimization: Because spatial regularization removes the standard DCF’s block-diagonal structure, the authors solve the normal equations iteratively with Gauss-Seidel.The weight construction ensures the symmetry and positive definiteness needed for convergence.
4. Our Tracking Framework
The tracking framework updates SRDCF online from newly extracted samples and detects targets across multiple scales. It interpolates detection scores to sub-grid precision and selects the scale with the highest maximal score.
- Training: At each frame, the model incorporates a new training sample centered at the current target location using a learning-rate update.The update maintains exponentially decaying sample weights and avoids storing all previous samples.
- Training: The tracker performs a fixed number of Gauss-Seidel iterations per frame to compute updated filter coefficients.The filter from the previous frame initializes the iterative optimization.
- Detection: Detection samples are extracted around the previous target location at multiple scale levels relative to the current target scale.Each sample is resized before feature computation, and the scale increment controls the sampled resolutions.
- Detection: Trigonometric-polynomial interpolation produces pixel-dense detection scores from DFT coefficients, and Newton’s method maximizes them at sub-grid locations.The grid maximum initializes Newton optimization, which typically converges in only a few iterations.
- Detection: The scale with the highest maximal detection score determines the updated target location and scale.Sub-grid score maximization is applied independently at each scale level.
- Computational cost: Excluding feature extraction, total complexity is O(dSMN log MN + SMNN_Ne + (d + K^2)dMNN_GS), dominated by filter optimization.N_Ne is the number of sub-grid detection iterations.
5. Experiments
The method is evaluated on four benchmark datasets using specified feature, sampling, windowing, and spatial-weight settings.
- Evaluation setup: The evaluation uses OTB-2013, OTB-2015, ALOV++, and VOT2014.These four datasets form the reported benchmark evaluation.
- Spatial-weight settings: The spatial weight function is quadratic, centered on the target, with µ = 0.1 and η = 3.Small Fourier coefficients are removed, leaving approximately 10 non-zero coefficients in w_hat.
- Feature and sampling settings: Samples use HOG features with 4×4-pixel cells on a square grid whose image area is 42 times the target area.The initial scale limits the maximum sample size to M = 50 cells, and samples are multiplied by a Hann window.
5.2. Baseline Comparison
The baseline study compares standard and spatial regularization under conventional and expanded sample sizes, showing that spatial regularization supports larger training regions without corrupting the target model and improves tracking performance.
- The evaluation compares standard and proposed regularization using both conventional and expanded sample sizes.The standard DCF uses uniform regularization weights w(m, n) = λ, with λ = 0.01.
- 7.0% gain in mean OP is obtained with spatial regularization compared to the standard DCF formulation.The comparison uses OTB-2013 and evaluates four methods within the tracking framework.
- Spatial regularization enables expanded training regions without corrupting positive training samples, unlike the standard DCF setting.The expanded region provides additional negative training patches for the filter.
- On OTB-2013, the CFLB baseline achieves 48.6% mean OP, while the matched SRDCF version achieves 54.3%, a 5.7% improvement.The comparison uses the same grayscale, scale, detection, and sample-size settings for both methods.
5.3. OTB-2013 Dataset
On OTB-2013, SRDCF is compared with state-of-the-art trackers across overall success, initialization robustness, attributes, and challenging qualitative sequences, with gains over leading alternatives.
- State-of-the-art Comparison: 78.1% mean OP is achieved by SRDCF on OTB-2013, 8.0% above MEEM’s 70.1%.The comparison covers all 50 videos and reports the top 10 trackers.
- State-of-the-art Comparison: 63.3% AUC is obtained by SRDCF, outperforming SAMF’s 57.7% by 5.6% in the success plot.The success plot ranks trackers by area under the curve over intersection-over-union thresholds.
- Robustness to Initialization: SRDCF shows a consistent performance gain over SAMF and MEEM in both temporal and spatial robustness evaluations.Temporal robustness and spatial robustness are evaluated using the TRE and SRE protocols.
- Robustness to Initialization: Qualitative comparisons report consistent SRDCF results under occlusions, fast motion, background clutter, and target rotations.The comparison uses the Soccer, Human6, and Tiger2 videos.
- Attribute Based Comparison: SRDCF outperforms existing trackers on 10 of the 11 annotated OTB-2013 attributes.The attributes include occlusion, deformation, fast motion, rotations, and background clutter.
- Attribute Based Comparison: SRDCF improves over MEEM by 3.3% for out-of-plane rotation and over DSST by 4.1% for scale variation.Standard DCF trackers particularly struggle with motion blur and fast motion because of restricted search areas.
5.4. OTB-2015 Dataset
On OTB-2015, SRDCF outperforms the strongest existing trackers in mean overlap precision and success-plot AUC, while the broader attribute analysis reports superior performance across the evaluated scenarios.
- Cross-dataset Comparison: SRDCF achieves the best overall results among 24 trackers on ALOV++ according to the survival-curve comparison.The mean F-score reported for SRDCF is 0.787.
- Dataset Comparison: SRDCF surpasses the best existing tracker by 8.2% in mean OP on the 100-video OTB-2015 dataset.SAMF and MEEM are the strongest existing methods, with mean OP values of 64.7% and 63.4%, respectively.
- Success-plot Comparison: 60.5% AUC is obtained by SRDCF, outperforming SAMF’s 54.8% by 5.7% in the success plot.MEEM achieves an AUC score of 53.8%.
5.5. ALOV++ Dataset
On ALOV++, SRDCF is evaluated with survival curves over 314 videos and achieves the best overall result among 24 compared trackers.
- Survival-curve Evaluation: The ALOV++ evaluation uses sorted per-video F-scores across 314 videos and 89,364 frames, where higher F-scores indicate better performance.Performance is summarized through survival curves and average F-scores.
- Survival-curve Evaluation: 0.787 mean F-score is achieved by SRDCF, compared with 0.708 for MEEM.The survival curves compare 24 trackers, including 19 trackers from the benchmark evaluation and five top methods from the OTB comparison.
- Survival-curve Evaluation: SRDCF achieves the best overall performance compared with the 24 evaluated trackers.The result is reported from the survival curves and average F-scores.
5.6. VOT2014 Dataset
On VOT2014, the tracker was evaluated against 38 challenge participants plus MEEM using accuracy, robustness, and final ranking. SRDCF achieved the top final rank.
- Evaluation: VOT2014 evaluates trackers by accuracy, robustness, and a final rank computed from both measures.Accuracy is based on overlap with ground truth, while robustness is determined by failure rate; trackers restart after failures.
- Evaluation: The comparison included 38 participating trackers and MEEM.
- Results: 8.26 was SRDCF's top final ranking score, outperforming DSST and SAMF.
6. Conclusions
The paper introduces SRDCF to overcome standard DCF limitations by spatially regularizing the filter, enabling larger training regions and efficient Fourier-domain learning. SRDCF outperformed existing trackers on all four benchmark datasets.
- Conclusions: SRDCF addresses standard DCF limitations by introducing spatial regularization into correlation-filter learning.
- Conclusions: Spatial regularization enables learning on larger image regions, producing a more discriminative appearance model.
- Conclusions: An efficient Fourier-domain optimization strategy uses the sparsity of regularization and the Gauss-Seidel method.
- Conclusions: SRDCF outperformed existing trackers on all four benchmark datasets.