Source-linked AI summary
Learning Spatial-Temporal Regularized Correlation Filters for Visual Tracking
Feng Li, Cheng Tian, Wangmeng Zuo, Lei Zhang, Ming-Hsuan Yang
TL;DR
DCF trackers must balance efficient computation with boundary-effect control and robust adaptation from multiple training images. STRCF introduces temporal regularization into a single-sample SRDCF model and solves it efficiently with ADMM. Across three benchmarks, it improves accuracy and speed over SRDCF and remains competitive with state-of-the-art CNN-based trackers.
Problem
DCF trackers are efficient but suffer boundary effects, while SRDCF and its multiple-image formulation increase complexity and hinder efficient online updating.
Method
STRCF combines spatial and temporal regularization in a single-sample SRDCF formulation, motivated by online Passive-Aggressive learning and solved with ADMM.
Results
STRCF achieves real-time tracking at ∼30FPS and a +5.7% average mean OP gain over SRDCF on two datasets, while DeepSTRCF reaches 84.2% mean OP on OTB-2015.
Takeaways & Limitations
STRCF provides a more robust appearance model under large appearance variations and performs favorably against state-of-the-art trackers.
Abstract
from arXiv · showhide
Discriminative Correlation Filters (DCF) are efficient in visual tracking but suffer from unwanted boundary effects. Spatially Regularized DCF (SRDCF) has been suggested to resolve this issue by enforcing spatial penalty on DCF coefficients, which, inevitably, improves the tracking performance at the price of increasing complexity. To tackle online updating, SRDCF formulates its model on multiple training images, further adding difficulties in improving efficiency. In this work, by introducing temporal regularization to SRDCF with single sample, we present our spatial-temporal regularized correlation filters (STRCF). Motivated by online Passive-Agressive (PA) algorithm, we introduce the temporal regularization to SRDCF with single sample, thus resulting in our spatial-temporal regularized correlation filters (STRCF). The STRCF formulation can not only serve as a reasonable approximation to SRDCF with multiple training samples, but also provide a more robust appearance model than SRDCF in the case of large appearance variations. Besides, it can be efficiently solved via the alternating direction method of multipliers (ADMM). By incorporating both temporal and spatial regularization, our STRCF can handle boundary effects without much loss in efficiency and achieve superior performance over SRDCF in terms of accuracy and speed. Experiments are conducted on three benchmark datasets: OTB-2015, Temple-Color, and VOT-2016. Compared with SRDCF, STRCF with hand-crafted features provides a 5 times speedup and achieves a gain of 5.4% and 3.6% AUC score on OTB-2015 and Temple-Color, respectively. Moreover, STRCF combined with CNN features also performs favorably against state-of-the-art CNN-based trackers and achieves an AUC score of 68.3% on OTB-2015.
1. Introduction
DCF trackers are efficient but face boundary effects and speed–accuracy trade-offs from spatial regularization and large training sets. STRCF adds temporal regularization to a single-sample SRDCF formulation, enabling efficient optimization while improving robustness and tracking performance.
- SRDCF inefficiency stems from scale estimation, spatial regularization, and formulation on a large training set.
- STRCF combines spatial and temporal regularization in a single-sample DCF model motivated by online Passive-Aggressive learning.
- STRCF performs more robustly under occlusion while adapting to large appearance variation, and experiments cover OTB-2015, Temple-Color, and VOT-2016.
- ∼30FPS tracking speed and +5.7% average mean OP gain over SRDCF were achieved on two datasets.
- ADMM efficiently solves STRCF, with each subproblem having a closed-form solution and empirical convergence within very few iterations.
2. Related Work
Related work develops DCF tracking through richer features, kernels, scale and part adaptation, and spatial regularization. These advances address boundary effects and appearance variation but can increase computational cost.
- DCF trackers evolved through kernelization, multi-channel filters, richer features, scale adaptation, and part-based methods.
- Periodic repetitions at boundary positions degrade DCF tracking, motivating spatial regularization methods.
- Historical samples can improve robust correlation filters but make online learning and model updating more computationally demanding.
3. Spatially Regularized DCF
SRDCF suppresses boundary effects through spatial regularization but becomes computationally expensive because it uses multiple training images and large linear systems. STRCF introduces temporal regularization with a single sample and solves the resulting convex model efficiently with ADMM.
- SRDCF limitations: SRDCF penalizes correlation-filter coefficients spatially, but its multiple-image formulation breaks the circulant structure and increases computational burden.The multiple-image formulation and spatial regularization make optimization more difficult than in current-frame-only CF trackers.
- SRDCF limitations: Large sparse linear systems and the slow-convergence Gauss-Seidel solver give SRDCF high complexity and a long first-frame startup time.The resulting system has dimension DMN × DMN, and the tracker needs substantial startup time to learn the filters.
- STRCF formulation: STRCF relaxes SRDCF’s multiple-image formulation to a single-image model by adding temporal regularization motivated by online passive-aggressive learning.The temporal term keeps the current filter close to the previous-frame filter while retaining spatial regularization.
- STRCF formulation: STRCF balances passive and aggressive updating, helping it remain robust to occlusion and large appearance variations while approximating SRDCF’s multiple-sample learning.Unlike SRDCF, it can stay close to the previous filter when recent samples are corrupted by occlusion.
- ADMM optimization: ADMM introduces an auxiliary variable, alternates closed-form subproblem updates, and efficiently solves the convex STRCF objective toward its global optimum.The method uses Fourier-domain decomposition and Sherman–Morrison updates; its overall cost is O(DMN log(MN)NI).
- ADMM optimization: The ADMM implementation has O(DMN log(MN)NI) overall cost and avoids SRDCF’s first-frame filter-initialization startup time.The filter subproblem costs O(DMN log(MN)), while the auxiliary-variable update costs O(DMN).
4. Experimental Results
The experiments evaluate STRCF against state-of-the-art correlation-filter and CNN trackers on OTB-2015, Temple-Color, and VOT-2016, including feature and regularization analyses.
- Experiments compare STRCF with state-of-the-art trackers using hand-crafted and CNN features on OTB-2015, then evaluate Temple-Color and VOT-2016.The study also analyzes temporal regularization and the hyper-parameter μ on OTB-2015.
- The implementation crops a square target-centered region, extracts HOG, CN, and CNN features, and weights them with a cosine window.The crop side length is 5√WH, where W and H are the target width and height.
- ADMM uses μ = 16, initial stepsize γ^(0) = 10, maximum γ_max = 100, and scale factor ρ = 1.2 throughout the experiments.
4.1. The OTB-2015 benchmark
On OTB-2015, STRCF is evaluated across benchmark protocols, tracker families, video attributes, qualitative sequences, and ablations, generally outperforming SRDCF while retaining higher speed.
- OTB-2015 contains 100 fully annotated sequences with 11 attributes, and evaluation uses OPE with overlap precision and success plots.Overlap precision counts bounding-box overlaps exceeding 0.5 in a sequence.
- Comparison with hand-crafted based trackers: STRCF surpasses SRDCF by 6.9% in mean OP with hand-crafted features and is competitive with ECO-HC.It also outperforms SRDCFDecon and several other correlation-filter trackers.
- Comparison with hand-crafted based trackers: STRCF (HOGCN) runs at 24.3 FPS versus SRDCF’s 5.8 FPS, while STRCF (HOG) reaches real-time speed at 31.5 FPS.The HOGCN configuration is nearly 4.2× faster than SRDCF, and the HOG configuration is 1.2× faster than BACF.
- Comparison with hand-crafted based trackers: 65.1% AUC ranks STRCF second among hand-crafted trackers, exceeding SRDCF by 5.4% and SRDCFDecon by 2.3%.
- Video Attribute Based Comparison: Qualitative evaluations on six sequences show STRCF performing favorably against ECO-HC, BACF, SRDCF, and SRDCFDecon.
- Video Attribute Based Comparison: STRCF outperforms most competing trackers except ECO-HC across all 11 video attributes, including rotation gains of 5.8% and 7.6% over SRDCF.The reported rotation comparisons cover in-plane and out-of-plane rotation, respectively.
- Comparison with deep feature-based trackers: DeepSTRCF achieves 84.2% mean OP, exceeding DeepSRDCF by 7.4% and C-COT by 1.2% on OTB-2015.DeepSTRCF runs at 5.3 FPS and ranks second in the deep-feature overlap success curves, exceeding DeepSRDCF by 5.2%.
- Ablative study: The ablation study compares SRDCF variants and STRCF using overlap success plots and visualizes temporal filter variation across frames on Shaking.
4.2. Internal Analysis of the proposed approach
On OTB-2015, temporal regularization improves STRCF over SRDCF variants by balancing passive and aggressive model updates. This produces more robust adaptation to gradual, sudden, and large appearance changes.
- 6% and 4.4% gains over SRDCF(-M) and SRDCF, respectively, are achieved by STRCF on OTB-2015.
- Temporal regularization improves STRCF by adaptively balancing aggressive and passive model updating.
- STRCF passively updates filters during small appearance changes, improving robustness compared with SRDCF.
- During occlusion, passive model learning makes STRCF less sensitive to slow appearance variations than SRDCF.
- During sudden illumination changes, aggressive model learning helps STRCF adapt better than SRDCF.
- The choice of regularization parameter µ significantly affects STRCF accuracy because it controls replacement of the previous filter with the current sample.
4.3. The Temple-Color Benchmark
On Temple-Color, STRCF outperforms SRDCF and related handcrafted-feature trackers, while DeepSTRCF achieves the strongest performance among the compared trackers.
- The benchmark contains 128 color sequences and compares STRCF and DeepSTRCF with state-of-the-art trackers using overlap success plots.
- 60.1% AUC is achieved by DeepSTRCF, the best-performing tracker among the competing Temple-Color trackers.
4.4. The VOT-2016 Benchmark
On VOT-2016, STRCF performs significantly better than BACF and SRDCF in EAO, while DeepSTRCF improves over DeepSRDCF.
- VOT-2016 evaluation uses accuracy, robustness, and expected average overlap across 60 challenging videos.
- 3.7% EAO is the gain of DeepSTRCF over DeepSRDCF on VOT-2016.
- STRCF performs significantly better than BACF and SRDCF in terms of EAO on VOT-2016.
5. Conclusion
The paper concludes that STRCF addresses SRDCF’s inefficiency by combining spatial and temporal regularization with single-sample learning. Experiments across three benchmarks show improved accuracy and speed with handcrafted features and favorable accuracy and robustness with deep features.
- STRCF introduces temporal regularization to single-sample SRDCF, approximating SRDCF with multiple training samples.
- An ADMM algorithm efficiently solves the STRCF model.
- Handcrafted-feature STRCF is superior to baseline SRDCF in accuracy and speed across three benchmarks.
- Deep-feature STRCF performs favorably against state-of-the-art trackers in accuracy and robustness.
- Future work will investigate compatibility with SAMF+AT, Staple+CA, and components used in ECO.