Source-linked AI summary
Discriminative Correlation Filter with Channel and Spatial Reliability
Alan Lukežič, Tomáš Vojíř, Luka Čehovin, Jiří Matas, Matej Kristan
TL;DR
Short-term tracking must localize targets despite substantial appearance and motion variation, while standard DCFs face search-region and target-shape limitations. CSR-DCF integrates spatially constrained filter support and channel reliability weighting through an efficient learning procedure. It achieves state-of-the-art benchmark performance with simple features while operating close to real time on a single CPU.
Problem
Circular correlation restricts search-region size and rectangular filter support can learn background for irregular or occluded objects, making robust short-term tracking difficult.
Method
CSR-DCF estimates a spatial reliability map for constrained filter support and channel reliability weights for per-channel response weighting during localization.
Results
CSR-DCF achieves state-of-the-art performance on OTB100, VOT2015, and VOT2016 while running close to real time on a single CPU.
Takeaways & Limitations
The spatial and channel reliability formulation is general and can be used in most modern correlation filters, including those using deep features.
Takeaways & Limitations
The derivation assumes a single channel, and the spatial prior relies on central pixels being more reliably associated with deformable objects.
Abstract
from arXiv · showhide
Short-term tracking is an open and challenging problem for which discriminative correlation filters (DCF) have shown excellent performance. We introduce the channel and spatial reliability concepts to DCF tracking and provide a novel learning algorithm for its efficient and seamless integration in the filter update and the tracking process. The spatial reliability map adjusts the filter support to the part of the object suitable for tracking. This both allows to enlarge the search region and improves tracking of non-rectangular objects. Reliability scores reflect channel-wise quality of the learned filters and are used as feature weighting coefficients in localization. Experimentally, with only two simple standard features, HoGs and Colornames, the novel CSR-DCF method -- DCF with Channel and Spatial Reliability -- achieves state-of-the-art results on VOT 2016, VOT 2015 and OTB100. The CSR-DCF runs in real-time on a CPU.
1 Introduction
Short-term visual tracking is challenging because targets undergo occlusion, motion, deformation, and appearance changes. CSR-DCF addresses DCF limitations by constraining filter support spatially and weighting channels by reliability.
- Short-term model-free tracking continuously localizes a target from a single appearance example.
- Occlusion, illumination change, fast motion, deformation, and background similarity make short-term tracking challenging.
- Circular correlation enables efficient FFT learning but ties filter and search-region sizes and introduces unrealistic wrapped-around training examples.
- Learning from larger regions increases detection range but can reduce DCF performance because of the larger background area.
- Published DCF methods assume axis-aligned rectangular targets, causing background learning and potential drift for irregular, hollow, or occluded objects.
- CSR-DCF uses a spatial reliability map to constrain the filter to trackable object parts and channel reliability scores to weight filter responses.
2 Related work
Related work develops DCF tracking through richer features, improved filter learning, scale adaptation, and part-based or spatially constrained representations. These advances trade tracking robustness or performance against computational cost and boundary artifacts.
- Feature improvements: DCF tracking progressed from grayscale templates toward HoG, color attributes, feature combinations, and convolutional features.Convolutional features improved performance but substantially reduced speed.
- Filter-learning improvements: Kernelized learning preserved high speed while improving tracking performance beyond standard DCF.This was described as the first successful theoretical extension of standard DCF.
- Filter-learning improvements: Scale adaptation, continuous-space learning, and multi-layer response combination addressed scale changes and feature-map limitations.Related methods also combined correlation responses with segmentation probability maps or appearance clusters.
- Part-based methods: Part-based correlation filters were proposed because a single target filter can be corrupted by occlusion or deformation.These methods combine multiple part responses or model target structure explicitly.
- Spatial constraints: Spatially constrained and zero-padded filter-learning methods addressed limited training regions and boundary artifacts.Spatial constraints penalize filter values outside the object boundary, while implicit zero padding increases the effective filter size.
3 Spatially constrained correlation filters
CSR-DCF constrains correlation filters spatially and treats feature channels independently, combining a binary reliability mask with channel weighting to improve localization. Its optimization alternates efficient frequency-domain updates, while the spatial map is estimated from color-based foreground–background modeling.
- Independent per-channel optimization avoids feature-scale differences causing weakly discriminative channels to dominate the final response.Standard multi-channel DCF sums channel responses, but HoG, colornames, and grayscale channels can differ in scale by orders of magnitude.
- The method combines spatially constrained filters with channel reliability weights that scale per-channel responses according to discriminative power during localization.The section introduces channel weights as scaling factors and presents the tracker as integrating constrained learning, spatial-map estimation, and channel-reliability estimation.
- A binary spatial reliability map m constrains the learned filter to retain values only at pixels deemed suitable for tracking.The constraint is h ≡ m ⊙ h, with zero-valued mask elements forcing corresponding filter values to zero.
- 3.1 Constrained correlation filter learning: The constrained optimization uses an augmented Lagrangian and alternating minimization, with closed-form subproblem updates and multiplier updates at each iteration.The penalty parameter can be updated as µ_i+1 = βµ_i.
- 3.1 Constrained correlation filter learning: The optimization requires only two Fourier-transform calls per iteration and has complexity O(D log D).The procedure is summarized as constrained filter optimization with features, an ideal response, and a binary mask as inputs.
- 3.2 Constructing spatial reliability map: The spatial map is estimated after localization from training-region observations using foreground/background color models, spatial priors, and posterior probabilities.The object prior uses a modified Epanechnikov kernel with σ clipped to [0.5, 0.9], producing center probability 0.9 and a uniform prior away from the center.
4 Experimental analysis
The experiments evaluate implementation speed, constrained-learning convergence, boundary constraints, and robustness to non-axis-aligned initialization. CSR-DCF’s spatial reliability constraints improve tracking robustness when background occupies the training region.
- Implementation: CSR-DCF runs at 13 frames per second on an Intel Core i7 3.4GHz desktop.
- Boundary constraints: TSC achieves 0.32 AUC on VOT2015, compared with 0.28 for TSR and 0.16 for TLB.The variants differ only in their constraint formulation.
- Robustness to non-axis-aligned initialization: CSR-DCF remains consistently above SRDCF and LBCF across initialization thresholds for non-axis-aligned targets.The evaluation counts trajectories that retain positive overlap after increasing numbers of frames.
- Robustness to non-axis-aligned initialization: CSR-DCF by far outperforms SRDCF and LBCF on weighted tracking length and related measures, indicating robustness to challenging non-axis-aligned initialization.The results associate this robustness with spatial constraints that reduce background influence during filter learning.
4.4 Spatial and channel reliability ablation study
The ablation study evaluates spatial and channel reliability within CSR-DCF using EAO, accuracy, and robustness on VOT2016. Removing channel reliability causes a substantial EAO decrease.
- Experimental design: The ablation varies channel reliability, spatial reliability-map type, and constrained optimization.Table 2 identifies these factors in the Chan., Spat., and Opt. columns.
- Channel reliability: 15% EAO performance drop results when channel reliability weights are removed relative to CSR-DCFc−.The no-channel-reliability variant is denoted CSR-DCFc−o−.
4.5 Spatial reliability map quality analysis
The estimated spatial reliability maps achieve nearly the same tracking performance as maps derived from ground-truth segmentation masks. Pixel-level differences therefore do not translate into meaningful tracking differences in the reported comparison.
- Quantitative comparison: CSR-DCF and CSR*-DCF obtain equal EAO and average accuracy on VOT2016.CSR*-DCF uses ground-truth segmentation masks, whereas CSR-DCF uses the proposed color-based spatial reliability map.
- Quantitative comparison: CSR*-DCF records one fewer failure than CSR-DCF across 60 sequences, corresponding to 0.02 lower average robustness.Robustness is reported as the average number of failures.
- Qualitative analysis: The estimated maps differ from ground-truth masks at pixel level but are nearly equivalent for tracking.The comparison evaluates map usefulness from a visual-tracking perspective rather than pixelwise identity.
- Qualitative analysis: For a basketball player, poorly segmented legs are treated as non-informative because they constantly move and do not improve correlation-filter localization.Figure 10 visualizes the estimated maps alongside ground-truth segmentation masks and tracking regions.
4.6 The OTB100 benchmark (Wu et al, 2015)
CSR-DCF is evaluated against broad tracker sets on OTB100, VOT2015, and VOT2016. It ranks first on all three reported benchmarks, including an EAO of 0.338 on VOT2016, despite using simple features.
- OTB100: CSR-DCF ranks first on OTB100 and outperforms reported top performers, SRDCF, and MUSTER.Its success-plot performance is slightly below SRDCF because of poorer scale estimation, while its average precision is better.
- VOT2015: CSR-DCF outperforms all trackers and achieves the top rank in the VOT2015 comparison.The comparison applies VOT2016 rules excluding trackers trained on related VOT sequences.
- VOT2016: 0.338 EAO is achieved by CSR-DCF while outperforming all 70 trackers on VOT2016.The tracker also outperforms correlation-filter methods without deep ConvNets and computationally intensive deep-feature trackers.
4.9 Per-attribute analysis
CSR-DCF achieves strong per-attribute tracking performance and competitive VOT performance while operating substantially faster than related trackers. Its measured speed is suitable for real-time CPU tracking.
- Per-attribute performance: CSR-DCF ranks among the top three trackers on five of six VOT2016 visual attributes and ranks first on four.The four top-ranked attributes are size change, occlusion, camera motion, and unassigned.
- Speed and performance: CSR-DCF performs on par with VOT2016’s best-performing CCOT while running 20 times faster.CCOT uses computationally intensive deep ConvNet features.
- Speed and performance: CSR-DCF is twice as fast as SRDCF while achieving approximately 25% better tracking results.
- Speed and performance: The speed comparison reports fps, EAO, average accuracy, and average failures for correlation trackers and Struck.
- Processing time: CSR-DCF averages approximately 13 frames-per-second, or 77 milliseconds per frame, on VOT2016.Target localization averages 35 milliseconds per frame, comprising translation estimation and scale-change estimation.
4.11 Qualitative evaluation
Qualitative examples illustrate how channel and spatial reliability support tracking through deformation, occlusion, and changing object appearance. The tracker adapts channel weighting and filter support to the target and its visible structure.
- Processing: A single CSR-DCF processing iteration is decomposed into target localization and visual-model update steps.
- Channel reliability: Changing octopus shape makes some feature channels more discriminative than others, especially among the HoG channels.The example visualizes channel reliability weights across HoG, grayscale-template, and Colornames channels.
- Channel reliability: CSR-DCF successfully tracks a gymnast undergoing substantial deformation and rotation with a well-expressed localization-response peak.The response peak indicates that discriminative channels outweigh less discriminative ones during localization.
- Spatial reliability: Spatial reliability maps fit the sprinter’s target and prevent the filter from learning the background.
- Occlusion: The tracker recovers and localizes a motorcyclist after repeated short-term full occlusions.The example attributes this behavior to robust learning with channel and spatial reliability maps and a large capture range.
5 Conclusion
CSR-DCF combines spatially constrained filter support with channel reliability weighting to address limitations of standard correlation filters. Experiments report state-of-the-art benchmark performance and speed competitive with deep-feature trackers despite using simple features.
- Contributions: Spatial reliability adapts filter support to trackable object regions, enabling arbitrary search-region sizes and avoiding the rectangular-shape assumption.The method estimates the map efficiently and constrains correlation-filter learning with it.
- Contributions: Channel reliability is estimated from the constrained least-squares solution and weights per-channel filter responses during localization.
- Results: CSR-DCF achieves state-of-the-art performance on OTB100, VOT2015, and VOT2016 while running close to real time on a single CPU.
- Results: Using HoG and Colornames, CSR-DCF performs on par with deep-ConvNet trackers but is significantly faster.
- Scope: The spatial and channel reliability formulation is presented as generalizable to most modern correlation filters, including those using deep features.
6 Appendix 1: Derivation of the augmented Lagrangian minimizer
The appendix derives the minimization steps for the augmented Lagrangian used in constrained correlation-filter learning. It rewrites the problem in vectorized Fourier form, derives gradient conditions, and handles the binary support mask in the solution.
- Vectorized formulation: The derivation rewrites the filter-learning formulation in fully vectorized form using the Fourier-coefficient matrix F.F is defined as a D × D orthonormal matrix of Fourier coefficients.
- Optimization setup: Equation (8) is minimized iteratively with respect to the constrained filter variables and related auxiliary quantities.
- Closed-form updates: The minimizer for ˆhc is obtained by setting the complex gradient of the augmented Lagrangian to zero and evaluating its partial gradients.
- Closed-form updates: The optimal h is derived by requiring the complex gradient with respect to h to vanish.
- Mask constraint: Because the support mask is binary, entries outside the mask are set to zero to satisfy h ≡ h⊙m and avoid division by zero.The appendix identifies this masked expression as the proximal solution.
Acknownledgements
The authors acknowledge funding from Slovenian and Czech research programs, support from Toyota Motor Europe, and discussions with dr. Rok Žitko.
- The work received support from Slovenian research agency programs and projects P2-0214, L2-6765, and J2-8175.
- Jiří Matas and Tomáš Voјíř were supported by Czech Science Foundation Project GACR P103/12/G084 and Toyota Motor Europe.
- The authors thank dr. Rok Žitko for discussions on complex differentiation.