Source-linked AI summary
Learning Background-Aware Correlation Filters for Visual Tracking
Hamed Kiani Galoogahi, Ashton Fagg, Simon Lucey
TL;DR
Conventional correlation filters adapt quickly but train mainly on shifted foreground patches, leaving real background variation underused. BACF learns from densely extracted background negatives alongside the target and updates the model online. Across multiple tracking benchmarks, it reports competitive accuracy and superior real-time speed against CF-based and deep trackers.
Problem
Conventional CF training relies on shifted foreground patches affected by boundary effects and lacks real negative background examples.
Method
BACF learns filters from real background patches together with the target and uses online adaptation for changing target and background appearance.
Results
BACF achieved competitive accuracy and superior tracking speed across extensive evaluations against recent CF-based and deep trackers.
Takeaways & Limitations
Learning from real patches with online adaptation improved robustness to deformation, scaling, and background clutter within the evaluated tracking benchmarks.
Takeaways & Limitations
The direct optimization in Eq. 9 remains intractable for real-time tracking because it requires solving T independent K ×K linear systems.
Abstract
from arXiv · showhide
Correlation Filters (CFs) have recently demonstrated excellent performance in terms of rapidly tracking objects under challenging photometric and geometric variations. The strength of the approach comes from its ability to efficiently learn - "on the fly" - how the object is changing over time. A fundamental drawback to CFs, however, is that the background of the object is not be modelled over time which can result in suboptimal results. In this paper we propose a Background-Aware CF that can model how both the foreground and background of the object varies over time. Our approach, like conventional CFs, is extremely computationally efficient - and extensive experiments over multiple tracking benchmarks demonstrate the superior accuracy and real-time performance of our method compared to the state-of-the-art trackers including those based on a deep learning paradigm.
1. Introduction
Correlation filters adapt quickly and efficiently, but conventional training relies on shifted foreground patches while discarding real background examples. BACF addresses this gap by learning from densely extracted background negatives, with efficient optimization and competitive real-time accuracy.
- Motivation: CF trackers learn and detect rapidly in the frequency domain while adapting online to photometric and geometric changes.Per-frame adaptation supports tracking under motion blur, scaling, and lighting variation.
- Motivation: Circularly shifted foreground patches suffer boundary effects and may not represent real negative patches in natural scenes.These effects can produce suboptimal training examples.
- Motivation: Limited shifted examples can overfit filters, while missing real negatives weakens discrimination against cluttered backgrounds and increases drift risk.The risk is especially relevant when target and background share visual cues.
- Contribution: BACF learns filters from real negative examples densely extracted from the background instead of relying only on shifted foreground patches.The method is proposed for real-time visual tracking.
- Contribution: An ADMM-based approach learns multi-channel filters with computational cost O(LKT log(T)).Here, T is the vectorized frame size, K the number of feature channels, and L the number of ADMM iterations.
- Evaluation: BACF achieved competitive accuracy against CF-based and deep trackers while running at approximately 40 FPS on a CPU.The tracker was evaluated on OTB50, OTB100, Temple-Color128, and VOT2015.
2. Prior Art
Prior CF trackers preserve frequency-domain efficiency but are affected by shifted-patch boundaries and discard background information. BACF instead uses patches from the entire frame, including background negatives, while retaining efficient Fourier-domain computation.
- Existing CFs: MOSSE established the speed advantage of CF tracking, and later methods extended CFs with multi-channel features such as HOG.Frequency-domain formulations retained computational efficiency for real-time applications.
- Existing CFs: Frequency-domain CFs are affected by boundary effects from shifted patches, which can lead to suboptimal training.These methods also discard background visual information from learning.
- Deep tracking: Deep features and deep architectures improved robustness in visual tracking but introduced a distinct alternative to hand-crafted-feature CFs.The cited discussion frames this trend as having both advantages and disadvantages.
- BACF: BACF extracts all possible positive and negative patches from the entire frame and uses Fourier-domain objective augmentation to perform the operations efficiently.The target patch is positive, while surrounding background patches serve as negatives.
3. Correlation Filters
Standard multi-channel CF learning formulates correlation-filter training as a regularized spatial-domain objective with circularly shifted foreground examples. BACF replaces this limited training set with cropped patches from the surrounding frame to obtain more robust and generalized filters.
- Standard formulation: The spatial-domain objective learns multi-channel filters by matching correlation outputs to a desired response under regularization.Each x_k and h_k denotes an image and filter channel, while K is the number of feature channels.
- Standard formulation: The circular-shift operator generates shifted training examples, with y(j) denoting the j-th desired-response element.The operator applies a j-step discrete circular shift to x_k.
- Standard formulation: Training on D −1 circularly shifted foreground patches can overfit the target and limit discrimination against real non-target patches.The shifted examples are generated by the circular-shift operator.
- BACF motivation: Tracking lacks abundant target-specific positive and negative samples, but the surrounding background provides usable negative training patches.The target is the only positive sample available at training time.
4. Background-Aware Correlation Filters
BACF learns a multi-channel filter from large-support samples so cropped patches include both target and real background examples. Its ADMM-based optimization and Fourier-domain implementation are designed to retain real-time efficiency.
- The objective learns a multi-channel background-aware filter from a large-support training sample and desired correlation response.The filter has smaller spatial size than the training sample, while the response peaks at the target.
- Circular shifting followed by cropping generates all D-sized patches across the frame, including the target as a positive example and background as negative examples.The cropping matrix P can be precomputed and implemented efficiently with a lookup table.
- The frequency-domain formulation introduces an auxiliary variable and over-complete channel representations, then solves the objective iteratively with ADMM.The subproblems for the auxiliary variable and filter have closed-form solutions.
- The optimization avoids direct cubic solving by exploiting sparse banded structure and decomposing the problem into T independent K-dimensional objectives.This decomposition reduces the structure of the Fourier-domain subproblem before further acceleration.
- O(TK) computation replaces the O(TK^3) matrix solve through the Sherman-Morrison lemma, enabling a real-time extension of the optimization.The reduced cost is described as much smaller than the original computation.
- Online adaptation combines the previous model with the current frame using an adaptation rate, while detection applies the updated filter across multiple resolutions for scale changes.Detection produces correlation outputs for multiple scales and uses interpolation to maximize detection scores.
5. Experiments
BACF is evaluated across standard tracking datasets, metrics, feature types, initialization settings, and competing trackers. The experiments show strong accuracy, robustness, and real-time speed, including competitive results against deep-feature and deep-learning methods.
- Evaluation Methodology: BACF is evaluated using success plots, AUC ranking, success rate at IoU > 0.50, and CPU tracking speed across four standard datasets.The datasets are OTB50, OTB100, TC128, and VOT2015.
- Comparison with HOG-based Trackers: BACF achieves the highest accuracy among HOG-based trackers on OTB50, OTB100, and TC128.Its AUCs are 67.78 on OTB50, 62.98 on OTB100, and 51.97 on TC128.
- Attribute Based Evaluation: BACF maintains superior performance across the reported OTB100 visual attributes, including challenging conditions such as occlusion, deformation, and motion blur.The evaluation reports six of eleven manually annotated attributes in the main results.
- Robustness to Initialization: BACF shows robustness under noisy spatial initialization and different temporal initialization frames on OTB100.These are measured using spatial robustness (SRE) and temporal robustness (TRE) success plots.
- Comparison with Deep Feature-based Trackers: BACF achieves competitive accuracy with deep-feature CF trackers while tracking 170 times faster than them.BACF obtains an average success rate of 76.0, compared with 77.8 for CCOT, 65.0 for HCF, and 72.9 for DeepSRDCF.
- Comparison with Deep Feature-based Trackers: BACF is at least 100 times faster than CCOT on every OTB100 video while matching or exceeding CCOT on 60 of 101 sequences.BACF outperforms CCOT on 37 videos, CCOT wins on 41, and both perform equally on 23.
- Evaluation on VOT2015: BACF achieves the best accuracy on VOT2015 with 0.56 accuracy, improving the accuracy obtained by SRDCF and DeepSRDCF by 5%.CCOT has the highest robustness at 0.82, followed by DeepSRDCF at 1.05 and SRDCF at 1.24.
- Comparison with Deep Feature-based Trackers: BACF reaches 85.4 accuracy while outperforming SINT, SiamFC, STCT, YCNN, and FCNT, and remains close to MDNet at 87.3.The paper reports superior real-time performance against the compared deep trackers.
6. Conclusion
The proposed background-aware correlation filter learns from real background patches and adapts to changing target and background appearance over time. It achieves competitive accuracy and superior tracking speed against recent CF-based and deep trackers, with consistent results in challenging videos.
- Qualitative comparisons show consistent tracking under illumination change, fast motion, and background clutter.
- BACF exploits real background patches together with the target patch to learn the tracker.
- Online adaptation updates the model as target and background appearance change over time.
- Learning from real patches improves robustness against deformation, scaling, and background clutter.
- The method achieves competitive accuracy and superior tracking speed compared with recent CF-based and deep trackers over extensive evaluation.