Source-linked AI summary
Adaptive Correlation Filters with Long-Term and Short-Term Memory for Object Tracking
Chao Ma, Jia-Bin Huang, Xiaokang Yang, Ming-Hsuan Yang
TL;DR
Object tracking must handle major appearance changes, but highly adaptive correlation filters can drift and lack long-term memory for recovery. The paper combines translation, scale, and conservative long-term filters with an incremental detector, and reports favorable efficiency, accuracy, and robustness against state-of-the-art methods. Its performance depends on feature choice and initialization, with distinct failures reported for deep versus handcrafted features and inaccurate spatial initialization.
Problem
Appearance variation can make tracking difficult, while highly adaptive correlation filters are prone to drift and cannot recover from failures without long-term target memory.
Method
The tracker combines adaptive translation and scale filters with a conservative long-term filter for failure detection and an incrementally learned detector for target recovery.
Results
The proposed algorithm performs favorably against state-of-the-art methods in efficiency, accuracy, and robustness on benchmark tracking evaluations.
Takeaways & Limitations
Combining short-term adaptation with long-term memory provides a robust tracking approach that explicitly handles tracking failures through target recovery.
Takeaways & Limitations
Performance depends on feature and initialization choices: deep features fail on dark-versus-bright appearance, handcrafted features drift in clutter, and inaccurate spatial initialization degrades localization.
Abstract
from arXiv · showhide
Object tracking is challenging as target objects often undergo drastic appearance changes over time. Recently, adaptive correlation filters have been successfully applied to object tracking. However, tracking algorithms relying on highly adaptive correlation filters are prone to drift due to noisy updates. Moreover, as these algorithms do not maintain long-term memory of target appearance, they cannot recover from tracking failures caused by heavy occlusion or target disappearance in the camera view. In this paper, we propose to learn multiple adaptive correlation filters with both long-term and short-term memory of target appearance for robust object tracking. First, we learn a kernelized correlation filter with an aggressive learning rate for locating target objects precisely. We take into account the appropriate size of surrounding context and the feature representations. Second, we learn a correlation filter over a feature pyramid centered at the estimated target position for predicting scale changes. Third, we learn a complementary correlation filter with a conservative learning rate to maintain long-term memory of target appearance. We use the output responses of this long-term filter to determine if tracking failure occurs. In the case of tracking failures, we apply an incrementally learned detector to recover the target position in a sliding window fashion. Extensive experimental results on large-scale benchmark datasets demonstrate that the proposed algorithm performs favorably against the state-of-the-art methods in terms of efficiency, accuracy, and robustness.
1 Introduction
Object tracking remains difficult under appearance variation, while highly adaptive correlation filters can drift and cannot recover from failures. The paper addresses this stability–adaptivity dilemma with complementary filters that combine short-term adaptation, scale estimation, and long-term memory.
- Appearance changes from deformation, sudden motion, illumination change, occlusion, and disappearance make object tracking challenging.
- High learning-rate correlation-filter updates preserve short-term appearance memory but can drift from noisy updates and fail to recover after prolonged occlusion.
- The proposed tracker uses translation, scale, and long-term filters to balance adaptive tracking with stable target memory.
- Histogram of local intensities complements HOG features to strengthen discrimination between the target and surrounding background for localization.
- An incrementally updated detector recovers the target when the tracking confidence falls below a threshold.
- Experiments compare the method with state-of-the-art trackers on OTB2013, OTB2015, and 10 additional challenging sequences.
2 Related Work
Related tracking-by-detection methods update classifiers from local samples but face sampling ambiguity and a stability–adaptivity dilemma. Correlation filters address these concerns through efficient Fourier-domain computation, contextual information, and extensions for richer tracking functionality.
- Tracking-by-Detection: Tracking-by-detection methods formulate each frame as detection in a local search window and incrementally learn target–background classifiers.
- Tracking-by-Detection: Small labeling errors in sampled patches can accumulate over time and cause tracker drift, motivating robust updates with ensemble, semi-supervised, multiple-instance, structure, and transfer learning.
- Correlation Filters: Correlation filters compute spatial correlation efficiently in the Fourier domain and use kernel tricks to improve accuracy without substantially increasing computational complexity.
- Correlation Filters: Correlation filters incorporate surrounding context, which can provide discriminative cues for inferring target position during heavy occlusion.
- Correlation Filters: Research extensions include kernelized, multi-channel, contextual, scale-estimation, and spatially regularized correlation filters.
3 Overview
The proposed tracker uses multiple correlation filters to address appearance change, scale variation, and recovery from tracking failures. Translation and scale filters adapt over short time horizons, while a conservative long-term filter estimates confidence and triggers recovery.
- Motivation: A single correlation filter cannot simultaneously handle appearance change, scale variation, and recovery because stability and adaptivity are difficult to balance in one module.
- Three Correlation Filters: The translation filter adapts to target and contextual appearance, the scale filter predicts scale variation, and the long-term filter maintains target memory.
- Tracking Pipeline: The long-term filter determines whether tracking has failed by testing its response against a threshold.
- Tracking Pipeline: The tracker applies the translation filter in a search window, then uses the scale filter to predict scale changes around the estimated position.
4 Tracking Components
The tracker combines kernelized correlation filters, multi-channel features, sequential scale estimation, and distinct short- and long-term updates to localize targets and detect failures. Its long-term filter supports failure prediction while conservative updating limits drift from noisy samples.
- Kernelized Correlation Filters: The method uses correlation filters over circularly shifted training samples, with Fourier-domain computation and kernelization improving discrimination while retaining O(n log n) complexity.The linear filter treats shifted patches as regression data; kernelization uses a kernel space without significantly increasing computational complexity.
- Scale Regression Model via Correlation Filters: The tracker estimates translation first and scale second, avoiding dense joint state evaluation and reducing translation-filter degradation from inaccurate scale estimates.Unlike updating with estimated scale at every frame, the method uses the ground-truth scale in the first frame and reports significantly better results than a joint-scale alternative.
- Long-Term and Short-Term Memory: A moving-average update with learning rate η emphasizes short-term adaptivity, but noisy samples can cause drift and prevent recovery from tracking failures.The paper addresses this limitation with a separate long-term filter updated only when its confidence exceeds a stability threshold Ts.
- Long-Term and Short-Term Memory: The conservatively updated long-term filter preserves appearance information over time, predicts failures from response confidence, and supports re-detection after tracking loss.On the lemming sequence, it identified failure after frame 360 with confidence scores generally below 0.15, unlike aggressively updated filters and LSTM hidden states.
5 Implementation Details
The tracker combines three correlation filters with an SVM re-detection module and evaluates performance using standardized OTB metrics and robustness plots.
- Tracker components: The proposed tracker uses translation, scale, and long-term correlation filters, plus an SVM module to recover targets after tracking failures.The filters estimate position, scale, and long-term appearance confidence, while re-detection handles failures.
- Translation estimation: The translation filter uses contextual cues and an enlarged bounding-box ratio of r = 2.8.The selected context ratio exceeds the fixed ratio r = 2.5 used by existing methods.
- Online detection: The detector uses positive samples above 0.5 overlap and negative samples below 0.1 overlap, with re-detection and acceptance thresholds of 0.15 and 0.38.The re-detection module activates at Tr = 0.15 and accepts detection results at Ta = 0.38.
- Scale estimation: The scale filter uses N = 21 feature-pyramid levels with scale factor α = 1.03, while the learning rate is η = 0.01.These parameters are empirically determined and fixed throughout the experiments.
- Evaluation: Evaluation reports overlap success and distance precision under OPE, TRE, and SRE protocols on OTB2013.The precision plots use distance precision at 20 pixels, while success plots use overlap success AUC.
6 Experimental Results
Experiments evaluate the tracker on benchmark datasets using overlap, localization, and speed measures, alongside comparisons with correlation-filter and online-classifier baselines.
- Datasets: The proposed algorithm is evaluated on a 100-video benchmark dataset and ten additional sequences for testing its re-detection module.The additional sequences specifically validate recovery from tracking failures.
- Evaluation metrics: Performance is measured with overlap success rate and distance precision rate, based on bounding-box overlap and center-location error.Distance precision is commonly reported at a 20-pixel threshold, while overlap success uses an overlap threshold of 0.5.
- Baselines: The comparison includes handcrafted-feature and deep-feature versions of the proposed method against 13 state-of-the-art trackers.Baselines include correlation-filter trackers, single-classifier trackers, and multiple-classifier trackers.
6.2 Overall Performance
Across OTB2013 and OTB2015 evaluations, the proposed tracker performs favorably against representative baselines on overlap success and distance precision, while retaining near-real-time speed. Its main limitations are weaker robustness under TRE/SRE protocols and sensitivity to inaccurate initialization.
- The proposed algorithm performs favorably against representative baselines on overlap success and distance precision across OTB2013 and OTB2015.The evaluation reports OPE, TRE, and SRE results using distance precision, overlap success, center error, and tracking speed.
- 81.3% versus 69.6% overlap success over MEEM shows the benefit of explicitly predicting scale variation.The comparison concerns the MEEM dataset.
- 81.3% versus 78.4% on OTB2013 and 70.1% versus 68.3% on OTB2015 are reported against MUSTer.These are overlap success rates for the proposed method versus MUSTer.
- 20 frames per second keeps the tracker close to real-time, although single-filter trackers are faster.The detector activates only when confidence falls below the re-detection threshold.
- Performance is weaker under TRE and SRE than under OPE, and inaccurate spatial initialization adversely affects learned-filter localization.
- Trackers with re-detection modules outperform trackers without them in the reported comparisons.
6.3 Attribute-Based Evaluation
Attribute-based and component analyses show that complementary features, separate scale handling, and correlation-filter-based re-detection support the proposed tracker's performance across challenging conditions.
- Attribute-Based Evaluation: The proposed algorithm achieves large gains over MUSTer in seven attributes, including deformation, occlusion, and in-plan rotation.Reported gains are 2.0% to 7.9% across illumination variation, out-of-plan rotation, occlusion, deformation, in-plan rotation, background clutter, and low resolution.
- Evaluation Setup: The ablations compare translation, scale, and re-detection alternatives under OPE using precision at 20 pixels and success AUC.
- Feature Analysis: CT-HOGHOI-VGGNet19 outperforms alternative translation-filter designs in distance precision and overlap success.It combines deep and handcrafted features.
- Scale Filter: HOG-only scale filters are selected for efficiency because adding HOI features does not improve scale-estimation accuracy.
- Re-Detection Module: The passive-aggressive SVM update performs slightly better than the support-vector update, while CF-CF and LSTM-SVM perform worse than CT-HOGHOI-CF-SVM.The LSTM-SVM comparison is attributed to limited training data.
6.5 Sensitivity Analysis
The tracking confidence is robust to a reasonable range of stability thresholds, supporting the selected threshold for conservative long-term-filter updates.
- The baseline CT-HOGHOI tracker fails after the 360-th frame without re-detection, motivating the long-term filter's failure-detection role.
- Confidence scores are not sensitive to stability thresholds between 0.2 and 0.5.The method sets Ts equal to Ta as 0.38.
6.6 Exploiting Contextual Cues
Contextual features, independently updated translation and scale filters, and online re-detection support robust tracking across challenging sequences, but feature choice remains sequence-dependent and failures persist.
- Context: The translation filter is sensitive to padding size when incorporating surrounding context.
- Feature Design: HOG and HOI features reduce sensitivity to illumination, clutter, rotation, and partial occlusion in the reported sequences.
- Qualitative Evaluation: The proposed tracker performs favorably in both translation and scale estimation across seven challenging sequences.The comparison includes MUSTer, KCF, STC, Struck, and TLD.
- Filter Updates: Independent scale and translation updates alleviate degradation from inaccurate scale estimation and drifting caused by scale change.
- Re-Detection: Online detection re-detects targets after failures involving heavy occlusion or disappearance from view.
- Failure Cases: Long-term occlusion in girl2 prevents re-detection when the target resembles surrounding people.
- Failure Cases: Deep features fail on singer2, while handcrafted features fail on soccer because their respective semantic and spatial cues are mismatched to those sequences.
6.8 MEEM Dataset
On the MEEM dataset, the proposed tracker is evaluated against correlation-filter and re-detection baselines across occlusion, illumination, contrast, and deformation challenges. It performs favorably overall, with more than 10% gains in overlap success and distance precision rate.
- Dataset and protocol: The MEEM dataset contains approximately 7500 frames spanning heavy occlusion, abrupt illumination changes, low contrast, and non-rigid deformation.The comparison includes ten sequences and excludes deep features for fair comparison.
- Quantitative results: More than 10% gains in both overlap success and distance precision rate are reported over state-of-the-art trackers.The comparison includes correlation-filter trackers such as MUSTer, KCF, STC, and DSST.
- Qualitative analysis: Correlation trackers without re-detection modules cannot recover targets from heavy occlusion in boxing1, boxing2, and ped1.The qualitative comparison includes our approach, MUSTer, KCF, STC, DSST, and TLD.
- Overall finding: The proposed tracker performs favorably against the state-of-the-art trackers in both qualitative and quantitative comparisons.The results emphasize the role of multiple correlation filters in robust tracking.
7 Conclusion
The paper combines three correlation filters, additional appearance features, and an online detector to address robust object tracking. Extensive experiments report favorable performance against state-of-the-art methods in efficiency, accuracy, and robustness.
- Core architecture: Three correlation filters capture short-term and long-term target appearance through translation, scale, and long-term filters.The design addresses the stability-adaptivity dilemma.
- Feature and update design: HOI features complement HOG features to improve localization, while context size and learning rates are investigated as design choices.The paper also provides an ablation study of these choices.
- Failure recovery: An incrementally learned online detector explicitly handles tracking failures by recovering target positions.The conclusion identifies this detector as a separate extension to the correlation-filter framework.
- Empirical conclusion: Extensive experiments show favorable performance against state-of-the-art methods in efficiency, accuracy, and robustness.
SUPPLEMENTARY DOCUMENT
The supplementary studies test alternative update schemes, localization perturbations, and LSTM training data. They find that exhaustive historical updates are impractical and do not improve performance, while modest box enlargement has little effect and limited training data produces a train–test gap.
- All-history update: Updating the correlation filter over all appearances from the first frame through the current frame uses a weighted average quadratic error objective.The scheme assigns each frame a nonnegative weight β_j and minimizes the resulting objective over the filter w.
- All-history update: The all-history update requires linearly increasing computation and memory as the frame count grows, making it impractical in practice.Its average speed is 2.5 fps versus 20.8 fps for the proposed method on OTB2013.
- All-history update: 83.5% distance precision and 62.0% overlap success for the all-history update remain below the proposed method’s 84.8% and 62.8%, respectively.The supplementary analysis states that the alternative update scheme does not improve performance.
- Localization robustness: Slightly enlarging ground-truth bounding boxes by a scaling factor of 1.1 does not significantly affect tracking performance.The perturbation study also evaluates eight spatial-shift directions and scaling factors from 0.8 to 1.2.
- LSTM training data: The baseline LSTM tracker shows a large training-versus-validation/test performance gap because of limited training data.The comparison uses OTB2013 for testing and remaining OTB2015 sequences for validation.
- Supplementary evaluation: Updating the translation filter with all tracked results does not improve OTB2013 performance under one-pass evaluation.The plots report distance precision at 20 pixels and overlap success using AUC.