Source-linked AI summary
Staple: Complementary Learners for Real-Time Tracking
Luca Bertinetto, Jack Valmadre, Stuart Golodetz, Ondrej Miksik, Philip Torr
TL;DR
Correlation Filters are robust but sensitive to deformation, while colour-based models handle shape variation yet can be vulnerable to illumination changes and weak discrimination. Staple combines complementary template and colour cues through independently solved ridge-regression models, and it runs faster than 80 FPS while outperforming more complex trackers across multiple benchmarks.
Problem
Tracking unfamiliar objects requires robustness to changing appearance without advance training examples, while existing cues have complementary weaknesses involving deformation, illumination, or discrimination.
Method
Staple combines a Correlation Filter with HOG features and a global colour histogram by solving two independent ridge-regression problems and combining their scores in a dense translation search.
Results
Staple runs at speeds exceeding 80 FPS and outperforms more complex trackers across multiple benchmarks, including a 23% average relative improvement over Struck on OTB.
Takeaways & Limitations
Staple offers a simple, fast tracking approach for applications requiring robustness to colour, illumination, and shape changes.
Takeaways & Limitations
Real-time performance depends on choosing feature and loss functions whose associated optimisation problems can be solved efficiently, and benchmark performance measures must be considered jointly.
Abstract
from arXiv · showhide
Correlation Filter-based trackers have recently achieved excellent performance, showing great robustness to challenging situations exhibiting motion blur and illumination changes. However, since the model that they learn depends strongly on the spatial layout of the tracked object, they are notoriously sensitive to deformation. Models based on colour statistics have complementary traits: they cope well with variation in shape, but suffer when illumination is not consistent throughout a sequence. Moreover, colour distributions alone can be insufficiently discriminative. In this paper, we show that a simple tracker combining complementary cues in a ridge regression framework can operate faster than 80 FPS and outperform not only all entries in the popular VOT14 competition, but also recent and far more sophisticated trackers according to multiple benchmarks.
1. Introduction
Staple addresses the difficulty of tracking unfamiliar objects whose appearance changes over time by combining complementary representations for robust, fast adaptation. The resulting tracker combines template and colour cues and achieves strong benchmark performance at above-real-time speed.
- 1. Introduction: Above-real-time operation is advantageous for computationally intensive applications including robotics, surveillance, video processing and augmented reality.The introduction identifies above-real-time processing as a practical requirement for these applications.
- 1. Introduction: Model adaptation is needed because an object’s appearance can vary significantly during a video, but learning from predictions can cause model drift.Small errors may accumulate, particularly when the object’s appearance changes.
- 1. Introduction: Staple combines complementary image-patch representations to improve robustness to both colour changes and deformations.The tracker uses template and pixel-wise learners within a ridge-regression framework.
- 1. Introduction: Staple combines a Correlation Filter using HOG features with a global colour histogram and runs faster than 80 FPS while outperforming many more complex trackers.The result is reported across multiple benchmarks.
2. Related Work
Prior tracking methods trade off computational efficiency, deformation robustness, drift prevention, and complementary-model fusion. Staple builds on these limitations by combining template and histogram scores directly while retaining real-time operation.
- 2. Related Work: Correlation Filters enable dense training with high-dimensional feature images in real time, but their circular-shift constraint makes them sensitive to deformation.Spatial regularisation can address this deficiency, but at the cost of real-time operation.
- 2. Related Work: Learning a deformable model from a single video with only first-frame location supervision is considered ambitious, motivating the use of a simple bounding box.The paper notes that richer deformable models are not necessarily rewarded by the reported evaluations.
- 2. Related Work: Alternative deformation strategies include pixel voting and deformable models, but the cited voting methods had not demonstrated competitive benchmark performance.HoughTrack and PixelTrack accumulate pixel votes to estimate the object’s extent.
- 2. Related Work: Drift-reduction methods modify training or supervision through optical flow, conservative models, multiple-instance learning, self-paced learning, boosting, or co-training.Among the cited methods, only MILTrack and TLD appeared in current benchmarks, and neither had strong results.
- 2. Related Work: Ensemble trackers combine estimates so that weaknesses of different trackers can be reciprocally compensated.Prior examples use complementary trackers in sampling frameworks or factorial HMMs.
- 2. Related Work: Staple directly combines two independently learned scores in a dense search, preserving real-time operation rather than using a more elaborate fusion framework.This design is presented as the paper’s distinction from prior multi-estimate approaches.
3. Proposed Approach
Staple formulates tracking as dense translation search using complementary template and histogram scores, learned with efficient ridge-regression updates. The combined score preserves correlation-filter speed while adding permutation-invariant appearance information.
- Tracking formulation: Staple selects the target rectangle by maximizing a model score over candidate locations in each frame.The score is evaluated on transformed image windows and optimized using the previous tracking history.
- Complementary scores: The proposed score is a linear combination of template and histogram scores, with weights γtmpl = 1 − α and γhist = α selected on a validation set.Both component models are trained to produce compatible object-versus-background scores.
- Complementary scores: The template learner uses a spatial feature image and correlation-filter machinery, while the histogram learner uses a separate colour-based feature representation.The template is a K-channel image; the histogram operates on an M-channel feature image defined on a different grid.
- Complementary scores: Histogram scoring is invariant to spatial permutations, capturing appearance information that circular-shift training loses.This invariance complements the spatial sensitivity of the template representation.
- Efficient evaluation: Both learners support dense sliding-window evaluation because their feature transforms commute with translation, enabling convolution for templates and an integral image for histograms.Figure 2 depicts model updates from training patches and dense responses in the next frame.
- Efficient optimization: The two independent ridge-regression problems exploit least-squares structure and quadratic regularization to obtain closed-form updates without memory growing with the number of examples.Correlation-filter parameters use a fast formulation, while histogram parameters require a general-matrix optimization that is accelerated separately.
4. Evaluation
Staple is evaluated on VOT14, VOT15, and OTB-13 using accuracy, robustness, success-plot, and speed analyses. It outperforms competing trackers while maintaining real-time operation, with performance affected by configuration choices such as patch size, learning rates, and score merging.
- VOT14 and VOT15: Staple significantly outperforms all VOT14 entries and many later trackers when accuracy and failure counts are considered jointly.VOT14 reports average accuracy, number of failures, and an overall ranking derived from both metrics.
- VOT14 and VOT15: 7% higher accuracy and 41% fewer failures than SRDCF, and 11% higher accuracy and 13% fewer failures than DATs, are reported on VOT14.Staple ranks best in accuracy and fourth in failures; trackers with fewer failures score at least 20% worse in accuracy.
- VOT14 and VOT15: +22% and +10% accuracy, together with +35% and +47% robustness to failures, are reported against DATs and DSST respectively on VOT15.The comparison uses the second-best VOT14 performer and the VOT14 winner as baselines.
- OTB-13: 23% average relative improvement over Struck is reported on OTB-13, while Staple also outperforms later and deep-convolutional trackers at a higher frame rate.The evaluation covers OPE, SRE, and TRE success plots; ACT is comparable in frame rate but performs substantially worse.
- Efficiency: Approximately 80 FPS is achieved by the MATLAB prototype, and speeds beyond 100 FPS are possible with HOG cells of size 2 × 2 at a small increase in failures.The speed–robustness trade-off is evaluated across HOG cell sizes, with accuracy following a similar trend.
- Parameter and merge-factor experiments: Maximum robustness occurs around ηtmpl = 0.01 and ηhist = 0.01, while accuracy is significantly influenced by the merge factor and is best around α = 0.3.The learning rates control replacement of earlier evidence, and dense-response merging outperforms interpolation of final estimates.
5. Conclusion
Staple combines template and histogram scores to address deformation and colour or illumination changes while preserving speed. Its simplicity and benchmark performance make it suitable for applications where computational resources are also needed.
- Staple combines template and histogram scores to improve robustness to colour, illumination, and shape changes.
- Staple outperforms significantly more complex state-of-the-art trackers across several benchmarks.
- Given its speed and simplicity, Staple is a logical choice for applications that require computational effort themselves.