Source-linked AI summary
CREST: Convolutional Residual Learning for Visual Tracking
Yibing Song, Chao Ma, Lijun Gong, Jiawei Zhang, Rynson Lau, Ming-Hsuan Yang
TL;DR
Existing DCF trackers separate feature extraction from filter learning and rely on empirical update operations, limiting end-to-end training and potentially causing drift. CREST reformulates DCFs as a differentiable one-layer convolutional network with residual learning for appearance changes. On benchmark datasets, CREST performs favorably against state-of-the-art trackers.
Problem
Existing DCF trackers separate filter learning from feature extraction and use empirical interpolation updates that provide little end-to-end-training benefit and can cause drift from noisy updates.
Method
CREST reformulates DCFs as a one-layer convolutional network integrating feature extraction, response-map generation, model update, and residual learning.
Results
CREST performs favorably against state-of-the-art trackers on benchmark datasets.
Takeaways & Limitations
The formulation enables differentiable filter updates through back propagation while residual layers account for target appearance changes during online tracking.
Abstract
from arXiv · showhide
Discriminative correlation filters (DCFs) have been shown to perform superiorly in visual tracking. They only need a small set of training samples from the initial frame to generate an appearance model. However, existing DCFs learn the filters separately from feature extraction, and update these filters using a moving average operation with an empirical weight. These DCF trackers hardly benefit from the end-to-end training. In this paper, we propose the CREST algorithm to reformulate DCFs as a one-layer convolutional neural network. Our method integrates feature extraction, response map generation as well as model update into the neural networks for an end-to-end training. To reduce model degradation during online update, we apply residual learning to take appearance changes into account. Extensive experiments on the benchmark datasets demonstrate that our CREST tracker performs favorably against state-of-the-art trackers.
1. Introduction
Visual tracking must build robust appearance models from extremely limited initial-frame data. CREST addresses limitations in DCF feature learning and empirical updates by integrating convolutional processing with residual learning and reports favorable benchmark performance.
- Visual tracking must handle background clutter, scale variation, motion blur, and partial occlusions while learning from usually one first-frame bounding box.
- Deep convolutional features have helped DCF trackers achieve state-of-the-art performance on recent tracking benchmark datasets.
- Existing DCF trackers learn filters independently from feature extraction, benefit little from end-to-end training, and use empirical interpolation weights that can cause drift through noisy updates.
- CREST reformulates DCFs as a one-layer convolutional network integrating feature extraction, response-map generation, and model update into an end-to-end form.
- Residual learning captures target appearance changes and is used to reduce rapid model degradation during online updating.
- CREST is extensively evaluated on benchmark datasets and performs favorably against state-of-the-art trackers.
2. Related Work
Correlation-filter trackers emphasize computational efficiency and dense response prediction, while CNN trackers explore learned visual representations and online models. CREST’s pipeline combines convolutional features with base and residual mappings to produce response maps.
- Tracking by Correlation Filters: Correlation-filter trackers regress circularly shifted input features to a Gaussian function and can encode target appearance without multiple target-appearance samples.
- Tracking by Correlation Filters: Correlation filters attract attention partly because they offer computational efficiency in the Fourier domain.
- Tracking by CNNs: CNN-based tracking methods use pretrained object-recognition networks with discriminative or regression models to represent and localize targets.
- CREST extracts convolutional features from current-frame and initial-frame search patches, then transforms them into response maps through base and residual mappings.
3. Convolutional Residual Learning
CREST reformulates discriminative correlation filters as a convolutional network and adds spatial and temporal residual learning to model appearance differences during tracking.
- 3.1. DCF Reformulation: CREST formulates DCF learning as a one-layer convolutional neural network that integrates feature extraction, response generation, and model update.The base layer uses a convolutional filter equivalent to the DCF correlation filter.
- 3.1. DCF Reformulation: The base layer minimizes the L2 loss between its convolutional output F(X) and the ground-truth label Y, with L2 regularization on the filter.The filter size is set to cover the target object.
- 3.2. Residual Learning: Residual layers learn the difference between the base-layer output and the desired mapping instead of stacking additional layers to approximate the mapping directly.The spatial residual network uses three layers with small filters, and the combined mappings generate the response map.
- 3.2. Residual Learning: A temporal residual from the first frame complements spatial residual learning when spatial residuals are ineffective, without requiring additional training data.The temporal input is extracted from the initial object appearance, and the spatiotemporal process contributes to response-map generation.
- 3.2. Residual Learning: Residual layers alleviate noisy base-layer responses under large appearance variations, making the final target response more robust.The base layer may fail to differentiate target and background under changes such as background clutter.
4. Tracking via CREST
CREST performs tracking by extracting features and response maps for initialization, detection, scale estimation, and periodic online model updates. It uses maximum-response localization, multi-scale search, and training pairs collected during tracking.
- Model Initialization: CREST initializes from a target-centered training patch using VGG-based feature extraction and randomly initialized base and residual layers.The layers become well initialized after a few steps.
- Online Detection: For each new frame, CREST extracts a search patch and locates the target at the maximum response value.The search patch matches the training-patch size and is processed to generate a response map.
- Scale Estimation: CREST estimates scale by evaluating resized search patches at different scales, normalizing candidate object sizes to the training-patch size.The target width and height are updated using the scaled object with the maximum response; β smooths the size update.
- Model Update: During online tracking, CREST forms training pairs from search patches and corresponding ground-truth response maps collected every T frames.These pairs are fed into the network for online model updates after target localization.
5. Experiments
Experiments evaluate CREST through ablations and comparisons on OTB-2013, OTB-2015, and VOT-2016. Residual integration improves the base layer, while CREST performs favorably against state-of-the-art trackers but is less effective than CCOT on motion-blur and fast-motion sequences.
- Experimental Setup: CREST is evaluated on OTB-2013, OTB-2015, and VOT-2016 using benchmark-specific one-pass metrics and large-scale video sequences.OTB-2013 and OTB-2015 contain 50 and 100 sequences; VOT-2016 includes 60 challenging videos.
- Ablation Studies: The CREST base layer achieves performance similar to HCFT on average when both use convolutional features without scale estimation.The comparison uses AUC and average distance precision scores on OTB-2013.
- Ablation Studies: Spatial residual learning produces obvious improvement over the base layer, whereas temporal residual contributes little to overall performance.The ablation evaluates integration of spatiotemporal residual layers using AUC and average distance precision.
- Quantitative Evaluation: On OTB-2013, CREST performs favorably on both distance precision and overlap success rate among the compared trackers.Figure 7 reports AUC and distance precision at 20 pixels for 50 sequences; MDNet is excluded because it uses tracking videos for training.
- Qualitative Evaluation: Qualitative evaluations attribute CREST’s accurate localization in challenging scenarios to jointly optimized residual layers that refine the base response map.The comparison covers background clutter, occlusion, fast motion, illumination variation, deformation, and out-of-view cases across twelve sequences.
6. Concluding Remarks
CREST reformulates correlation filters as a differentiable one-layer convolutional network that unifies feature extraction, response generation, and model updating. Residual layers account for target appearance changes and refine response maps to reduce model degradation.
- CREST formulates the correlation filter as a one-layer convolutional neural network, called the base layer.
- The base layer integrates convolutional feature extraction, correlation response-map generation, and model updating for end-to-end training and prediction.
- Because the convolutional layer is fully differentiable, its filters can be updated through online backpropagation.
- Spatiotemporal residual layers capture target appearance changes and refine response maps by reducing noisy values.
- Experiments on standard benchmarks indicate that CREST performs favorably against state-of-the-art trackers.