Source-linked AI summary
Transferring Rich Feature Hierarchies for Robust Visual Tracking
Naiyan Wang, Siyi Li, Abhinav Gupta, Dit-Yan Yeung
TL;DR
Visual tracking lacks the abundant labeled data typically used to train CNNs, often offering only one labeled example in the first frame. The paper transfers offline-learned object features to online tracking with fine-tuning and structured pixel-wise outputs, reporting substantial improvement over state-of-the-art trackers, including a 0.602 versus 0.529 AUC result on an open benchmark.
Problem
Visual tracking typically has only one labeled example in the first video frame, limiting the labeled data available for applying CNNs.
Method
The paper pre-trains a CNN to distinguish objects from non-objects, transfers its generic features to online tracking, fine-tunes them, and produces pixel-wise probability maps.
Results
0.602 versus 0.529 AUC on the open benchmark’s overlap rate curve, with more than 10% improvement over the best state-of-the-art baseline tracker.
Takeaways & Limitations
The paper demonstrates that transferring high-level CNN feature hierarchies and using structured outputs can support visual tracking with limited labeled training data.
Takeaways & Limitations
The tracker is likely to drift when background distractors exist or the target is occluded, because similarly appearing distractors may be recognized as the target.
Abstract
from arXiv · showhide
Convolutional neural network (CNN) models have demonstrated great success in various computer vision tasks including image classification and object detection. However, some equally important tasks such as visual tracking remain relatively unexplored. We believe that a major hurdle that hinders the application of CNN to visual tracking is the lack of properly labeled training data. While existing applications that liberate the power of CNN often need an enormous amount of training data in the order of millions, visual tracking applications typically have only one labeled example in the first frame of each video. We address this research issue here by pre-training a CNN offline and then transferring the rich feature hierarchies learned to online tracking. The CNN is also fine-tuned during online tracking to adapt to the appearance of the tracked target specified in the first video frame. To fit the characteristics of object tracking, we first pre-train the CNN to recognize what is an object, and then propose to generate a probability map instead of producing a simple class label. Using two challenging open benchmarks for performance evaluation, our proposed tracker has demonstrated substantial improvement over other state-of-the-art trackers.
1. Introduction
The paper addresses visual tracking’s limited labeled data by transferring CNN features learned offline and adapting them online. It introduces object-focused pre-training and pixel-wise structured outputs, reporting improved benchmark performance.
- Motivation: Visual tracking often provides only one labeled target example in the first video frame, making direct large-scale CNN application infeasible.The paper frames tracking as a one-pass, model-free single-object setting with online adaptation to appearance changes.
- Approach: The proposed approach pre-trains a CNN offline and transfers generic object features to online tracking, where the model is fine-tuned for the target.This uses auxiliary data to learn an invariant representation before adapting from the video’s limited supervision.
- Objectness pre-training: The CNN is pre-trained to distinguish objects from non-objects rather than reconstructing images or performing categorical classification.The design targets overfitting and drifting during online tracking using object-level annotations.
- Structured output: The CNN outputs a pixel-wise probability map indicating whether each input pixel belongs to an object’s bounding box.This structured output provides an induced structured loss and computational scalability.
- Evaluation: 0.602 versus 0.529 AUC on the open benchmark’s overlap rate curve demonstrates the reported improvement.The paper also reports more than 10% improvement over the best state-of-the-art baseline tracker.
2. Related Work
Prior visual tracking methods largely use Bayesian motion and appearance models or generative and discriminative frameworks. Early deep-learning trackers were constrained by limited training data and showed comparable or inferior performance.
- Conventional tracking: Most existing trackers fit a Bayesian framework that separates motion modeling from appearance modeling.Much work focuses on improving the appearance model because it is crucial to tracking performance.
- Generative trackers: Generative trackers model the target-generation process and search for the most probable candidate tracking result.Representative approaches use principal component analysis, sparse coding, or dictionary learning.
- Discriminative trackers: Discriminative trackers learn to separate foreground from background using classifiers, including boosting, multiple-instance learning, structured output SVM, and Gaussian process regression.The paper presents generative and discriminative approaches as generally complementary.
- Deep learning for tracking: Deep-learning tracking methods using ensembles or pools of CNNs showed comparable or inferior results to state-of-the-art trackers because sufficient training data was unavailable.The paper identifies this as evidence that deep learning’s power had not yet been fully exploited for visual tracking.
3. Our Tracker
SO-DLT transfers generic object features learned offline into online tracking, then adapts them to the target through fine-tuning and two collaboratively updated CNNs. Its structured probability-map output supports localization, scale selection, and robustness to occlusion and clutter.
- 3.1 Overview: Training combines offline objectness pre-training with online fine-tuning and tracking, adapting generic features to the target specified in the first frame.The tracker maintains two CNNs that collaborate when determining each frame's result.
- 3.1 Overview: The structured-output CNN uses seven convolutional layers, three fully connected layers, multi-scale pooling, and a 50 × 50 probability map instead of a single output.Each output pixel corresponds to a local region used for localization.
- 3.2. Objectness Pre-training: ImageNet 2014 detection pre-training uses 478,807 training bounding boxes with randomized padding, scaling, and negative examples to learn generic object features.The pre-trained CNN is evaluated on a held-out detection validation set for object presence and localization.
- 3.3.1 Bounding Box Determination: Online tracking crops candidate regions around the previous estimate, obtains probability maps, searches multiple scales, and reports the target as missing if no scale reaches the threshold.The center is estimated from thresholded probability maps, while scale is selected by a subsequent search in the corresponding region.
- 3.3.1 Bounding Box Determination: A candidate bounding box is scored by aggregating probability-map values inside it, with ϵ balancing bounding-box scale; integral images make confidence calculation efficient.Several ϵ values are averaged for robust estimation.
- 3.3.2 Differentially-paced Fine-tuning: Differentially paced updates use CNNS for aggressive short-term adaptation and CNNL for conservative long-term adaptation, reducing drift from inaccurate updates.CNNS suppresses background objects that trigger the CNN, while CNNL updates only under high confidence.
4. Experiments
The experiments evaluate SO-DLT on a 50-sequence benchmark and a challenging non-rigid tracking dataset, using standardized tracking metrics and comparisons with state-of-the-art trackers. SO-DLT achieves strong overall performance but still exhibits drift in specific conditions.
- CVPR2013 Visual Tracker Benchmark: The CVPR2013 benchmark contains 50 fully annotated sequences covering varied challenging tracking scenarios.
- Evaluation Setting and Metrics: The evaluation uses one-pass error with success plots scored by AUC and precision plots scored by precision at threshold 20.Success rates are computed from overlap thresholds, while precision uses central pixel error.
- CVPR2013 Visual Tracker Benchmark: SO-DLT outperforms TGPR by 13.8% on success plots and 6.9% on precision plots.The comparison averages curves over all 50 test sequences.
- Quantitative Results: At overlap threshold 0.5, SO-DLT outperforms the nearest baseline by 20.6% in average success rate.
- Attribute-Based Analysis: SO-DLT obtains the highest ranking score for all 11 annotated attributes, including illumination variation, rotation, out-of-view, and scale variation.The authors associate these results with CNN representation and describe fine-tuning as helping correct drift after occlusion disappears.
- Non-rigid Object Tracking Dataset: On the non-rigid tracking dataset, SO-DLT outperforms PixelTracker and tracks every sequence to the end except the volleyball sequence.Because targets are highly deformable, evaluation uses central pixel error rather than bounding-box overlap.
- Discussion: The tracker can drift when similar distractors appear during occlusion or when the initial bounding box poorly specifies an irregular target.The discussion proposes combining high-level and low-level feature trackers as a possible remedy.
5. Conclusion
The paper transfers high-level CNN feature hierarchies to visual tracking and reports significant improvement over state-of-the-art trackers. It introduces structured output and objectness pre-training to address limited tracking labels.
- The paper transfers high-level CNN feature hierarchies to visual tracking and reports significant improvement over state-of-the-art trackers.
- It formulates tracking with a structured output CNN rather than proposal classification.
- The CNN is pre-trained on ImageNet detection to learn object localization and then transferred and fine-tuned during online tracking.