Source-linked AI summary
Online Tracking by Learning Discriminative Saliency Map with Convolutional Neural Network
Seunghoon Hong, Tackgeun You, Suha Kwak, Bohyung Han
TL;DR
Visual tracking must handle challenging appearance and motion changes while retaining precise localization. The paper combines a pre-trained CNN, online SVM, target-specific saliency maps, and generative Bayesian tracking, reporting substantial gains over state-of-the-art trackers and target-segmentation capability.
Problem
Visual tracking remains difficult under occlusion, pose variation, illumination change, fast motion, and background clutter, motivating robust target appearance modeling.
Method
The method uses pre-trained CNN features, an online SVM, back-propagated target-specific saliency maps, and a generative appearance model within sequential Bayesian filtering.
Results
The proposed algorithm achieves substantial performance gains over existing state-of-the-art trackers and shows capability for target segmentation.
Takeaways & Limitations
Target-specific saliency maps improve localization while enabling pixel-level target segmentation.
Takeaways & Limitations
Segmentation evaluation used a subset of typically short sequences because accurate annotation is labor intensive and time consuming.
Abstract
from arXiv · showhide
We propose an online visual tracking algorithm by learning discriminative saliency map using Convolutional Neural Network (CNN). Given a CNN pre-trained on a large-scale image repository in offline, our algorithm takes outputs from hidden layers of the network as feature descriptors since they show excellent representation performance in various general visual recognition problems. The features are used to learn discriminative target appearance models using an online Support Vector Machine (SVM). In addition, we construct target-specific saliency map by backpropagating CNN features with guidance of the SVM, and obtain the final tracking result in each frame based on the appearance model generatively constructed with the saliency map. Since the saliency map visualizes spatial configuration of target effectively, it improves target localization accuracy and enable us to achieve pixel-level target segmentation. We verify the effectiveness of our tracking algorithm through extensive experiment on a challenging benchmark, where our method illustrates outstanding performance compared to the state-of-the-art tracking algorithms.
1. Introduction
The paper addresses difficult visual tracking by combining pre-trained CNN representations with online discriminative learning and target-specific saliency maps. The method aims to improve localization while supporting pixel-level target segmentation.
- Visual tracking remains difficult under occlusion, pose variation, illumination change, fast motion, and background clutter.
- CNNs offer strong visual representations, but their use in tracking is limited by unsuitable offline classifiers, difficult online training, and weak spatial localization in top-layer features.
- The proposed tracker uses a large-scale-image-classification CNN with an online SVM to learn target appearance discriminatively against background.
- Back-propagating SVM-relevant CNN features constructs a target-specific saliency map that highlights regions discriminating target from background.
- The saliency-map strategy improves target localization despite the shift-invariant properties of CNN-based features and enables pixel-level target segmentation.
2. Overview of Our Algorithm
The algorithm combines pre-trained CNN features and an online SVM to generate target-specific saliency maps, then tracks objects using a saliency-based generative model and Bayesian filtering.
- A pre-trained CNN extracts feature descriptors that capture target semantics but may lose spatial information through pooling.
- Aggregated saliency maps preserve spatial target information for tracking, including meaningful shape and part configuration.
- An online SVM separates target from background and guides back-propagation of target-relevant features to produce saliency maps for positive samples.
- A generative appearance model accumulates target observations from saliency maps, and sequential Bayesian filtering uses convolution-based likelihoods for localization.
- The framework combines discriminative online SVM learning with generative appearance modeling to use benefits from both approaches.
3. Proposed Algorithm
The algorithm combines pre-trained CNN features, an online SVM, and target-specific saliency maps to localize and track targets. Saliency maps preserve discriminative spatial information and support Bayesian localization with a generative appearance model.
- CNN Features and Online SVM: A pre-trained CNN extracts feature descriptors from target proposals, while an online SVM classifies proposals as target or background.The method uses CNN representations and discriminative online learning rather than training the CNN online.
- Target-Specific Saliency Map Estimation: The saliency map addresses localization errors caused by spatial pooling, which can discard target spatial configuration in CNN features.The resulting map reveals meaningful target structure such as shape and parts.
- Target-Specific Saliency Map Estimation: Target-specific saliency maps back-propagate SVM-relevant CNN information to highlight discriminative target regions within each image.Only feature dimensions associated with positive SVM weights are selected, and their gradients with respect to the image form the target-specific signal.
- Target-Specific Saliency Map Estimation: Positive proposals are projected into image space and aggregated by pixelwise maximum gradient magnitude to construct the target-specific saliency map.Considering only positive examples suppresses erroneous background activations; strong activations typically expose target areas and spatial layouts.
- Target Localization with Saliency Map: Sequential Bayesian filtering uses the saliency map as observation, with a prior predicted from the previous state and a likelihood computed from target appearance similarity.The state transition uses estimated target displacement and Gaussian noise based on positive samples classified by the online SVM.
- Target Localization with Saliency Map: A generative appearance model averages recent target filters, and convolution between this model and the current saliency map produces the observation likelihood for localization.Each target filter is extracted from a saliency map near the optimal target location, and recent filters are retained for model updates.
4. Experiments
Experiments evaluate the tracker on benchmark sequences using quantitative, qualitative, and segmentation analyses. The method outperforms comparison trackers, while its saliency-map-based generative model improves localization over the SVM-only variant.
- Experimental setting: Experiments follow established protocols on 50 benchmark sequences, measuring success rate with AUC and precision with center-location error at 20 pixels.The implementation uses R-CNN features, 120 candidate samples per frame, and specified SVM, generative-model, and GrabCut settings.
- Bounding-box tracking: The proposed tracker outperforms all compared trackers with substantial margins in both success-rate and precision evaluations.Results are summarized in the average success and precision plots over all 50 sequences.
- Bounding-box tracking: The full algorithm achieves non-trivial performance improvement over the SVM-only variant, indicating that the saliency-map-based generative model helps localize the target.Attribute-wise evaluations also report that the full algorithm is generally better than its reduced version.
- Attribute analysis: Attribute-wise results show consistent superiority over other methods in almost all annotated tracking challenges.The tables report average success-rate and precision scores for individual attributes.
- Segmentation: Segmentation evaluation uses nine selected sequences with ground-truth annotations, and the proposed method outperforms other trackers with a substantial margin.The segmentation comparison includes the full method, bounding-box variants, and trackers whose boxes are treated as segmentation masks.
- Qualitative results: Qualitative comparisons show superior performance alongside target-specific saliency maps and GrabCut-based segmentation results across selected sequences.Figure 5 presents tracking comparisons, saliency maps, and segmentation for seven sequences.
5. Conclusion
The proposed tracker combines pre-trained CNN features with online discriminative learning and a target-specific saliency map for tracking and target segmentation.
- Pre-trained CNN convolutional features serve as generic object descriptors, while an online SVM learns discriminative appearance models.
- Back-propagation from the CNN and learned appearance model produces a saliency map that highlights discriminative target regions spatially.
- Sequential Bayesian filtering uses the target-specific saliency map as its observation, and the algorithm achieves substantial gains over state-of-the-art trackers while supporting target segmentation.