Source-linked AI summary
SANet: Structure-Aware Network for Visual Tracking
Heng Fan, Haibin Ling
TL;DR
CNN-based trackers can be sensitive to similar distractors because they mainly focus on inter-class classification. SANet models object self-structure with RNNs integrated into CNN features and outperforms other state-of-the-art methods on OTB100, TC-128, and VOT2015.
Problem
Existing CNN-based trackers may misclassify objects against similar distractors because their learned representations mainly focus on inter-class classification.
Method
SANet uses RNNs to encode object self-structure and fuses CNN and RNN features with skip concatenation during learning.
Results
The proposed tracker outperforms other state-of-the-art methods on the OTB100, TC-128, and VOT2015 benchmarks.
Takeaways & Limitations
Encoding object self-structure improves the tracker's ability to distinguish objects from similar distractors and background.
Abstract
from arXiv · showhide
Convolutional neural network (CNN) has drawn increasing interest in visual tracking owing to its powerfulness in feature extraction. Most existing CNN-based trackers treat tracking as a classification problem. However, these trackers are sensitive to similar distractors because their CNN models mainly focus on inter-class classification. To address this problem, we use self-structure information of object to distinguish it from distractors. Specifically, we utilize recurrent neural network (RNN) to model object structure, and incorporate it into CNN to improve its robustness to similar distractors. Considering that convolutional layers in different levels characterize the object from different perspectives, we use multiple RNNs to model object structure in different levels respectively. Extensive experiments on three benchmarks, OTB100, TC-128 and VOT2015, show that the proposed algorithm outperforms other methods. Code is released at http://www.dabi.temple.edu/~hbling/code/SANet/SANet.html.
1. Introduction
SANet addresses the vulnerability of CNN-based visual trackers to similar distractors by modeling an object’s self-structure with RNNs. It further fuses CNN and RNN features through skip concatenation and is evaluated on three tracking benchmarks.
- Introduction: CNN-based trackers can misclassify objects and backgrounds when similar distractors are present because they mainly learn inter-class classification.
- Introduction: SANet uses RNNs to encode an object’s self-structure during learning, improving discrimination of both inter-class background objects and intra-class similar distractors.
- Introduction: SANet adopts skip concatenation to fuse CNN and RNN features, supplying richer information and improving tracking performance.
- Introduction: Experiments on OTB100, TC-128 and VOT2015 show that SANet outperforms other state-of-the-art tracking methods.
2. Related Work
Related work spans discriminative and generative visual tracking, deep networks for learned representations, and RNNs for modeling image structure. SANet addresses intra-class distractors by integrating object self-structure modeled with RNNs into CNN-based tracking.
- Visual tracking: Discriminative trackers [7] [20] classify objects against changing backgrounds, whereas generative trackers model object appearance.Discriminative approaches use foreground and background information with P-N learning, multiple instance learning, or correlation filters [7] [20].
- Visual tracking: Low-level handcrafted features make existing approaches vulnerable to blur, occlusion, deformation, scale changes, and illumination variations in complex environments.Learned high-level features are proposed as a way to improve object-appearance representation.
- Deep networks in tracking: Deep networks improve feature extraction for tracking, but limited training data can prevent CNN-based trackers from learning robust representations.Prior work includes CNN-based human tracking, stacked-autoencoder tracking, and online learning with a CNN pool [30].
- RNNs on image processing: RNNs extend sequential prediction to multidimensional image tasks [18] and model object self-structure by capturing long-range dependencies among image units.Applications include image classification, scene labeling [39], and person re-identification [44].
- RNNs on image processing: Unlike prior tracking work, SANet models the object’s own structure with RNNs, integrates it into CNN classification, and uses it to distinguish intra-class distractors.Prior work models spatial relationships between the object and surrounding background to produce a confidence map for regularizing correlation filters; SANet’s RNNs are integrated with CNNs and trained on video sequences.
3. RNNs for Object Self-Structure Modeling
This section adapts recurrent neural networks to model object self-structure in two-dimensional images. It approximates the image’s undirected cyclic graph with four directed acyclic graphs and defines corresponding forward and backward propagation.
- RNNs for Object Self-Structure Modeling: RNN hidden states store progressively observed inputs, enabling modeling of long-range contextual dependencies among sequence elements.The hidden state combines the current input, previous hidden state, and bias through a nonlinear activation, while outputs use a second activation.
- RNNs for Object Self-Structure Modeling: Image self-structure forms an undirected cyclic graph, so standard sequence RNNs cannot be directly applied to two-dimensional image data.The method addresses the loopy topology by approximating it with several directed acyclic graphs and using variant RNNs to model the target object.
- RNNs for Object Self-Structure Modeling: The proposed graph RNN follows a directed acyclic graph topology, with each vertex’s hidden state depending on its current input and all predecessor vertices.Forward propagation traverses the graph from its start point, while backward propagation processes vertices in reverse order using successor information; errors arise from directed and propagated sources.
- RNNs for Object Self-Structure Modeling: The undirected cyclic graph is decomposed into four directed acyclic graphs oriented southeast, southwest, northwest, and northeast.Each graph has corresponding matrix parameters and bias terms, and its forward and backward passes produce structure-aware representations and errors propagated to the preceding convolutional layer.
4. Proposed Tracking Algorithm
SANet combines convolutional features with recurrent layers that model object structure at multiple levels, using skip concatenation to improve feature representation. Tracking uses particle filtering, adaptive updates, hard-negative mining, and bounding-box refinement.
- Network Architecture: SANet processes 107×107 RGB inputs through three convolutional layers, recurrent layers after each pooling layer, two fully connected layers, and a classification layer.Each recurrent layer models object structure at its corresponding feature level.
- Network Architecture: Skip concatenation fuses pooling and recurrent features to provide subsequent convolutional layers with additional information.
- Tracking and Updating: Particle filtering samples N candidates around the previous target position, evaluates their positive scores p(c_i), and selects the candidate with the highest score.The positive score represents the candidate’s probability of belonging to the target class.
- Tracking and Updating: Short-term updates occur when the tracked result’s positive score p(O) falls below threshold θ, whereas long-term updates otherwise retrain the network using collected positive samples.These updates address appearance changes from lighting variation and deformation, following [36].
- Training: Hard-negative mining, adopted from [36] and object detection, supplies informative distracting negatives to reduce drift caused by redundant samples and insufficient effective negatives.
5. Experiments · 6. Conclusion
SANet uses multi-level recurrent structure modeling within a CNN and outperforms competing trackers across OTB100, TC-128, and VOT2015, while structure information improves accuracy and robustness. The experiments also show that multi-domain learning contributes to tracking performance.
- 5.1. Implementation details: The Matlab implementation runs at around 1 FPS on a 3.7 GHz Intel i7 Core with an NVIDIA GTX TITAN Z GPU and samples 300 candidates per frame.Candidates are sampled in translation and scale from a Gaussian distribution.
- 5.1. Implementation details: SANet models image-unit dependencies with three independent RNNs attached to the first, second, and third pooling layers, using hidden dimensions matching their channel counts.RNN learning rates start at 10^-3 and decay exponentially at rate 0.9; convolutional parameters follow [36].
- 5. Experiments: The benchmark evaluations use precision and success plots on OTB100 and TC-128, while VOT2015 uses accuracy, robustness, and expected average overlap under its re-initialization protocol.OTB100 contains 100 videos, TC-128 contains 128 color sequences, and VOT2015 contains 60 sequences.
- 5.2. Evaluation on OTB: On OTB100, SANet outperforms other state-of-the-art trackers in both precision and success plots, combining few misses at mild thresholds with tight boxes at strict thresholds.The comparison includes deep CNN tracker MDNet [36] and uses the benchmark’s precision and success evaluations.
- 5.2. Evaluation on OTB: Removing multi-domain learning reduces SANet’s OTB100 scores to 0.922 in precision ranking and 0.688 in success ranking, demonstrating the strategy’s effectiveness.All other training and experimental conditions remain the same in this comparison.
- 5.3. Evaluation on TC-128: SANet outperforms state-of-the-art trackers in both precision and success on TC-128 and consistently leads the other three trackers across most attributes.The comparisons use Figure 5 for overall precision and success and Figure 6 for attribute-level analysis.
- 5.4. Evaluation on VOT2015: SANet outperforms other trackers on VOT2015 in accuracy, robustness, and expected average overlap, including improvements over MDNet [36].Table 1 reports the accuracy and robustness comparison, while Figure 7 visualizes expected-overlap ranks.