Source-linked AI summary
Deep Attentive Tracking via Reciprocative Learning
Shi Pu, Yibing Song, Chao Ma, Honggang Zhang, Ming-Hsuan Yang
TL;DR
Visual tracking needs classifiers that remain focused on robust target features despite appearance changes, while existing methods mainly generate frame-specific feature weights with extra attention modules. The paper trains classifiers directly with gradient-derived attention maps used as regularization, and reports favorable benchmark performance, including larger gains than attentive features in its ablation.
Problem
Tracking objects through large appearance changes requires attention that remains robust over time, but existing approaches mainly use additional modules to generate feature weights.
Method
Reciprocative learning computes attention maps from forward classification and backward input derivatives, then couples them with classification loss to train the deep classifier.
Results
The tracker performs favorably against state-of-the-art methods across benchmark datasets and gains 3.2% in DP and 3.3% in OS over baseline with reciprocative learning.
Takeaways & Limitations
The classifier itself learns to attend to target regions robust to appearance changes, while attention maps are used during training rather than testing.
Abstract
from arXiv · showhide
Visual attention, derived from cognitive neuroscience, facilitates human perception on the most pertinent subset of the sensory data. Recently, significant efforts have been made to exploit attention schemes to advance computer vision systems. For visual tracking, it is often challenging to track target objects undergoing large appearance changes. Attention maps facilitate visual tracking by selectively paying attention to temporal robust features. Existing tracking-by-detection approaches mainly use additional attention modules to generate feature weights as the classifiers are not equipped with such mechanisms. In this paper, we propose a reciprocative learning algorithm to exploit visual attention for training deep classifiers. The proposed algorithm consists of feed-forward and backward operations to generate attention maps, which serve as regularization terms coupled with the original classification loss function for training. The deep classifier learns to attend to the regions of target objects robust to appearance changes. Extensive experiments on large-scale benchmark datasets show that the proposed attentive tracking method performs favorably against the state-of-the-art approaches.
1 Introduction
Visual attention can help tracking-by-detection classifiers focus on robust target features, but existing attention modules learn frame-specific feature weights that may not remain reliable over time. The paper proposes reciprocative learning, which trains the classifier itself with attention-derived regularization.
- Motivation: Existing attentive trackers mainly add modules that generate feature weights for selecting discriminative features.These weights are learned from single frames and may not support robust attention across long temporal spans.
- Motivation: Single-frame feature weights may fail under large appearance changes, while slight weight inaccuracies can worsen misclassification.The paper therefore investigates how deep classifiers can attend to target objects over time.
- Proposed approach: Reciprocative learning directly trains an attentive classifier through forward classification and backward derivatives with respect to the input.The backward step computes partial derivatives without updating network parameters.
- Proposed approach: Attention maps are coupled with classification loss as regularization terms so deep classifiers learn to attend to temporally robust target features.This differs from approaches that only use separate attention modules to weigh features.
- Evaluation: Experiments on benchmark datasets report favorable performance against state-of-the-art trackers.The paper presents this as a contribution of the proposed tracking framework.
2 Related Work
Visual attention has been used across computer vision, including tracking, where spatial weighting in correlation filters can suppress boundary effects. Related deep tracking methods emphasize attentive features with additional modules, whereas this paper uses network-output derivatives as training regularizers.
- Visual attention: Visual attention originates in cognitive neuroscience and focuses perception on the most pertinent subset of sensory data.It has been applied to image classification, image captioning, pose estimation, and visual tracking.
- Visual attention in tracking: Correlation-filter trackers use cosine or Gaussian spatial weights to suppress Fourier-transform boundary effects, which can be interpreted as attention maps.Weights closer to target centers receive greater emphasis for localization.
- Deep attentive tracking: Deep attentive trackers generally generate feature weights with additional attention modules and train on the resulting attentive features.This implements attention through feature selection within end-to-end models.
3 Proposed Method
The method derives sample-specific attention maps from classifier-score gradients and incorporates them into the training loss. Positive and negative attention constraints encourage target-focused, discriminative classifiers, while attention maps are computed during training rather than testing.
- 3.1 Attention Exploitation: For each input sample, the network produces class scores, and the derivative of a class score with respect to the input defines an attention map.The map indicates how input pixels affect the class score.
- 3.1 Attention Exploitation: The attention map is sample-specific because it is computed from the network output derivative at that input image.This follows from the Taylor-based interpretation of the score function.
- 3.1 Attention Exploitation: Forward propagation obtains the class score, while backward propagation computes its derivative with respect to the input using the chain rule.The backward operation computes attention without requiring a separate attention module.
- 3.2 Attention Regularization: The binary tracker forms positive and negative attention maps, encouraging target pixels to receive high positive attention and low negative attention for positive samples.The corresponding negative-sample regularization reverses the class-specific objective.
- 3.2 Attention Regularization: The training loss adds attention regularization to cross-entropy, with λ balancing the two components.The regularization controls attention-map means and variances for positive and negative examples.
- 3.3 Reciprocative Learning: During training, attention maps are recomputed for every sample and progressively encourage coverage of the target rather than only a few discriminative regions.The accompanying visualization reports broader target coverage with reciprocative learning.
4 Tracking Process
The tracker initializes, detects, and updates online using sampled candidates and IoU-based labels. Reciprocative learning progressively focuses attention on the whole target, improving confidence and resistance to visually similar occlusions.
- Tracking Process: The proposed tracker requires no offline training and operates through model initialization, online detection, and model update.These components define the online tracking process.
- Model initialization: During initialization, N1 samples around the initial target are labeled by whether their IoU with ground truth exceeds 0.5, then used for H1 update iterations.
- Online detection: For online detection, the tracker samples N2 candidates near the previous prediction, selects the highest-scoring candidate, and refines its location with bounding-box regression.
- Model update: For model updates, N2 samples near the predicted target are labeled by IoU with the predicted box and used to update the fully connected layers every T frames for H2 iterations.
- Reciprocative learning: As tracking proceeds, reciprocative learning expands attention over the target and helps distinguish it from similar occluders, whereas the baseline drifts during occlusion.The comparison is illustrated on the Girl sequence, including Frames #432 and #467.
5 Experiments
Experiments evaluate the method through implementation details, ablations, and benchmark comparisons. Reciprocative learning improves the baseline across parameter settings and outperforms feature-weighting attention in the reported OTB ablation, while performing favorably against state-of-the-art trackers across OTB and VOT datasets.
- Experimental setup: The experiments use a baseline with fixed convolutional feature layers and incrementally updated fully connected classifier layers, following standard OTB and VOT evaluation protocols.OTB reports DP and OS under one-pass evaluation, while additional measures include CLE and OS0.5.
- Ablation studies: Reciprocative learning consistently improves the baseline across λ values from 0 to 8, with top performance generally occurring for λ between 3 and 5.The reported experiments fix λ = 5 afterward.
- Ablation studies: Reciprocative learning improves the baseline more than attentive features, with gains of 3.2% versus 2.1% in DP and 3.3% versus 1.5% in OS.The comparison is on OTB-2013 and contrasts attentive classifiers with an additional attention module generating feature weights.
- Overall performance: On OTB-2013, the method performs well against state-of-the-art trackers and improves substantially over the MDNet baseline in distance precision and overlap success.The evaluation compares 57 trackers and also reports CLE and OS0.5.
- Overall performance: On OTB-2015, the tracker has lower center location error than CCOT on all benchmark sequences but lower overlap success than CCOT.The authors attribute the overlap-success difference to sparse sampling for scale estimation.
- Overall performance: On VOT-2016, the tracker performs comparably with CCOT under EAO and better than the other compared trackers, with all compared trackers exceeding the reported state-of-the-art threshold.
6 Concluding Remarks
The proposed reciprocative learning scheme trains tracking classifiers using attention maps derived during forward and backward propagation. The resulting classifier attends to temporally robust features and directly classifies samples at test time.
- The method computes classification loss during forward propagation and partial derivatives with respect to each input sample during backward propagation to form attention maps.These attention maps are used during classifier training rather than generated during inference.
- Attention maps serve as a regularization term coupled with the original classification loss to train discriminative classifiers.This distinguishes the method from approaches that add separate modules to generate feature weights.
- The trained classifier attends to robust features over a long temporal span and directly classifies each input sample during testing.No attention maps are generated in the test stage.