Dual Deep Network for Visual Tracking

I. INTRODUCTION

• Tracking remains difficult under appearance variation, abrupt motion, occlusion, and background clutter; hand-crafted low-level features have limited ability to capture target semantics.Deep ConvNets offer learned representations, but tracking often has only one labeled first-frame box, while online learning with more data raises computational cost; prior work also did not use information across layer levels.Source: §I · pp. 1–2
• DNT combines high-level semantic context and low-level spatial structure from two ConvNet layers to represent targets for visual tracking.The authors report in preliminary experiments that simply stacking three layers performs worse than using two layers alone.Source: §I · p. 2
• LoG boundary maps add shape information to hierarchical deep features, and ICA-R refines the target contour for a better observation likelihood model.The resulting feature maps incorporate low-level texture and salient target contours.Source: §I · p. 2
• Center-shifted random patches augment positive training samples, helping the dual network discriminate the foreground target from background clutter.The paper describes the tracker as using a self-supervised learning framework.Source: §I · p. 2
• Stochastic and periodic online updates are presented to address drift and occlusion and reduce noisy updates from inaccurate training samples.The tracking framework also includes a module to detect drift and occlusion.Source: §I · p. 2
• The introduction reports substantial improvement over state-of-the-art methods on two large-scale benchmark datasets and says ablation studies assess each component.It does not provide quantitative results in this passage.Source: §I · p. 2

II. RELATED WORK

• Tracking methods are broadly generative, selecting the most probable candidate region, or discriminative, classifying target against background.Examples include appearance modeling with PCA and sparse coding for generative methods, and classification approaches for discriminative methods.Source: §II · pp. 2–3
• Correlation filters, including variants using HOG or color-attribute features, are described as efficient and effective, with hand-crafted features improving tracking accuracy.The related work also cites optical-flow SVM, multiple-instance learning, and structured-output SVM approaches.Source: §II · p. 3
• Deep tracking approaches include online ConvNet classification, pretrained-CNN SVMs, layer-wise feature combinations, online feature selection, and recurrent networks for long-range context.The related work notes that classifier schemes requiring many positive and negative samples are impractical for online tracking.Source: §II · p. 3
• The paper identifies three gaps in deep trackers: pretrained features may not adapt to a sequence, pooled or strided features lose texture detail, and surrounding background or similar objects are underused.It proposes online sequence-specific adaptation, incorporating boundary and shape information, and using patch translations for online training and model updates.Source: §II · p. 3
• Offline deep feature maps can lose texture through pooling or strided convolution, while boundary and shape information can improve candidate localization.The paper proposes integrating hand-crafted boundary and shape information with deep representations.Source: §II · p. 3
• The paper proposes online adaptation for each sequence because features learned on other datasets may not fit the target appearance, and patch translations for training and model updates.These measures address the stated limitations of pretrained features and insufficient use of surrounding background and similar objects.Source: §II · p. 3

A. Deep Feature Extraction with ICA-R

• The method extracts conv4-3 and conv5-3 features from ImageNet-pretrained 16-layer VGG for hierarchical representations.The network has 13 convolutional and 3 fully connected layers; these features are also fed into the dual network.Source: A. Deep Feature Extraction with ICA-R · p. 3
• A Laplacian of Gaussian filter produces boundary maps that are integrated with VGG and dual-network features to form coarse prior maps for localization.The integrated maps combine deep semantic features with low-level texture information, and boundary-weighted dual-network maps provide more accurate target locations.Source: A. Deep Feature Extraction with ICA-R · p. 3
• ICA-R uses VGG prior maps as references and dual-network maps as mixed signals, applying a fast one-unit algorithm to extract target-relevant maps.The method seeks a projection that maximizes negentropy; a Newton-like learning method iteratively updates the projection vector.Source: A. Deep Feature Extraction with ICA-R · p. 3
• ICA-R compensates for Gaussian labels that may not capture accurate target contours by transforming Gaussian shape toward target shape, using boundary and texture cues.The resulting ICA-R maps are fed into the Bayesian tracking observation model.Source: A. Deep Feature Extraction with ICA-R · p. 4

B. Dual Deep Network

• The adaptation module comprises two same-structure networks with different learned weights, each using three convolutional layers with 128, 64, and 64 kernels sized 7 × 7, 5 × 5, and 3 × 3.The final layer's feature map highlights salient objects; the network is supervised with first-frame ground truth and updated using similarity between the current and last best-tracked target appearances.Source: B. Dual Deep Network · p. 4
• Random patches are center-shifted versions of the base patch that include more background clutter or similar nearby objects, helping distinguish the target, suppress distractors, and augment scarce training data.Patches sampled beyond image boundaries receive zero padding.Source: B. Dual Deep Network · p. 4
• Figure 3 shows heat maps for center-shifted patches and illustrates the dual network's robustness to discriminating the target from background or distractors at different locations.The figure's example uses a 30-pixel coordinate offset.Source: B. Dual Deep Network · p. 4, Figure 3 · p. 4
• The similarity-ranking loss trains the network to distinguish the target from surrounding background and suppress similar-looking distractors, reducing the risk of tracking drift.The paper contrasts its supervised tracking framework with, which uses query, tracked, and random patches for unsupervised image classification.Source: B. Dual Deep Network · p. 4

C. Target location estimation with occlusion and drifts

• The tracker estimates Z_t=(x, y, σ) by maximizing a Bayesian posterior that combines candidate-target likelihood with a Gaussian location-and-scale motion prior.The maximum-scoring candidate is selected across the two streams.Source: C. Target location estimation with… · pp. 4–5
• Candidate confidence sums target heat-map values over canonical-size maps, then weights confidence by candidate area to adapt the observation model to scale variation.Larger candidates are weighted more when confidence is high, and less when their maps contain more zeros.Source: C. Target location estimation with… · p. 5
• Final localization combines confidence from both hierarchical ICA-R maps with regularization and selects the maximum-scoring candidate across the two streams.The two map-specific confidence scores are denoted by k = {1, 2}.Source: C. Target location estimation with… · p. 5

Occlusion and Drifts

• A threshold on normalized deviation of current confidence from its average flags drift or heavy occlusion; high confidence suggests distractors, while a small denominator indicates occlusion.A proper update can help recover from drift; under occlusion, the tracker considers surrounding background when updating the dual network.Source: C. Target location estimation with… · p. 5

Online Update

• The dual network is updated stochastically to distinguish targets from background or similar objects, and periodically to adapt to target appearance and shape variation.The method maintains the best tracked target features from recent frames, based on the observation that feature representations of two best-tracked frames should be similar.Source: D. Online Update · p. 5

Stochastic Update

• During occlusion or drift, stochastic updates use the last best-tracked patch, current patch, and random patches with Gaussian target-map labels to refine target appearance.This self-supervised process also updates the last best-tracked target; its losses select a best target from recent frames, update background appearance only in the current frame, and penalize center-shifted random patches. The estimated location is held when unreliable.Source: D. Online Update · pp. 5–6

Periodic Update

• At fixed intervals, e.g., every 10 frames, periodic updates use Gaussian target-map labels to supervise target appearance using the first and last best-tracked frames.The first frame is included to preserve its rich information in the learned model.Source: D. Online Update · p. 6

• The section covers implementation details and experimental settings for the proposed DNT tracker.Source: §IV · p. 6
• Quantitative results are reported on OTB50, OTB100, and VOT2015.Source: §IV · p. 6
• Qualitative results are also presented on OTB50, OTB100, and VOT2015.Source: §IV · p. 6

A. Implementation Details

• The tracker runs at 5 frames per second on a 3.4GHz CPU and TITAN GPU, with online SGD training for 50 iterations on the first frame.Training uses an initial learning rate of 10^-6, momentum 0.6, and weight decay β = 0.005.Source: A. Implementation Details · p. 6
• Each frame samples 600 target candidates; λ is set to 0.4, θ to 0.45, and K to 10 for stochastic updates.Changing λ from 0.2 to 0.8 altered OTB50 performance by less than 2%; θ was chosen empirically to handle occlusion and drifts, and K selects the best-tracked patch from the last K frames.Source: A. Implementation Details · p. 6
• On OTB50, OURS scores 0.907 precision and 0.664 overlap, below MDNet’s 0.948 and 0.708 but above HCF’s 0.891 and 0.605.The comparisons cover 11 trackers under OPE, SRE, and TRE.Source: Figure 5 · p. 6, A. Implementation Details · p. 6

Data Set and Evaluation Settings

• The OTB50 evaluation uses precision and success scores under OPE, SRE, and TRE, with rankings based on 20-pixel center error and success-plot AUC.SRE perturbs the initial bounding box, while TRE varies the sequence starting frame; both assess robustness beyond OPE.Source: B. Evaluation on OTB Data Set · pp. 6–7

Quantitative Evaluation

• DNT gains 1.8% on precision and 9.75% on overlap over state-of-the-art trackers in OTB50 OPE, with the highest success plots across OPE, SRE, and TRE.In TRE, DNT performs worse than HCF when the target is missing or partly occluded in intermediate frames.Source: B. Evaluation on OTB Data Set · p. 7
• Random-patch updates during occlusion or drift consume most of DNT’s runtime; removing them can increase speed but decreases precision.The runtime comparison covers deep ConvNet-based trackers.Source: B. Evaluation on OTB Data Set · p. 7, Table III · p. 7
• On OTB50 attribute evaluations, DNT performs favorably on most precision attributes but fares worse in some occlusion, motion-blur, and background-clutter cases; success is weaker on low-resolution sequences.The reported explanations are background activations equal to or stronger than the target for some precision exceptions, and weak boundary priors for ICA-R on low-resolution sequences.Source: Table I · p. 7, Table II · p. 7, B. Evaluation on OTB Data Set · p. 8
• On OTB100, DNT estimates location and scale accurately and outperforms the other six selected trackers in almost all attributes.The evaluation covers 100 sequences and compares the seven trackers selected for OTB50 OPE precision above 0.7 and success above 0.5.Source: B. Evaluation on OTB Data Set · p. 8, Figure 6 · p. 8

Qualitative Evaluation

• In qualitative examples, DNT tracks challenging targets accurately through large scale changes and weak activations, using semantic features and boundary guidance to localize them.The examples also report tracking despite fast motion and distractors, with fine-grained spatial details aiding localization when higher-layer semantics are weak.Source: B. Evaluation on OTB Data Set · pp. 8–9

C. Evaluation on VOT2015

• VOT2015 evaluates 60 sequences using accuracy (bounding-box overlap) and robustness (tracking-failure probability), reinitializing after failures.The tracker is reinitialized with the ground-truth location when estimated overlap with ground truth is zero.Source: C. Evaluation on VOT2015 · p. 9
• DNT ranked 3rd in the VOT2015 challenge and performed favorably against other state-of-the-art trackers in the baseline and Overall evaluations.The reported accuracy and robustness plots cover the top 24 trackers.Source: C. Evaluation on VOT2015 · p. 9
• Small targets resembling surrounding distractors caused drifting when the dual network could not extract semantic information; lower CNN layers could track them better.In these cases, fine-grained spatial information was more useful.Source: C. Evaluation on VOT2015 · p. 9
• Long-term occlusions in the soldier and sheep sequences led to failures because the method lacked an additional detection mechanism to learn the appearance model.Source: C. Evaluation on VOT2015 · p. 9

D. Ablation Study

• The full DNT tracker performs much better than variants without boundary information, ICA-R, random-sample updates, or stochastic updates.The study also evaluates boundary guidance and random-sample updates across OTB50 tracking attributes in OPE; no attribute-specific values are provided here.Source: D. Ablation Study · p. 10
• Using conv4-3 and conv5-3 together outperforms a single layer, combining semantic and fine-grained cues while preserving clear target outlines.The feature-combination ablation is reported in Table VII.Source: D. Ablation Study · p. 10, Table VII · p. 10
• Adding conv3-3 yields worse performance than the base tracker because target outlines are difficult to preserve in that layer.For this experiment, λ in (9) is set to [0.2, 0.3, 0.5] for conv3-3, conv4-3, and conv5-3, respectively.Source: D. Ablation Study · pp. 10–11
• Features from higher layers outperform those from lower layers, indicating that semantic appearance plays an important role in visual tracking.Source: D. Ablation Study · p. 11

V. CONCLUSION

• The method combines higher-layer semantic and lower-layer spatial features with boundary information to form a coarse target prior map.The prior guides ICA-R extraction of a target appearance model.Source: §V · p. 11
• A dual network adapts and fine-tunes offline-learned VGG ConvNet features for each target, using a self-supervised scheme for updates.The ConvNet features are learned offline on a large-scale data set.Source: §V · p. 11
• A course-to-fine strategy locates targets, and experiments on OTB and VOT2015 demonstrate accuracy and robustness against state-of-the-art algorithms.Source: §V · p. 11