Source-linked AI summary
Distractor-aware Siamese Networks for Visual Object Tracking
Zheng Zhu, Qiang Wang, Bo Li, Wei Wu, Junjie Yan, Weiming Hu
TL;DR
Siamese trackers can miss semantic distractors because imbalanced training data yields less discriminative features, motivating a more adaptive approach. DaSiamRPN addresses this through distractor-aware offline learning, incremental inference adaptation, and local-to-global search for long-term tracking, with reported gains across short- and long-term benchmarks.
Problem
Siamese trackers mainly discriminate foreground from non-semantic background, leaving semantic distractors insufficiently represented and limiting robustness in cluttered or changing scenes.
Method
DaSiamRPN uses semantic negative pairs and distractor-aware feature learning offline, incremental domain adaptation during inference, and local-to-global search for long-term tracking.
Results
DaSiamRPN outperforms state-of-the-art trackers on short- and long-term benchmarks, including a 9.6% relative gain in VOT2016 and 35.9% in UAV20L.
Takeaways & Limitations
The proposed framework achieves state-of-the-art tracking accuracy while operating at 160 FPS on short-term datasets and 110 FPS on long-term datasets.
Takeaways & Limitations
The conclusion states that the overall system speed is still far from being real-time.
Abstract
from arXiv · showhide
Recently, Siamese networks have drawn great attention in visual tracking community because of their balanced accuracy and speed. However, features used in most Siamese tracking approaches can only discriminate foreground from the non-semantic backgrounds. The semantic backgrounds are always considered as distractors, which hinders the robustness of Siamese trackers. In this paper, we focus on learning distractor-aware Siamese networks for accurate and long-term tracking. To this end, features used in traditional Siamese trackers are analyzed at first. We observe that the imbalanced distribution of training data makes the learned features less discriminative. During the off-line training phase, an effective sampling strategy is introduced to control this distribution and make the model focus on the semantic distractors. During inference, a novel distractor-aware module is designed to perform incremental learning, which can effectively transfer the general embedding to the current video domain. In addition, we extend the proposed approach for long-term tracking by introducing a simple yet effective local-to-global search region strategy. Extensive experiments on benchmarks show that our approach significantly outperforms the state-of-the-arts, yielding 9.6% relative gain in VOT2016 dataset and 35.9% relative gain in UAV20L dataset. The proposed tracker can perform at 160 FPS on short-term benchmarks and 110 FPS on long-term benchmarks.
1 Introduction
The paper identifies imbalanced training data and limited online adaptation as key weaknesses in Siamese trackers, then proposes DaSiamRPN for distractor-aware short- and long-term tracking.
- 1 Introduction: Siamese trackers balance accuracy and speed but struggle with semantic distractors, online appearance changes, and out-of-view or fully occluded targets.Their features mainly separate foreground from non-semantic background, while many methods use fixed models and local search.
- 1 Introduction: Imbalanced training data makes learned representations less discriminative because easy non-semantic backgrounds overwhelm semantic negative pairs.The paper explicitly analyzes this imbalance as an obstacle to feature learning.
- 1 Introduction: 9.6% relative gain in VOT2016 Expected Average Overlap and 35.9% relative gain in UAV20L demonstrate strong short- and long-term benchmark performance.The tracker runs at 160 FPS on short-term datasets and 110 FPS on long-term datasets.
- 1 Introduction: DaSiamRPN learns distractor-aware features offline and explicitly suppresses distractors during online inference.The framework combines semantic negative-pair generation and a distractor-aware module that transfers general embeddings to the current video domain.
- 1 Introduction: A local-to-global search strategy extends DaSiamRPN to long-term tracking and improves handling of out-of-view and full-occlusion challenges.The strategy complements distractor-aware feature learning for long-term scenarios.
2 Related Work
Related work covers Siamese similarity trackers, learned visual features, and long-term tracking frameworks that combine local tracking with detection or verification.
- Siamese Networks based Tracking: Siamese trackers locate targets through similarity comparison, progressing from fixed deep similarity functions to fully convolutional and attention-based variants.The cited progression includes SINT, SiamFC, and RASNet.
- Features for Tracking: Visual-feature research spans distractor suppression, unsupervised autoencoding, temporal constraints, CNN classifiers, and unified feature-learning and tracking frameworks.These approaches differ in feature source and how tightly representation learning is coupled to tracking.
- Long-term Tracking: Traditional long-term trackers either match local keypoints with geometric models or combine a short-term tracker with a detector.Examples include TLD, MUSTer, and a DSST-plus-CNN detector framework.
- Siamese Networks based Tracking: Figure 1 compares response heatmaps from SiamFC, SiamRPN, SiamRPN+ trained with distractors, and DaSiamRPN.The figure uses search images in panel (a) and tracker heatmaps in panels (b-e).
3 Distractor-aware Siamese Networks
DaSiamRPN addresses weak discrimination against semantic distractors by balancing training pairs, incorporating contextual distractors during inference, and extending search for long-term tracking.
- Siamese trackers learn strong foreground–background discrimination but can confuse semantic distractors and omit useful background information during nearest-neighbor matching.
- Distractor-aware Training: Semantic negative pairs from the same and different categories make representations more fine-grained and reduce drift toward arbitrary objects during out-of-view or occlusion challenges.
- Distractor-aware Incremental Learning: DaSiamRPN generates contextual distractors with NMS and reranks top-k proposals using a distractor-aware objective that integrates hard negatives into similarity matching.
- Distractor-aware Incremental Learning: Linear cross-correlation enables distractor-aware reranking at comparable speed to SiamRPN, while incremental learning updates target and distractor templates.
- DaSiamRPN for Long-term Tracking: For long-term tracking, DaSiamRPN switches from short-term tracking to iterative local-to-global search when failures occur, using detection scores to identify tracking quality.
4 Experiments
Experiments evaluate DaSiamRPN across short-term and long-term benchmarks, showing strong accuracy, robustness, and real-time speed. Ablations and attribute analyses associate gains with distractor-aware learning and the long-term search strategy.
- VOT2016 and VOT2017: DaSiamRPN achieves an EAO of 0.411 on VOT2016, outperforming state-of-the-art methods by relative 9.6% at 160 FPS.The VOT2016 comparison includes 70 other trackers; ECO scores 0.375 and the SiamRPN baseline scores 0.3441.
- VOT2016 and VOT2017: DaSiamRPN ranks first on VOT2017 with an EAO of 0.326 and a real-time EAO of 0.326, exceeding CSRDCF++ by relative 53.8%.The VOT2017 evaluation compares against 51 state-of-the-art trackers.
- UAV benchmarks: DaSiamRPN outperforms other trackers on UAV20L, obtaining an AUC of 0.617 and a precision score of 0.838.The reported relative gains include 35.9% and 41.8% for success comparisons, and 34.3% and 35.8% for precision comparisons.
- Attribute analysis: DaSiamRPN performs strongly under full occlusion and background clutter, outperforming SiamRPN by relative 153.1% and 393.2%, respectively.The attribute analysis also covers out-of-view and partial occlusion.
- UAV benchmarks: On UAV123, DaSiamRPN achieves a success score of 0.586, compared with 0.527 for SiamRPN and 0.525 for ECO.The method outperforms the other compared trackers in both success and precision scores.
- Ablation analysis: Ablations show that detection data, negative pairs, distractor-aware learning, and long-term tracking each improve benchmark performance.On UAV20L, the long-term module raises AUC from 49.8% to 61.7%; on VOT2016, detection data raises EAO from 0.344 to 0.368.
5 Conclusions
The paper proposes a distractor-aware Siamese framework for accurate and long-term visual tracking. It combines offline feature learning, inference-time embedding adaptation, and a local-to-global search strategy, while reporting state-of-the-art accuracy but a stated speed limitation.
- The framework uses distractor-aware feature learning during offline training to increase the networks’ discriminative power.
- During inference, a distractor-aware module transfers the general embedding to the current video domain.
- A local-to-global search strategy extends DaSiamRPN to long-term tracking.
- The tracker achieves state-of-the-art accuracy on comprehensive short-term and long-term benchmarks.
- The conclusion states that the overall system speed is still far from being real-time.
6 Supplementary Material
Supplementary experiments visualize DaSiamRPN’s discriminative response maps, benchmark performance across tracking attributes, qualitative robustness, and runtime efficiency.
- DaSiamRPN generates more discriminative response maps than SiamFC, SiamRPN, and SiamRPN+ across different videos.
- DaSiamRPN achieves leading performance across all 12 UAV20L attributes, ranked by area under curve of the success plot.
- The supplementary results report success plots across 12 UAV123 attributes and all 11 OTB2013 and OTB2015 challenge attributes.
- 160 FPS is reported on Titan X, with convolution operations forming the computational bottleneck while remaining more efficient than traditional deep learning trackers.
- DaSiamRPN handles challenging sample videos while SiamRPN and SiamFC tend to drift to distractors, and PTAV fails in the second and last videos.