Source-linked AI summary
Siamese Cascaded Region Proposal Networks for Real-Time Visual Tracking
Heng Fan, Haibin Ling
TL;DR
One-stage Siamese-RPN trackers can degrade with similar distractors and large scale variation, motivating a more robust tracking design. C-RPN cascades RPNs across Siamese-network layers, uses hard-negative sampling, feature transfer, and progressive regression, and achieves state-of-the-art real-time results across six benchmarks.
Problem
One-stage Siamese-RPN tracking is limited by similar semantic distractors, underused low-level spatial features, and inaccurate single-step localization under changing target scales.
Method
C-RPN cascades RPNs from high-level to low-level Siamese features, trains successive stages on hard examples, transfers features across layers, and progressively refines anchors through multiple regressions.
Results
C-RPN consistently achieves state-of-the-art results and runs in real time across OTB-2013, OTB-2015, VOT-2016, VOT-2017, LaSOT, and TrackingNet.
Takeaways & Limitations
C-RPN provides a real-time tracker with more robust distractor discrimination and more accurate localization across the evaluated benchmarks.
Takeaways & Limitations
The Siamese network layer notation is defined in inverse order, with conv-N through conv-1 corresponding to low-level through high-level layers.
Abstract
from arXiv · showhide
Region proposal networks (RPN) have been recently combined with the Siamese network for tracking, and shown excellent accuracy with high efficiency. Nevertheless, previously proposed one-stage Siamese-RPN trackers degenerate in presence of similar distractors and large scale variation. Addressing these issues, we propose a multi-stage tracking framework, Siamese Cascaded RPN (C-RPN), which consists of a sequence of RPNs cascaded from deep high-level to shallow low-level layers in a Siamese network. Compared to previous solutions, C-RPN has several advantages: (1) Each RPN is trained using the outputs of RPN in the previous stage. Such process stimulates hard negative sampling, resulting in more balanced training samples. Consequently, the RPNs are sequentially more discriminative in distinguishing difficult background (i.e., similar distractors). (2) Multi-level features are fully leveraged through a novel feature transfer block (FTB) for each RPN, further improving the discriminability of C-RPN using both high-level semantic and low-level spatial information. (3) With multiple steps of regressions, C-RPN progressively refines the location and shape of the target in each RPN with adjusted anchor boxes in the previous stage, which makes localization more accurate. C-RPN is trained end-to-end with the multi-task loss function. In inference, C-RPN is deployed as it is, without any temporal adaption, for real-time tracking. In extensive experiments on OTB-2013, OTB-2015, VOT-2016, VOT-2017, LaSOT and TrackingNet, C-RPN consistently achieves state-of-the-art results and runs in real-time.
1. Introduction
One-stage Siamese-RPN tracking is efficient but can fail with similar distractors, underused low-level features, and large scale changes. C-RPN addresses these issues with cascaded hard-negative sampling, cross-level feature transfer, and progressive multi-step localization, achieving state-of-the-art real-time results across six benchmarks.
- Problem and Motivation: One-stage Siamese-RPN may drift to similar semantic distractors because its training samples are imbalanced and dominated by easy negatives.The resulting classifier degrades when encountering difficult similar background.
- Problem and Motivation: Siamese-RPN uses only final-layer features, whose high-level semantics may insufficiently distinguish targets from same-category or semantically similar background.C-RPN instead targets both semantic and spatial information across network layers.
- Problem and Motivation: Single-step regression from coarse predefined anchors is insufficient for accurate localization when the target scale is unknown or changes substantially.C-RPN progressively adjusts anchor locations and sizes through multiple regression stages.
- Contribution: C-RPN cascades RPNs from high-level to low-level layers, filtering easy negatives so successive stages train on harder, more balanced examples.This sequential process is intended to make later classifiers more discriminative against difficult distractors.
- Contribution: The feature transfer block fuses high-level features into low-level RPNs, while multi-regression progressively refines anchor boxes for more discriminative tracking and accurate localization.These mechanisms jointly exploit semantic and spatial information across layers.
- Contribution: C-RPN consistently achieves state-of-the-art results and runs in real time across OTB-2013, OTB-2015, VOT-2016, VOT-2017, LaSOT, and TrackingNet.The paper evaluates the tracker on six popular benchmarks.
2. Related Work
Related tracking work includes deep CNN and Siamese approaches, while C-RPN combines multi-level features with cascaded RPN stages rather than using independent models or a single-stage design.
- Deep tracking: Deep convolutional networks have been introduced into visual tracking and have demonstrated strong performance.The related work surveys deep tracking methods based on learned feature representations.
- Siamese tracking: Siamese trackers learn or use similarity functions to match a target template against candidate regions, often emphasizing efficient inference without online model updates.SiamFC is cited as an efficient fully convolutional Siamese tracker running at 80 fps.
- C-RPN positioning: C-RPN differs from prior multi-level-feature trackers by fusing features across layers within a feature transfer block for each RPN.Earlier approaches cited here use multi-level features separately in independent models.
- C-RPN positioning: C-RPN also differs from one-stage Siamese-RPN by cascading RPNs for hard-negative sampling and progressively refining anchor boxes through multi-regression.These design choices address data imbalance and target localization within the tracking pipeline.
3. Siamese Cascaded RPN (C-RPN)
C-RPN cascades RPNs across Siamese-network layers, combining hard-negative filtering, multi-level feature fusion, and progressive anchor refinement for robust classification and localization.
- Architecture: C-RPN extracts template and search-region features with a Siamese network, then supplies them to a sequence of cascaded RPNs.Each RPN performs anchor classification and localization.
- Multi-level features: C-RPN uses features from multiple Siamese-network levels rather than only the final layer to improve robustness.The approach leverages high-level semantic and low-level spatial information.
- Cascaded RPN: Easy negative anchors are filtered between stages, so later RPNs train on harder examples and become more discriminative against difficult distractors.The remaining anchors form the next stage's training set.
- Progressive localization: C-RPN progressively adjusts anchor locations and sizes using preceding regression outputs, providing better initialization for subsequent regressors.This replaces fixed anchors with stage-dependent refinement for more accurate localization.
- Training: C-RPN is trained end-to-end with a multi-task loss combining classification and regression losses across its RPN stages.Anchor ratios are set to [0.33, 0.5, 1, 2, 3] during training.
- Feature transfer block: The feature transfer block fuses high-level features into each lower-level RPN through dimension matching, summation, ReLU, and interpolation.The resulting maps share a common resolution for classification and regression.
4. Experiments
Experiments across six tracking benchmarks show that C-RPN consistently achieves strong accuracy and real-time efficiency, while ablations support the contributions of cascading, negative-anchor filtering, and feature transfer.
- OTB-2013 and OTB-2015: C-RPN achieves the best OTB-2013 and OTB-2015 precision scores, reaching 0.675 and 0.663 versus SiamRPN's 0.658 and 0.637.These correspond to improvements of 1.9% and 2.6% over the one-stage baseline.
- VOT-2016 and VOT-2017: C-RPN significantly outperforms SiamRPN and other trackers on VOT-2016, leading in both accuracy and robustness while remaining efficient.VOT-2016 evaluates performance using EAO, which combines accuracy and robustness, and EFO for normalized speed.
- VOT-2016 and VOT-2017: C-RPN obtains 0.289 EAO on VOT-2017 versus SiamRPN's 0.243 and achieves the best real-time EAO of 0.273.It also shows competitive performance against LSART and CFWCR.
- LaSOT: On LaSOT, C-RPN leads both protocols with SUC scores of 0.459 and 0.455, while running at around 23 fps.It outperforms MDNet and SiamFC on success and is substantially faster than MDNet.
- TrackingNet: On TrackingNet, C-RPN leads PRE, NPRE, and SUC with scores of 0.619, 0.746, and 0.669, running at around 32 fps.The reported scores exceed those of the second-best MDNet on all three metrics.
- Ablation Experiment: Ablations show that the second stage provides larger gains than the third, while negative-anchor filtering and FTB further improve benchmark performance.Adding stages can improve results but reduces speed from 48 to 23 fps; filtering and FTB improve LaSOT SUC and VOT-2017 EAO.
5. Conclusion
The paper concludes that C-RPN improves visual tracking through cascaded hard-negative sampling, cross-layer feature transfer, and progressive box refinement. Across six benchmarks, it reports state-of-the-art results while operating in real time.
- Conclusion: C-RPN uses a cascade architecture to perform hard-negative sampling and handle complex background such as similar distractors more robustly.Its stages are cascaded from high-level to low-level Siamese-network layers.
- Conclusion: The feature transfer block fuses high-level semantic and low-level spatial features to improve representation discriminability.
- Conclusion: Multiple regressions progressively refine the target bounding box, leading to more accurate localization.
- Conclusion: Across six popular benchmarks, C-RPN consistently achieves state-of-the-art results and runs in real time.