Source-linked AI summary
SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks
Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, Junjie Yan
TL;DR
Siamese trackers face an accuracy gap and have struggled to benefit from deep networks because strict translation invariance is not preserved. SiamRPN++ addresses this with spatial-aware sampling plus layer-wise and depth-wise correlation designs, achieving state-of-the-art results across large tracking benchmarks while remaining efficient.
Problem
Siamese trackers have an accuracy gap and cannot directly obtain expected gains from deeper networks, with the analysis identifying destroyed strict translation invariance as a core restriction.
Method
The paper uses spatial-aware sampling to break the spatial-invariance restriction and adds layer-wise feature aggregation and depth-wise separable correlation to a ResNet-driven Siamese tracker.
Results
SiamRPN++ obtains state-of-the-art tracking results on VOT2018, LaSOT, TrackingNet, and other large benchmarks while running in real time.
Takeaways & Limitations
The results support deep Siamese tracking with the proposed training strategy and architecture, combining improved tracking performance with efficient inference.
Takeaways & Limitations
The Siamese formulation retains intrinsic restrictions on strict translation invariance and structure symmetry.
Abstract
from arXiv · showhide
Siamese network based trackers formulate tracking as convolutional feature cross-correlation between target template and searching region. However, Siamese trackers still have accuracy gap compared with state-of-the-art algorithms and they cannot take advantage of feature from deep networks, such as ResNet-50 or deeper. In this work we prove the core reason comes from the lack of strict translation invariance. By comprehensive theoretical analysis and experimental validations, we break this restriction through a simple yet effective spatial aware sampling strategy and successfully train a ResNet-driven Siamese tracker with significant performance gain. Moreover, we propose a new model architecture to perform depth-wise and layer-wise aggregations, which not only further improves the accuracy but also reduces the model size. We conduct extensive ablation studies to demonstrate the effectiveness of the proposed tracker, which obtains currently the best results on four large tracking benchmarks, including OTB2015, VOT2018, UAV123, and LaSOT. Our model will be released to facilitate further studies based on this problem.
NLPR, CASIA
The passage identifies Junliang Xing with NLPR, CASIA.
- Junliang Xing is affiliated with NLPR, CASIA.
VIPL, ICT
The passage identifies Junjie Yan with SenseTime Research.
- Junjie Yan is affiliated with SenseTime Research.
1. Introduction
The introduction identifies strict translation invariance as a core restriction limiting deep Siamese tracking, then presents sampling and aggregation designs that enable SiamRPN++ to achieve strong accuracy and efficiency across major benchmarks.
- Motivation: Siamese trackers formulate tracking as cross-correlation between target-template and search-region features, balancing accuracy and efficiency.
- Motivation: A notable accuracy gap remains, and directly training Siamese trackers with deeper architectures such as ResNet has not produced the expected gains.
- Analysis: The analysis attributes the restriction to destroyed strict translation invariance, with padding in deep networks identified as one contributing issue.
- Method: A spatial-aware sampling strategy breaks the spatial-invariance restriction and enables a ResNet-driven Siamese tracker with improved performance.
- Architecture: Layer-wise aggregation combines representations from multiple levels, while depth-wise separable correlation produces multiple semantic similarity maps and reduces parameters.
- Results: 35 FPS and best tracking results on five large benchmarks are reported for SiamRPN++, while a MobileNet variant runs at 70 FPS with competitive performance.
2. Related Work
Related work describes tracking advances, Siamese cross-correlation methods, and the field’s reliance on relatively shallow architectures despite the broader progress of deep networks.
- Siamese Trackers: The section situates Siamese trackers within research emphasizing balanced tracking accuracy and efficiency.
- Tracking Research: New benchmark datasets and improved methodologies have accelerated visual-tracking research and enabled standardized algorithm comparisons.
- Siamese Trackers: Siamese trackers use a Y-shaped network with template and search branches to formulate tracking as cross-correlation.
- Deep Architectures: Modern architectures including ResNet and MobileNet have advanced computer vision tasks such as detection, segmentation, and pose estimation.
- Deep Architectures: Deep visual trackers typically use no more than five convolutional layers tailored from AlexNet or VGGNet because shallow features support accurate localization.
3. Siamese Tracking with Very Deep Networks
The paper identifies broken strict translation invariance as a core obstacle to using very deep networks in Siamese tracking, then addresses it with spatial-aware sampling and a ResNet-based architecture. Layer-wise aggregation and depth-wise correlation further improve feature use while reducing correlation-module cost.
- Analysis of Siamese Networks: Siamese tracking seeks the most similar search-region patch through cross-correlation between target and search features.The target patch is treated as an exemplar, and the network learns a similarity map in an embedding space.
- Analysis of Siamese Networks: Strict translation invariance and structure symmetry are intrinsic constraints of the Siamese matching function.The translation condition preserves efficient training and inference, while symmetry supports similarity learning.
- Analysis of Siamese Networks: Padding in deep networks breaks strict translation invariance, producing spatial bias and preventing straightforward use of architectures such as ResNet.The paper addresses this issue with spatial-aware sampling, while treating RPN feature asymmetry separately.
- Spatial-aware Sampling: ±64 pixels is vital for training a ResNet-50 SiamRPN, whereas zero shift reduces VOT2018 performance to 0.14.Simulation experiments also show that increasing translation ranges alleviates center bias and that 32-shift heatmaps better match test-object location distributions.
- Deep Siamese Architecture: The proposed framework uses modified ResNet features and multiple SiamRPN blocks whose outputs are fused into dense predictions.The architecture reduces later-block strides, uses dilated convolutions, crops the template feature, and jointly fine-tunes the deep network end-to-end.
- Layer-wise and Depth-wise Aggregation: Depth-wise cross-correlation has 10 times fewer parameters than UP-XCorr while maintaining comparable performance and producing semantically interpretable channels.Layer-wise aggregation combines representations from multiple levels, while depth-wise outputs associate channels with categories such as cars, persons, and faces.
4. Experimental Results
Experiments show that SiamRPN++ benefits from deeper backbones, layer-wise aggregation, and depthwise correlation, achieving strong results across multiple tracking benchmarks while retaining real-time operation.
- Layer-wise Feature Aggregation: 0.414 EAO on VOT2018 is achieved by aggregating conv3, conv4, and conv5 features, 4.0% above the single-layer baseline.The combined aggregation improves both accuracy and robustness, with gains between 3.1% and 1.3% on VOT and OTB.
- LaSOT and UAV123 Datasets: 49.6% AUC is achieved on LaSOT, with relative improvements of 23.7% in normalized distance precision and 24.9% in AUC over MDNet.On UAV123, SiamRPN++ obtains a success score of 0.613, compared with 0.586 for DaSiamRPN and 0.525 for ECO.
- TrackingNet Dataset: 73.3% AUC, 69.4% precision, and 80.0% normalized precision are achieved on TrackingNet, exceeding DaSiamRPN by 9.5%, 10.3%, and 6.6%.The reported results are best on all three metrics and are presented as evidence of generalization ability.
5. Conclusions
SiamRPN++ is a unified end-to-end deep Siamese tracker that combines multi-layer aggregation with depthwise correlation. It achieves state-of-the-art results in real-time on VOT2018 and on large datasets including LaSOT and TrackingNet.
- Contributions: SiamRPN++ combines multi-layer representation aggregation with depthwise correlation to end-to-end train a deep Siamese tracker.The architecture reduces computation cost and redundant parameters while improving convergence.
- Results: SiamRPN++ achieves state-of-the-art results on VOT2018 in real-time and on LaSOT and TrackingNet, demonstrating reported generalizability across large tracking datasets.The conclusion attributes these results to the effectiveness of the unified framework.