PaperlayerView exact revision on arXiv

Source-linked AI summary

Deeper and Wider Siamese Networks for Real-Time Visual Tracking

Zhipeng Zhang, Houwen Peng

arXiv:1901.01660v3cs.CV

TL;DR

The paper asks how Siamese trackers can benefit from deeper and wider backbones when direct substitutions fail. It introduces nopadding residual modules and architectures with controlled receptive fields and stride, achieving reported gains over original SiamFC and SiamRPN while retaining real-time speed. The main scope boundary is that performance can remain limited when depth changes also reduce stride suitability or feature-channel capacity.

  • Problem

    Siamese trackers rely on shallow AlexNet backbones, while direct replacement with deeper architectures can fail because receptive field, stride, and padding impair tracking.

  • Method

    The paper designs cropping-inside residual units and deeper or wider Siamese backbones that remove padding-affected features and control receptive field size and network stride.

  • Results

    The proposed architectures yield up to 9.8%/5.7% AUC, 23.3%/8.8% EAO, and 24.4%/25.0% EAO relative improvements over original SiamFC/SiamRPN on OTB-15, VOT-16, and VOT-17.

  • Takeaways & Limitations

    Effective Siamese backbones require padding removal and coordinated control of receptive field size and stride, rather than simply increasing depth or width.

  • Takeaways & Limitations

    CIResNet-43 gains are limited because its stride changes to 4 and its output channels fall to 256 versus 512 in other compared networks.

Abstract

from arXiv · show

Siamese networks have drawn great attention in visual tracking because of their balanced accuracy and speed. However, the backbone networks used in Siamese trackers are relatively shallow, such as AlexNet [18], which does not fully take advantage of the capability of modern deep neural networks. In this paper, we investigate how to leverage deeper and wider convolutional neural networks to enhance tracking robustness and accuracy. We observe that direct replacement of backbones with existing powerful architectures, such as ResNet [14] and Inception [33], does not bring improvements. The main reasons are that 1)large increases in the receptive field of neurons lead to reduced feature discriminability and localization precision; and 2) the network padding for convolutions induces a positional bias in learning. To address these issues, we propose new residual modules to eliminate the negative impact of padding, and further design new architectures using these modules with controlled receptive field size and network stride. The designed architectures are lightweight and guarantee real-time tracking speed when applied to SiamFC [2] and SiamRPN [20]. Experiments show that solely due to the proposed network architectures, our SiamFC+ and SiamRPN+ obtain up to 9.8%/5.7% (AUC), 23.3%/8.8% (EAO) and 24.4%/25.0% (EAO) relative improvements over the original versions [2, 20] on the OTB-15, VOT-16 and VOT-17 datasets, respectively.

1. Introduction

The paper investigates why deeper and wider backbones fail to improve Siamese tracking when directly substituted for AlexNet, then proposes architectures that address the identified structural issues. The resulting designs improve SiamFC and SiamRPN while preserving real-time speed.

  • Siamese trackers commonly use shallow AlexNet backbones despite modern deeper networks offering stronger feature embeddings.
  • Directly replacing AlexNet with VGG, Inception, or ResNet can reduce performance as network depth or width increases.Figure 1 reports AUC against depth and width for SiamFC on OTB-13.
  • Receptive field size, network stride, and padding affect tracking accuracy by changing feature discriminability, localization precision, and position bias.The paper identifies these factors as preventing Siamese trackers from benefiting from deeper architectures.
  • The proposed cropping-inside residual units remove padding-affected features, while the new deeper and wider architectures control receptive field size and stride.The architectures are built by stacking the proposed units and are applied to SiamFC and SiamRPN.
  • Up to 9.8%/5.7% AUC, 23.3%/8.8% EAO, and 24.4%/25.0% EAO relative improvements are reported over original SiamFC/SiamRPN on OTB-15, VOT-16, and VOT-17.The improvements are attributed solely to the proposed network architectures, which remain lightweight and real-time.
  • The work contributes a systematic study of backbone factors and new deeper and wider architectures based on nopadding residual units.The authors report clear improvements over baseline trackers and provide code and models.

2. Background on Siamese Tracking

SiamFC uses a fully convolutional Siamese architecture to compare an exemplar of the target with a larger search image. The shared ConvNet produces feature maps whose cross-correlation forms a response map for locating the target.

  • The exemplar image z contains the target object, while the larger search image x represents the search area in later video frames.
  • Both images pass through a ConvNet ϕ with parameters θ, producing feature maps for matching.
  • The response map is fθ(z, x) = ϕθ(z) ⋆ϕθ(x) + b · 1, formed by cross-correlating the two feature maps and adding a bias term.
  • The maximum response value is used to estimate the target location through an exhaustive search of z over x.
  • The ConvNet is trained offline on random video image pairs and ground-truth labels using a logistic loss.
  • This work studies how to replace the commonly used shallow AlexNet backbone with a more advanced ConvNet to improve robustness and accuracy.

3. Analysis of Performance Degradation

The analysis attributes performance degradation in deeper Siamese backbones to interacting structural factors, especially stride, receptive field, and padding. It derives design guidelines that preserve localization precision and matching consistency.

  • 3.1. Analysis: The study compares network stride, padding, receptive field, output feature size, and width in addition to depth and width.
  • 3.1. Analysis: Increasing stride to 16 significantly degrades performance, indicating a preference for mid-level features with stride 4 or 8 for localization.
  • 3.1. Analysis: The effective maximum receptive field is about 60%∼80% of the exemplar size, while fields larger than the exemplar can sharply reduce performance.
  • 3.1. Analysis: Padding creates inconsistent exemplar and search embeddings because border features may include zero-padding, degrading similarity matching near image boundaries.
  • 3.2. Guidelines: The proposed analysis recommends removing padding or cropping padding-affected features to address perceptual inconsistency between Siamese streams.
  • 3.2. Guidelines: Network stride, receptive field, and output feature size should be designed jointly because changing one alters the others and affects feature discrimination.

4. Deeper and Wider Siamese Networks

The paper builds deeper and wider Siamese-tracking backbones from cropping-inside residual units, controlling padding effects, stride, and receptive-field size while preserving real-time application.

  • Cropping-Inside Residual Units: Cropping-inside residual (CIR) units remove features affected by zero-padding after residual addition, preventing padding-related position bias.The units augment residual blocks with cropping, while CIR-D units modify downsampling to remove padding effects.
  • Cropping-Inside Residual Units: Downsampling CIR units change convolutional stride from 2 to 1 and relocate downsampling to avoid progressively removing outer image content.Without this change, deeper networks could produce noisy or incomplete features because cropped features would miss signals from outermost input pixels.
  • Wider Networks: CIR-Inception and CIR-NeXt units widen the residual design through multiple feature transformations and branches for constructing wide backbones.The CIR-Inception structure uses a 1×1 convolution in the shortcut and merges branch features by concatenation.
  • Architecture Design: The architectures control stride, unit placement, and receptive-field size by stacking CIR units across stages; stride 8 yields three stages, while stride 4 yields two.The design goal is to keep the final-layer receptive field within the derived range.
  • Deeper and Wider Networks: The resulting deeper and wider families include CIResNet, CIResInception, and CIResNeXt variants with distinct depths and widths.CIResNet-22 uses three stages and 22 weighted convolution layers, while CIResNet-43 uses two stages to keep its receptive field within range; the latter has 6.07G FLOPs.
  • Applications: Replacing the original five-layer AlexNet backbones with the designed networks is the only modification made to SiamFC and SiamRPN.This isolates the effect of the proposed backbone architectures in the two representative Siamese trackers.

5. Experiments

Experiments evaluate the proposed architectures in SiamFC and SiamRPN across OTB and VOT benchmarks, baseline comparisons, state-of-the-art comparisons, and ablations. The results show substantial gains from deeper and wider backbones and identify CIR units, stride, receptive field, and feature size as important design factors.

  • Experimental Setup: The experiments evaluate the proposed networks in SiamFC and SiamRPN on OTB-2013, OTB-2015, and VOT-17, with speed measured on a GeForce GTX 1080 GPU.Table 4 compares network architectures with the original AlexNet-based frameworks.
  • Comparison with Baselines: 9.8% AUC and 24.4% EAO relative gains are obtained by SiamFC with CIResNet-22 over the original AlexNet baseline, while SiamRPN achieves 4.4% and 23.3% gains.These results are reported on OTB-2015 and VOT-17, respectively, for the corresponding metrics.
  • Comparison to State-of-the-Art Trackers: SiamFC+ and SiamRPN+ are compared with state-of-the-art trackers on OTB-2013, OTB-2015, VOT-15, VOT-16, and VOT-17.The enhanced trackers use CIResNet-22 as their backbone.
  • VOT-16: On VOT-16, SiamRPN+ ranks first in EAO, surpassing CCOT by 3.9 points and VITAL by 4.7 points, while SiamFC+ improves over its baseline by 6.0 EAO points.The comparison uses the top-10 trackers from the challenge.
  • VOT-17: On VOT-17, SiamRPN+ reaches 3.01 EAO at 150 fps, while SiamFC+ exceeds CSRDCF++ by 2.2 EAO points among real-time comparisons.SiamRPN+ is slightly below LSART in EAO but runs 150 times faster.
  • Ablation Study: Replacing CIR units with original residual units reduces CIResNet-22 performance by 8.8 points, from 0.301 to 0.213, supporting the importance of padding removal.The ablation attributes the effect to eliminating position bias in learning.
  • Ablation Study: The proposed CIR-D unit outperforms both direct original downsampling and post-addition cropping, whose cropping can remove internal features and lose input information.The comparison evaluates three downsampling settings in SiamFC.

6. Discussions

The paper frames backbone design around adapting deep and wide architectures to Siamese tracking rather than directly importing classification networks. It identifies padding removal and control of receptive field size and stride as key design principles.

  • Network Architectures: Residual and Inception-style designs provide the deeper and wider architecture patterns that this work modifies for Siamese networks.ResNets support depth through identity mappings, while Inception modules increase width through multiple feature transformations.
  • Network Architectures: Two backbone principles are removing internal padding and controlling receptive field size and network stride.Both factors are reported to have significant impact on tracking performance.
  • Siamese Trackers: Siamese trackers use similarity matching between a target exemplar and candidate image patches, with follow-up methods improving precision or localization.The paper positions its backbone study within this established Siamese tracking framework.
  • Network Architectures: Deep and wide architectures are adapted to Siamese tracking instead of used as direct backbone replacements.The related work distinguishes this architectural approach from prior work based on data augmentation and feature fusion.

7. Conclusion

The paper designs deep and wide network architectures for Siamese trackers after finding that direct backbone replacement does not improve tracking. Experiments on five benchmarks demonstrate the effectiveness of the proposed architectures and competitive performance across five datasets.

  • 7. Conclusion: The proposed work designs deep and wide network architectures specifically for Siamese trackers.This design is motivated by the failure of direct replacement with existing powerful backbones to bring improvements.
  • 7. Conclusion: Receptive field size, network padding, and stride are identified as crucial causes affecting the success of backbone replacement.The conclusion reports these factors as the key causes determined through the paper’s study.
  • 7. Conclusion: Five benchmarks demonstrate the effectiveness of the proposed architectures, which achieve competitive performance on five datasets.The conclusion attributes the reported evaluation outcome to the proposed architecture design.
Loading 1901.01660v3…