PaperlayerView exact revision on arXiv

Source-linked AI summary

Deformable Siamese Attention Networks for Visual Object Tracking

Yuechen Yu, Yilei Xiong, Weilin Huang, Matthew R. Scott

arXiv:2004.06711v2cs.CV

TL;DR

Siamese trackers perform well but lack online template updates and compute target and search features independently, limiting adaptation and contextual discrimination. SiamAttn introduces deformable self- and cross-attention plus region refinement, achieving new state-of-the-art results across six benchmarks, including higher EAO than SiamRPN++ on VOT 2016 and 2018.

  • Problem

    Siamese trackers cannot update target templates online and compute target and search features independently, limiting robustness to appearance variation, deformations, occlusions, distractors, and complex backgrounds.

  • Method

    SiamAttn jointly applies deformable self-attention and cross-attention, then uses a region refinement module with depth-wise cross correlations on attentional features.

  • Results

    SiamAttn achieves new state-of-the-art results on six benchmarks and improves SiamRPN++ from 0.464→0.537 and 0.415→0.470 EAO on VOT 2016 and 2018.

  • Takeaways & Limitations

    The proposed attention strengthens target discriminability and robustness to appearance variations, complex backgrounds, and distractors while retaining real-time running speed.

Abstract

from arXiv · show

Siamese-based trackers have achieved excellent performance on visual object tracking. However, the target template is not updated online, and the features of the target template and search image are computed independently in a Siamese architecture. In this paper, we propose Deformable Siamese Attention Networks, referred to as SiamAttn, by introducing a new Siamese attention mechanism that computes deformable self-attention and cross-attention. The self attention learns strong context information via spatial attention, and selectively emphasizes interdependent channel-wise features with channel attention. The cross-attention is capable of aggregating rich contextual inter-dependencies between the target template and the search image, providing an implicit manner to adaptively update the target template. In addition, we design a region refinement module that computes depth-wise cross correlations between the attentional features for more accurate tracking. We conduct experiments on six benchmarks, where our method achieves new state of-the-art results, outperforming the strong baseline, SiamRPN++ [24], by 0.464->0.537 and 0.415->0.470 EAO on VOT 2016 and 2018. Our code is available at: https://github.com/msight-tech/research-siamattn.

1. Introduction

Visual tracking remains difficult under deformation, motion, occlusion, and complex backgrounds. SiamAttn addresses limitations of offline, independently computed Siamese features with deformable attention and refinement for more accurate tracking.

  • Visual object tracking must handle deformations, motions, and occlusions in complex backgrounds while remaining fast, accurate, and robust.
  • Siamese trackers match target templates to search regions using cross-correlation within an RPN-based detection framework.
  • Offline training prevents online template updates, limiting tracking under appearance changes, deformations, and occlusions; independent feature computation also discards useful background context.
  • SiamAttn jointly computes deformable self-attention and cross-attention to capture spatial and channel context and encode target-search interdependencies into the target template.
  • The region refinement module uses depth-wise cross correlations to improve representations and generate bounding boxes and object masks.
  • 0.464→0.537 and 0.415→0.470 EAO: SiamAttn improves on SiamRPN++ on VOT 2016 and 2018 while maintaining real-time speed with ResNet-50.

2. Related Work

Related work spans online-updatable correlation-filter trackers, offline Siamese trackers, and attention-based approaches. SiamAttn extends this trajectory by jointly using self-attention and cross-branch attention with deformable operations.

  • Correlation-filter trackers offer efficiency and expansibility but can improve the tracking object online at the cost of representation ability.
  • Siamese trackers learn from massive offline data to measure target-search similarity, with later work adding RPNs, distractor-aware training, deeper networks, and related extensions.
  • Online template-update methods and deep online discriminative frameworks were developed to improve target discriminability against distractors and complex backgrounds.
  • Attention-based tracking research models channel-wise or contextual relationships, but earlier Siamese attention computed target and search attentions separately.
  • SiamAttn jointly combines self-attention and cross-branch attention with deformable operations to enhance target discriminative representation.

3. Deformable Siamese Attention Networks

SiamAttn combines deformable self-attention and cross-attention to enhance Siamese tracker features with spatial, channel-wise, and template-search context. A lightweight refinement module then predicts a single tracking region with bounding-box and mask outputs.

  • Architecture: SiamAttn comprises a deformable Siamese attention module, Siamese RPN, and region refinement module.The attention module enhances features before proposal generation and refinement.
  • Deformable Siamese Attention: The DSA module applies spatial and channel self-attention to capture global context and emphasize interdependent channel-wise features.Self-attention is computed on both target and search branches, with spatial attention modeling positions and channel attention modeling feature dependencies.
  • Deformable Siamese Attention: Confidence maps with DSA enhance target-background discriminability compared with maps computed without the module.The visualization compares search images, confidence maps without DSA, and confidence maps with DSA.
  • Deformable Siamese Attention: Cross-attention aggregates contextual interdependencies between template and search features, encoding target information into the search representation.The resulting cross-attentional features are combined with self-attentional features by element-wise summation; target features are computed similarly.
  • Deformable Siamese Attention: Deformable attention samples feature maps at variable locations, helping the tracker attend to objects undergoing complex geometric transformations.A 3 × 3 deformable convolution is applied to computed attentional features to produce more accurate, discriminative, and robust features.
  • Region Refinement Module: The refinement module uses lightweight convolutional heads to predict a single tracking region's bounding box and class-agnostic binary mask efficiently.The bounding-box head predicts a 4-tuple, while the mask head predicts a 64 × 64 binary mask; no classification head is used because tracking is class-agnostic.

4. Experiments and Results

Experiments evaluate SiamAttn on six tracking benchmarks using standard datasets, metrics, and implementation settings. The method achieves strong results across benchmarks, with ablations attributing gains primarily to Siamese attention and region refinement.

  • Evaluation Setup: SiamAttn is evaluated on OTB-2015, UAV123, VOT2016, VOT2018, LaSOT, and TrackingNet.The experiments include an ablation study to assess individual components.
  • State-of-the-Art Comparisons: 0.537 EAO on VOT2016 surpasses SiamRPN++ and SiamMask by 7.3% and 9.5%, while VOT2018 EAO improves from 0.387 to 0.470 at 33 fps.On VOT2016, the tracker also reports 0.68 accuracy and 0.14 robustness; on VOT2018, accuracy is 0.63.
  • State-of-the-Art Comparisons: 64.8% normalized precision on LaSOT exceeds SiamRPN++'s 56.9%, while TrackingNet results improve over DiMP-50 by 1.2% in success and 1.6% in normalized precision.On LaSOT, success also rises from 49.5% to 56.0%.
  • Ablation Study: On VOT2016, adding self-attention and cross-attention improves EAO by 4.7% and 4.9%, respectively, producing 0.537 EAO versus the 0.464 SiamRPN++ baseline.The region refinement module further increases EAO, while joint attention improves robustness and accuracy.
  • Ablation Study: Removing all deformable layers reduces EAO to 0.516, while inference reaches 45 fps without the mask head and 33 fps on VOT benchmarks with rotated boxes.These ablations suggest Siamese attention and refinement contribute more strongly than deformable layers.

5. Conclusion

SiamAttn combines deformable self-attention and cross-attention with region refinement for visual object tracking. Experiments on six benchmarks report new state-of-the-art results at real-time speed.

  • SiamAttn introduces a deformable Siamese attention mechanism combining self-attention and cross-attention.The mechanism is presented as the paper’s central contribution for visual object tracking.
  • Self-attention enhances target discriminability and robustness against large appearance variations, complex backgrounds, and distractors.
  • A region refinement module further increases tracking accuracy.
  • Extensive experiments on six benchmarks report new state-of-the-art results with real-time running speed.

S1. Qualitative Results

Qualitative VOT2018 results show SiamAttn tracking and segmenting most objects across varied sizes, motions, deformations, and complex backgrounds.

  • SiamAttn tracks and segments most VOT2018 objects despite differences in size, motion, deformation, and background complexity.
Loading 2004.06711v2…