PaperlayerView exact revision on arXiv

Source-linked AI summary

SparseTT: Visual Tracking with Sparse Transformers

Zhihong Fu, Zehua Fu, Qingjie Liu, Wenrui Cai, Yunhong Wang

arXiv:2205.03776v1cs.CV

TL;DR

Transformer self-attention can overlook relevant targets and emphasize background in visual-tracking search regions. SparseTT uses sparse attention and a double-head predictor to address this issue, outperforming state-of-the-art approaches across multiple datasets while operating at 40 FPS.

  • Problem

    Transformer self-attention models global dependencies but can under-focus targets and over-focus background in search regions, degrading tracking.

  • Method

    SparseTT combines a sparse Transformer target focus network with a double-head predictor for foreground-background classification and target-box regression.

  • Results

    SparseTT outperforms state-of-the-art approaches on LaSOT, GOT-10k, TrackingNet, and UAV123 while running at 40 FPS.

  • Takeaways & Limitations

    SparseTT provides a real-time sparse Transformer tracking baseline with reported applicability across multiple datasets.

Abstract

from arXiv · show

Transformers have been successfully applied to the visual tracking task and significantly promote tracking performance. The self-attention mechanism designed to model long-range dependencies is the key to the success of Transformers. However, self-attention lacks focusing on the most relevant information in the search regions, making it easy to be distracted by background. In this paper, we relieve this issue with a sparse attention mechanism by focusing the most relevant information in the search regions, which enables a much accurate tracking. Furthermore, we introduce a double-head predictor to boost the accuracy of foreground-background classification and regression of target bounding boxes, which further improve the tracking performance. Extensive experiments show that, without bells and whistles, our method significantly outperforms the state-of-the-art approaches on LaSOT, GOT-10k, TrackingNet, and UAV123, while running at 40 FPS. Notably, the training time of our method is reduced by 75% compared to that of TransT. The source code and models are available at https://github.com/fzh0917/SparseTT.

1 Introduction

Visual tracking remains difficult under deformation, occlusion, and scale variation, while Transformer self-attention can overemphasize background. SparseTT addresses this with target-focused sparse attention and a double-head predictor.

  • Motivation: Target deformation, partial occlusion, and scale variation remain major obstacles to accurate visual tracking.Tracking errors can accumulate across time slices and lead to tracking failure.
  • Motivation: Transformer self-attention can under-focus targets while over-focusing background, blurring foreground-background boundaries and degrading tracking.This limitation arises from self-attention’s global perspective in search regions.
  • Method: SparseTT concentrates on the most relevant search-region information with a sparse Transformer to make targets more discriminative and bounding boxes more accurate.The approach is intended to handle deformation, occlusion, and scale variation.
  • Method: The target focus network highlights relevant target features for more accurate state estimation.It is a central component of the proposed sparse Transformer-based siamese tracking framework.
  • Results: SparseTT combines a target focus network and a sparse Transformer-based siamese framework, with experiments reporting superiority across four tracking datasets at 40 FPS.The cited contribution passage names LaSOT, GOT-10k, TrackingNet, and UAV123.

2 Related Work

Transformer trackers replace local cross-correlation with global feature interaction, but vanilla self-attention can neglect the most relevant search-region information. SparseTT adapts sparse attention to focus on relevant targets and suppress distracting background.

  • Siamese Trackers: Cross-correlation measures template-search similarity but performs local linear matching that may fall into local optima.This limitation motivates alternatives to conventional siamese tracking pipelines.
  • Transformer Trackers: TransT and DTT replace cross-correlation with Transformers, while TrDiMP retains cross-correlation for similarity measurement.The cited passage contrasts these tracker designs within siamese-like pipelines.
  • Sparse Attention: Self-attention captures global information and long-range dependencies but lacks focus on the most relevant search-region information.Sparse attention is introduced to address this drawback in Transformer trackers.
  • Sparse Attention: SparseTT adapts a sparse Transformer into an end-to-end siamese tracker whose sparse attention suppresses distracting background more efficiently.Its target focus network uses an encoder-decoder sparse Transformer.

3 Method

SparseTT uses a siamese sparse-Transformer architecture that focuses attention on relevant search-region information, then predicts foreground classification and target boxes with two specialized heads.

  • Architecture: The siamese architecture combines a weight-shared feature extractor, sparse-Transformer target focus network, and double-head predictor.The target focus network generates target-focused features, while the predictor discriminates foreground from background and outputs target bounding boxes.
  • Target Focus Network: The target focus network uses an encoder-decoder design: the encoder processes target-template features, and the decoder transforms search-region features into target-focused features.Spatial position encoding is added to target-template features before encoder processing.
  • Decoder: Each decoder layer first applies sparse multi-head self-attention to search features, then naive multi-head cross-attention between encoded templates and search features.The decoder also receives search-region positional information or the previous decoder output together with encoded target-template features.
  • Sparse Multi-Head Self-Attention: Sparse self-attention retains only the K most similar pixel values for each position, replacing other attention weights with zero before multiplying by the values.This design focuses foreground features and makes foreground edges more discriminative than naive attention.
  • Double-Head Predictor: The double-head predictor contains a two-layer fully connected head and a convolutional head, jointly supporting foreground-background classification and bounding-box regression.Training uses a weighted sum of classification and regression losses; focal loss is used for classification and IoU loss for box regression.

4 Experiments

Experiments evaluate SparseTT’s training configuration, ablations, and performance across multiple visual-tracking benchmarks. The method achieves strong results on long-term, challenging, aerial, and general tracking datasets while maintaining efficient training and inference.

  • Training Settings: The model is trained for 20 epochs with AdamW on six tracking and detection datasets, using sampled frame pairs as template-search examples.Training uses 600,000 image pairs per epoch, except 300,000 from GOT-10k, with a maximum frame-index difference of 100.
  • Training Settings: 60 hours of training on 4 NVIDIA RTX 2080 Ti GPUs is 4× faster than TransT’s 240-hour training process.The learning rate decreases after epochs 10 and 15; batch size is 32.
  • Ablation Study: Two encoder layers and two decoder layers are selected because additional layers reduce performance, with three decoder layers also substantially slowing inference.The encoder ablation attributes the decline beyond two layers to possible overfitting; the decoder study reports a similar pattern.
  • Ablation Study: Sparse attention performs better than naive multi-head attention, and K = 32 gives the best performance in the SMSA ablation.When K = H′W ′, SMSA becomes naive MSA; the experiments use K = 32.
  • Benchmark Results: On LaSOT, SparseTT achieves the best success, precision, and normalized precision, and performs best on deformation, partial occlusion, and scale-variation subsets.LaSOT’s test split contains 280 sequences averaging more than 2500 frames; the challenging-subset evaluation also covers rotation and viewpoint change.
  • Benchmark Results: SparseTT surpasses competing trackers on GOT-10k, UAV123, TrackingNet, and OTB2015, including settings with limited annotated training data and difficult aerial imagery.It achieves state-of-the-art performance on UAV123, the best success metric on TrackingNet, and comparable performance while outperforming TransT on OTB2015.

5 Conclusions

The method combines sparse self-attention with a double-head predictor to improve visual tracking, outperform state-of-the-art approaches across multiple datasets, and reduce training time while maintaining real-time speed.

  • Sparse self-attention highlights relevant search-region information instead of relying on global context, helping identify potential targets.The method addresses vanilla self-attention's tendency to neglect relevant target information.
  • A double-head predictor improves foreground-background classification and target-box regression accuracy.
  • The method significantly outperforms state-of-the-art approaches on multiple datasets while running at real-time speed.The cited conclusion reports superiority across multiple datasets without specifying individual scores.
  • Training time is only 25% of TransT's training time.This corresponds to a 75% reduction compared with TransT.
Loading 2205.03776v1…