PaperlayerView exact revision on arXiv

Source-linked AI summary

SwinTrack: A Simple and Strong Baseline for Transformer Tracking

Liting Lin, Heng Fan, Zhipeng Zhang, Yong Xu, Haibin Ling

arXiv:2112.00995v3cs.CV

TL;DR

Transformer tracking has largely used CNN features, leaving Transformer-based representation learning under-explored. SwinTrack uses Transformer-based representation learning and feature fusion with a motion token, achieving state-of-the-art performance across multiple benchmarks.

  • Problem

    Existing tracking methods mainly use Transformer for fusing CNN features, leaving its potential for feature representation learning under-explored.

  • Method

    SwinTrack is a fully attentional Siamese tracker using Transformer for representation learning and feature fusion, augmented with a lightweight historical-trajectory motion token.

  • Results

    SwinTrack achieves state-of-the-art performance on multiple tracking benchmarks, with the motion token providing clear gains at negligible computation.

  • Takeaways & Limitations

    SwinTrack demonstrates the effectiveness of attention-based representation learning and fusion for tracking and provides a baseline for future Transformer-tracking research.

  • Takeaways & Limitations

    Alternative motion-token constructions remain possible, while multi-token temporal representations were ineffective in early experiments and may require more precise temporal modeling.

Abstract

from arXiv · show

Recently Transformer has been largely explored in tracking and shown state-of-the-art (SOTA) performance. However, existing efforts mainly focus on fusing and enhancing features generated by convolutional neural networks (CNNs). The potential of Transformer in representation learning remains under-explored. In this paper, we aim to further unleash the power of Transformer by proposing a simple yet efficient fully-attentional tracker, dubbed SwinTrack, within classic Siamese framework. In particular, both representation learning and feature fusion in SwinTrack leverage the Transformer architecture, enabling better feature interactions for tracking than pure CNN or hybrid CNN-Transformer frameworks. Besides, to further enhance robustness, we present a novel motion token that embeds historical target trajectory to improve tracking by providing temporal context. Our motion token is lightweight with negligible computation but brings clear gains. In our thorough experiments, SwinTrack exceeds existing approaches on multiple benchmarks. Particularly, on the challenging LaSOT, SwinTrack sets a new record with 0.713 SUC score. It also achieves SOTA results on other benchmarks. We expect SwinTrack to serve as a solid baseline for Transformer tracking and facilitate future research. Our codes and results are released at https://github.com/LitingLin/SwinTrack.

1 Introduction

SwinTrack is proposed as a fully attentional Siamese tracker that uses Transformer architectures for both representation learning and feature fusion. It also introduces a lightweight motion token encoding recent target trajectories, and reports strong performance across multiple benchmarks.

  • Method: SwinTrack applies the Swin Transformer to Siamese tracking, extending Transformer use from feature fusion to feature representation learning.Its fusion module uses homogeneous concatenation without a query-based decoder.
  • Results: 0.713 SUC is achieved by SwinTrack-B-384 on LaSOT while running at around 45 fps.SwinTrack-T-224 achieves 0.672 SUC at around 96 fps.
  • Motion token: The motion token represents the historical target trajectory within a local temporal window and is incorporated into the fusion decoder.It provides temporal motion information with negligible computation to improve tracking robustness.
  • Method: As a pure Transformer framework, SwinTrack enables feature interactions within template and search-region learning and their fusion.The paper contrasts this with pure CNN-based and hybrid CNN-Transformer frameworks.
  • Results: SwinTrack achieves state-of-the-art performance on multiple benchmarks, supported by extensive experiments on five large-scale benchmarks.The framework is presented as a simple and strong baseline for fully attentional tracking.

2 Related Work

Prior tracking work developed Siamese matching frameworks and progressively added proposals, deeper backbones, and multi-stage or anchor-free designs. Transformer-based trackers mainly enhance or fuse convolutional features, whereas SwinTrack uses Transformer for both representation learning and feature fusion in a pure Transformer architecture.

  • Siamese Tracking: Siamese tracking formulates tracking as a matching problem and learns a generic matching function offline.Subsequent methods incorporated RPNs for scale variation and explored deeper backbones, multi-stage architectures, and anchor-free designs.
  • Transformer in Vision: Transformer originated in NLP, then showed promise in vision through object detection and Transformer-based backbone representation learning.Vision Transformer (ViT) was reported to attain excellent performance compared with convolutional alternatives.
  • Transformer in Tracking: Tracking methods have used Transformer to enhance Siamese features, exploit temporal features, and integrate spatio-temporal modeling with model updating.These approaches target improved tracking performance and robustness through feature fusion, temporal information, or dedicated tracking architectures.
  • Transformer in Tracking: Unlike prior hybrid CNN-Transformer trackers, SwinTrack is pure Transformer-based, using Transformer for both representation learning and feature fusion.This design enables exploration of better features for robust tracking.

3 Tracking via Vision-Motion Transformer

SwinTrack is a vision-motion integrated Transformer tracker in a Siamese framework, combining Swin-Transformer feature extraction, concatenation-based vision fusion, motion-token modeling, and classification-regression heads. Its motion token embeds sampled historical target trajectories, while concatenated attention shares operations and weights to reduce computation and preserve symmetric branch metrics.

  • Framework: SwinTrack comprises a Swin-Transformer backbone, a vision-motion encoder-decoder, and a localization head within a Siamese tracking framework.The tracker processes template and search-region image patches as paired inputs.
  • Encoder fusion: Concatenating template and search tokens enables shared self-attention and cross-attention, reducing computation and parameters through operation and weight sharing.This design also makes the metric between the two Siamese branches symmetric.
  • Motion modeling: The motion token embeds a fixed-length sample of the target’s historical bounding-box trajectory, emphasizing recent observations while reducing redundancy.Its construction uses coordinate-embedding lookups and token concatenation, resulting in negligible FLOPs.
  • Decoder: The decoder combines encoder outputs with the motion token through multi-head cross-attention and a feed-forward network to produce a vision-motion representation.The representation is subsequently passed to the tracking head.
  • Prediction head: The head uses separate three-layer perceptron branches for classification and bounding-box regression, predicting response maps with 1 and 4 channels, respectively.The classification branch outputs rcls ∈R(Hx×Wx)×1, while regression outputs rreg ∈R(Hx×Wx)×4.

4 Experiments

SwinTrack achieves strong results across five tracking benchmarks with both lightweight and larger configurations, while ablations show benefits from the Swin Transformer backbone, united positional encoding, and varifocal loss. The experiments also report efficient tracking speed and compare alternative feature-fusion and decoder designs.

  • Model: SwinTrack-T-224 uses a Swin Transformer-Tiny backbone pretrained on ImageNet-1k, while SwinTrack-B-384 uses Swin Transformer-Base pretrained on ImageNet-22k.Their template/search sizes are [112 × 112]/[224 × 224] and [192 × 192]/[384 × 384], respectively.
  • Benchmark results: 0.672 SUC and 0.708 PRE are achieved by SwinTrack-T-224 on LaSOT, matching or exceeding several Transformer-based and more complex trackers.On TNL2k, SwinTrack-B-384 sets a new state-of-the-art with 0.559 SUC.
  • Benchmark results: 0.840 SUC is achieved by SwinTrack-B-384 on TrackingNet, exceeding STARK-ST101 at 0.820 SUC and TransT at 0.814 SUC.SwinTrack-T-224 achieves 0.811 SUC on the same benchmark.
  • Benchmark results: 0.724 mAO is achieved by SwinTrack-B-384 on GOT-10k, while SwinTrack-T-224 obtains 0.713 mAO and both outperform the cited Transformer-based counterparts.The GOT-10k evaluation uses the training split only, as required by the dataset protocol.
  • Efficiency: SwinTrack-T-224 is 3× and 2× faster than STARK-ST50 at 32 fps and 42 fps, respectively, and SwinTrack-B-384 is faster than STARK-ST101 and STARK-ST50.The comparison indicates that the larger SwinTrack configuration retains a speed advantage over the cited STARK models.
  • Ablation studies: The Swin Transformer backbone improves performance over ResNet-50 by 2.5% SUC on LaSOT and 5.1% SUC on LaSOText, while united positional encoding adds 0.8-1.9 absolute percentage points.Varifocal loss outperforms BCE without reducing efficiency; cross-attention fusion and a target-query decoder instead produce inferior empirical results.

5 Conclusion

SwinTrack is presented as a simple, strong Transformer-tracking baseline whose representation learning and feature fusion use attention. A motion token further improves robustness by providing historical object trajectory information.

  • SwinTrack is introduced as a simple and strong baseline for Transformer tracking.
  • Both representation learning and feature fusion in SwinTrack are implemented with attention mechanisms.
  • The proposed motion token enhances tracker robustness by providing the historical object trajectory.The motion token also demonstrates the Transformer model’s flexibility in architectural design.

Appendix · A Positional Encoding

The appendix explains why SwinTrack needs positional encoding and adopts untied positional encoding, extending it to multidimensional and concatenation-based fusion settings. It also incorporates relative positional bias and specifies head- and module-level parameter sharing.

  • A Positional Encoding: Transformer requires positional encoding because self-attention is permutation-invariant and cannot understand the order of input tokens.
  • A Positional Encoding: SwinTrack selects TUPE’s untied positional encoding through comparison experiments and generalizes it to arbitrary dimensions for compatibility with tracker components.
  • A Positional Encoding: Untied encoding removes token–position correlation terms and uses separate learnable projection matrices U^Q and U^K for positional embeddings.
  • A Positional Encoding: In the multi-head variant, positional embeddings are shared across heads, while U^Q and U^K differ by head.
  • A Positional Encoding: Relative positional bias adds a learnable scalar b_{j−i} for each relative offset, shared across layers but distinct for each head in multi-head attention.
  • A Positional Encoding: For n-dimensional inputs, the method allocates one positional embedding matrix per dimension and sums corresponding embedding vectors, together with relative positional bias.
  • A Positional Encoding: For concatenation-based fusion, untied absolute positional encoding is concatenated according to real positions, while relative-bias indices include query and key origin pairs.
  • A Positional Encoding: Untied positional-encoding parameters are shared separately within the encoder and decoder, with decoder query-origin indexing fixed.

B The Effect of Pre-training Datasets

SwinTrack-T-224 uses ImageNet-1k, whereas SwinTrack-B-384 uses ImageNet-22k under the default configurations. Comparing pre-training datasets shows model-size- and benchmark-dependent gains and degradations.

  • Pre-training configurations: SwinTrack-T-224 adopts ImageNet-1k pre-training, while SwinTrack-B-384 adopts ImageNet-22k.The variants follow settings derived from Swin Transformer.
  • Benchmark effects: For SwinTrack-B-384, ImageNet-22k pre-training improves LaSOT by +2.2% and GOT-10k by +3.0%.The larger model contains 91M parameters.
  • Benchmark effects: For SwinTrack-B-384, ImageNet-22k slightly degrades TrackingNet by -0.6% but improves LaSOText by +0.2%.These results compare pre-training datasets under the ablation-study settings.
  • Benchmark effects: For SwinTrack-T-224, ImageNet-22k causes a -0.9% degradation on LaSOText and a -1.4% change on GOT-10k.The smaller model has 23M parameters, and the GOT-10k results use the full dataset.

C Comparison with Newly Released Transformer-based Trackers

SwinTrack remains competitive against newly released Transformer-based trackers across four challenging benchmarks and achieves the best performance on the reported benchmark. Its fully attentional architecture provides a significant bounding-box accuracy advantage, while attribute analysis on LaSOT reveals no obvious shortcomings except viewpoint change.

  • Comparison with Newly Released Transformer-based Trackers: SwinTrack remains competitive with STARK, SBT, ToMP, MixFormer, AiATrack, Unicorn, and OSTrack across four challenging benchmarks.The comparison is reported in Table 6.
  • Comparison with Newly Released Transformer-based Trackers: SwinTrack obtained the best performance on the benchmark evaluated through success and precision plots.The comparison includes SwinTrack-T-224, SwinTrack-B-384, TransT, STARK, MixFormer, AiATrack, and ToMP.
  • Comparison with Newly Released Transformer-based Trackers: The fully attentional architecture gives SwinTrack a significant advantage in bounding-box accuracy over other trackers.
  • Comparison with Newly Released Transformer-based Trackers: LaSOT Test-set attribute analysis finds no obvious SwinTrack shortcomings except viewpoint change.The analysis uses success AUC scores under different attributes.

D Results on UAV123 and VOT Benchmark

SwinTrack is evaluated on UAV123, VOT2020, and VOT-STB2022. It matches AiATrack on UAV123, while SwinTrack-T-224 outperforms the larger SwinTrack-B-384 on both VOT benchmarks.

  • Benchmark evaluation: The evaluation covers three additional benchmarks: UAV123, VOT2020, and VOT-STB2022.UAV123 contains 123 aerial video sequences, while VOT2020 contains 60 videos with segmentation masks.
  • UAV123: On UAV123, SwinTrack performs on par with the state-of-the-art tracker AiATrack.The comparison results are reported in Table 7.
  • VOT2020: On VOT2020, SwinTrack-T-224 performs better than the larger SwinTrack-B-384.Because SwinTrack is bounding-box-only, the comparison uses trackers that also produce bounding boxes.
  • VOT-STB2022: On VOT-STB2022, SwinTrack-T-224 also achieves better performance, with no comparison because the benchmark is newly released.The VOT-STB2022 results are reported in Table 9.

E Quantitative Analysis of the Effectiveness of Motion Token

Motion-token analysis on LaSOT shows improved tracker robustness in success and precision plots, while attribute-based results indicate assistance in recovering from failure states.

  • LaSOT test-set analysis: The motion token improves tracker performance by boosting robustness in LaSOT success and precision plots.The analysis uses the LaSOT test set’s success plot and precision plot.
  • LaSOT test-set analysis: Attribute-based success AUC results further show that the motion token assists recovery from failure states.This analysis is reported across different LaSOT test-set attributes.

F Response Visualization for Qualitative Analysis

Response-map heatmaps for SwinTrack-B-384 qualitatively demonstrate strong discriminative power on challenging LaSOText sequences involving fast motion, full occlusion, and hard distractors.

  • Qualitative response visualization: Heatmaps from the IoU-aware classification branch show SwinTrack-B-384 responses on LaSOText sequences with fast motion, full occlusion, and hard distractors.The visualization highlights the tracker’s discriminative response behavior under challenging conditions where similar objects or occlusion can produce multi-peak maps in many trackers.

G Failure Case

SwinTrack exhibits failures under combined visual and motion challenges, distractor occlusion, and non-semantic targets. The paper visualizes representative cases from LaSOText and VOT-STB2022.

  • Failure causes: Representative failures include low resolution, fast motion, and background clutter occurring together in one case.The examples use SwinTrack-B-384 on LaSOText and VOT-STB2022.
  • Failure causes: A second failure occurs when a distractor causes fast occlusion of the target.
  • Failure causes: A third failure involves a non-semantic target.This case is shown on VOT-STB2022.
Loading 2112.00995v3…