Source-linked AI summary
Separable Self and Mixed Attention Transformers for Efficient Object Tracking
Goutam Yelluru Gopal, Maria A. Amer
TL;DR
Transformer attention's computational complexity limits its use in lightweight Siamese tracking and on constrained hardware. SMAT combines a hybrid separable mixed-attention backbone with a separable self-attention prediction head. It outperformed related lightweight trackers on six challenging benchmarks and ran at 158 fps on GPU.
Problem
Transformer attention complexity limits its use in lightweight Siamese trackers and constrains their speed on resource-limited hardware.
Source: §1 · p. 1Method
SMAT uses a CNN–ViT backbone with separable mixed attention to fuse template and search features, plus a separable self-attention head for global feature modeling.
Source: §1 · pp. 1–2, §2 · p. 2Results
SMAT outperformed related lightweight trackers on six challenging benchmarks and ran at 158 fps on GPU.
Source: §5 · p. 8Takeaways & Limitations
Abstract
from arXiv · showhide
The deployment of transformers for visual object tracking has shown state-of-the-art results on several benchmarks. However, the transformer-based models are under-utilized for Siamese lightweight tracking due to the computational complexity of their attention blocks. This paper proposes an efficient self and mixed attention transformer-based architecture for lightweight tracking. The proposed backbone utilizes the separable mixed attention transformers to fuse the template and search regions during feature extraction to generate superior feature encoding. Our prediction head performs global contextual modeling of the encoded features by leveraging efficient self-attention blocks for robust target state estimation. With these contributions, the proposed lightweight tracker deploys a transformer-based backbone and head module concurrently for the first time. Our ablation study testifies to the effectiveness of the proposed combination of backbone and head modules. Simulations show that our Separable Self and Mixed Attention-based Tracker, SMAT, surpasses the performance of related lightweight trackers on GOT10k, TrackingNet, LaSOT, NfS30, UAV123, and AVisT datasets, while running at 37 fps on CPU, 158 fps on GPU, and having 3.8M parameters. For example, it significantly surpasses the closely related trackers E.T.Track and MixFormerV2-S on GOT10k-test by a margin of 7.9% and 5.8%, respectively, in the AO metric. The tracker code and model is available at https://github.com/goutamyg/SMAT
1. Introduction
Transformer attention can limit lightweight tracking on resource-constrained hardware. SMAT addresses this with a hybrid backbone and a separable transformer prediction head.
1. Introduction
- Standard attention's computational cost limits transformer use in lightweight Siamese trackers and constrains CPU deployment.Source: §1 · p. 1
- SMAT cascades CNN and ViT blocks, using mixed attention to fuse template and search features and a separable self-attention head to model global context for localization.Source: §1 · pp. 1–2
- The proposed tracker backbone uses a cascaded arrangement of CNN and ViT blocks, combining convolutional learning of spatially local representations with transformer modeling of long-range dependencies.Source: §1 · p. 1
- SMAT runs beyond real-time speed on a CPU.Source: §1 · p. 2
2. Related Work
Prior work applies transformers to Siamese tracking and lightweight trackers, but approaches differ in where they use transformer modules. SMAT distinguishes itself through joint feature fusion and concurrent transformer-based backbone and head modules.
2. Related Work
- LightTrack [35], FEAR [2], Stark-Lightning [34], HiFT [3], SiamHFFT, and HCAT [4] represent lightweight Siamese tracking approaches using efficient architectures or transformer-based fusion.Source: §2 · p. 2
- Many lightweight transformer trackers use CNN backbones for feature extraction and transformers only for relation modeling [3] [4] [34].Source: §2 · p. 2
- MixFormerV2 [9] and E.T.Track are identified as the closest comparisons: one has a fully transformer-based backbone, the other an exemplar-transformer prediction head.Source: §2 · p. 2
- SMAT exchanges template and search information during feature extraction, unlike two-stream approaches [2] [3] [4] [34] [35], and uses a transformer head for localization unlike [2] [3] [4] [9] [34].Source: §2 · p. 2
- SMAT's prediction head has 1.8M parameters versus 5.7M for E.T.Track; SMAT also trains once rather than through repeated pruning and distillation rounds [9].Source: §2 · p. 2
- The proposed contribution combines a separable mixed-attention backbone with a separable self-attention head, deploying both efficiently in lightweight tracking for the first time.Source: §2 · p. 2
3. The Proposed Method
The proposed method specifies a backbone, prediction head, and training loss for processing template and search features and estimating target state.
3.1. Proposed Mixed Attention ViT-based Backbone
- CNN inverted-residual blocks extract spatially local features and pool them into lower-dimensional inputs for the ViT block.Source: §3.1 · p. 3
- Shared CNN projections concatenate template and search tokens for separable mixed attention, modeling both within-region and cross-region interactions.Source: §3.1 · p. 3
- Elementwise attention operations reduce latency versus dense matrix-to-matrix attention; the block then splits and reprojects template and search features.Source: §3.1 · p. 3
- Pixel-wise cross-correlation fuses the backbone’s template and search features into an encoding that is sent to the prediction head.Source: §3.1 · pp. 3–4
3.2. Proposed Self-Attention Prediction Head
- Two cascaded convolutional-transformer branches use global self-attention for classification and bounding-box regression, producing a score map, offsets, and box size.Source: §3.2 · p. 4, Figure 3 · p. 4
3.3. Loss function for Training
- Training combines weighted focal loss for classification with ℓ1 and generalized IoU losses for regression, weighted by separate hyperparameters.Source: §3.3 · p. 4
4. Experimental and Ablation Results
SMAT is evaluated across six tracking benchmarks, with experiments examining its implementation, training, benchmark performance, component ablations, attribute-specific results, and attention maps. The reported comparisons show strong results across datasets alongside trade-offs in speed and attention design.
4.1. Implementation Details
- The tracker uses 128 × 128 template and 256 × 256 search inputs, two CNN-based IR blocks, two ViT blocks, and a 128-channel fused encoding.The classification and regression branches use 2 and 4 transformer layers, respectively.Source: §4.1 · p. 4
4.2. Training and Inference Details
- Training combines GOT10k, LaSOT, TrackingNet, and COCO, with 300 epochs of 6 × 10^4 image pairs and no additional finetuning of hyperparameters derived from [37].Inference uses the first-frame annotation as the template and does not update the model.Source: §4.2 · pp. 4–5
4.3. Comparison to the Related Work
- SMAT outperforms related lightweight trackers on all six test datasets; on GOT10k-test it exceeds E.T.Track and MixFormerV2 by 6.8% in AO, 8.8% in SR0.50, and 12.4% in SR0.75 on average.Table 1 compares the trackers across GOT10k-test, TrackingNet-test, LaSOT-test, NfS30, UAV123, and AVisT.Source: §4.3 · pp. 5–6, Table 1 · p. 6
4.4. Ablation Study
- Mixed attention approximates explicit self- and cross-attention on concatenated template and search features, yielding the best speed-performance trade-off among the fusion variants.Variant C performs best on both datasets but is 35% slower than SMAT on CPU; variant A is 1.05× faster but has lower AUC on GOT10k and LaSOT.Source: §4.4 · pp. 6–7, Table 2 · p. 7
- Replacing the transformer prediction head with a fully convolutional head lowers performance by 3.5% in GOT10k-test AO and 2.3% in LaSOT-test AUC.The paper attributes the difference to global contextual modeling of encoded features by the proposed head.Source: §4.4 · p. 7, Table 3 · p. 7
- Separable attention is 15.6% faster than standard transformer attention, with 1.7% higher LaSOT AUC but 1.4% lower GOT10k AUC.The comparison replaces the separable backbone blocks with standard transformer-based MobileViT blocks.Source: §4.4 · p. 7, Table 4 · p. 7
4.5. Attribute-based analysis
- SMAT performs best on 8 of 14 LaSOT attributes and second-best on 5, but performs worse than related trackers on Fast Motion.The reported AUC advantages include 4.3% for Illumination Variation and 3.7% for Deformation over MixFormerV2-S [9].Source: §4.5 · pp. 7–8, Table 5 · p. 8
4.6. Visualizing the attention maps
- Attention-map examples show SMAT locating targets under appearance challenges, with maps focusing on visual cues around a partially occluded target.The examples include aspect-ratio change, illumination variation, deformation, and partial occlusion.Source: §4.6 · p. 8
5. Conclusion
SMAT combines separable mixed attention in its backbone with a self-attention predictor head, and outperforms related lightweight trackers across six challenging benchmarks. Its 3.8M-parameter design exceeds real-time speed on CPU and runs at 158 fps on GPU.
5. Conclusion
- The separable mixed-attention backbone exchanges information between template and search regions, producing improved encoding over a two-stream tracking pipeline.Source: §5 · p. 8
- Figure 5 pairs predicted bounding boxes with attention maps, where red marks larger values and blue marks smaller values.Source: Figure 5 · p. 8
- 3.8M-parameter SMAT outperformed related lightweight trackers on six challenging benchmarks, exceeding real-time speed on CPU and reaching 158 fps on GPU.Source: §5 · p. 8