Source-linked AI summary
Transformer Tracking with Cyclic Shifting Window Attention
Zikai Song, Junqing Yu, Yi-Ping Phoebe Chen, Wei Yang
TL;DR
Existing transformer trackers use pixel-level attention that weakens positional information and object integrity. This paper proposes multi-scale cyclic shifting window attention with spatial regularization and redundant-computation removal, reporting stronger performance than previous state-of-the-art trackers across five challenging benchmarks.
Problem
Existing transformer trackers apply pixel-level attention to flattened features, which weakens relative positional information and ignores target-object integrity.
Method
The method uses multi-scale cross-window attention with cyclic shifts, a spatially regularized mask, and strategies that remove redundant computations.
Results
The tracker performs better than previous state-of-the-art trackers on five challenging benchmarks.
Takeaways & Limitations
Window-level attention preserves object integrity and location information while cyclic shifting expands samples for more accurate tracking.
Takeaways & Limitations
Window-level attention reduces the attention-map resolution, producing coarser similarity scores and differently sized attention outputs across heads.
Abstract
from arXiv · showhide
Transformer architecture has been showing its great strength in visual object tracking, for its effective attention mechanism. Existing transformer-based approaches adopt the pixel-to-pixel attention strategy on flattened image features and unavoidably ignore the integrity of objects. In this paper, we propose a new transformer architecture with multi-scale cyclic shifting window attention for visual object tracking, elevating the attention from pixel to window level. The cross-window multi-scale attention has the advantage of aggregating attention at different scales and generates the best fine-scale match for the target object. Furthermore, the cyclic shifting strategy brings greater accuracy by expanding the window samples with positional information, and at the same time saves huge amounts of computational power by removing redundant calculations. Extensive experiments demonstrate the superior performance of our method, which also sets the new state-of-the-art records on five challenging datasets, along with the VOT2020, UAV123, LaSOT, TrackingNet, and GOT-10k benchmarks.
1. Introduction
The paper addresses information loss from pixel-level transformer attention by introducing multi-scale cyclic shifting window attention for visual object tracking. The design preserves object integrity and positional information while improving attention efficiency and accuracy.
- Existing transformer trackers match every template pixel with every search-region pixel, weakening relative positional information and object integrity.
- The proposed architecture elevates attention from pixels to indivisible windows, preserving location information within each window.
- Multi-head multi-scale attention measures relevance among partitioned windows at different scales for cross-window matching.
- Cyclic shifts expand window samples while preserving positional information and object integrity, improving the accuracy of window attention.
- A spatially regularized attention mask addresses boundary performance drops caused by cyclic shifts, while optimization strategies remove redundant computations.
2. Related Work
Related work covers correlation-filter, deep-network, Siamese, and transformer-based tracking approaches. The paper positions its method against prior transformer trackers and draws inspiration from shifted-window vision transformers.
- Correlation-filter trackers learn Fourier-domain filters from circularly shifted patches to discriminate the target from background.
- Deep neural network trackers use convolutional backbones and classification heads to extract features and locate targets.
- Siamese trackers compare template and search-region branches through cross-correlation but cannot effectively explore their semantic correlation.
- Transformers extend tracking beyond cross-correlation by using self-attention and cross-attention to model relations between template and search-region features.
- Swin Transformer uses shifted windows with pixel-level attention inside each window, whereas the proposed approach applies window-level attention across template and search-region windows.
3. Method
CSWinTT matches template and search-region features with multi-scale window-level cross-attention, using cyclic shifts to preserve positional and object-integrity information. Spatial regularization and computational optimizations address boundary artifacts and redundant shifting costs.
- Architecture: CSWinTT uses a ResNet-50 backbone, transformer matching module, and bounding-box estimation head for template-search tracking.Backbone features are partitioned into window sequences before transformer matching.
- Window attention: Multi-scale window partitioning extracts ri × ri patches from template and search feature maps for separate attention heads.The resulting window sequences are concatenated and processed with query-key-value multi-head attention.
- Cyclic shifting: Cyclic shifts expand each r × r base window into (2r−1)^2 samples while preserving positional information and object integrity.Shifts translate samples by single-pixel offsets, with boundaries warped cyclically; duplicate computations are removed separately.
- Spatial regularization: Spatial regularization penalizes shifted samples more as they move farther from the base sample, reducing boundary artifacts caused by cyclic shifts.The spatial mask is added directly to the attention score.
- Efficient computation: Three optimizations remove redundant computation by eliminating Query shifts, halving duplicated shifting periods, and using matrix-coordinate permutations.These changes target the increased complexity introduced by cyclic shifting.
4. Experiments
Experiments evaluate CSWinTT against established trackers across five benchmarks and analyze its components, window scales, visualization, and computational optimizations. The results show strong tracking performance, benefits from cyclic shifting and multi-scale windows, and improved but still lower speed than the original transformer after optimization.
- State-of-the-art Comparison: CSWinTT records 66.2% AUC and 70.9% Precision on LaSOT, outperforming the previous best result from STARK-ST50.
- State-of-the-art Comparison: CSWinTT ranks first on TrackingNet with 81.9% AUC and 86.7% normalized precision, including a 0.7% relative AUC improvement over TransT.
- Ablation Study: On UAV123, cyclic shifts improve AUC by 15.3% over window-level attention and outperform the original transformer by 3.5%.
- Ablation Study: Multi-scale windows reach 70.5% AUC on UAV123, exceeding the best single-window result of 70.0%.
- Ablation Study: Optimization raises tracking speed from around 1 FPS without optimization to 12.4 FPS, although the method remains slower than the original transformer.
- Ablation Study: Attention heat maps compare the original and proposed transformers, including cases with occlusion and similar distractors.
5. Conclusion
The proposed transformer tracker uses multi-scale cyclic shifting window attention to preserve object integrity and location information. Experiments across five challenging benchmarks show better performance than previous state-of-the-art trackers.
- The tracker uses multi-scale cyclic shifting window attention to preserve object integrity and location information during cross-window attention.
- Spatially regularized attention masks address boundary performance drops caused by cyclic shifts.
- Computational optimization removes redundant computations to improve the efficiency of window attention.
- Experiments on five challenging benchmarks show better performance than previous state-of-the-art trackers.
- The authors report stronger discrimination for cyclic shifting window attention than for original pixel-level attention in tracking.