Source-linked AI summary
AiATrack: Attention in Attention for Transformer Visual Tracking
Shenyuan Gao, Chunluan Zhou, Chao Ma, Xinggang Wang, Junsong Yuan
TL;DR
Transformer trackers can suffer from noisy and ambiguous attention weights because query-key correlations are computed independently. The paper introduces AiA to refine those correlations and AiATrack to reuse features and exploit temporal references, achieving state-of-the-art performance on six benchmarks at 38 fps.
Problem
Independent query-key correlation computation can produce erroneous correlations and noisy, ambiguous attention weights in Transformer visual tracking.
Method
AiA seeks consensus among correlation vectors to refine attention, while AiATrack combines it with cached-feature reuse and target-background embeddings for temporal references.
Results
AiATrack achieves state-of-the-art performance on six tracking benchmarks while running at 38 frames per second (fps).
Takeaways & Limitations
AiA improves correlation quality for feature aggregation and information propagation, and AiATrack effectively utilizes long-term and short-term temporal references.
Abstract
from arXiv · showhide
Transformer trackers have achieved impressive advancements recently, where the attention mechanism plays an important role. However, the independent correlation computation in the attention mechanism could result in noisy and ambiguous attention weights, which inhibits further performance improvement. To address this issue, we propose an attention in attention (AiA) module, which enhances appropriate correlations and suppresses erroneous ones by seeking consensus among all correlation vectors. Our AiA module can be readily applied to both self-attention blocks and cross-attention blocks to facilitate feature aggregation and information propagation for visual tracking. Moreover, we propose a streamlined Transformer tracking framework, dubbed AiATrack, by introducing efficient feature reuse and target-background embeddings to make full use of temporal references. Experiments show that our tracker achieves state-of-the-art performance on six tracking benchmarks while running at a real-time speed.
1 Introduction
Transformer tracking relies on self- and cross-attention, but independent query-key correlations can produce noisy weights. AiATrack refines these correlations with AiA and improves temporal-reference use through feature reuse and target-background embeddings.
- Transformer tracking uses self-attention to enhance reference and search features and cross-attention to bridge their correlations for target prediction.
- Independent query-key correlation computation can create erroneous links from imperfect features or distracting background patches, producing noisy and ambiguous attention weights.
- AiA refines raw correlations by seeking consensus among correlation vectors, enhancing relevant query-key pairs and suppressing irrelevant ones.It can be inserted into self-attention for feature aggregation and cross-attention for information propagation.
- AiATrack reuses cached encoded features, selects high-quality short-term references with an IoU prediction head, and uses target-background embeddings to exploit temporal references.
- 38 frames per second (fps) and state-of-the-art results on six prevailing benchmarks are reported for AiATrack.The experiments cover various tracking scenarios and are conducted without bells and whistles.
2 Related Work
Related work has applied Transformers and attention to visual tracking while addressing attention quality and temporal adaptation. AiATrack instead combines correlation refinement with efficient encoded-feature reuse and target-background embedding assignment.
- Transformer-based trackers use attention to aggregate information and have achieved impressive performance in visual tracking.
- Prior model-adaptation methods generate or ensemble updated reference features, but often resize and re-encode reference frames, reducing computational efficiency.
- Parameter-optimization approaches require sophisticated sparse-update strategies to meet real-time requirements, whereas AiATrack reuses encoded features efficiently.
- Existing attention research targets issues in conventional attention, which commonly serves as a dynamic information aggregator in spatial and temporal domains.
- Correlation-as-feature methods recalibrate matching results with handcrafted algorithms or learnable blocks; this paper introduces the insight into Transformer attention for tracking.
3 Method
AiATrack refines attention correlations with an inner attention module that seeks consensus among correlation vectors, then integrates this mechanism into a Transformer tracker using temporal references and target-background embeddings.
- 3.1 Attention in Attention: Conventional attention computes each query-key correlation independently, allowing feature imperfections and background distractors to produce noisy, ambiguous attention weights.These errors can impair feature aggregation in self-attention and information propagation in cross-attention.
- 3.1 Attention in Attention: AiA inserts an inner attention module before softmax to refine the correlation map by aggregating correlations and seeking global consistency among correlation vectors.The residual correlation map is added to the original map before the outer attention produces weighted value combinations.
- 3.1 Attention in Attention: The inner module treats columns of the correlation map as queries, keys, and values, applies dimension-reducing transformations and normalization, and generates a residual correlation map.Normalized correlation vectors are processed with scaled dot-product attention; an output transform and identical connection adjust the aggregated correlations.
- 3.1 Attention in Attention: AiA can be inserted into both self-attention and cross-attention blocks, with shared AiA parameters across parallel attention heads.This supports feature enhancement within frames and reference-to-search information propagation in the Transformer tracker.
- 3.2 Proposed Framework: AiATrack uses a long-term initial-frame reference and an ensemble of short-term intermediate references, whose encoded features are reused rather than recomputed during model updates.An IoU prediction head selects high-quality short-term references, while the two-branch decoder retrieves information from long-term and short-term references.
- 3.2 Proposed Framework: Learnable target and background embeddings are assigned by reference-frame region and attached to reference features as values for cross-attention.They enrich reused appearance features with contextual cues while distinguishing target from background regions.
4 Experiments
Experiments evaluate AiATrack across six benchmarks, attribute splits, and component ablations, showing strong tracking performance, real-time operation, and benefits from AiA and temporal-reference designs.
- Results and Comparisons: AiATrack is evaluated on three large-scale and three small-scale tracking benchmarks using RTX 2080 Ti experiments.The evaluated datasets are LaSOT, TrackingNet, GOT-10k, NfS30, OTB100, and UAV123.
- Results and Comparisons: 1.9% AUC and 3.6% precision improvements over KeepTrack are reported on LaSOT, with faster runtime.The tracker also achieves the best performance on all LaSOT attribute splits.
- Results and Comparisons: 82.7% AUC and 87.8% normalized precision are achieved on TrackingNet, surpassing previously published trackers.TrackingNet evaluates short-term tracking using an online evaluation server.
- Ablation Studies: Removing reference-background features drastically reduces performance, while target-background embeddings improve the framework over the corresponding variant.This ablation indicates that contextual background features are useful for visual tracking.
- Ablation Studies: Applying AiA to both self-attention and cross-attention improves all metrics by 1.7∼2.7% over the basic framework on two LaSOT subsets.Removing positional encoding reduces performance, supporting its role in providing spatial cues for consensus seeking.
- Ablation Studies: AiA outperforms simply adding attention blocks and a similarly costly convolutional bottleneck through global, dynamically weighted correlation refinement.The convolutional alternative still provides positive effects, indicating that correlation refinement itself is useful.
- Ablation Studies: Increasing the short-term reference ensemble from 1 to 3 steadily improves performance, whereas larger ensembles provide little further benefit and barely affect speed.The default ablative ensemble size is 3.
5 Conclusion
The paper concludes that AiA improves Transformer attention for visual tracking by refining correlations, while AiATrack uses feature reuse and embedding assignment to exploit temporal references. Extensive experiments demonstrate the method’s superiority, with potential relevance to other Transformer-based video tasks.
- Conclusion: AiA enhances appropriate correlations and suppresses erroneous ones by seeking consensus among correlation vectors.The module is presented as an improvement to the attention mechanism for Transformer visual tracking.
- Conclusion: AiATrack uses efficient feature reuse and embedding assignment mechanisms to utilize temporal references.These mechanisms form the streamlined Transformer tracking framework described in the conclusion.
- Conclusion: Extensive experiments demonstrate the superiority of the proposed method.
- Conclusion: The AiA module could also benefit video object segmentation, video object detection, and multi-object tracking.This is stated as a potential application to related tasks using Transformer-based feature aggregation or information propagation.
AiATrack: Attention in Attention for Transformer Visual Tracking (Supplementary Material)
The supplementary material provides additional experimental and analytical details for the proposed method.
- Supplementary Material: Additional details about the experiments and analyses are provided in the supplementary material.
6 Additional Experiment Details
Additional experiments specify the end-to-end prediction and training setup, compare AiA structures, and report competitive VOT2020 performance. The selected AiA structure balances accuracy and computational cost while correlation refinement remains effective.
- Target Prediction: The anchor-free prediction head outputs probability maps for target-box corners, whose coordinates are obtained from their expected positions.The two maps correspond to the top-left and bottom-right corners.
- Training Objective: The network jointly minimizes GIoU, L1, and IoU prediction losses for bounding-box and IoU estimation.The ground truths are the bounding box b and IoU i, with separate trade-off weights.
- Training Details: Training uses LaSOT, TrackingNet, GOT-10k, and COCO, with synthetic video clips generated from COCO and temporally ordered reference and search frames.For efficiency, only one short-term reference frame is sampled during training.
- AiA Structures: AiAv1 is used in AiATrack, while AiAv2 and AiAv3 vary normalization, value transformation, and post-aggregation connections.AiAv3 corresponds to a typical vanilla Transformer self-attention structure.
- AiA Structures: Linear transformation of values can improve performance, but it is removed to balance performance against computational cost; layer normalization and identical connections are not key components.The experiments support correlation refinement with an additional attention module.
- VOT2020 Results: 5% EAO margin over STARK is reported for AiATrack on VOT2020 under the anchor-based evaluation protocol.Masks are generated with Alpha-Refine, and overall performance is ranked by Expected Average Overlap.
7 Additional Visualization Results
Additional visualizations examine AiATrack across LaSOT attributes and challenging sequences. The reported analyses cover varied scenarios and qualitative comparisons with representative state-of-the-art trackers.
- Attribute Analysis: AiATrack shows encouraging LaSOT performance across background clutter, camera motion, deformation, and other challenging attributes.The attribute analysis reports AUC scores in the figure legend.
- Qualitative Comparisons: Qualitative comparisons include KeepTrack and STARK across eight challenging sequences.The sequences include bird-17, goldfish-8, sepia-13, shark-2, sheep-3, squirrel-8, tiger-4, and turtle-8.