AiATrack: Attention in Attention for Transformer Visual Tracking

• Independent query-key correlations can create erroneous links and noisy, ambiguous attention weights, motivating consensus-based refinement.Source: §1 · p. 2
• AiA applies inner attention to raw correlations, enhancing relevant pairs and suppressing erroneous ones for self-attention feature aggregation and cross-attention information propagation.Source: §1 · p. 2
• AiATrack reuses cached encoded features, selects high-quality short-term references with an IoU prediction head, and uses learnable target-background embeddings to exploit temporal references efficiently.Source: §1 · p. 3
• AiATrack achieves state-of-the-art results on six tracking benchmarks at 38 fps.Source: §1 · p. 3

2.1 Visual Tracking

• Transformer-based visual trackers use attention to capture global context, but conventional attention limits their potential, motivating the proposed AiA module.Source: §2.1 · p. 3
• Prior trackers address appearance change through reference-feature generation or ensembling or inference-time optimization, whereas AiATrack reuses encoded features and adds target-background embeddings.Generation and ensembling may require resizing and re-encoding reference frames, while optimization methods need sophisticated sparse updates for real-time use; the embeddings distinguish target and background and provide contextual cues, unlike [33].Source: §2.1 · pp. 3–4

2.2 Attention Mechanism

• Unlike prior work addressing other conventional-attention issues, AiA addresses noisy and ambiguous attention by seeking consensus among correlations with a global receptive field.Source: §2.2 · p. 4

2.3 Correlation as Feature

• Prior work recalibrates correlations as features using handcrafted algorithms [49,5] or learnable blocks, while AiA introduces this insight into attention as a unified block for feature aggregation and information propagation.Correlations refer to pixel or region matching results, obtainable through squared difference, cosine similarity, or inner product.Source: §2.3 · p. 4

3.1 Attention in Attention

• AiA refines the correlation map by using consistency among correlation vectors to enhance relevant query-key correlations and suppress erroneous ones.Conventional attention computes each query-key correlation independently, which can produce noisy, ambiguous attention and impair feature aggregation or information propagation. AiA applies an inner attention to correlation vectors before softmax, seeking consensus with a global receptive field; it can be inserted into self- and cross-attention blocks.Source: §3.1 · pp. 5–6

3.2 Proposed Framework

• AiATrack encodes search and reference-frame features, then uses separate long- and short-term cross-attention branches to propagate reference information to the search frame.The long-term branch retrieves information from the initial frame, while the short-term branch addresses changes in target and background appearance; the decoder omits self-attention, and both branches use AiA-equipped cross-attention.Source: §3.2 · pp. 6–8
• Learnable target and background embeddings label reference regions and are attached to reference features as contextual cues for cross-attention.A target prediction head estimates bounding-box corners, while an IoU prediction head assesses predicted boxes to determine whether a search frame becomes a short-term reference.Source: §3.2 · p. 8

3.3 Tracking with AiATrack

• AiATrack updates short-term references by reusing cached encoded features and embedding maps when a predicted box’s IoU exceeds a predefined threshold.It uniformly samples cached short-term references while always including the latest, closest frame, and removes the oldest when the cache reaches its maximum size.Source: §3.3 · p. 9

4.2 Results and Comparisons

• On NfS30, AiATrack improves AUC by 2.7% over STARK; it outperforms KeepTrack by 0.9% on UAV123 and achieves comparable performance on saturated OTB100.Table 2 reports comparisons on commonly used small-scale datasets in terms of AUC.Source: Table 2 · p. 11, §4.2 · pp. 11–12
• AiATrack achieves the best AUC performance on every LaSOT attribute split, with significant improvements in scenarios including background clutter, camera motion, and deformation.The figure presents attribute-based evaluation on LaSOT in terms of AUC.Source: Figure 5 · p. 11, §4.2 · p. 11
• On LaSOT, AiATrack beats KeepTrack by 1.9% AUC and 3.6% precision; TrackingNet scores 82.7% AUC and 87.8% normalized precision, while GOT-10k success improves 2.3% over STARK and TrDiMP.The paper reports that AiATrack runs much faster than KeepTrack on LaSOT and surpasses all previously published trackers on TrackingNet; GOT-10k evaluation follows its one-shot protocol.Source: §4.2 · p. 11

4.3 Ablation Studies

• Removing background context sharply degrades performance, while target-background embeddings improve every metric over the variant without them by distinguishing target from background while preserving context.The ablations were conducted on the LaSOT test set and its extension, totaling 430 videos.Source: §4.3 · p. 12
• Removing the short-term reference branch or using one cross-attention branch instead of separate long- and short-term branches lowers performance, supporting the two-branch design.The authors attribute the single-branch drop to unreliable short-term references disturbing the robust long-term reference.Source: §4.3 · p. 12
• AiA improves performance when inserted into encoder self-attention, decoder cross-attention, or both; the final model improves all metrics by 1.7∼2.7% over the basic framework on both LaSOT subsets.Removing positional encoding from the final model lowers performance; positional encoding supplies spatial cues for consensus seeking.Source: §4.3 · pp. 12–13
• Simply adding attention blocks yields little benefit, while a convolutional bottleneck helps but underperforms AiA, which uses global receptive fields and dynamic weights to refine correlations.The bottleneck uses a similar computational cost but aggregates locally with fixed weights; AiA flexibly seeks consensus among correlation vectors.Source: §4.3 · p. 13, Table 4 · p. 14
• Increasing the short-term reference ensemble from one to three improves performance, while further increases add little benefit and barely affect speed; query-perspective refinement reaches 68.5% LaSOT AUC.The default ensemble size for ablations is three; the paper also visualizes correlation maps from the perspective of keys.Source: Table 5 · p. 14, §4.3 · p. 14

• AiA enhances appropriate correlations and suppresses erroneous ones by seeking consensus among all correlation vectors.Source: §5 · p. 14
• AiATrack is a streamlined Transformer tracking framework that uses efficient feature reuse and embedding assignment to fully utilize temporal references.Source: §5 · p. 14
• The authors report that extensive experiments demonstrate the proposed method’s superiority.Source: §5 · p. 14
• The authors suggest AiA could benefit Transformer-based feature aggregation and information propagation in video object segmentation [33], video object detection, and multi-object tracking [51] [42] [64].Source: §5 · p. 14

• The supplementary material adds experimental details and analyses of the proposed method.Source: AiATrack: Attention in Attention for… · p. 20

6.1 Target Prediction

• The anchor-free prediction head outputs probability maps for the top-left and bottom-right box corners, with coordinates obtained from the maps.The coordinate equations use spatially weighted sums over the corner probability maps.Source: §6.1 · p. 20, §6.2 · p. 20

6.2 Training Objective

• The network jointly minimizes bounding-box and IoU prediction errors using GIoU loss, L1 loss, and squared IoU prediction error.The three terms are weighted by λ_giou, λ_l1, and λ_mse, respectively.Source: §6.2 · p. 20

6.3 Training Strategy

• Offline training uses LaSOT, TrackingNet, GOT-10k, and COCO, with synthetic COCO video clips and one short-term reference frame sampled per training example.Search frames are later than reference frames, and affine transforms jitter reference and search-frame sizes and locations to simulate tracking and reduce padding-related position bias [28]. Training uses AdamW [38], with separate backbone and other-component learning rates and fixed initial ResNet-50 layers.Source: §6.3 · pp. 20–21

6.4 Different Structures of the AiA Module

• AiAv1 is used in AiATrack; layer normalization and identical connection are not key components, while value linear transformation improves performance but is omitted to reduce computational cost.The explored structures are AiAv1, AiAv2, and AiAv3; AiAv3 is a vanilla Transformer self-attention structure. Correlation refinement experiments validate the effectiveness of adding an attention module to conventional attention.Source: §6.4 · pp. 21–22, Table 6 · p. 21, Figure 6 · p. 21

6.5 Results on VOT

• 5% in EAO: AiATrack outperforms STARK on VOT2020 under the anchor-based evaluation protocol.The evaluation uses Alpha-Refine to generate masks because VOT2020 ground truths are segmentation masks; overall performance is ranked by EAO.Source: §6.5 · p. 22, Table 7 · p. 22

7.1 Attribute Analysis

• On LaSOT, AiATrack shows encouraging performance under background clutter, camera motion, and deformation, suggesting potential in challenging scenarios.Fig. 7 presents attribute analysis with AUC scores in its legend.Source: §7.1 · p. 22, Figure 7 · p. 23

7.2 Qualitative Comparisons

• Qualitative comparisons visualize AiATrack, KeepTrack, and STARK on eight challenging sequences.The sequences are bird-17, goldfish-8, sepia-13, shark-2, sheep-3, squirrel-8, tiger-4, and turtle-8.Source: §7.2 · p. 22, Figure 8 · p. 24