Source-linked AI summary
Generalized Relation Modeling for Transformer Tracking
Shenyuan Gao, Chunluan Zhou, Jun Zhang
TL;DR
Existing one-stream trackers let template tokens interact with all search tokens through every encoder layer, although early background interactions can cause target confusion. GRM adaptively divides tokens to select appropriate template–search interactions, using attention masking and Gumbel-Softmax for parallel, end-to-end learning. The method reports outstanding results on six challenging benchmarks and state-of-the-art performance with real-time speed.
Problem
Global template–search interaction in one-stream tracking can create undesired relations with background or distractors when early representations are insufficiently discriminative.
Method
GRM adaptively divides template and search tokens into categories, restricting template interaction to suitable search tokens while using attention masking and Gumbel-Softmax.
Results
GRM achieves outstanding results on six challenging visual tracking benchmarks and reports 2.4% higher AO and 2.5% higher success rate than OSTrack on GOT-10k with aligned settings.
Takeaways & Limitations
GRM provides a more flexible attention-based relation-modeling formulation that includes two-stream and one-stream pipelines as degenerate forms.
Takeaways & Limitations
Token division can mistakenly select distractor tokens during partial occlusion, causing temporary tracker drift; model updates are left for future work.
Abstract
from arXiv · showhide
Compared with previous two-stream trackers, the recent one-stream tracking pipeline, which allows earlier interaction between the template and search region, has achieved a remarkable performance gain. However, existing one-stream trackers always let the template interact with all parts inside the search region throughout all the encoder layers. This could potentially lead to target-background confusion when the extracted feature representations are not sufficiently discriminative. To alleviate this issue, we propose a generalized relation modeling method based on adaptive token division. The proposed method is a generalized formulation of attention-based relation modeling for Transformer tracking, which inherits the merits of both previous two-stream and one-stream pipelines whilst enabling more flexible relation modeling by selecting appropriate search tokens to interact with template tokens. An attention masking strategy and the Gumbel-Softmax technique are introduced to facilitate the parallel computation and end-to-end learning of the token division module. Extensive experiments show that our method is superior to the two-stream and one-stream pipelines and achieves state-of-the-art performance on six challenging benchmarks with a real-time running speed.
1. Introduction
Transformer trackers progressed from separate two-stream relation modeling to one-stream joint modeling, but global template–search interaction can confuse targets with background. GRM addresses this with adaptive token division and flexible relation modeling, supported by attention masking and end-to-end token learning.
- One-stream trackers jointly model cross-relations and self-relations through unified attention, enabling earlier template–search interaction.
- Global interaction can create undesired cross-relations with background distractors before features become sufficiently discriminative.These relations may inject target information into distractors and degrade template representations.
- GRM adaptively selects suitable search tokens for template interaction while blocking inappropriate search-token interactions.Its generalized formulation can represent both two-stream and one-stream pipelines as degenerate cases.
- The method divides tokens into three categories to enable more flexible interaction between template and search regions.
- Attention masking enables parallel computation, while Gumbel-Softmax supports end-to-end learning of the discrete token division module.
- Extensive experiments report outstanding results for GRM on six challenging visual tracking benchmarks.
2. Related Works
Related work spans two-stream Siamese trackers, one-stream Transformer trackers, dynamic architecture design, and token organization methods for efficient vision Transformers. These lines of research motivate joint relation modeling and adaptive token organization for tracking.
- Two-stream Siamese trackers separately extract template and search features, then model cross-relations with additional correlation modules.Deep neural-network backbones are commonly used to obtain discriminative features.
- One-stream Transformer trackers concatenate template and search tokens to jointly model cross-relations and self-relations with self-attention.They can use pretrained models for cross-relation modeling instead of randomly initialized correlation modules.
- Neural architecture search methods dynamically design efficient backbones, prediction heads, or correlation modules during training.The resulting network architectures are frozen after training.
- Token organization methods exploit attention flexibility to accelerate vision Transformer inference.SparseTT and OSTrack use attention weights from self-attention, with SparseTT aggregating features from the most similar tokens.
3. Method
The method generalizes Transformer tracking relation modeling by adaptively dividing search tokens, allowing selective template interaction while preserving search-token communication. It implements these relations with masked parallel attention and learns token assignments end to end.
- Preliminary: The one-stream pipeline concatenates template and search tokens, then jointly updates them through multi-head attention and feed-forward blocks in each encoder layer.The final search tokens are reshaped into a spatial feature map for tracking.
- Motivation: Global template interaction can propagate target information into background or distractor tokens and degrade template representations when early features are insufficiently discriminative.These effects can make precise target identification more difficult.
- Generalized Relation Modeling: The method divides search tokens between ES and EA while fixing template tokens in ET, blocking template interaction with ES but retaining interactions among search tokens.ET interacts with ET and EA; ES interacts with ES and EA; EA can aggregate from all tokens.
- Generalized Relation Modeling: This design degenerates to two-stream modeling when all search tokens belong to ES and to one-stream modeling when all belong to EA.The generalized formulation can therefore adapt its relation pattern across layers.
- Adaptive Token Division: Search-token categories are selected by comparing predicted probabilities, with πi,0 assigned to ES and πi,1 assigned to EA.The categorization is represented as a binary assignment for each search token.
- Adaptive Token Division: An attention mask combines the three category-specific attention operations into one computation, enabling GPU parallelization with only a minor increase in computation.The mask indicates whether each token may aggregate information from another token according to the relation rules.
4. Experiments
Experiments evaluate GRM across six tracking benchmarks, compare token-division variants, and analyze layer-wise behavior. Results show strong benchmark performance and support adaptive division with attention masking.
- Results and Comparisons: GRM is evaluated against state-of-the-art trackers on six challenging visual-tracking benchmarks.The comparisons cover GOT-10k, TrackingNet, LaSOT, AVisT, NfS30, and UAV123.
- Results and Comparisons: 2.4% in AO and 2.5% in success rate are the reported improvements over OSTrack on GOT-10k under aligned model settings.
- Results and Comparisons: 84.0% in success score and 83.3% in precision score are achieved on TrackingNet, overtaking previously published trackers.
- Ablation Studies: The best token-division variant preserves search-token interaction while allowing template interaction with selected search tokens.The three-category design alleviates the limitations of dividing search tokens into only ET and ES, while the ES and EA design achieves the best overall performance among the ablations.
- Ablation Studies: Adaptive token division is applied after the first encoder layer, and maximum pooling outperforms average pooling for target-aware representation aggregation.The first encoder layer is considered unsuitable because its raw embedding projections have not yet been refined by feature aggregation.
- Ablation Studies: 45 fps with attention masking compares favorably with 33 fps using separate attention operations.The masking strategy is introduced to avoid separate attention operations becoming a running-speed bottleneck.
- Ablation Studies: Layer-wise visualizations show intermediate, two-stream, and one-stream relation patterns at the 3rd, 7th, and 11th encoder layers.The 3rd layer selects target-relevant tokens for EA, the 7th selects none, and the 11th selects all search tokens as EA.
5. Conclusion
The paper presents a generalized relation modeling method that adaptively divides tokens, combining the strengths of two-stream and one-stream Transformer tracking while allowing more flexible interactions.
- GRM inherits the strengths of two-stream and one-stream Transformer tracking pipelines while providing more flexible relation modeling.The method adaptively determines which search tokens interact with template tokens.
Appendix
The appendix supplements the paper with discussions of limitations, failure cases, differences from a pioneering work, and possible future research directions.
- The appendix discusses limitations and failure cases, differences with a pioneering work, and potential avenues for further exploration.
A. Limitations and Failure Cases
Adaptive token division can fail when distractor tokens are mistakenly selected to interact with the template, particularly during partial occlusion, causing temporary tracker drift.
- Distractor tokens may be mistakenly selected for interaction with the template, causing the tracker to fail.The paper identifies this as a typical failure mode of adaptive token division.
- When partial occlusion makes the target and distractor difficult to distinguish, the tracker can drift in subsequent frames.The target is partially occluded by a distractor in the illustrated failure case.
B. Differences with DynamicViT
The paper distinguishes GRM from DynamicViT by emphasizing different objectives and token-selection behavior: GRM targets relation modeling, whereas DynamicViT fixes a token ratio for inference acceleration.
- GRM and DynamicViT are motivated differently: GRM aims to prevent relation-modeling problems, while DynamicViT targets inference acceleration with an accuracy drop.
- DynamicViT discards a fixed token ratio during inference, whereas GRM adaptively assigns tokens to categories according to prediction scores.
C.1. Supervision of Token Division
The authors base GRM’s token-division design on exploratory foreground-selection work and use Gumbel-Softmax for end-to-end optimization. Ground-truth bounding-box supervision is identified as an uninvestigated auxiliary signal.
- Early prototypes learned rough foreground classification from feature vectors using prediction modules in each encoder layer.These modules were directly supervised by ground-truth bounding boxes but did not affect global cross-relation modeling during training.
- Gumbel-Softmax enables end-to-end optimization of GRM’s token division.The method achieves satisfactory results on multiple benchmarks.
- Auxiliary ground-truth bounding-box supervision is not investigated for the token-division modules.The authors conjecture that combining implicit and explicit supervision could improve interpretability and tracking performance.
C.2. Form of Relation Modeling
Most encoder layers converge to two-stream or one-stream relation modeling, while the ordering of these forms across layers lacks an obvious pattern. Attempts to force intermediate forms are sensitive across benchmarks.
- Most encoder layers permanently degenerate to two-stream or one-stream relation modeling forms.The corresponding token-division modules could be skipped during inference, potentially reducing computation overhead and increasing running speed.
- The layer-wise relation-modeling pattern has no obvious progression from earlier to later encoder layers.The irregular pattern may depend on initialization weights from pretrained models.
- Regularization intended to make all layers use the intermediate form produced sensitive results without on-par performance across all benchmarks.The authors suggest degenerated forms may generalize better because attention blocks handle stationary relation-modeling situations.
C.3. Alternative to Discrete Categorization
The authors identify continuous attention-weight scaling as an alternative to strict discrete token categorization. This alternative can avoid Gumbel-Softmax and supports token- or region-level weighting designs.
- Continuous prediction-module estimates can scale raw attention weights instead of enforcing strict token categorization.This alternative bypasses the non-differentiability caused by strict constraints and eliminates the need for Gumbel-Softmax.
- Region-level scaling weights may complement token-level attention weights more effectively.The authors propose hierarchical attention-weight design as a direction for future investigation.
- Hierarchical attention-weight design is presented as a possible future avenue for improving relation modeling in Transformer trackers.