Source-linked AI summary
Transformer Tracking
Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, Huchuan Lu
TL;DR
Correlation-based fusion can lose semantic information and global context, making high-accuracy real-time tracking challenging. TransT replaces correlation with attention-based fusion using self-attention and cross-attention, and achieves strong benchmark performance, including a 2.2% higher AO than SiamR-CNN for TransT-GOT.
Problem
Correlation is a local linear matching process that can lose semantic information and fail to use global context, while high-accuracy real-time tracking remains challenging.
Method
TransT uses an attention-only feature fusion network with ego-context self-attention and cross-feature cross-attention modules between template and search-region features.
Results
TransT achieves the best reported performance on LaSOT, TrackingNet, and GOT-10k comparisons, while TransT-GOT scores 2.2% higher than SiamR-CNN in GOT-10k AO.
Takeaways & Limitations
Attention-based fusion can replace correlation for combining template and search-region features while supporting object localization and bounding-box regression.
Abstract
from arXiv · showhide
Correlation acts as a critical role in the tracking field, especially in recent popular Siamese-based trackers. The correlation operation is a simple fusion manner to consider the similarity between the template and the search region. However, the correlation operation itself is a local linear matching process, leading to lose semantic information and fall into local optimum easily, which may be the bottleneck of designing high-accuracy tracking algorithms. Is there any better feature fusion method than correlation? To address this issue, inspired by Transformer, this work presents a novel attention-based feature fusion network, which effectively combines the template and search region features solely using attention. Specifically, the proposed method includes an ego-context augment module based on self-attention and a cross-feature augment module based on cross-attention. Finally, we present a Transformer tracking (named TransT) method based on the Siamese-like feature extraction backbone, the designed attention-based fusion mechanism, and the classification and regression head. Experiments show that our TransT achieves very promising results on six challenging datasets, especially on large-scale LaSOT, TrackingNet, and GOT-10k benchmarks. Our tracker runs at approximatively 50 fps on GPU. Code and models are available at https://github.com/chenxin-dlut/TransT.
1. Introduction
Visual tracking seeks accurate target localization despite challenges such as occlusion, similar-object interference, and motion blur. TransT replaces correlation-based fusion with attention to integrate template and search-region features more effectively.
- Visual object tracking predicts a target’s position and shape in each video frame for applications including robot vision, surveillance, and unmanned driving.
- TransT produces more robust and accurate tracking results under occlusion, similar-object interference, and motion blur than two state-of-the-art trackers.
- High-accuracy, real-time tracking remains challenging because videos contain occlusion, deformation, and similar-object interference.
- Correlation performs local linear matching, losing semantic information and limiting nonlinear interactions between template and region-of-interest features.
- TransT introduces attention-based fusion with self-attention ego-context augmentation and cross-attention cross-feature augmentation.
- The framework combines feature extraction, attention-only Transformer-like fusion, and prediction heads for classification and regression.
2. Related Work
Siamese trackers commonly rely on correlation, but correlation can underuse global context and lose semantic information. TransT instead directly fuses template and search-region features with attention, avoiding correlation.
- Siamese-based trackers such as SiamFC, SiamRPN, and ATOM commonly use correlation to integrate template information into regions of interest.
- Correlation-based networks may underuse global context and lose semantic information, contributing to local optima and imprecise target-boundary prediction.
- Transformer attention captures global information by determining which parts of an input sequence are important.
- Earlier tracking attention methods improved selection or enhancement but still relied heavily on correlation to fuse template and search-region features.
- TransT directly fuses template and search-region features with a new attention-based network without any correlation operation.
3. Transformer Tracking
TransT replaces correlation-based fusion with an attention-based network that separately enhances template and search features before adaptively fusing them. Its concise architecture uses a Siamese-like backbone, repeated attention fusion layers, and a prediction head for classification and anchor-free box regression.
- Overall Architecture: TransT extracts template and search-region features separately, enhances and fuses them, then predicts foreground/background labels and normalized bounding-box coordinates.The architecture contains a backbone, feature fusion network, and prediction head.
- Feature Fusion Network: Two ECAs and two CFAs form a fusion layer, which repeats N times before an additional CFA produces the fused feature map.The implementation uses N = 4 fusion-layer repetitions.
- Ego-Context Augment: ECA uses residual multi-head self-attention with spatial positional encodings to integrate information from different positions within each feature map.The attention output is added residually to the input.
- Cross-Feature Augment: CFA uses residual multi-head cross-attention and an FFN to adaptively fuse features from the two branches.Queries come from the branch containing CFA, while keys and values come from the other branch.
- Attention Visualization: Attention visualizations show deeper fusion strengthens target location in search self-attention and target-boundary focus in search cross-attention.Template features increasingly store target-boundary information, while search features retain spatial information.
4. Experiments
Experiments evaluate TransT across large-scale and smaller tracking benchmarks, ablate post-processing and fusion choices, and compare attention-based fusion with correlation. The reported results show strong accuracy, including state-of-the-art performance without post-processing and approximately 50 fps operation.
- Evaluation on TrackingNet, LaSOT and GOT-10k Datasets: TransT achieves the best performance on LaSOT among compared trackers except SiamR-CNN, while running at 50 fps versus SiamR-CNN’s less than 5 fps.Attribute-based evaluation also reports that TransT performs much better than competing trackers across all LaSOT attributes.
- Ablation Study and Analysis: TransT without post-processing still achieves state-of-the-art performance, while window-penalty post-processing further improves accuracy across almost all reported metrics.The ablation compares TransT with TransT-np, the version without post-processing.
- Ablation Study and Analysis: The proposed Transformer-like fusion performs better than the original Transformer structure by a large margin in comparisons with and without post-processing.The comparison keeps the other tracker components, training data, and strategy unchanged.
- Ablation Study and Analysis: Compared with correlation, attention-based fusion establishes long-distance feature associations and produces semantically rich features rather than only a similarity map.The analysis attributes these properties to global information aggregation between template and search-region features.
- Evaluation on Other Datasets: On NFS, TransT reaches an AUC score of 65.7%, exceeding PrDiMP’s 63.5% by 2.2%; it is comparable on OTB2015 and best on UAV123.The NFS result is calculated from the reported 63.5% baseline and 2.2% gain.
5. Conclusions
The paper proposes a Transformer-like tracking framework that fuses template and search-region features solely through attention. Its reported experiments show significantly better performance than state-of-the-art algorithms while maintaining real-time speed.
- 5. Conclusions: TransT uses ego-context self-attention and cross-feature cross-attention to fuse template and search-region features and extract richer semantic information.The fusion network is designed to replace correlation for object localization and bounding-box regression.
- 5. Conclusions: Experiments across many benchmarks report significantly better performance than state-of-the-art algorithms while the tracker runs at real-time speed.The conclusion characterizes the framework as simple, high-performance, and Transformer-like.