Source-linked AI summary
Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking
Boyu Chen, Peixia Li, Lei Bai, Lei Qiao, Qiuhong Shen, Bo Li, Weihao Gan, Wei Wu, Wanli Ouyang
TL;DR
Existing visual trackers depend on customized interaction modules and architecture-specific prior knowledge, limiting a general tracking framework. SimTrack serializes exemplar and search images for joint processing in a transformer backbone and adds a foveal window to preserve target detail. It reports strong benchmark performance, including 70.5% AUC on LaSOT and 55.6% AUC on TNL2K, while remaining competitive with specialized trackers.
Problem
Existing tracking approaches rely on customized sub-modules and prior knowledge for architecture selection, while transformer-based generic simplification has not been investigated in VOT.
Method
SimTrack serializes exemplar and search images into tokens for joint feature learning and interaction in a one-branch transformer backbone, supplemented by a foveal window.
Results
70.5% AUC on LaSOT, 55.6% AUC on TNL2K, 83.4% AUC on TrackingNet, 69.8% AO on GOT-10k, and 71.2% on UAV123 are reported.
Takeaways & Limitations
SimTrack provides a simpler and more generic tracking framework that achieves competitive performance without specialized interaction modules.
Abstract
from arXiv · showhide
Exploiting a general-purpose neural architecture to replace hand-wired designs or inductive biases has recently drawn extensive interest. However, existing tracking approaches rely on customized sub-modules and need prior knowledge for architecture selection, hindering the tracking development in a more general system. This paper presents a Simplified Tracking architecture (SimTrack) by leveraging a transformer backbone for joint feature extraction and interaction. Unlike existing Siamese trackers, we serialize the input images and concatenate them directly before the one-branch backbone. Feature interaction in the backbone helps to remove well-designed interaction modules and produce a more efficient and effective framework. To reduce the information loss from down-sampling in vision transformers, we further propose a foveal window strategy, providing more diverse input patches with acceptable computational costs. Our SimTrack improves the baseline with 2.5%/2.6% AUC gains on LaSOT/TNL2K and gets results competitive with other specialized tracking algorithms without bells and whistles.
1 Introduction
SimTrack uses a transformer backbone to jointly learn features and interaction, simplifying visual tracking by removing specialized interaction modules. A foveal window preserves target details lost during down-sampling, while experiments show strong results across tracking benchmarks.
- Motivation: Existing tracking methods use customized transformer heads and task-specific prior knowledge, but transformer-based simplification had not been investigated in VOT.The paper frames a general-purpose tracking architecture as a way to simplify frameworks and reduce reliance on hand-designed components.
- SimTrack architecture: SimTrack serializes exemplar and search images, concatenates their tokens, and processes them jointly in a one-branch transformer backbone.Search features from the backbone go directly to target localization without an interaction module.
- Foveal window strategy: The foveal window produces more diverse target patches around the exemplar center to reduce information loss from transformer down-sampling.The strategy assigns higher patch-sampling frequency to the image centre, where target-relevant information is concentrated.
- SimTrack architecture: SimTrack removes the existing interaction head from Siamese tracking while retaining feature interaction throughout the backbone.The architecture uses joint feature learning and interaction rather than a separate, hand-designed interaction stage.
- Results: 70.5% AUC on LaSOT, 55.6% AUC on TNL2K, 83.4% AUC on TrackingNet, 69.8% AO on GOT-10k, and 71.2% on UAV123 are reported for SimTrack.The experiments cover multiple tracking datasets and report state-of-the-art performances for these benchmarks.
2 Related Work
Visual tracking has evolved from Siamese two-branch similarity matching toward transformer-based interaction. SimTrack instead places generic, bidirectional interaction throughout a pretrained backbone rather than using specialized post-backbone or hand-designed modules.
- Transformer architectures: Vision transformers convert image patches into tokens and use self-attention to model global dependencies across the input.This architecture has supported transformer applications in classification and detection, motivating broader vision-task simplification.
- Siamese tracking: Siamese trackers extract exemplar and search features in separate branches, then use cross-correlation or proposal networks for target localization.Later methods replace global cross-correlation with more structured interaction, while transformer trackers strengthen exemplar-search information exchange.
- Transformer tracking: Most prior tracking methods introduce exemplar-search interaction after their backbones, whereas some add hand-designed interaction modules inside the backbone.These approaches retain specialized interaction designs despite moving interaction closer to feature extraction.
- SimTrack distinction: SimTrack differs by using a single generic backbone with pretrained vision-transformer interaction in every block and bidirectional information flow.The comparison contrasts SimTrack with methods whose interaction is limited to selected blocks or flows only from exemplar to search.
3 Proposed Method
SimTrack simplifies tracking by concatenating exemplar and search tokens before a unified transformer backbone that jointly extracts features and models their interaction. A foveal window supplies additional target detail, while a predictor localizes the target from backbone search features.
- Architecture: SimTrack uses a transformer backbone and predictor, with the backbone jointly extracting features and interacting between exemplar and search representations.The target-relevant search features are passed to a corner predictor for target localization.
- Baseline Model: The baseline replaces STARK-S’s Res50 backbone with ViT while retaining its separate transformer-head interaction pipeline.The baseline crops an exemplar from the first frame and treats subsequent frames as search inputs.
- Image Serialization: Input images are reshaped into flattened patches, linearly projected into tokens, and serialized as exemplar and search sequences before backbone processing.The exemplar and search patch counts are determined by their spatial resolutions and patch size P.
- Joint Feature Extraction and Interaction: Unlike the baseline, SimTrack concatenates exemplar and search sequences before every backbone layer, enabling cross-image attention during feature learning.Cross-attention terms allow exemplar and search features to influence one another in each layer, removing the need for an additional interaction head.
- Position Embedding: Distinct position embeddings identify the exemplar and search tokens, using pretrained embeddings for the search image and a target-aware learnable embedding for the exemplar.The exemplar embedding is computed from patch position and the target-area ratio within each patch.
- Foveal Window Strategy: The foveal window strategy crops a smaller exemplar region to produce diverse patches containing additional target information alongside the original patches.The strategy addresses detail loss caused by down-sampling while maintaining acceptable computational costs.
4 Experiments
Experiments across five tracking benchmarks show that SimTrack achieves competitive or state-of-the-art performance with a simplified framework, while ablations support its efficiency, initialization, foveal window, and backbone interaction choices.
- State-of-the-art comparisons: On LaSOT, Sim-L/14 reaches 70.5% AUC and 79.7% normalized precision, while Sim-B/16 reaches 69.3% AUC and 78.5% normalized precision.Sim-B/16∗ also outperforms all compared trackers with a simpler framework and lower computation costs.
- State-of-the-art comparisons: On TNL2K, ViT-B/16 exceeds the highest compared AUC of 52.0% by 2.8 points, and ViT-L/14 further raises performance to 55.6% AUC.The paper reports that SimTrack performs best among the compared trackers on this benchmark.
- Ablation study: Using the same backbone as STARK-SV, SimTrack improves LaSOT/TNL2K AUC by 3.7/3.1, 2.5/2.6, and 1.3/1.6 points for ViT-B/32, ViT-B/16, and ViT-L/14.The gains occur with similar or fewer computation costs.
- Ablation study: Training for 200 epochs matches the baseline's 66.8% AUC after 500 epochs, and MAE initialization gives the best performance among the tested pre-training weights.The authors attribute the training difference to avoiding a randomly initialized transformer head.
- Ablation study: The foveal window adds 0.8 AUC points on TNL2K, while reducing information interaction causes successive 2.5-point AUC drops.These analyses support using detailed target patches and comprehensive interaction across the backbone.
5 Conclusions
SimTrack presents a simple visual object tracking framework that streamlines the pipeline and achieves compelling results against established baselines, while leaving room for architectural and training improvements.
- SimTrack uses a transformer backbone for joint feature learning and information interaction, eliminating most specialization in current tracking methods.
- The framework obtains compelling results against well-established baselines on five tracking benchmarks.
- The architecture and training techniques remain open to further optimization for performance improvements.
6 Appendix
The appendix visualizes how SimTrack’s backbone develops target-relevant attention and examines implementation and input-resolution choices. Its attention maps are compared with a separately processed baseline under matched training settings.
- Visualization: SimTrack directly exposes target-relevant attention maps from its transformer backbone, whereas the baseline requires post-training feature calculations.
- Visualization: A(sl) is computed by applying softmax to attention scores between search and exemplar or search tokens.
- Visualization: The target-relevant portion of A(sl) is averaged across exemplar-token dimensions, reshaped into a spatial map, and up-sampled to search-image size.
- Visualization: SimTrack’s attention can quickly and gradually focus on a more accurate and comprehensive target area.
- Training Details: The training configuration uses 500 epochs, 6 × 10^4 image pairs per epoch, batch size 256, AdamW, and weight decay 10^-4.
- Input Resolution: Increasing the input resolution from 224 × 224 to higher tested resolutions helps improve tracking accuracy.
- Visualization: Figure 5 compares exemplar images, search images, and attention maps across transformer layers for the baseline and SimTrack.