Source-linked AI summary
Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking
Ning Wang, Wengang Zhou, Jie Wang, Houqaing Li
TL;DR
Existing trackers largely overlook temporal relationships among successive frames, despite tracking challenges such as occlusion, deformation, appearance changes, and temporal error accumulation. The paper introduces a modified transformer with parallel encoder and decoder branches inside Siamese-like pipelines, using feature reinforcement and temporal cue propagation. The resulting framework improves two popular tracking pipelines and achieves several new state-of-the-art records across prevalent benchmarks.
Problem
Existing trackers largely overlook temporal relationships among successive frames, while occlusion, deformation, appearance changes, and temporal error accumulation make tracking challenging.
Method
A modified transformer uses parallel encoder and decoder branches within Siamese-like tracking pipelines to reinforce templates and propagate temporal cues to current search features.
Results
The transformer improves Siamese and DCF tracking pipelines, with several new state-of-the-art records reported across seven prevalent tracking benchmarks.
Takeaways & Limitations
The work demonstrates the tracking potential of frame-wise relationships by integrating transformer-based temporal modeling into popular tracking frameworks.
Takeaways & Limitations
The reported baseline comparison uses a larger search area and probabilistic IoUNet, so the baseline differs from standard DiMP.
Abstract
from arXiv · showhide
In video object tracking, there exist rich temporal contexts among successive frames, which have been largely overlooked in existing trackers. In this work, we bridge the individual video frames and explore the temporal contexts across them via a transformer architecture for robust object tracking. Different from classic usage of the transformer in natural language processing tasks, we separate its encoder and decoder into two parallel branches and carefully design them within the Siamese-like tracking pipelines. The transformer encoder promotes the target templates via attention-based feature reinforcement, which benefits the high-quality tracking model generation. The transformer decoder propagates the tracking cues from previous templates to the current frame, which facilitates the object searching process. Our transformer-assisted tracking framework is neat and trained in an end-to-end manner. With the proposed transformer, a simple Siamese matching approach is able to outperform the current top-performing trackers. By combining our transformer with the recent discriminative tracking pipeline, our method sets several new state-of-the-art records on prevalent tracking benchmarks.
1. Introduction
Visual tracking remains difficult because occlusion, deformation, appearance changes, and temporal error accumulation challenge online tracking, while existing methods largely overlook mutual reasoning across successive frames. The paper introduces a transformer-assisted Siamese-like framework that bridges frames through parallel encoder and decoder branches and reports strong benchmark performance.
- Occlusion, deformation, appearance changes, and temporal error accumulation make online visual tracking challenging.
- Most trackers process frames independently or use limited motion priors and incremental updates, without mutual reasoning across video frames.
- The framework introduces a transformer to bridge isolated frames and convey temporal cues across them within visual tracking.
- The encoder and decoder occupy parallel branches in a Siamese-like pipeline, supporting template-based model generation and current-frame search.
- The transformer encoder reinforces multiple template features, while the decoder propagates template features and spatial masks to enhance search representations.
- The transformer generalizes across Siamese and DCF pipelines, with results reported on seven prevalent tracking benchmarks and several new state-of-the-art records.
2. Related Work
Related tracking methods often treat tracking as per-frame detection, leaving temporal characteristics underused. The paper positions transformer-based frame relationship modeling as a distinct approach for transferring temporal information across video frames.
- Siamese trackers use template matching, with later improvements including region proposal networks and attention mechanisms.
- DCF methods formulate tracking as ridge regression and generate responses by convolving a learned discriminative kernel with the search area.
- Most existing methods frame tracking as per-frame object detection and therefore do not adequately exploit temporal characteristics.
- Prior temporal approaches use graph neural networks, spatial-temporal regularization, or optical flow, whereas this work models frame-wise relationships with a transformer.
- Vision transformers had been applied to image-level tasks, while this paper uses transformer structure to transfer temporal information across video frames.
3. Revisting Tracking Frameworks
Siamese and DCF trackers can be expressed through a shared two-branch pipeline: templates generate a tracking model, and the search branch localizes the target. Their response computation differs in how the tracking model is obtained and applied.
- The Siamese-like pipeline assigns the top branch to tracking-model generation and the bottom branch to target localization.
- Siamese matching feeds exemplar and search patches through a weight-sharing CNN and cross-correlates their feature maps to produce a response map.
- In Siamese tracking, the exemplar represents the target while the search patch covers a larger area in subsequent frames.
- Deep-learning DCF optimizes a tracking model with ridge regression and generates responses by convolving that model with search features.
- The DCF formulation jointly considers target matching and background discrimination because its template region is larger than the Siamese exemplar.
- DiMP optimizes the DCF ridge-regression objective with a meta-learner end-to-end and reports state-of-the-art performance.
4. Transformer for Visual Tracking
The framework adapts a transformer to Siamese-like tracking by separating encoder and decoder branches, propagating temporal context, and producing enhanced template and search features for tracking models.
- Architecture: The transformer is modified for tracking by separating encoder and decoder branches within the Siamese-like pipeline.The design also uses shared self-attention weights across branches to align template and search embeddings.
- Transformer Encoder: Self-attention mutually reinforces multiple template features to generate high-quality encoded representations for tracking-model construction.The encoded template feature is reshaped for use in Siamese or DCF tracking pipelines.
- Transformer Decoder: Cross-attention establishes pixel-to-pixel correspondence between template and search features to propagate temporal context across frames.The decoder transfers both feature information and spatial masks from previous templates to the current search patch.
- Transformer Decoder: Spatial attention produces a reinforced search feature that better highlights the potential target area.The transformed mask serves as an attention weight for the search feature.
- Tracking Integration: The framework generates tracking responses using either a Siamese cross-correlation pipeline or a DCF pipeline with transformer-enhanced features.During online tracking, the template ensemble is updated every 5 frames and retains at most 20 templates.
- Tracking Effects: The transformer helps a classic Siamese pipeline perform against recent DiMP and further improves DiMP's suppression of distracting background confidence.The response-map comparison reports more effective suppression of distracting-object confidences with the transformer.
5. Experiments
Experiments evaluate the transformer-assisted trackers through ablations and comparisons across seven tracking benchmarks. The complete transformer improves Siamese and DiMP baselines, while TrSiam and TrDiMP achieve strong benchmark performance with real-time efficiency.
- Implementation: 35 FPS for TrSiam and 26 FPS for TrDiMP demonstrate real-time operation on a single Nvidia GTX 1080Ti GPU.Both trackers use probabilistic IoUNet for target scale estimation after response-map localization.
- Ablation Study: The GOT-10k ablation evaluates transformer components on 180 test videos without overlap between training and test object classes.This protocol assesses generalization to unseen object classes while avoiding test-set hyperparameter fine-tuning.
- Ablation Study: 5.3% and 2.1% AO gains are obtained by the complete transformer for the Siamese and DiMP baselines, respectively.Feature propagation alone yields 4.3% and 1.4% AO gains, while mask propagation independently improves both pipelines and reduces training errors.
- State-of-the-art Comparisons: 63.9% AUC at 26 FPS makes TrDiMP competitive with SiamR-CNN at 64.8% AUC and 4.7 FPS on LaSOT.On VOT2018, both TrSiam and TrDiMP clearly outperform all participant trackers according to the EAO comparison.
- State-of-the-art Comparisons: 71.1% and 70.8% AUC are achieved by TrDiMP and TrSiam, respectively, on OTB-2015, while the trackers set new state-of-the-art records on NfS.On UAV123, TrDiMP performs on par with PrDiMP-50.
6. Conclusion
The work introduces a transformer to bridge isolated video frames and convey temporal cues across them. Applied to two popular trackers, it delivers consistent improvements and several state-of-the-art records.
- Careful modifications make the classic transformer architecture suitable for visual tracking.
- Two popular trackers gain consistent performance improvements with the proposed transformer.
- The method sets several new state-of-the-art records on prevalent tracking datasets.
A. Ablation Study
The online tracker primarily tunes template sampling interval and ensemble size, while other tracking hyper-parameters remain inherited from DiMP.
- The online tracking stage uses template sampling interval and template ensemble size as its only involved hyper-parameters.
- Sampling templates every 5 frames produces promising results.
- Increasing the template ensemble memory size steadily improves performance.
- Other tracking-related hyper-parameters follow the DiMP baseline without modification.
A.2. Improvements upon Baselines
The transformer consistently improves both Siamese and DiMP tracking pipelines across seven benchmarks, with especially clear gains on several challenging datasets.
- The transformer consistently improves the Siamese baseline across seven tracking datasets.
- 5.3% AUC gain is achieved on GOT-10k, with additional gains of 4.7%, 3.3%, and 3.0% on NfS, LaSOT, and TrackingNet.
- 1.6% AUC gain is achieved on OTB-2015 over the Siamese baseline.
- TrDiMP is compared against DiMP, whose baseline already includes memory updates, probabilistic IoUNet, and a larger search area.
- The attention decoder uses sparse cross-attention, selecting several correlated keys to propagate context between patches.
B.2. Response Visualization
Response visualizations show how feature and mask transformations reinforce target evidence and suppress distracting background responses.
- DiMP baseline response maps can be misled by distracting objects in challenging scenarios.
- Figure 9 visualizes encoder self-attention and decoder cross-attention maps.
- Feature transformation reinforces target representations in the search region and restrains background response values.
- Mask transformation propagates spatial attention from previous templates to the current search region, suppressing background objects.
C. Results on VOT2019
On the challenging VOT2019 benchmark, TrDiMP achieves competitive accuracy, lower failure rates, and the highest reported EAO among the compared trackers.
- The evaluation compares accuracy, robustness, and expected average overlap across state-of-the-art methods on VOT2019.
- 0.397 EAO places TrDiMP above the VOT2019 challenge winner DRNet at 0.395.
- TrDiMP maintains similar accuracy to DiMP-50 while exhibiting a much lower failure rate.
- TrDiMP significantly surpasses SiamRPN++, SiamDW, and SiamMask among recent deep trackers using the ResNet-50 backbone.
- A reported failure case shows TrDiMP failing to track an occluded target.
E. Attribute Analysis
Attribute-based evaluation on LaSOT shows strong performance across several challenging scenarios, while transformer assistance improves the simple TrSiam baseline.
- 3.3% AUC improvement over the TrSiam baseline is reported with the proposed transformer on LaSOT.
- TrSiam shows good results under motion blur, background clutter, low resolution, and viewpoint change.
- The simple TrSiam does not use complex online model optimization techniques and is described as more efficient than recent approaches.
- The LaSOT attribute evaluation uses success-plot AUC scores to compare tracking performance across scenarios.