Source-linked AI summary
Target-Aware Tracking with Long-term Context Attention
Kaijie He, Canlong Zhang, Sheng Xie, Zhixin Li, Zhiwen Wang
TL;DR
Target-only Siamese templates omit contextual information, while tracking must contend with appearance changes, rapid movement, and similar-object attraction. TATrack uses long-term context attention to fuse target and context across frames; ablations report that location encoding matters, while multiple templates reduce speed.
Problem
Siamese trackers commonly use target-only templates without context, while tracking faces appearance changes, rapid movement, and attraction from similar objects.
Source: Introduction · p. 1Method
TATrack uses LCA to fuse target and background information across frames, with target-aware features and confidence-based dynamic template updates.
Source: Introduction · p. 1, Long-term Contextual Attention (LCA) · p. 2, Conclusion · p. 7Results
Removing location codes significantly degrades metrics across three datasets, while removing Gaussian localization information causes greater degradation than removing ltrb.
Source: LCA Ablation · p. 6, Box-embedding · p. 6Takeaways & Limitations
LCA combines cross-image feature extraction and correlation to support target-aware tracking, alongside dynamic template updating.
Source: Conclusion · p. 7Using multiple templates, especially previous templates containing background, makes the model run slowly.
Source: Limitations · p. 7
Abstract
from arXiv · showhide
Most deep trackers still follow the guidance of the siamese paradigms and use a template that contains only the target without any contextual information, which makes it difficult for the tracker to cope with large appearance changes, rapid target movement, and attraction from similar objects. To alleviate the above problem, we propose a long-term context attention (LCA) module that can perform extensive information fusion on the target and its context from long-term frames, and calculate the target correlation while enhancing target features. The complete contextual information contains the location of the target as well as the state around the target. LCA uses the target state from the previous frame to exclude the interference of similar objects and complex backgrounds, thus accurately locating the target and enabling the tracker to obtain higher robustness and regression accuracy. By embedding the LCA module in Transformer, we build a powerful online tracker with a target-aware backbone, termed as TATrack. In addition, we propose a dynamic online update algorithm based on the classification confidence of historical information without additional calculation burden. Our tracker achieves state-of-the-art performance on multiple benchmarks, with 71.1\% AUC, 89.3\% NP, and 73.0\% AO on LaSOT, TrackingNet, and GOT-10k. The code and trained models are available on https://github.com/hekaijie123/TATrack.
Introduction
Siamese trackers commonly use target-only templates, leaving them challenged by appearance changes, rapid movement, and similar-object interference. TATrack addresses this with cross-frame target-context attention and confidence-based template updates.
Introduction
- Target-only templates provide insufficient appearance information to distinguish the target from similar background objects or handle severe appearance changes.The introduction also identifies rapid movement and static models that do not perceive changes in target state over time.Source: Introduction · p. 1
- LCA fuses a target template, historical frame, and current search frame through self- and cross-attention, extracting target location and surrounding-state information.LCA modules alternate with SWA modules to form a backbone designed for tracking.Source: Introduction · pp. 1–2
- TATrack selects historical templates using classification confidence, aiming to update online without the extra computation of two-stage approaches.The introduction contrasts this with update methods requiring two-stage inference or training.Source: Introduction · p. 2
- The paper presents LCA, a tracking-oriented backbone, and confidence-based online updating as its three main contributions.It reports comprehensive ablation and comparison experiments to evaluate the tracker.Source: Introduction · p. 2
Related Work
Prior trackers use classification-oriented backbones and varied online-update strategies, each with stated limitations. TATrack instead emphasizes target-aware feature extraction and uses historical classification confidence for template updates.
Related Work
- Classification-derived backbones lack target-background awareness, which conflicts with tracking’s need to distinguish interferers.The cited progression includes AlexNet, ResNet, and Transformer backbones in Siamese trackers.Source: Tracker Backbone · p. 2
- TATrack proposes a target-aware backbone focused on target features and auxiliary positioning information.This contrasts with directly using feature extractors designed for image classification.Source: Tracker Backbone · p. 2
- Earlier online-update methods face stated costs or constraints, while TATrack uses inference-time historical classification confidence to update templates without additional cost.The discussion cites mini-branch training and inference overhead, template contamination, secondary training, and repeated inference among prior approaches.Source: Online Update · p. 2
Method
TATrack centers on LCA, which fuses target, previous-template, and search-image information for target-aware feature extraction and correlation. The method combines this attention with a Swin-style backbone, target-state encoding, and confidence-based template updates.
Long-term Contextual Attention
- LCA jointly processes the target template, previous template, and search image with self- and cross-attention, reinforcing target features while suppressing interference.It uses target-state information, including appearance and relative position in the previous template, to help exclude similar objects.Source: Long-term Contextual Attention (LCA) · pp. 2–3, Figure 2 · p. 3
- Inter-image location encoding partitions relative-position attention into nine source-to-destination regions, preserving distinctions among target, previous-template, and search features.The method extends positional encoding to the concatenated multi-image input and divides relative position encoding into 9 independent regions.Source: Long-term Contextual Attention (LCA) · p. 3
Overall Architecture
- The Swin-style backbone alternates shift-window attention with LCA, and a neck stacks LCAs to refine target-state information before search features reach the heads.The backbone processes target, previous-template, and search features, while the regression and classification heads use search-image features.Source: Figure 3 · p. 4, Backbone · p. 4, Neck · p. 4
Neck
- The neck encodes prior target location and size with a Gaussian localization map and an MLP embedding of four-side box distances, while training randomly dithers previous-template regions.The position and size embeddings are summed to form the box embedding.Source: Neck · p. 4
Head and Loss
- The final LCA passes search features attending to the target and previous templates to classification and regression heads, trained with varifocal and generalized IoU losses.The heads produce classification and bounding-box regression maps from the search feature map.Source: Neck · p. 4, Head and Loss · p. 4
Online Update Strategy
- Dynamic updates select historical templates by classification-confidence thresholds; mean thresholds underperform on LaSOT amid prolonged disappearance or occlusion, motivating a penalty formula that relies more on prior scores.The penalty threshold is reported as more stable than the mean formula and helps resist erroneous updates during long target disappearance or occlusion.Source: Online Update Strategy · p. 5
Experiments
Experiments report state-of-the-art performance on TrackingNet and LaSOT and examine the contributions of LCA, localization information, background-bearing templates, and dynamic template updates. The ablations also identify a runtime limitation from using multiple templates.
Evaluation setup
- The experiments use joint training on TrackingNet, LaSOT, COCO, and GOT-10k; GOT-10k evaluation follows its training-set-only rule.GOT-10k results were obtained by submitting raw tracking results for online evaluation.Source: GOT-10k · p. 5, Implementation details · p. 5
Benchmark comparison
- TATrack-L achieves state-of-the-art performance on TrackingNet, and the p-mean algorithm achieves state-of-the-art performance on LaSOT.Table 2 compares results on GOT-10k, TrackingNet, and LaSOT.Source: TrackingNet · p. 5, LaSOT · p. 6, Table 2 · p. 6
LCA ablation
- Removing LCA degrades metrics across TrackingNet, LaSOT, and GOT-10k, while removing location codes also significantly degrades all three datasets.Adding two LCA layers to TATrack-S directly improves performance; without location encodings, constructing associations across images becomes difficult.Source: LCA Ablation · p. 6
Box embedding
- Removing Gaussian localization information causes greater performance degradation than removing ltrb box embedding.The authors report that localization information is more important in templates containing background; ltrb may also carry previous-template box errors into current-frame predictions.Source: Box-embedding · p. 6
Template background
- Templates containing background outperform templates without background across all reported metrics, including when the target template is not updated.The authors chose a compromise between target-only and background-bearing templates, citing performance flatness and the potential of background context.Source: With or Without Background · pp. 6–7, Table 4 · p. 7
Update strategy
- On LaSOT, a historical-mean threshold with a penalty is more stable against prolonged low-quality templates, while showing no significant negative impact on short-series datasets.The unpenalized historical mean performs well on GOT-10k but its threshold is too low to resist long-term target disappearance on LaSOT.Source: Update Experiment · p. 7, Table 5 · p. 7
LCA visualization
- LCA feature-map visualizations show that it locates the target in previous templates and search images and excludes interfering objects layer by layer.The visualizations average target, previous-frame, and search features across channels into response maps.Source: Visualization of LCA · p. 7, Figure 4 · p. 7
Limitations
- Using multiple templates, especially previous templates containing background, makes the model run low.The authors identify reducing computation while mining background-bearing templates as a direction for future work.Source: Limitations · p. 7
Conclusion
The conclusion presents Long-Term Contextual Attention (LCA) as a way to fuse target and background information across multiple time frames. It also describes TATrack as a target-aware tracker built around LCA, alongside a dynamic-threshold online template update algorithm.
Conclusion
- LCA fuses target and background information across multiple time frames.The module is presented as a fusion module.Source: Conclusion · p. 7
- TATrack uses LCA’s feature-extraction and correlation capabilities to build a target-aware tracker.Source: Conclusion · p. 7
- A dynamic-threshold online algorithm is proposed to update templates.Source: Conclusion · p. 7