Source-linked AI summary
Continuous Token-Level Spatio-Temporal Context Modeling for Visual Object Tracking
Ding Xia, Meiqin Liu, Jing Zhou, Jian Lan
TL;DR
Existing trackers often rely on discrete sampling to extract spatio-temporal cues, while one-stream approaches also struggle to model relations across search frames. TLCTrack continuously updates salient tokens and uses MUA, SSTC, and TSTB to process spatial and temporal context. It reports superior performance over previous state-of-the-art trackers across five benchmarks.
Problem
Existing approaches use discrete sampling for spatio-temporal cues, and one-stream methods hinder relation modeling across search frames.
Source: §I · p. 1Method
TLCTrack models token-level spatio-temporal context through continuously updated salient tokens and uses MUA, SSTC, and TSTB to integrate and refine spatial and temporal features.
Source: §I · pp. 1–2Results
Across five benchmarks, TLCTrack achieves superior performance over previous state-of-the-art trackers; on GOT10k, TLCTrack384 achieves 77.5% AO, outperforming previous state-of-the-art trackers.
Source: B. Results and Comparisons · p. 5, §IV · p. 6Takeaways & Limitations
The reported results support continuous salient-token updating as a framework for modeling spatio-temporal context in visual tracking.
Source: §I · p. 1, §IV · p. 6
Abstract
from arXiv · showhide
Spatio-temporal context has become increasingly crucial for visual tracking. However, most existing approaches extract spatio-temporal cues via discrete sampling strategies, which inherently deviate from the continuity of spatio-temporal context, thereby deteriorating tracking performance. To address this challenge, we propose TLCTrack, a novel tracking framework that models token-level spatio-temporal context through continuously updated salient tokens, enabling more accurate target representation. Specifically, TLCTrack incorporates three components: Masked Unidirectional Attention (MUA), Spatial Salient Token Collection (SSTC), and Temporal Salient Token Bank (TSTB) modules. By explicitly integrating spatio-temporal context, MUA extracts discriminative targetaware spatial features in the search region. To avoid the negative impact of background on feature learning, SSTC progressively suppresses background interference, thereby enhancing target spatial representation. Finally, TSTB captures high-quality spatio-temporal information through continuous salient token updates. Extensive experiments on five benchmarks demonstrate that our method achieves superior performance over state-of-the-art trackers. Code and models are available at https://github.com/xiading123/TLCTrack.
I. INTRODUCTION
Existing tracking pipelines struggle to model spatio-temporal context continuously, particularly in one-stream and sampling-based approaches. TLCTrack addresses this with continuously updated salient tokens and modules for integrating temporal information and refining target representations.
I. INTRODUCTION
- One-stream methods based on template-search pairs hinder relation modeling across search frames and fail to capture spatio-temporal context.The paper identifies this limitation as a factor limiting tracking performance.Source: §I · p. 1
- Feature-update and template-update approaches use sampling-based discrete updates, which miss continuous context and incur computational burden.Fig. 1 contrasts offline learning, discrete update strategies, and token-level continuous updating.Source: §I · p. 1, Figure 1 · p. 1
- TLCTrack models token-level spatio-temporal context by continuously collecting and updating salient tokens across a video sequence.This progressive refinement is intended to capture high-quality target information.Source: §I · pp. 1–2
- MUA integrates salient-token context into search features, SSTC suppresses noisy background regions, and TSTB updates salient tokens while filtering invalid temporal features.The paper presents these modules as supporting target-aware spatial features and high-quality spatio-temporal information.Source: §I · p. 2
- The attention comparison shows TLCTrack focusing on target-relevant regions and achieving better tracking performance than OSTrack.The figure contrasts search results and attention maps at time t.Source: §I · p. 2, Figure 2 · p. 2
II. METHOD
TLCTrack combines a backbone with spatial and temporal salient-token modules and a prediction head. Its MUA uses unidirectional attention to integrate temporal context into search features while preserving temporal-token independence.
A. Overview
- The architecture combines a backbone with MUA, SSTC for collecting spatial salient tokens, TSTB for temporal salient tokens, and a prediction head.TSTB collects and updates temporal tokens; backbone search features then feed the prediction head.Source: A. Overview · p. 2
B. Masked Unidirectional Attention Module
- MUA uses unidirectional masking to regulate relations among template, search, and temporal tokens.Template tokens use self-attention, search tokens can aggregate from all three categories, and temporal tokens remain invariant.Source: B. Masked Unidirectional Attention… · p. 2
- MUA integrates temporal and template information into search features while keeping temporal tokens unchanged across frames.The design maintains temporal-feature independence and supports discriminative target-aware spatial features for localization.Source: B. Masked Unidirectional Attention… · p. 2
C. Spatial Salient Token Collection Module
SSTC progressively collects spatial salient tokens to improve target representations, addressing noise that masked relation modeling does not fully block.
C. Spatial Salient Token Collection Module
- SSTC progressively collects spatial salient tokens, filtering background noise to improve target representations and tracking robustness; masked relation modeling leaves residual noise.The module is intended to enhance target spatial representations.Source: C. Spatial Salient Token Collection… · pp. 2–3
- A score vector a quantifies candidate search-token importance, and high-score tokens are collected into Es.The token keeping ratio is ρ = Ns/N, with Is denoting the high-score token indices.Source: C. Spatial Salient Token Collection… · pp. 2–3
- A learnable-query adapter integrates Q and X; an auxiliary head predicts from X′′ and supplies gradients to train the adapter and Q.An auxiliary loss is added to the overall loss with discount factor λaux.Source: C. Spatial Salient Token Collection… · p. 3
- Stopping gradients through X before equation (8) decouples token collection from feature extraction.Source: C. Spatial Salient Token Collection… · p. 3
D. Temporal Salient Token Bank Module
TSTB combines long- and short-term salient tokens, updating them to preserve temporal information and recent appearance cues for future prediction and robustness against distractors.
D. Temporal Salient Token Bank Module
- TSTB forms Et by concatenating long-term Elt and short-term Est tokens, initialized from template tokens with both token counts equal to Nz.Source: D. Temporal Salient Token Bank Module · p. 3
- Long-term Elt is updated from current Et using mt; high-score indices Ilt are selected using the prediction-head score map S.Source: D. Temporal Salient Token Bank Module · p. 3
- For Est, TSTB aggregates spatial tokens Es with prediction results into F′, then selects tokens using ms and high-score indices Ist.The backbone feature map F is constructed by padding and reshaping Es, and Sbin is the binary score map converted from S.Source: D. Temporal Salient Token Bank Module · pp. 3–4
- Updated long- and short-term token sets are concatenated to form E′t.Source: D. Temporal Salient Token Bank Module · p. 4
- Elt preserves high-quality temporal information across frames while Est selectively captures the latest target appearance; together they maintain spatio-temporal context for future prediction.The bank adaptively integrates both token sets to improve robustness against distractors.Source: D. Temporal Salient Token Bank Module · pp. 3–4
E. Head and Loss
The heads produce classification and bounding-box outputs, while the training objective combines classification and regression losses.
E. Head and Loss
- Each head uses three convolutional branches to output a classification score map, local offset, and bounding-box size.The classification loss is focal loss; regression uses L1 and GIoU losses.Source: E. Head and Loss · p. 4
- The auxiliary-head loss enters the overall loss with a 0.5 discount factor.The stated loss weights are λL1 = 5 and λiou = 2.Source: E. Head and Loss · p. 4
- Table II compares model parameters, FLOPs, inference speed, and memory overhead on an RTX 3090 GPU.Source: Table II · p. 4
A. Implementation Details
Implementation uses ViT-Base with SSTC at selected blocks, two input-size variants, and multi-frame training that iteratively updates temporal tokens.
A. Implementation Details
- TLCTrack uses a ViT-Base backbone, with SSTC at blocks 4, 7, and 10 and keeping ratio ρ = 0.8.The two variants use template/search sizes of 128×128/256×256 and 192×192/384×384.Source: A. Implementation Details · p. 4
- Training uses TrackingNet, LaSOT, COCO, and GOT10k splits, with batches simulating context transmission across three templates and four search frames.Temporal tokens are initialized from two templates, then updated sequentially as the third template is paired with each search frame.Source: A. Implementation Details · p. 4
- The model trains for 300 epochs, except for GOT10k, which uses 100 epochs.Training uses two RTX 4090 GPUs with total batch size 64; inference inputs the template and current search frame sequentially.Source: A. Implementation Details · p. 5
B. Results and Comparisons
Across five benchmarks, TLCTrack reports state-of-the-art results on GOT10k, TNL2K, NFS, and OTB100, and competitive performance on TrackingNet.
B. Results and Comparisons
- 77.5% AO on GOT10k and 60.1% AUC / 62.7% P on TNL2K are reported as state-of-the-art; NFS reaches 70.7% AUC.The NFS result is reported as outperforming previous state-of-the-art approaches.Source: B. Results and Comparisons · p. 5
- TLCTrack384 runs at 67 fps on TrackingNet, versus 43 fps for ARTrack384, while TLCTrack is competitive against most prior trackers.Source: B. Results and Comparisons · p. 5
- The GOT10k test-split examples show TLCTrack maintaining stable target localization in qualitative comparisons with leading trackers.Source: Figure 5 · p. 5
- On OTB100, TLCTrack surpasses state-of-the-art methods across a benchmark with eleven common challenges.The figure presents the state-of-the-art comparison.Source: B. Results and Comparisons · p. 5
C. Qualitative Results
Qualitative comparisons show TLCTrack maintaining accurate, stable localization under cluttered backgrounds and target appearance variation.
C. Qualitative Results
- Under clutter and target appearance changes, OSTrack and MixFormer fail while TLCTrack maintains accurate, stable localization.The authors attribute the observed robustness to leveraging spatio-temporal context via salient tokens.Source: C. Qualitative Results · p. 5
D. Ablation Study and Analysis
TLCTrack achieves superior performance over previous state-of-the-art trackers on five benchmarks. GOT10k ablations examine its component progression and the effects of SSTC’s keeping ratio and auxiliary-loss weight.
D. Ablation Study and Analysis
- #2 gains 1.3% AO over baseline #1 after adding TSTB; MUA outperforms naive cross-attention, and SSTC brings the model to 75.4% AO.The ablation describes these modules as a progressive design with strong synergy.Source: D. Ablation Study and Analysis · p. 5, Table III · p. 6
- ρ = 0.8 gives peak performance; lower ratios may overlook target-aware spatial features, while higher ratios fail to adequately suppress background noise.ρ = 1 means SSTC is not adopted.Source: D. Ablation Study and Analysis · p. 5, Table III · p. 6
- λaux = 0.5 gives optimal performance; appropriate weighting helps SSTC learn discriminative spatial representations, while improper weights degrade its generalization.The authors link the learned representations to accurate collection of Es.Source: D. Ablation Study and Analysis · pp. 5–6, Table III · p. 6
IV. CONCLUSION
- TLCTrack achieves superior performance over previous state-of-the-art trackers on five benchmarks.The conclusion also reports remarkable robustness and capability.Source: §IV · p. 6