Source-linked AI summary
ODTrack: Online Dense Temporal Token Learning for Visual Tracking
Yaozong Zheng, Bineng Zhong, Qihua Liang, Zhiyi Mo, Shengping Zhang, Xianxian Li
TL;DR
Visual trackers commonly rely on sparse reference-search image pairs, limiting temporal associations across video frames. ODTrack models arbitrary-length video streams through autoregressive temporal token propagation, achieving state-of-the-art results on seven benchmarks while avoiding intricate online updates. Its global video-level approximation remains constrained by GPU resources and computational cost.
Problem
Offline image-pair tracking uses sparse reference and search frames, limiting temporal correlations and continuous cross-frame associations.
Method
ODTrack reformulates tracking as autoregressive token-sequence propagation over arbitrary-length video streams, using temporal tokens to capture target trajectories.
Results
ODTrack achieves state-of-the-art tracking results on seven benchmarks.
Takeaways & Limitations
Iterative token propagation avoids complex online update strategies and enables more efficient model representation and computation.
Takeaways & Limitations
The global video-level approximation is constrained by GPU resources and is not yet cost-effective, motivating lower-complexity and lightweight transformer modeling.
Abstract
from arXiv · showhide
Online contextual reasoning and association across consecutive video frames are critical to perceive instances in visual tracking. However, most current top-performing trackers persistently lean on sparse temporal relationships between reference and search frames via an offline mode. Consequently, they can only interact independently within each image-pair and establish limited temporal correlations. To alleviate the above problem, we propose a simple, flexible and effective video-level tracking pipeline, named \textbf{ODTrack}, which densely associates the contextual relationships of video frames in an online token propagation manner. ODTrack receives video frames of arbitrary length to capture the spatio-temporal trajectory relationships of an instance, and compresses the discrimination features (localization information) of a target into a token sequence to achieve frame-to-frame association. This new solution brings the following benefits: 1) the purified token sequences can serve as prompts for the inference in the next video frame, whereby past information is leveraged to guide future inference; 2) the complex online update strategies are effectively avoided by the iterative propagation of token sequences, and thus we can achieve more efficient model representation and computation. ODTrack achieves a new \textit{SOTA} performance on seven benchmarks, while running at real-time speed. Code and models are available at \url{https://github.com/GXNU-ZhongLab/ODTrack}.
Introduction
ODTrack addresses the limited temporal associations of offline image-pair tracking by modeling video streams with autoregressive token propagation. It uses arbitrary-length video inputs to densely associate target context across frames and reports state-of-the-art results on seven benchmarks.
- Introduction: Current trackers often sample only reference and search frames, limiting continuous cross-frame associations.Offline methods typically extract features from two frames, match or fuse target information, and predict the current localization.
- Introduction: Online updating methods add temporal information but still use sparse frames and do not effectively explore propagation across search frames.These methods may update templates or model parameters, while relying on additional update frames and decision mechanisms.
- Introduction: ODTrack reformulates tracking as autoregressive token-sequence propagation over video streams, densely associating contextual relationships across frames.The framework extends inputs from image pairs to arbitrary-length video streams and captures spatio-temporal target trajectories.
- Introduction: Temporal token propagation compresses discriminative target features into prompts that guide inference on future frames and avoid complex online updates.The token sequence iteratively carries past target information into subsequent frame inference.
- Introduction: ODTrack achieves state-of-the-art tracking results on seven benchmarks.The reported benchmarks include LaSOT, TrackingNet, GOT10K, LaSOText, VOT2020, TNL2K, and OTB100.
Related Work
Prior visual trackers mainly rely on Siamese image-pair matching, while online-update methods introduce complex optimization and manual rules. ODTrack instead propagates temporal tokens across video frames to model trajectory information and simplify temporal tracking.
- Related Work: Popular trackers are dominated by the Siamese paradigm, which performs tracking through image-pair matching.
- Related Work: Transformer-based and other architectural enhancements improve feature interactions, but much prior work remains centered on pairwise tracking.Related approaches include prediction heads, cross-correlation modules, stronger backbones, and attention mechanisms.
- Related Work: Single-object tracking has comparatively limited research using spatio-temporal trajectory information, unlike multi-object tracking.
- Related Work: Online-update trackers use adaptive templates or parameter optimization to incorporate temporal cues.Examples include template fusion and sophisticated online model-parameter optimization.
- Related Work: These methods involve complex loss functions, multi-stage training, manual update rules, and limited propagation of temporal cues across search frames.
- Related Work: ODTrack introduces dense token-based context propagation to model target motion and trajectory information while avoiding intricate update processes.Its framework uses video-level inputs and temporal token propagation for visual tracking.
Approach
ODTrack extends visual tracking from sparse image-pair matching to video-level modeling with autoregressive temporal token propagation. It samples video clips, compresses target trajectory information into tokens, and propagates them to guide subsequent-frame inference while avoiding complex online updates.
- Video-level tracking: ODTrack reformulates visual tracking as video-level token sequence propagation, addressing the limited inter-frame associations of image-pair trackers.The framework accepts reference and search frames of arbitrary lengths and models target spatio-temporal trajectory relationships.
- Video-level tracking: The framework extends the tracker input from reference-search pairs to multiple reference frames, search frames, and a temporal token sequence.Its formulation is B ← Ψ : {R1, R2, ..., Rk, S1, S2, ..., Sn, T}.
- Video sequence sampling: ODTrack samples multiple frames across a larger interval to capture richer long-term spatio-temporal trajectory information than short-term image-pair sampling.The resulting video clips can contain arbitrary numbers of reference and search frames.
- Temporal token attention: Two temporal token attention mechanisms inject video content and temporal token vectors into attention to extract and integrate instance-level inter-frame correlations.The mechanisms are based on compression-propagation and include concatenated and separated token attention.
- Temporal token propagation: Each temporal token compresses target appearance, localization, and trajectory information, then serves as a prompt for inferring the next video frame.The token from frame t is propagated autoregressively to subsequent frames.
- Temporal token propagation: Online iterative token propagation avoids complex update strategies used by prior trackers and enables more efficient model representation and computation.The paper contrasts this design with methods requiring additional branches, optimization processes, and update decision rules.
Experiments
ODTrack evaluates video-level token propagation across multiple tracking benchmarks, ablations, visualizations, and efficiency settings. It reports state-of-the-art or competitive results while examining propagation designs, sampling choices, and computational limitations.
- Implementation Details: 32 fps inference is achieved on an NVIDIA 2080Ti, with three reference frames incorporated at equal intervals during inference.Search frames and temporal token vectors are processed frame by frame.
- Comparison with the SOTA: 77.0% AO on GOT10K surpasses ARTrack’s 75.5% AO, supporting the benefit of video-level sample strategy.The authors attribute this benefit to releasing the potential of video-level modeling.
- Comparison with the SOTA: Against ARTrack on LaSOT, ODTrack-B gains 0.6%, 1.5%, and 1.5% in AUC, PNorm, and P score, respectively.Attribute evaluation also reports better performance on multiple challenge attributes.
- Comparison with the SOTA: Compared with SeqTrack on TrackingNet, ODTrack improves success, normalized precision, and precision score by 1.2%, 1.3%, and 1.3%, respectively.The authors interpret these results as evidence of strong generalization capabilities.
- Comparison with the SOTA: On LaSOText, ODTrack reaches 52.4% AUC, 63.9% PNorm, and 60.1% P score, exceeding ARTrack by 0.5%, 1.9%, and 1.6%, respectively.The paper associates these results with more stable localization in complex scenarios.
- Comparison with the SOTA: ODTrack-B and ODTrack-L achieve EAO scores of 58.1% and 60.5% on VOT2020 mask evaluations, and best performance on TNL2K and OTB100.The VOT2020 evaluation uses Alpha-Refine for segmentation-mask prediction.
- Ablation Study: Removing token propagation decreases LaSOT AUC by 1.2%, while increasing search-clip length from 2 to 3 improves AUC by 0.3%.The ablations also report gains when the sampling range expands from 200 to 1200, whereas overly long clips impose a learning burden.
- Visualization and Limitation: The temporal token attention visualization shows tokens propagating and attending to object motion trajectories during tracking.The authors connect this behavior with accurate target localization.
Conclusion
ODTrack reformulates visual tracking as autoregressive token propagation across video frames, using video sampling and temporal attention to model spatio-temporal relationships while avoiding intricate online updates. Experiments report promising results on seven tracking benchmarks.
- ODTrack reformulates visual tracking as a token propagation task that densely associates contextual relationships across video frames autoregressively.
- The framework introduces video sequence sampling and two temporal token propagation attention mechanisms for video-level spatio-temporal modeling.
- The proposed mechanisms simplify video-level modeling and avoid intricate online update strategies.
- ODTrack achieves promising results on seven tracking benchmarks.