Source-linked AI summary
Explicit Visual Prompts for Visual Object Tracking
Liangtao Shi, Bineng Zhong, Qihua Liang, Ning Li, Shengping Zhang, Xianxian Li
TL;DR
Visual trackers struggle to exploit consecutive-frame context while managing template updates for changing target appearances. EVPTrack propagates spatio-temporal information with tokens and uses spatio-temporal and multi-scale prompts in a transformer encoder, achieving competitive real-time performance across six benchmarks. Its training-video length is constrained by limited GPU memory, restricting full exploitation of spatio-temporal information.
Problem
Existing trackers often rely on complex template updating while insufficiently exploiting context between consecutive frames, creating a when-and-how-to-update dilemma.
Method
EVPTrack propagates spatio-temporal information with tokens and generates explicit spatio-temporal and multi-scale prompts fused with image tokens by transformer encoders.
Results
EVPTrack achieves state-of-the-art results on six benchmarks, including 72.7% success score (AUC) on the LaSOT test set.
Takeaways & Limitations
EVPTrack achieves competitive real-time performance by exploiting spatio-temporal and multi-scale information without complicated template-updating strategies.
Takeaways & Limitations
Limited GPU memory restricts training-video length, hindering the tracker from fully exploiting spatio-temporal information.
Abstract
from arXiv · showhide
How to effectively exploit spatio-temporal information is crucial to capture target appearance changes in visual tracking. However, most deep learning-based trackers mainly focus on designing a complicated appearance model or template updating strategy, while lacking the exploitation of context between consecutive frames and thus entailing the \textit{when-and-how-to-update} dilemma. To address these issues, we propose a novel explicit visual prompts framework for visual tracking, dubbed \textbf{EVPTrack}. Specifically, we utilize spatio-temporal tokens to propagate information between consecutive frames without focusing on updating templates. As a result, we cannot only alleviate the challenge of \textit{when-to-update}, but also avoid the hyper-parameters associated with updating strategies. Then, we utilize the spatio-temporal tokens to generate explicit visual prompts that facilitate inference in the current frame. The prompts are fed into a transformer encoder together with the image tokens without additional processing. Consequently, the efficiency of our model is improved by avoiding \textit{how-to-update}. In addition, we consider multi-scale information as explicit visual prompts, providing multiscale template features to enhance the EVPTrack's ability to handle target scale changes. Extensive experimental results on six benchmarks (i.e., LaSOT, LaSOT\rm $_{ext}$, GOT-10k, UAV123, TrackingNet, and TNL2K.) validate that our EVPTrack can achieve competitive performance at a real-time speed by effectively exploiting both spatio-temporal and multi-scale information. Code and models are available at https://github.com/GXNU-ZhongLab/EVPTrack.
Introduction
EVPTrack addresses the limitations of fixed and dynamically updated templates by propagating spatio-temporal information with tokens and generating explicit visual prompts. It combines spatio-temporal and multi-scale information, achieving state-of-the-art results across six benchmarks, including 72.7% LaSOT success AUC.
- Fixed-template trackers ignore frame-to-frame associations, while dynamic-template trackers require careful update design to balance timely updates and redundant information.
- EVPTrack propagates spatio-temporal information between consecutive frames with tokens instead of template updating mechanisms.
- EVPTrack generates spatio-temporal and multi-scale prompts, then feeds them with image tokens into an Image-Prompt Encoder for relational modeling.Multi-scale prompts use fine-grained template features to model targets at different scales.
- 72.7% success score (AUC) was achieved on the LaSOT test set, while the tracker attained state-of-the-art results on six benchmarks.
Related Work
Prior trackers use initial or dynamic templates to match targets, but dynamic approaches involve complex update modules and strategies. EVPTrack instead propagates spatio-temporal information with tokens to avoid these update strategies.
- Initial-template trackers match search frames using information from the target’s first frame, whereas dynamic-template trackers update target information online.
- Dynamic-template methods include online model updating, memory networks, and temporal adaptive convolution for exploiting historical or consecutive-frame information.
- Existing dynamic-template approaches require complex update modules and strategies and cannot fully utilize context between consecutive frames.
- EVPTrack uses tokens to propagate spatio-temporal information, avoiding complex update strategies.
Method
EVPTrack is an end-to-end tracker that propagates spatio-temporal information through tokens and converts spatio-temporal and multi-scale features into explicit visual prompts. These prompts are fused with image features to support target appearance and scale changes without elaborate update modules.
- Framework: EVPTrack combines an Image-Prompt Encoder, Spatio-Temporal Encoder, and Prompt Generator in a simple end-to-end tracking framework.The Image-Prompt Encoder fuses explicit prompts with image features, while the other modules propagate information and generate prompts.
- Inputs: The tracker inputs template and search images together with spatio-temporal tokens from the previous frame.The template comes from the target region in the first frame, while the search image comes from a subsequent candidate region.
- Inference and Training: Explicit prompts guide feature extraction and fusion directly, reducing computational burden and avoiding additional elaborate update modules.The search features are ultimately used to locate the target, with prediction trained using classification, L1, and GIoU losses.
- Spatio-Temporal Encoder: The Spatio-Temporal Encoder fuses template, search, and spatio-temporal tokens to produce new tokens for propagating information across successive frames.At the first frame, spatio-temporal tokens are initialized from template tokens; later, template tokens from the encoder output become the new propagated tokens.
- Spatio-Temporal Encoder: Continuous token propagation exploits spatio-temporal information without relying on intermittent template updates or deciding when to update them.The mechanism is designed to avoid heavy accumulation of errors during long-term tracking.
- Prompt Generators: EVPTrack generates multi-scale and spatio-temporal prompts to capture appearance changes and maintain tracking under target scale variation.The multi-scale generator aggregates features from multiple template resolutions, while the spatio-temporal generator average-pools propagated tokens and applies a fully connected network.
Experiments
EVPTrack achieves strong results across multiple tracking benchmarks and attribute challenges while maintaining real-time efficiency. Ablations show that combining spatio-temporal and multi-scale prompts improves tracking, although training sequence length is constrained by GPU memory.
- Benchmark Results: EVPTrack-384 obtains 72.7% AUC, 82.9% P, and 80.3% Pnorm on LaSOT.
- Benchmark Results: EVPTrack-384 reaches 53.7% AUC and 61.9% P on LaSOText, outperforming EVPTrack-224 by 5%.The authors attribute the disparity to insufficient background context at lower resolution.
- Benchmark Results: EVPTrack-384 achieves 84.4% AUC, 89.1% P, and 84.2% Pnorm on TrackingNet, while also obtaining 76.6% AO and 86.7% SR0.5 on GOT-10k.
- Benchmark Results: The tracker achieves state-of-the-art performance on TNL2K and performs best among compared trackers on UAV123.EVPTrack-224 obtains 57.5% AUC on TNL2K.
- Ablation Studies: Adding spatio-temporal and multi-scale prompts improves the baseline, with the best performance achieved when both prompt types are used.Multi-scale prompts provide fine-grained template features, while spatio-temporal prompts exploit information between consecutive frames.
- Ablation Studies: Increasing training sequence length from 4 to 8 raises LaSOT AUC by 0.5%, but increasing it from 8 to 16 decreases AUC by 1.2%.The authors attribute the decrease partly to reduced generalization when only two videos are used per iteration and identify GPU memory as a limitation.
- Efficiency: EVPTrack-224 runs at over 71fps, while EVPTrack-384 uses 2x fewer FLOPs than SeqTrack-B384 and achieves 1.2% higher LaSOT AUC.
Conclusions
EVPTrack propagates spatio-temporal information through tokens and represents spatio-temporal and multi-scale information as explicit visual prompts. Experiments on six visual tracking benchmarks show competitive real-time performance.
- EVPTrack propagates spatio-temporal information between consecutive frames, eliminating the need to focus on when-to-update.
- The framework abstracts spatio-temporal and multi-scale information into explicit visual prompts.
- The prompts are fused with image tokens through transformer encoders without customized additional modules, avoiding complicated updating strategies.
- Experiments on six visual tracking benchmarks show competitive performance at real-time speed.