Source-linked AI summary
ARTrackV2: Prompting Autoregressive Tracker Where to Look and How to Describe
Yifan Bai, Zeyang Zhao, Yihong Gong, Xing Wei
TL;DR
Visual trackers must determine where to look while adapting to how the target appears across frames, but prior methods often separate trajectory and appearance modeling or use inefficient within-frame autoregression. ARTrackV2 introduces a unified autoregressive framework that jointly evolves trajectory and appearance with parallel within-frame processing. It reports state-of-the-art benchmark performance alongside substantial speed improvements, including 86.1% AUC on TrackingNet and approximately 3× to 5× speed improvements for the top-performing model.
Problem
Prior tracking approaches commonly focus on trajectory or appearance separately, overlook temporal dependencies, and may use inefficient intra-frame autoregression or hand-tuned appearance updates.
Method
ARTrackV2 uses a unified generative framework with inter-frame autoregression to jointly evolve trajectory and reconstruct appearance, while a pure encoder processes within-frame tokens in parallel.
Results
ARTrackV2 achieves state-of-the-art benchmark performance with 86.1% AUC on TrackingNet and approximately 3× to 5× speed improvements for its top-performing model.
Takeaways & Limitations
The framework combines continuous trajectory-appearance modeling with simpler, more efficient tracking without intra-frame autoregression or hand-tuned appearance-update parameters.
Abstract
from arXiv · showhide
We present ARTrackV2, which integrates two pivotal aspects of tracking: determining where to look (localization) and how to describe (appearance analysis) the target object across video frames. Building on the foundation of its predecessor, ARTrackV2 extends the concept by introducing a unified generative framework to "read out" object's trajectory and "retell" its appearance in an autoregressive manner. This approach fosters a time-continuous methodology that models the joint evolution of motion and visual features, guided by previous estimates. Furthermore, ARTrackV2 stands out for its efficiency and simplicity, obviating the less efficient intra-frame autoregression and hand-tuned parameters for appearance updates. Despite its simplicity, ARTrackV2 achieves state-of-the-art performance on prevailing benchmark datasets while demonstrating remarkable efficiency improvement. In particular, ARTrackV2 achieves AO score of 79.5\% on GOT-10k, and AUC of 86.1\% on TrackingNet while being $3.6 \times$ faster than ARTrack. The code will be released.
1. Introduction
ARTrackV2 extends generative visual tracking by jointly modeling trajectory and appearance over time, while simplifying inference through parallel token processing and improving benchmark performance and speed.
- Motivation: Previous trackers commonly emphasize either motion prediction or appearance modeling, often overlooking temporal dependencies across frames.Some appearance-update methods use heuristic rules or learnable modules.
- Contribution: ARTrackV2 jointly evolves trajectory and appearance in a unified generative framework using appearance prompts that reconstruct the target over successive frames.A masking strategy prevents appearance tokens from relying directly on trajectory tokens when reconstructing appearance.
- Results: 3.6× speed improvement over ARTrack accompanies 71.6% AUC on LaSOT and 84.9% AUC on TrackingNet for the base model.
- Results: 73.6% AUC on LaSOT and 86.1% AUC on TrackingNet are reported for the top-performing model, with approximately 3× to 5× speed improvements.
- Contribution: Sequence data augmentation is introduced alongside time-autoregressive joint trajectory-appearance modeling to improve accuracy.
- Efficiency: Parallel processing of all within-frame tokens eliminates less efficient intra-frame autoregression while preserving inter-frame autoregression.The pure encoder architecture also supports end-to-end, single-stage training without hand-tuned template-update parameters.
2. Related Work
Related tracking methods rely heavily on template matching and specialized subtasks, while recent generative approaches recast tracking as sequence generation but still leave appearance evolution and tuning challenges.
- Tracking Framework: Template-matching trackers typically split tracking into specialized subtasks and add complex postprocessing, potentially overlooking temporal dependencies.
- Appearance Modeling: Appearance-modeling methods use discriminative scores or likelihoods for dynamic templates, often requiring hand-tuned update intervals and thresholds.
- Appearance Modeling: ARTrackV2 extends sequence generation to jointly model trajectory and appearance evolution with end-to-end single-stage training.
3. Method
ARTrackV2 jointly models trajectory and appearance across video frames while retaining inter-frame autoregression. It removes intra-frame autoregression through parallel encoding, uses appearance reconstruction and confidence guidance, and trains with sequence-aware objectives and reverse augmentation.
- Joint trajectory-appearance autoregression: ARTrackV2 jointly models trajectory and appearance evolution in an autoregressive framework, using appearance tokens as dynamic templates across frames.The initial template remains static, while appearance tokens are updated to describe the target’s temporal appearance continuously.
- Appearance reconstruction and confidence: Appearance tokens are reconstructed from the current search region, and a confidence token predicts IoU to measure appearance-token quality.The reconstruction decoder updates appearance tokens for subsequent frames, while the confidence module uses IoU as its indicator.
- Pure encoder architecture: A pure Transformer encoder processes visual, trajectory, and command tokens within each frame in parallel, eliminating intra-frame autoregression while preserving inter-frame autoregression.Tokens are formed from template and search patches, trajectory prompts, command tokens, and positional and identity embeddings.
- Appearance reconstruction and confidence: An attention-masking strategy restricts appearance-token interactions so the model does not rely only on predicted localization when learning appearance evolution.
- Sequence augmentation: Reverse augmentation expands sequence-level training data by inverting video sequences while preserving their time-series structure.Fixed- and random-interval sampling reduced accuracy because they disrupted natural temporal progression and induced spurious temporal interactions.
- Training and inference: The tracker is trained end-to-end with a structured sequence objective combining cross-entropy, SIoU, MSE, and IoU L1 losses, then propagates trajectory, appearance, and confidence tokens during inference.The loss weights balance the contributions of the four terms, while autoregressive propagation continues across subsequent frames.
4. Experiments
ARTrackV2 is evaluated across four tracking benchmarks and ablations, showing strong accuracy alongside substantially improved efficiency. Its experiments attribute these gains to a pure encoder, autoregressive appearance evolution, reconstruction choices, and sequence-level training.
- Main Results: ARTrackV2-L384 outperforms state-of-the-art trackers across all GOT-10k metrics under the one-shot protocol.ARTrackV2 is trained exclusively on the GOT-10k training split for this evaluation.
- Main Results: ARTrackV2-L384 establishes state-of-the-art performance in three metrics on TrackingNet, while ARTrackV2384 achieves the best AUC.The test set contains 511 videos covering diverse object categories.
- Main Results: 49 FPS: ARTrackV2-L384 sets a new state-of-the-art on LaSOT while running over 5× faster than SeqTrack-L384.ARTrackV2256 matches ARTrack384 despite using lower input resolution.
- Efficiency: 79.5% AO: ARTrackV2-L384 achieves a new GOT-10k state-of-the-art, while ARTrackV2256 reaches competitive performance at 94 FPS.Removing intra-frame autoregression yields nearly 3× higher inference efficiency without compromising accuracy.
- Ablation Studies: Feature reconstruction improves AO by approximately 1.1% compared with image reconstruction.The experiments identify feature reconstruction as more effective for appearance evolution.
- Ablation Studies: IoU is a better reconstruction indicator than confidence, distance, or visibility because it more closely aligns with the tracking evaluation metric.The study reports that visibility produces an unsatisfactory distribution.
5. Conclusion
ARTrackV2 unifies trajectory evolution and appearance reconstruction in an end-to-end generative tracker. It jointly reads out localization and retells appearance while using parallel within-frame processing to improve efficiency.
- ARTrackV2 jointly evolves target trajectory and reconstructs appearance through a unified generative framework.The framework propagates trajectory-appearance prompts into successive frames to strengthen inter-frame autoregression.
- Parallel processing of all within-frame tokens eliminates less efficient intra-frame autoregression.The pure encoder architecture preserves inter-frame autoregression while processing frame tokens in parallel.
- ARTrackV2 is presented as an end-to-end tracker with improvements in both performance and efficiency.
Appendix
The appendix collects supplementary benchmark comparisons and ablation studies for ARTrackV2. It covers LaSOT attributes, additional datasets, sequence format, trajectory-appearance evolution, masking, and visualizations.
- The appendix provides comprehensive attribute results on the LaSOT dataset.
- It reports performance on the TNL2K, NFS, and UAV123 datasets.
- Supplementary ablations examine sequence format, trajectory-appearance evolution, and the appearance-model masking ratio.
- The appendix also includes additional cross-attention maps and image visualizations.
A. LaSOT Attributes Results
ARTrackV2 is evaluated across LaSOT attributes and additional tracking benchmarks, with analyses of sequence formats. The reported comparisons emphasize robustness across challenging scenarios and the superiority of the selected coordinate format.
- LaSOT Attributes Results: ARTrackV2 shows notable gains on LaSOT scenarios involving Full Occlusion and Partial Occlusion.The comparison evaluates AUC separately across video attributes.
- Additional Benchmarks: ARTrackV2-L384 consistently outperforms other trackers on TNL2K, NFS, and UAV123 without compromising inference speed.These benchmarks cover multimodal pedestrian videos, high-frame-rate videos, and long-term aerial videos.
- Sequence Format Study: The study compares three coordinate sequence formats for representing target localization.The formats include corner coordinates, center-width-height coordinates, and paired top-left/bottom-right coordinates.
- Additional Benchmarks: State-of-the-art comparisons are also presented for TNL2K, NFS, and UAV123.
- Sequence Format Study: The [xmin, ymin, xmax, ymax] format surpasses the alternative formats.The [x, y, w, h] variant with separate vocabularies performs better than a unified-vocabulary variant, but multiple vocabularies introduce additional interaction.
D. Trajectory and Appearance Evolution
The trajectory-appearance evolution study tests the contribution of modeling both components. Removing either component reduces performance, while removing both converts continuous tracking into frame-level template matching.
- Removing either trajectory evolution or appearance evolution causes a notable performance decline.
- Jointly modeling trajectory and appearance is identified as important for optimal tracking performance.
- Removing both evolution mechanisms produces a more pronounced accuracy decline and breaks time-continuous tracking.The resulting variant becomes a frame-level template-matching approach.
E. Masking Ratio for Appearance Modeling
ARTrackV2 uses exceptionally high masking for appearance modeling, with accuracy improving as the masking ratio rises up to 90%.
- 90% masking ratio marks the point up to which model accuracy consistently improves.The study attributes this behavior to temporal information being less redundant and more relevant than textual and image cues.
F. More Visualization of Appearance Tokens
The appearance-token visualizations examine cross-attention between appearance tokens and search regions, while highlighting both challenging-case adaptability and limitations in token visualization quality.
- Appearance tokens are visualized through cross-attention maps between the tokens and search regions.Figure 8 shows search regions and templates alongside appearance-token attention visualizations.
- The visualizations indicate adaptability and versatility, particularly in challenging scenarios.
- Low-resolution appearance-token representations produce blurry grayscale images with unclear boundaries.The visualization uses t-SNE to reduce high-dimensional tokens into one-dimensional spaces for grayscale rendering.
G. Visualization of Image Reconstruction
ARTrackV2 reconstructs target appearance in the image pixel domain to visualize appearance evolution across variation, out-of-view, and illumination scenarios.
- Image-domain reconstruction provides an intuitive analysis of appearance evolution beyond feature-domain cross-attention maps.Figure 9 compares target appearances with reconstructed appearance images at fixed frame intervals.
- The tracker captures long-term appearance evolution and avoids erroneous appearance propagation that could mislead localization.When the target reappears, its appearance is accurately captured and faithfully reconstructed.
- ARTrackV2 adapts reconstructed appearance as targets move out of view and as illumination gradually changes.