Source-linked AI summary
Autoregressive Queries for Adaptive Tracking with Spatio-TemporalTransformers
Jinxia Xie, Bineng Zhong, Zhiyi Mo, Shengping Zhang, Liangtao Shi, Shuxiang Song, Rongrong Ji
TL;DR
Visual tracking needs richer spatio-temporal modeling than approaches built around hand-crafted aggregation components provide. AQATrack uses autoregressive queries and spatio-temporal fusion to combine static and instantaneous appearance, achieving competitive or state-of-the-art results across reported benchmarks, while longer-term modeling remains constrained by GPU memory.
Problem
Most top-performing trackers rely on hand-crafted spatio-temporal aggregation components, leaving rich information for changing target appearance underexplored.
Method
AQATrack learns spatio-temporal information with learnable autoregressive queries in a sliding window and fuses them with initial-template features through STM.
Results
AQATrack achieves state-of-the-art or competitive performance across six tracking benchmarks, including 71.4% and 72.7% AUC for AQATrack-256 and AQATrack-384 on LaSOT.
Takeaways & Limitations
The results support autoregressive query-based modeling as an effective way to capture target state changes and motion trends for robust tracking.
Takeaways & Limitations
Longer-term spatio-temporal modeling has not been thoroughly explored because of GPU memory limitations.
Abstract
from arXiv · showhide
The rich spatio-temporal information is crucial to capture the complicated target appearance variations in visual tracking. However, most top-performing tracking algorithms rely on many hand-crafted components for spatio-temporal information aggregation. Consequently, the spatio-temporal information is far away from being fully explored. To alleviate this issue, we propose an adaptive tracker with spatio-temporal transformers (named AQATrack), which adopts simple autoregressive queries to effectively learn spatio-temporal information without many hand-designed components. Firstly, we introduce a set of learnable and autoregressive queries to capture the instantaneous target appearance changes in a sliding window fashion. Then, we design a novel attention mechanism for the interaction of existing queries to generate a new query in current frame. Finally, based on the initial target template and learnt autoregressive queries, a spatio-temporal information fusion module (STM) is designed for spatiotemporal formation aggregation to locate a target object. Benefiting from the STM, we can effectively combine the static appearance and instantaneous changes to guide robust tracking. Extensive experiments show that our method significantly improves the tracker's performance on six popular tracking benchmarks: LaSOT, LaSOText, TrackingNet, GOT-10k, TNL2K, and UAV123.
1. Introduction
Visual object tracking must handle camera movement, deformation, occlusion, distractions, and continually changing target appearance. AQATrack addresses limited spatio-temporal modeling with autoregressive queries and fusion of static and instantaneous appearance information.
- Visual tracking estimates an arbitrary target’s position and shape from its initial status in video sequences despite changing conditions.The cited challenges include camera movement, self-deformation, occlusion, and similar-object distractions.
- Existing trackers often struggle to discriminate targets using only static appearance from the initial template.Only a few models explore spatio-temporal information to capture appearance changes.
- AQATrack uses learnable autoregressive queries in a sliding window to capture instantaneous target appearance changes.Temporal attention enables interaction among existing queries to generate a query for the current frame.
- A spatio-temporal fusion module combines the initial static appearance with instantaneous changes to guide target localization.The module is designed to aggregate spatio-temporal information without hand-designed components.
- 71.4% and 72.7% AUC are achieved by AQATrack-256 and AQATrack-384, respectively, on the long-term LaSOT benchmark.The authors report state-of-the-art performance on six challenging benchmarks.
2. Related Work
Related tracking research has emphasized spatial feature extraction, appearance or state updates, and query-based representations for changing objects. AQATrack combines these directions through a spatial encoder, temporal decoder, and spatio-temporal fusion module.
- Visual object tracking based on spatial features: Spatial-feature trackers commonly use powerful backbones, including AlexNet and ResNet, to extract representations from template and search images.
- Tracking combining spatio-temporal information: Spatio-temporal tracking methods model target changes through dynamic templates or learned state and motion features.These approaches update appearance representations to improve matching between template and search images.
- AQATrack framework: Figure 2 organizes the framework into a spatial encoder, temporal decoder, spatio-temporal fusion module, and prediction head.The temporal decoder learns autoregressive target queries incorporating temporal information.
- The utilization of query: Query-based methods use queries to collect information about changing object states in video tasks such as segmentation, multiple-object tracking, and detection.
3. Method
AQATrack combines a spatial encoder, a temporal decoder with autoregressive queries and temporal attention, and a parameter-free spatio-temporal fusion module. The resulting features support center-based target prediction using classification, IoU, and L1 losses.
- 3.1. Overview: The tracker consists of a spatial encoder, temporal decoder, spatio-temporal fusion module, and prediction head.The encoder extracts spatial representations, the decoder learns temporal changes, STM combines spatial and temporal information, and the head predicts tracking results.
- 3.2. Spatial Encoder: The spatial encoder gradually downsamples images using 4×4 patch embeddings, eight MLP layers, and two merging layers.This design is intended to preserve correlations between patches better than performing large downsampling at once.
- 3.3. Temporal Decoder: The temporal decoder combines spatial features with temporal queries through temporal attention, multi-head attention, and feedforward layers.Its inputs are spatial features from the encoder and temporal queries.
- 3.3. Temporal Decoder: Autoregressive target queries represent previous and current-frame information, with current queries propagated through a sliding window of prior queries.Qcur captures the current target state, while Qpre contains queries passed from earlier frames; the window length is m.
- 3.3. Temporal Decoder: Temporal attention uses Qcur to generate queries from Qall, enabling interaction among existing queries to model target-state changes and motion trends.This mechanism is designed because self-attention gives equal attention to all queries and cannot isolate pure temporal information from Qpre.
- 3.4. Spatio-temporal Information Fusion Module: STM computes dot-product similarities between search spatial features and temporal information, then uses them to enhance spatial features elementwise.The operation is parameter-free, and the similarity scores highlight locations where the target may be located.
4. Experiments
Experiments evaluate AQATrack across six tracking benchmarks and analyze its implementation, efficiency, components, and temporal modeling choices. The tracker achieves strong benchmark performance, while ablations show benefits from spatio-temporal modules and temporal attention.
- Results and Comparisons: AQATrack was evaluated against state-of-the-art trackers on LaSOT, LaSOText, GOT-10k, TNL2K, UAV123, and TrackingNet.The evaluation follows benchmark-specific protocols, including one-shot training for GOT-10k.
- Results and Comparisons: 71.4% and 72.7% AUC scores were achieved by AQATrack-256 and AQATrack-384 on LaSOT, respectively.AQATrack-384 surpassed other trackers, while AQATrack also performed competitively across LaSOT challenges.
- Results and Comparisons: 62.2% Pnorm was achieved on LaSOText by AQATrack-256, exceeding ARTrack by 1.4%.AQATrack-384 also outperformed previous trackers across all three reported metrics.
- Results and Comparisons: 57.8% and 59.3% AUC scores were obtained on TNL2K by AQATrack-256 and AQATrack-384, respectively.Both model variants are reported as strong performers on this 700-video test dataset.
- Ablation Study and Analysis: Removing spatio-temporal modules reduced LaSOT AUC to 70.5%, 0.9% below the full AQATrack, while replacing temporal attention also lowered performance.A self-attention substitute achieved 70.7% AUC, 80.9% Pnorm, and 77.8% P, with AUC 0.9% below the designed temporal attention.
5. Conclusion
AQATrack models continuous spatio-temporal information with temporal attention and autoregressive queries, achieving competitive tracking performance. Longer-term modeling remains insufficiently explored because of GPU memory limitations.
- AQATrack models continuous spatio-temporal information using a temporal decoder with temporal attention and temporal queries.
- The temporal queries capture motion trends that help discriminate the target from similar objects.
- The method achieves competitive performance compared to previous state-of-the-art trackers.
- GPU memory limitations prevented thorough exploration of longer-term spatio-temporal modeling.