Source-linked AI summary
Similarity-Guided Layer-Adaptive Vision Transformer for UAV Tracking
Chaocan Xue, Bineng Zhong, Qihua Liang, Yaozong Zheng, Ning Li, Yuanliang Xue, Shuxiang Song
TL;DR
Lightweight ViT trackers contain redundant, repetitive representations, making complete architectures cumbersome for efficiency-sensitive UAV tracking. The paper introduces similarity-guided layer adaptation, which disables similar layers and selects one representative layer, producing SGLATrack. Across six benchmarks, SGLATrack delivers state-of-the-art tracking speed with competitive precision while reducing model computation.
Problem
Complete ViT architectures are too cumbersome for UAV tracking, where limited onboard resources require both high precision and efficiency.
Method
Similarity-guided layer adaptation disables redundant ViT layers and dynamically selects one optimal subsequent layer using representation similarity.
Results
SGLATrack achieves state-of-the-art performance and speed across six benchmarks, including 224.7 FPS for SGLATrack-DeiT* and 236.9 FPS for SGLATrack-EVA on GPU.
Takeaways & Limitations
Layer adaptation provides an accuracy-speed trade-off, with at most a 0.4% AUC loss alongside nearly 22% higher speed, 27% fewer parameters, and 30% fewer Flops.
Abstract
from arXiv · showhide
Vision transformers (ViTs) have emerged as a popular backbone for visual tracking. However, complete ViT architectures are too cumbersome to deploy for unmanned aerial vehicle (UAV) tracking which extremely emphasizes efficiency. In this study, we discover that many layers within lightweight ViT-based trackers tend to learn relatively redundant and repetitive target representations. Based on this observation, we propose a similarity-guided layer adaptation approach to optimize the structure of ViTs. Our approach dynamically disables a large number of representation-similar layers and selectively retains only a single optimal layer among them, aiming to achieve a better accuracy-speed trade-off. By incorporating this approach into existing ViTs, we tailor previously complete ViT architectures into an efficient similarity-guided layer-adaptive framework, namely SGLATrack, for real-time UAV tracking. Extensive experiments on six tracking benchmarks verify the effectiveness of the proposed approach, and show that our SGLATrack achieves a state-of-the-art real-time speed while maintaining competitive tracking precision. Codes and models are available at https://github.com/GXNU-ZhongLab/SGLATrack.
1. Introduction
UAV tracking requires both high precision and efficiency because aerial perspectives are challenging and onboard resources are limited. SGLATrack addresses this tension by adapting lightweight ViT layers according to representation similarity.
- UAV tracking supports applications including visual surveillance, path planning, and border security, but aerial perspectives make it more challenging than generic object tracking.
- Limited UAV power and computing resources impose strict efficiency requirements alongside the need for high tracking precision.
- Existing ViT trackers improve tracking representations but remain too cumbersome for direct real-time UAV deployment.
- Layer-by-layer analysis finds that tiny ViT features change substantially in shallow layers before reaching saturation, after which subsequent layers have smaller effects on predictions.
- SGLATrack disables redundant layers while retaining an optimal representative layer, achieving a better accuracy-speed trade-off across six benchmarks.
2. Related Works
Prior UAV-tracking methods trade efficiency, robustness, or precision, while ViT acceleration techniques can incur accuracy losses. Recent adaptive trackers reduce computation, but their layer-selection mechanisms introduce additional complexity or rely on difficult input-complexity estimates.
- DCF-based trackers are efficient but struggle with robustness, whereas Siamese trackers provide deep representations while retaining limitations from early CNNs.
- AVTrack dynamically skips layers using internal classifiers, but estimating input complexity is difficult and classifiers attached to every layer are redundant.
- ViT trackers improve modeling capability but face speed limitations on resource-constrained edge devices.
- Low-rank approximation and quantization can improve efficiency at the cost of significant accuracy loss, while fixed pruning is another acceleration direction.
3. Method
SGLATrack adapts one-stream ViT trackers by identifying redundant deeper layers and retaining a similarity-selected representative layer. Its selection module is trained with layer-wise similarity and dynamically disables remaining layers during inference.
- ViT-based tracking framework: The standard one-stream tracker tokenizes template and search images, transforms their concatenated tokens through ViT layers, and predicts the target box from final search features.The prediction head H produces the bounding-box result from the final search features.
- Layer redundancy: Cosine similarity between consecutive-layer search features increases in deeper layers, while AUC rises rapidly before slowing, indicating saturation and redundant deep layers.These observations suggest that some deep layers can be disabled without significant precision loss.
- Similarity-guided layer adaptation: SGLATrack retains one subsequent layer whose output maximizes similarity to saturated features instead of directly pruning all deeper layers.The approach addresses the precision drop that can result from removing repetitive but still useful representations.
- Similarity-guided layer adaptation: A selection MLP receives a portion of saturated features and outputs probabilities for subsequent layers, retaining the highest-probability layer while disabling the others.The layer-wise similarity loss penalizes low-similarity choices and encourages high-similarity ones.
- SGLATrack architecture: SGLATrack restructures existing ViTs into an efficient layer-adaptive architecture with a one-stream backbone, prediction head, and selection module.The saturated-layer index l* controls the precision-speed trade-off, while the selection module determines the retained subsequent layer.
4. Experiments
Experiments across six UAV tracking benchmarks evaluate SGLATrack’s accuracy, speed, ablations, qualitative robustness, and embedded deployment. The results show competitive tracking with substantial efficiency gains and real-time operation.
- Performance evaluation: SGLATrack-DeiT* establishes new AUC scores on DTB70, UAV123, UAVTrack112, and UAVTrack112 L while maintaining top-three precision across all datasets.On UAVTrack112, it exceeds AVTrack by 2.1% AUC and 2.5% precision.
- Speed evaluation: 223.2 FPS, 224.7 FPS, and 236.9 FPS are achieved by SGLATrack-ViT, SGLATrack-DeiT*, and SGLATrack-EVA, respectively, on GPU.The family also demonstrates competitive real-time CPU speeds.
- LA on different backbones: 0.2% AUC decrease is the best-case loss from layer adaptation, while SGLATrack-EVA gains this efficiency with only a slight performance drop.The comparison covers three different ViT backbones on UAV123 and UAVTrack112.
- LA on different backbones: 0.4% AUC loss accompanies a nearly 22% speed increase, 27% parameter reduction, and 30% Flops decrease in the worst reported case.This result is reported for SGLATrack-ViT on UAVTrack112.
- Qualitative results: SGLATrack-DeiT* can maintain robust tracking with many disabled layers and sometimes outperform comparison trackers in challenging UAV123 scenarios.The qualitative comparison includes AVTrack, TCTrack++, and HiFT.
- Embedded deployment: 33 FPS is sustained by SGLATrack-DeiT* on an NVIDIA Jetson TX2 4GB without TensorRT or other acceleration techniques.The embedded deployment directly tests practicability under real-world conditions.
5. Conclusion
SGLATrack explores layer redundancy in tiny ViTs and uses similarity-guided adaptive layer selection to support real-time UAV tracking across six benchmarks.
- SGLATrack studies layer redundancy through layer-by-layer feature changes and result variations in tiny ViTs.
- Its selection module disables redundant layers and retains an optimal subsequent layer to balance accuracy and speed.
- The module uses a layer-wise similarity loss to focus the model on targets when many layers are disabled.
- Experiments report state-of-the-art tracking performance and speed on six benchmarks.