Source-linked AI summary
Learning Occlusion-Robust Vision Transformers for Real-Time UAV Tracking
You Wu, Xucheng Wang, Xiangyang Yang, Mengyuan Liu, Dan Zeng, Hengzhou Ye, Shuiwang Li
TL;DR
UAV tracking needs ViT trackers that remain robust during frequent target occlusions without sacrificing real-time efficiency. ORTrack learns occlusion-robust representations through spatial-Cox-process masking, while AFKD distills a compact student; experiments report state-of-the-art performance across multiple benchmarks.
Problem
Single-stream ViT trackers for UAV tracking lack effective strategies for frequent target occlusions, while UAV platforms also require computational efficiency.
Method
ORTrack enforces target-feature invariance to spatial-Cox-process random masking, and AFKD adaptively transfers teacher features to a compact student according to tracking difficulty.
Results
ORTrack achieves state-of-the-art performance across multiple UAV tracking benchmarks, while ORR improves existing trackers and AFKD increases speed with minimal accuracy reduction.
Takeaways & Limitations
The proposed components can be integrated into ViT-based tracking frameworks to improve occlusion robustness and support more efficient real-time UAV tracking.
Takeaways & Limitations
The reported real-time performance applies to platforms similar to or more advanced than the authors' evaluation platform.
Abstract
from arXiv · showhide
Single-stream architectures using Vision Transformer (ViT) backbones show great potential for real-time UAV tracking recently. However, frequent occlusions from obstacles like buildings and trees expose a major drawback: these models often lack strategies to handle occlusions effectively. New methods are needed to enhance the occlusion resilience of single-stream ViT models in aerial tracking. In this work, we propose to learn Occlusion-Robust Representations (ORR) based on ViTs for UAV tracking by enforcing an invariance of the feature representation of a target with respect to random masking operations modeled by a spatial Cox process. Hopefully, this random masking approximately simulates target occlusions, thereby enabling us to learn ViTs that are robust to target occlusion for UAV tracking. This framework is termed ORTrack. Additionally, to facilitate real-time applications, we propose an Adaptive Feature-Based Knowledge Distillation (AFKD) method to create a more compact tracker, which adaptively mimics the behavior of the teacher model ORTrack according to the task's difficulty. This student model, dubbed ORTrack-D, retains much of ORTrack's performance while offering higher efficiency. Extensive experiments on multiple benchmarks validate the effectiveness of our method, demonstrating its state-of-the-art performance. Codes is available at https://github.com/wuyou3474/ORTrack.
1. Introduction
UAV tracking requires both occlusion robustness and computational efficiency, yet existing single-stream ViT trackers struggle with occlusions. ORTrack addresses this gap through occlusion-robust representation learning, while AFKD produces a faster compact student tracker.
- UAV tracking must balance accuracy with restricted aerial-platform energy and computational resources.
- Single-stream ViT trackers are effective for generic tracking but remain vulnerable to occlusions caused by buildings, mountains, and trees.
- ORTrack learns occlusion-robust target representations by enforcing feature invariance to random masking modeled by a spatial Cox process.
- The masking-based representation objective integrates without additional inference computation, while AFKD adaptively distills ORTrack into a more compact tracker according to task difficulty.
- ORTrack and ORTrack-D target the precision–efficiency trade-off required for real-time UAV tracking, with evaluations reporting state-of-the-art real-time performance.
2. Related work
Prior work has developed occlusion-robust representations and knowledge-distillation methods, but unified ViT-based occlusion robustness for UAV tracking remains underexplored. This work positions random masking and adaptive distillation as targeted responses to those gaps.
- Occlusion-Robust Feature Representation: Earlier occlusion-robust vision methods used handcrafted features, appearance models, motion analysis, sensor fusion, and later deep networks.
- Occlusion-Robust Feature Representation: Occlusion-robust representations have supported applications including action recognition, pose estimation, object detection, and visual tracking.
- Occlusion-Robust Feature Representation: Prior work had not broadly explored learning occlusion-robust ViTs in a unified UAV-tracking framework; this study uses spatial-Cox-process random masking to simulate occlusions.
- Knowledge Distillation: Knowledge distillation transfers knowledge from a complex teacher to a simpler student to reduce computational resources and memory while maintaining performance.
- Knowledge Distillation: Existing tracking distillation methods were mainly Siamese-based and tailored to specific frameworks, limiting adaptation to other settings.
3. Method
ORTrack trains a teacher with spatial Cox process masking to learn occlusion-robust target representations, then trains a smaller student through adaptive feature-based knowledge distillation. The masking and representation-learning components operate during teacher training, while inference uses the unmasked template without added masking cost.
- Framework: ORTrack uses two sequential training phases: occlusion-robust representation learning for the teacher, followed by adaptive knowledge distillation for the student.The teacher is trained first, after which its weights are fixed while the student is trained.
- Occlusion-Robust Representations: Spatial Cox process masking simulates occlusion with spatially varying and randomly varying patterns rather than uniformly masking each position.Its bell-shaped intensity concentrates masking behavior toward the central template area, while the random intensity produces diverse masking ratios.
- Occlusion-Robust Representations: The teacher minimizes mean squared error between feature representations of the original and randomly masked target templates.Template and search-image tokens are tracked separately so the loss compares the corresponding target-template representations.
- Inference: During inference, ORTrack inputs only the original template and search image, so random template masking adds no computational cost.The framework is independent of the particular ViT and can use efficient ViTs.
- Adaptive Feature-Based Knowledge Distillation: The student uses a smaller ViT backbone and feature-based distillation, transferring intermediate representations from the fixed teacher.A self-similar architecture avoids extra alignment techniques, and MSE penalizes differences between teacher and student backbone features.
- Adaptive Feature-Based Knowledge Distillation: AFKD weights feature transfer according to tracking difficulty measured by the deviation of the student’s GIoU loss from its average value.The adaptive loss uses a slope-and-intercept function of this deviation to quantify task difficulty.
4. Experiments
Experiments across four UAV benchmarks compare ORTrack with lightweight and deep trackers, then examine occlusion robustness, component effects, masking operators, generalizability, transfer to existing trackers, and qualitative tracking behavior. ORTrack achieves strong accuracy-speed trade-offs, while ORR improves robustness and AFKD increases efficiency with limited accuracy loss.
- Experimental Setup: Experiments cover four UAV benchmarks and compare ORTrack with 26 state-of-the-art trackers, including lightweight and generic deep trackers.The evaluation uses official codes and hyperparameters on DTB70, UAVDT, VisDrone2018, and UAV123.
- State-of-the-art Comparison: ORTrack trackers exceed 82.0% average precision and 62.0% average success, while ORTrack-DeiT reaches 85.6% and 65.0%, respectively, among competing lightweight trackers.ORTrack-DeiT also achieves the highest average precision and success rate in this comparison.
- State-of-the-art Comparison: On VisDrone2018, ORTrack-DeiT surpasses deep trackers in precision and GPU speed, running 4, 6, and 4 times faster than AQATrack, HIPTrack, and ROMTrack.The comparison uses precision and GPU speed against 14 state-of-the-art deep trackers.
- Occlusion Robustness: ORTrack-DeiT achieves 85.0% precision on the partial-occlusion subset of VisDrone2018, ranking second and only 0.2% behind AQATrack.The comparison also includes ORTrack-DeiT* without the occlusion-robust enhancement.
- Ablation Study: ORR raises baseline precision by 2.7%–4.8% and success by 2.1%–3.1% on UAVDT, while AFKD improves GPU speed by over 30.0% with slight precision and success reductions.ORTrack-DeiT receives a 36.0% GPU-speed improvement after AFKD, while its precision and success reductions remain slight.
- Ablation Study: The spatial-Cox masking operator outperforms alternative masking and mixing methods, with mC producing 7.0% and 4.6% improvements on VisDrone2018.The best result using SAM is only comparable to the mU masking operator.
- Ablation Study: AFKD provides over 1.1% additional gains in LaSOT Pnorm and P compared with non-adaptive distillation, indicating stronger generalizability.The non-adaptive loss is represented by ϖ(Liou; α, 0).
- Generalization and Qualitative Results: Adding ORR to ARTrack, GRM, and DropTrack improves both precision and success by more than 1.2% across two datasets, while qualitative results show successful tracking in challenging sequences.With ORR, attention remains more focused and feature maps more consistent across masking ratios.
5. Conclusion
ORTrack learns occlusion-robust representations through spatial-Cox-process masking and improves efficiency with adaptive feature-based knowledge distillation. Across multiple UAV tracking benchmarks, ORTrack-DeiT achieves state-of-the-art performance.
- ORTrack learns Occlusion-Robust Representations by enforcing feature invariance to random masking modeled by a spatial Cox process.
- Adaptive Feature-Based Knowledge Distillation is proposed to enhance tracking efficiency.
- Extensive experiments across multiple UAV tracking benchmarks validate the method’s effectiveness.
- ORTrack-DeiT achieves state-of-the-art performance.