Source-linked AI summary
HiFT: Hierarchical Feature Transformer for Aerial Tracking
Ziang Cao, Changhong Fu, Junjie Ye, Bowen Li, Yiming Li
TL;DR
Aerial trackers must combine robust localization with real-time efficiency despite limited resources, while existing approaches struggle with single-layer features, multi-level computation, or transformer generalization. HiFT uses a hierarchical feature transformer to fuse shallow spatial and deep semantic features, learning a discriminative tracking-tailored space. Results across four aerial benchmarks and real-world aerial tests validate its effectiveness and practicability at real-time speed.
Problem
Existing aerial trackers either rely on a single similarity map that can degrade localization or separately process multiple maps with intractable computation for mobile platforms.
Method
HiFT feeds multi-level convolutional features into a hierarchical transformer that interactively fuses spatial and semantic cues and learns their interdependencies.
Results
HiFT achieves promising performance on four aerial benchmarks and demonstrates real-time effectiveness and practicability in real-world aerial tests.
Takeaways & Limitations
HiFT provides an efficient aerial-tracking architecture with a precision-speed trade-off and comparable performance to trackers using deeper backbones.
Abstract
from arXiv · showhide
Most existing Siamese-based tracking methods execute the classification and regression of the target object based on the similarity maps. However, they either employ a single map from the last convolutional layer which degrades the localization accuracy in complex scenarios or separately use multiple maps for decision making, introducing intractable computations for aerial mobile platforms. Thus, in this work, we propose an efficient and effective hierarchical feature transformer (HiFT) for aerial tracking. Hierarchical similarity maps generated by multi-level convolutional layers are fed into the feature transformer to achieve the interactive fusion of spatial (shallow layers) and semantics cues (deep layers). Consequently, not only the global contextual information can be raised, facilitating the target search, but also our end-to-end architecture with the transformer can efficiently learn the interdependencies among multi-level features, thereby discovering a tracking-tailored feature space with strong discriminability. Comprehensive evaluations on four aerial benchmarks have proven the effectiveness of HiFT. Real-world tests on the aerial platform have strongly validated its practicability with a real-time speed. Our code is available at https://github.com/vision4robotics/HiFT.
1. Introduction
Aerial tracking must balance robustness against challenging visual conditions with the limited computation available on UAVs. HiFT addresses this trade-off by fusing multi-level features through a lightweight hierarchical transformer and reports strong benchmark and real-world performance.
- Aerial tracking supports UAV applications but remains difficult under fast motion, low resolution, frequent occlusion, and limited computational resources.
- Lightweight CNNs lack robust features, whereas larger kernels and deeper backbones improve representation at the cost of efficiency and practicability.
- HiFT feeds shallow spatial features and deep semantic features into a transformer to learn their interdependencies and construct a discriminative tracking-tailored feature space.
- A feature modulation layer and a new classification label further exploit hierarchical features to improve accuracy on small objects.
- Evaluations on four aerial benchmarks validate HiFT against state-of-the-art trackers, including trackers with deeper backbones.
- Real-world tests on an aerial platform demonstrate HiFT’s efficiency and effectiveness in practical scenarios.
2. Related Works
Prior visual tracking methods trade representation quality, global context, or robustness against computational efficiency. HiFT instead uses a lightweight hierarchical transformer to combine multi-level features and global context for aerial tracking.
- DCF-based trackers offer high efficiency but handcrafted features limit robustness under varied tracking conditions.
- Siamese trackers formulate tracking as template-search similarity matching and benefit from end-to-end learning, while deeper backbones sacrifice efficiency.
- Lightweight backbones lack global context, deep CNNs are not sufficiently real-time onboard UAVs, and multi-level feature methods impose unaffordable computation.
- Transformers have expanded from machine translation into computer vision through attention-based architectures.
- In single-object tracking, object queries generalize poorly to arbitrary targets and transformers struggle with low-resolution aerial objects.
- HiFT replaces object queries with deeper-layer feature maps, combines shallow and deep features, and uses a lightweight encoder-decoder suited to mobile platforms.
3. Proposed Method
HiFT combines multi-level Siamese features with a hierarchical transformer that encodes spatial relationships and decodes semantic information before classification and regression.
- Architecture: HiFT contains feature extraction, hierarchical feature transformation, and classification-and-regression submodules using features from the last three layers.The lightweight AlexNet backbone supplies template and search features, while the transformer operates on hierarchical representations.
- Hierarchical Feature Transformer: The transformer encodes high-resolution features to learn cross-layer dependencies and spatial information, especially for objects at different scales.Its decoder then aggregates semantic information from the low-resolution feature map.
- Hierarchical Feature Transformer: Feature maps are formed by applying convolution and cross-correlation to corresponding template and search features, then reshaping them before transformation.The resulting maps are supplemented with learnable positional encoding before entering the transformer.
- Feature Encoding: Multi-head attention combines encoded hierarchical features so interdependencies and global context enrich the high-resolution representation.The attention computation uses normalized feature representations and scaled dot-product attention.
- Feature Encoding: A modulation layer further exploits internal spatial information between feature levels, enriching encoded features for distinguishing objects from complex backgrounds.It uses feed-forward processing and global average pooling, with γ1 as a learning weight.
- Classification and Regression: The classification and regression network uses convolutional layers, two classification branches, pseudo-random negative-label constraints, and a weighted loss combining three losses.The overall loss combines cross-entropy, binary cross-entropy, and IoU loss through λ1, λ2, and λ3.
4.1. Implementation Details
HiFT is trained for 70 epochs with selective AlexNet fine-tuning, specified template/search sizes, and a compact one-encoder, two-decoder transformer.
- Training Setup: Training lasts 70 epochs, with the last three AlexNet layers fine-tuned during the final 60 epochs while the first two remain frozen.The learning rate is initialized at 5×10−4 and decreased logarithmically from 10−2 to 10−4.
- Model Configuration: Template and search images use sizes 3 × 127 × 127 and 3×287×287, respectively.The feature transformer contains one encoder layer and two decoder layers.
- Training Data: Image pairs for training are extracted from COCO, ImageNet VID, GOT-10K, and Youtube-BB.
4.2. Evaluation Metrics
Tracking performance is evaluated with one-pass precision and success-rate metrics based on center location error and bounding-box IoU.
- Metrics: One-pass evaluation uses precision and success rate to assess tracking performance.
- Success Rate: Success rate is measured from the IoU between ground-truth and estimated bounding boxes across thresholds.The success plot shows the percentage of frames whose IoU exceeds each predefined threshold.
- Precision: Precision is evaluated using center location error between the estimated target location and the ground truth.
4.3. Evaluation on Aerial Benchmarks
HiFT is evaluated across four aerial benchmarks, attribute conditions, ablations, and speed–accuracy comparisons. It achieves leading tracking performance while maintaining real-time efficiency and outperforming or matching deeper-backbone trackers.
- Overall Performance: HiFT ranks first on UAV123, with precision 0.787 and success rate 0.589, exceeding SiamRPN++ and ECO in precision.Its precision surpasses SiamRPN++ (0.769) and ECO (0.752) by 2.3% and 4.7%, respectively.
- Overall Performance: HiFT achieves the best UAV20L precision 0.763 and success rate 0.566, outperforming SiamRPN++ and DaSiamRPN.The precision gains over the second- and third-best trackers are 9.6% and 14.7%.
- Overall Performance: On UAV123@10fps, HiFT obtains the best precision 0.754 and success rate 0.574, supporting stable performance under stronger motion.The benchmark is downsampled from 30 FPS, making strong motion more severe than in UAV123.
- Attribute-based Comparison: HiFT ranks first across low-resolution, scale-variation, occlusion, and fast-motion attributes, improving the second-best result by around 10% for fast motion.The reported comparisons associate these results with global contextual information, robust features, and multi-scale feature maps.
- Ablation Study: Replacing object queries with feature maps raises baseline precision by 10.47%, while the full hierarchical feature transformer improves performance by 24.88%.Adding the original transformer with object queries instead lowers precision by about 2.29% and success rate by 3.67%.
- Comparison to Trackers with Deeper Backbone: HiFT achieves 0.783 average precision at 129.87 FPS and reaches 127 FPS on UAV20L, balancing robustness with real-time efficiency.On UAV20L, its precision of 0.763 exceeds SiamRPN++ with ResNet-50 at 0.749, while running 1.8 times faster.
5. Real-World Tests
HiFT was implemented on an NVIDIA AGX Xavier UAV platform and tested in varied real-world conditions, including day and night scenes with multiple tracking challenges.
- HiFT was implemented on an embedded NVIDIA AGX Xavier processor to assess practicability in real-world aerial tracking.
- Three in-the-wild tests covered day and night scenes involving occlusion, viewpoint change, low resolution, camera motion, small objects, and similar nearby objects.
6. Conclusion
HiFT combines multi-level features and global contextual information through a lightweight hierarchical transformer, achieving a favorable precision-speed trade-off for real-world aerial tracking.
- HiFT streamlines the use of global contextual information and multi-level features through a hierarchical feature transformer.
- Low-resolution semantic information and high-resolution spatial details help the transformed feature discriminate objects from clutter through a lightweight structure.
- The modulation layer and new classification label allow the feature transformer to reach its full potential.
- HiFT achieves an excellent precision-speed trade-off and can be used in real-world aerial tracking scenarios.
- HiFT achieves comparable performance to trackers with deeper backbones.