Source-linked AI summary
Visible-Thermal UAV Tracking: A Large-Scale Benchmark and New Baseline
Pengyu Zhang, Jie Zhao, Dong Wang, Huchuan Lu, Xiang Ruan
TL;DR
RGB-T tracking lacks paired training data despite the complementary strengths of visible and thermal modalities. The paper constructs VTUAV and proposes HMFT, whose hierarchical fusion combines multiple fusion types; experiments analyze their complementarity, while HMFT remains limited under heavy or full occlusion.
Problem
RGB-T tracking is bottlenecked by scarce paired training samples, as recent benchmarks mainly provide test sequences.
Method
The paper constructs VTUAV with diverse RGB-T UAV data and proposes HMFT, which hierarchically combines image, feature, and decision fusion.
Results
Experiments across several datasets show the effectiveness and complementarity of different fusion types, with HMFT improving MSR and MPR in ablation results.
Takeaways & Limitations
VTUAV supports comprehensive short-term, long-term, and pixel-level RGB-T evaluation with challenge-specific attribute analysis.
Takeaways & Limitations
HMFT often misses targets under heavy or full occlusion because online noisy samples can bias filter learning, limiting re-detection ability.
Abstract
from arXiv · showhide
With the popularity of multi-modal sensors, visible-thermal (RGB-T) object tracking is to achieve robust performance and wider application scenarios with the guidance of objects' temperature information. However, the lack of paired training samples is the main bottleneck for unlocking the power of RGB-T tracking. Since it is laborious to collect high-quality RGB-T sequences, recent benchmarks only provide test sequences. In this paper, we construct a large-scale benchmark with high diversity for visible-thermal UAV tracking (VTUAV), including 500 sequences with 1.7 million high-resolution (1920 $\times$ 1080 pixels) frame pairs. In addition, comprehensive applications (short-term tracking, long-term tracking and segmentation mask prediction) with diverse categories and scenes are considered for exhaustive evaluation. Moreover, we provide a coarse-to-fine attribute annotation, where frame-level attributes are provided to exploit the potential of challenge-specific trackers. In addition, we design a new RGB-T baseline, named Hierarchical Multi-modal Fusion Tracker (HMFT), which fuses RGB-T data in various levels. Numerous experiments on several datasets are conducted to reveal the effectiveness of HMFT and the complement of different fusion types. The project is available at here.
1. Introduction
RGB-T tracking addresses visible-modality weaknesses by combining illumination-robust thermal cues with visible information, but progress is constrained by scarce, low-diversity training data. The paper introduces VTUAV and HMFT to support broader evaluation and hierarchical multi-modal fusion.
- Thermal imagery is insensitive to illumination variation but may fail when target and background have similar temperatures.
- Existing RGB-T benchmarks provide only 284 unique short-term sequences overall, forcing training on test sets or synthetic data with limited generalization and training gaps.
- VTUAV provides high-diversity RGB-T UAV data for short-term tracking, long-term tracking, and segmentation mask prediction, with frame- and sequence-level attribute annotations.
- HMFT unifies image, feature, and decision fusion in a hierarchical framework, with short-term and long-term versions and analysis across several RGB-T datasets.
- Existing datasets span grayscale-thermal, visible-thermal, synthetic, and short-term collections, while VTUAV emphasizes high-quality training pairs and evaluation across three tracking perspectives.
- Experiments report that fusion at multiple levels provides complementary contributions to RGB-T tracking performance.
3. VTUAV Benchmark
VTUAV is collected by UAV across diverse categories, scenes, weather, and times, and provides hierarchical attributes plus bounding-box and mask annotations. It evaluates tracking and mask prediction with modality-aware metrics and standard one-pass evaluation.
- Large-scale sequences with high diversity: VTUAV uses professional UAV and thermal-camera capture to obtain long, diverse sequences in extreme conditions including night, fog, and wind.
- Generic object and scene category: Targets span 5 super-classes and 13 sub-classes across 15 scenes in two cities, covering settings such as roads, parks, beaches, courts, and schools.
- Large-scale sequences with high diversity: 325 sequences are captured during daytime and 175 at night, including windy, cloudy, and foggy conditions.
- Hierarchical attributes: The benchmark annotates 13 challenges hierarchically at sequence and frame levels while preserving sequence continuity and allowing multiple or absent frame labels.
- High-quality Annotation: Expert annotations include modality-specific bounding boxes, segmentation masks, and frame-level attributes, totaling 326,961 boxes, 24,464 masks, and 430,960 attributes across 301,678 frames.
- Evaluation Metrics: Trackers use one-pass evaluation with MSR and MPR, while mask prediction is assessed using Jaccard index and F-score.
4. Hierarchical Multi-modal Fusion Tracker
HMFT unifies complementary image, discriminative feature, and adaptive decision fusion to combine shared, modality-specific, and response-level cues for RGB-T tracking.
- Architecture: HMFT contains Complementary Image Fusion (CIF), Discriminative Feature Fusion (DFF), and Adaptive Decision Fusion (ADF) in one hierarchical framework.CIF learns shared patterns, DFF combines heterogeneous representations, and ADF combines responses from complementary and discriminative classifiers.
- Complementary Image Fusion: CIF uses a shared ResNet50 backbone and divergence loss to constrain visible-thermal feature distributions, producing complementary features for representation and scale estimation.The visible and thermal features are concatenated into an overall complementary feature after block-level processing.
- Discriminative Feature Fusion: DFF extracts each modality separately, then learns channel-wise modality weights from globally pooled features to form a weighted fused feature map.The modality weights are generated through fully connected layers followed by softmax normalization.
- Adaptive Decision Fusion: ADF estimates modality confidence with a self-attention Modality Aggregation Module and uses generated spatial weight maps to combine two response maps.The final response is RF = Rd ⊙Ed + Rc ⊙Ec, combining discriminative and complementary branch responses.
- Training: The implementation uses DiMP with truncated ResNet50 and trains the branches, DFF, and ADF in successive stages with selected components fixed during later stages.The multi-step process first trains both backbones, then DFF and classifiers, and finally ADF, IoU prediction modules, and classifier fine-tuning.
5. Experimental Analysis for RGB-T Tracking
HMFT combines multiple RGB-T fusion strategies and achieves strong short- and long-term tracking performance. Ablations and qualitative comparisons show benefits from image, feature, and adaptive decision fusion.
- Overall Performance: 62.7% MSR and 75.8% MPR make real-time HMFT the top performer on the evaluated RGB-T benchmark.HMFT also achieves state-of-the-art performance on GTOT and RGBT210, and top-three results on RGBT234.
- Long-term Evaluation: 29.8% and 29.4% relative improvements in MSR and MPR make HMFT LT a new long-term RGB-T baseline over HMFT.HMFT LT combines a global detector with a tracker switcher for target rediscovery.
- Ablation Study: 3.2% and 4.4% MSR gains show that complementary image fusion and discriminative feature fusion outperform single-modality tracking.The gains are reported for DiMP+CIF and DiMP+DFF, respectively.
- Ablation Study: 2.6% MSR and 2.7% MPR improvements show the benefit of adaptive decision fusion in the final HMFT model.Simply averaging complementary and discriminative responses slightly decreases performance, whereas ADF adapts their combination.
- Qualitative Analysis: HMFT tracks targets accurately under occlusion, camera movement, and scale variation, while competing trackers sometimes miss targets or misestimate scale.The comparison is qualitative and presented in Figure 5.
6. Experimental Results on VTUAV-V Subset
VTUAV-V evaluates RGB trackers on short- and long-term subsets. Transformer-based trackers lead short-term results, while long-term performance declines and redetection mechanisms improve rankings.
- Dataset and Protocol: VTUAV-V contains only visible images and evaluates RGB trackers with SR and PR rather than selecting the maximum across modalities.The subset supports short-term and long-term RGB tracking evaluation.
- Short-term Evaluation: 64.9% SR and 75.3% PR make STARK the top-performing short-term RGB tracker on VTUAV-V.Transformer-based trackers obtain the strongest short-term results.
- Long-term Evaluation: All trackers perform worse on the long-term subset than on the short-term subset.STARK leads SR, while LTMU leads PR in the long-term evaluation.
- Long-term Evaluation: Global redetection mechanisms significantly improve the rankings of LTMU, GlobalTrack, and SPLT in long-term tracking.These trackers use global detection to recover targets after loss.
7. Conclusion
The paper releases a diverse, high-resolution RGB-T benchmark spanning multiple tracking and prediction tasks, and introduces HMFT to combine complementary fusion strategies.
- Conclusion: VTUAV addresses limited RGB-T training data with diverse, high-resolution paired images captured under varied conditions.The benchmark includes short-term tracking, long-term tracking, and pixel-level prediction.
- Conclusion: VTUAV is a unified RGB-T dataset covering short-term tracking, long-term tracking, and pixel-level prediction.This broadens evaluation beyond a single tracking setting.
- Conclusion: Thirteen sequence- and frame-level challenges support scene-specific tracker analysis and attribute-aware evaluation.The annotations are designed to exploit attribute effectiveness.
- Conclusion: HMFT combines image, feature, and decision fusion, with results on three benchmarks showing complementary effects among these strategies.The conclusion also links performance to adequate training data.
1. Attribute Description of VTUAV
VTUAV provides an attribute description intended to support attribute-aware tracker analysis. The supplied material identifies the table but does not enumerate individual attributes.
- Attribute Description: Comprehensive attribute annotation is intended to exploit the effectiveness of attribute-aware trackers.The claim is stated as the purpose of the attribute description.
- Attribute Description: The paper frames VTUAV attributes as a resource for analyzing tracker performance across challenges.No individual attribute definitions are available in the supplied passages.
- Attribute Description: Table 1 presents the attribute description for VTUAV.The supplied passage names the table but does not list its attribute categories.
2. Attribute-based Comparison on VTUAV
HMFT performs well across most short-term and long-term attributes, but full occlusion remains a weakness. Its long-term version benefits substantially from global mechanisms, especially for out-of-view and occlusion challenges.
- HMFT performs satisfactorily on all attributes except fully occlusion, where it is inferior to FSRPN.The paper attributes this to HMFT’s online learning noise during target absence, whereas FSRPN relies on its initial template for re-detection.
- HMFT LT outperforms the short-term HMFT by a large margin across most long-term challenges.The result supports the necessity of a global detection component for difficult long-term cases.
- More than 10% promotion in both MSR and MPR is attributed to the global mechanism when targets are out-of-view or fully occluded.
3. Mask Evaluation
The mask evaluation compares short-term trackers with video object segmentation methods under different initialization settings. Short-term trackers outperform the VOS methods on segmentation mask prediction.
- Three short-term trackers and two VOS methods are compared for segmentation mask prediction.Short-term trackers receive an initial bounding box, whereas VOS methods receive an initial mask.
- Short-term trackers outperform VOS methods on segmentation mask prediction.
- Short-term trackers predict masks in a small search region, which avoids dis...
4. Analysis on Different Alignment Methods
The analysis evaluates image alignment for visible-thermal tracking and reports that the proposed alignment method is effective for image registration. The paper suggests incorporating alignment modules into future trackers.
- Image alignment is applied to the initial frame of every sequence.
- The proposed alignment method is effective for image registration on GTOT and RGBT234.
- Future trackers can design an alignment module to achieve better results.