Source-linked AI summary
Exploring Fusion Strategies for Accurate RGBT Visual Object Tracking
Zhangyong Tang, Tianyang Xu, Hui Li, Xiao-Jun Wu, Xuefeng Zhu, Josef Kittler
TL;DR
RGB and TIR offer complementary tracking information, while data scarcity and the challenge of extending strong single-modality trackers leave multimodal fusion as an open problem. The paper compares three fusion levels and proposes adaptive decision-level fusion with RGB-only network training. It reports strong results across RGBT benchmarks, including a new state of the art on VOT-RGBT2020.
Problem
RGB and TIR have complementary strengths and weaknesses, but limited annotated RGBT data and the challenge of extending single-modality trackers make effective fusion difficult.
Source: §1 · pp. 1–2Method
The paper compares pixel-, feature-, and decision-level fusion, then proposes adaptive decision-level fusion using a Siamese network trained on RGB data and deployed on both RGB and TIR.
Source: §1 · p. 2, §5 · p. 11Results
DFAT achieves a new state of the art on VOT-RGBT2020 and demonstrates effectiveness and robustness on VOT-RGBT2019 and GTOT.
Source: §1 · p. 2, §5 · p. 11Takeaways & Limitations
Among the explored fusion levels, decision-level fusion performs best with only visible data used for training.
Source: §5 · p. 11
Abstract
from arXiv · showhide
We address the problem of multi-modal object tracking in video and explore various options of fusing the complementary information conveyed by the visible (RGB) and thermal infrared (TIR) modalities including pixel-level, feature-level and decision-level fusion. Specifically, different from the existing methods, paradigm of image fusion task is heeded for fusion at pixel level. Feature-level fusion is fulfilled by attention mechanism with channels excited optionally. Besides, at decision level, a novel fusion strategy is put forward since an effortless averaging configuration has shown the superiority. The effectiveness of the proposed decision-level fusion strategy owes to a number of innovative contributions, including a dynamic weighting of the RGB and TIR contributions and a linear template update operation. A variant of which produced the winning tracker at the Visual Object Tracking Challenge 2020 (VOT-RGBT2020). The concurrent exploration of innovative pixel- and feature-level fusion strategies highlights the advantages of the proposed decision-level fusion method. Extensive experimental results on three challenging datasets, \textit{i.e.}, GTOT, VOT-RGBT2019, and VOT-RGBT2020, demonstrate the effectiveness and robustness of the proposed method, compared to the state-of-the-art approaches. Code will be shared at \textcolor{blue}{\emph{https://github.com/Zhangyong-Tang/DFAT}.
1. Introduction
RGB and TIR provide complementary information, but their distinct weaknesses and limited annotated RGBT data complicate robust fusion. The paper compares pixel-, feature-, and decision-level strategies and proposes adaptive decision-level fusion for RGBT tracking.
1. Introduction
- RGB provides detailed scene information but is vulnerable to appearance changes, whereas TIR is more robust to lighting but has less discriminative power.The paper identifies illumination, blur, and occlusion as RGB challenges, and thermal crossover and insufficient texture as TIR limitations.Source: §1 · p. 1
- Limited annotated RGBT data constrains feature-level fusion because networks trained on small datasets risk overfitting and may lack discriminative representations.The paper notes that the permitted RGBT training data are limited, including at most 65 sequences across the cited datasets.Source: §1 · p. 2
- The paper compares pixel-, feature-, and decision-level fusion, finding that even primitive decision-level fusion outperforms the other two under limited-data conditions.The authors also note that decision-level fusion has been effective in other multimodal tasks.Source: §1 · p. 2
- DFAT uses online adaptive weights to address RGB–TIR classification bias, and trains its Siamese tracking network using RGB sequences alone.The paper attributes higher RGB response scores in most sequences to differences between RGB and TIR imaging principles.Source: §1 · p. 2
- DFAT defines a new state of the art on VOT-RGBT2020, while results on VOT-RGBT2019 and GTOT demonstrate promising performance.The paper also reports that a variant of the tracker won the VOT-RGBT 2020 Challenge.Source: §1 · p. 2
2. Related work
Related work spans single-modality RGB and TIR trackers and multimodal approaches that fuse information at pixel, feature, or decision level. The paper compares these options and describes its own image-fusion, attention-based, and adaptive decision-level strategies.
2.1. Tracking with Single Modality
- RGB tracking research includes Siamese trackers, while many TIR trackers rely on handcrafted features because of limited data for learning TIR representations.SiamRPN++ uses a ResNet-50 backbone and spatially-aware sampling; cited TIR approaches include trackers based on motion, edges, and HOG features.Source: §2.1 · p. 3
- TIR-related work also includes larger datasets, including MLSSNet's 430-sequence dataset and MM-Net's dataset of over 1,100 video sequences.MM-Net's dataset covers 30 classes.Source: §2.1 · p. 3
2.2. Tracking with Multiple Modalities
- Prior multimodal work uses pixel-level addition or concatenation, feature-level adaptive fusion and selection, and less commonly decision-level fusion of response maps.Some feature-level approaches use multilevel features, channel suppression, or modality-specific and shared representations; decision-level methods include reliability-based weighting.Source: §2.2 · pp. 3–4
- The paper uses MDLatLRR to separate salient and detailed image information for pixel fusion, and channel attention for feature selection rather than weighted random selection.It also considers spatial attention for cross-modal feature fusion.Source: §2.2 · p. 4
- The paper reports that decision-level fusion performs best among its explored strategies, with adaptive weights learned from classification results to address an RGB–TIR gap.The paper contrasts its ResNet-50 feature extractor with the VGGNet used in JMMAC and notes that its fusion comparison also includes pixel-level fusion.Source: §2.2 · p. 4
3. Proposed Method
This section establishes SiamRPN++ as the single-modality baseline, then specifies pixel-, feature-, and decision-level fusion and template updating. It presents decision-level fusion as the stronger alternative.
3.1. Baseline Tracker
- SiamRPN++ extracts multi-layer features, computes classification and regression outputs, and averages the multimodal results before postprocessing.Its tracking template is ordinarily fixed from the first frame; its offline training uses a translation-sampling strategy for deeper networks.Source: §3.1 · p. 5
3.2. Fusion Mechanism
- Pixel-level fusion uses MDLatLRR to separate salient and detailed information, fuse low-frequency parts by weighted averaging, and fuse detail parts using nuclear-norm-based adaptive weights.The fused image is passed to the tracker; the described pipeline selects the level-2 output.Source: §3.2 · p. 5, Figure 3 · p. 5
- Feature-level fusion selects channels using saliency scores, retains the top 80%, and applies spatial or channel attention to compute cross-modal fusion scalars.These operations follow the neck module; less-discriminative channels are suppressed.Source: §3.2 · pp. 5–6, Figure 4 · p. 6
- Decision-level fusion averages regression outputs and adaptively weights classification outputs before normalization, using ReLU suppression and score modulation to address RGB–TIR score bias.Modulation includes shrinking scores to a scale similar to the original and applying a scaling factor; regression results are fused similarly.Source: Figure 5 · p. 6, §3.2 · p. 6
3.3. Template Update
- Template updating linearly interpolates current and historical embeddings from three neck modules using a predefined learning rate, with updates every 10 frames to prevent overfitting.Source: §3.3 · p. 6
4. Experiments
Experiments compare pixel-, feature-, and decision-level fusion on VOT-RGBT2019, then evaluate DFAT across VOT-RGBT2019, VOT-RGBT2020, and GTOT. They report the strongest overall results for decision-level fusion and analyze its components and scaling factor.
4.1. Implementation Details; 4.2. Experimental Setup
- DFAT runs at 20 FPS; experiments use VOT-RGBT2019, VOT-RGBT2020, and GTOT, with benchmark-specific tracking measures and protocols.The network uses a ResNet-50 backbone pretrained on ImageNet and is trained on COCO and YouTube-VOS; TIR images are triplicated to match RGB input channels.Source: §4.1 · p. 7, §4.2 · p. 7
4.3. Results of Self-Comparison
- Pixel-level MDLatLRR fusion improves over single-modality baselines, but most fused-image results remain below the original RGB-and-TIR baseline.Level 3 outperforms level 2, and using TIR as the second input yields better results at both levels.Source: §4.3 · pp. 7–8, Table 1 · p. 7
- Feature-level fusion with less discriminative channels removed achieves the best feature-fusion robustness (0.2987) and EAO (0.3788).The selection rate is 20%, retaining 80% of channels.Source: §4.3 · p. 8, Table 2 · p. 9
- Decision-level fusion before normalization is more effective than after normalization, and DFAT reaches 0.3986 EAO, improving EAO by 1.09% and failure rate by 2.49%.Equal-weight fusion before normalization reaches 0.3877 EAO; the authors report that their debiasing module significantly reduces bias.Source: §4.3 · p. 8, Table 3 · p. 9
4.4. Comparing to State-of-the-art Trackers; 4.6. Evaluation on the VOT-RGBT2020 Dataset
- DFAT ranks marginally best in accuracy and second in EAO on VOT-RGBT2019, while reaching 0.4178 and setting a new state-of-the-art on VOT-RGBT2020.On VOT-RGBT2020, the ’W/O bias’ variant won the challenge; DFAT ranked first and that variant second.Source: §4.4 · p. 8, §4.6 · pp. 10–11, Table 4 · p. 11
4.5. Self-Analysis
- On VOT-RGBT2019, combining modalities raises the baseline EAO from 0.3189 to 0.3433, while template updating and debiasing jointly produce the best ablation result.DFAT gains 1.13% over ’W/O bias’ and 3.30% over ’W/O update’; on GTOT, it reaches a 0.723 success rate, 3.6% above baseline.Source: §4.5 · pp. 9–10, Figure 7 · p. 9, Figure 8 · p. 9
- With regression-branch fusion weights set to 0.5, scaling factor 0.47 yields the best reported VOT-RGBT2019 result (0.4080), while 0.44 yields the worst (0.3884).The authors report DFAT is relatively insensitive to the scaling-factor value.Source: §4.5 · p. 10, Figure 10 · p. 10
5. Conclusion
The paper compares three fusion levels and concludes decision-level fusion performs best when training uses visible data only. DFAT combines dynamic RGB–TIR debiasing with periodic template updating, and results across three datasets support its effectiveness and robustness.
5. Conclusion
- Decision-level fusion performs best among the tested fusion levels when the tracker is trained using visible data only.The tracker uses SiamRPN++ trained on RGB data to embed both RGB and TIR modalities.Source: §5 · p. 11
- DFAT dynamically debiases RGB and TIR contributions to classification, scales classification scores relative to regression, and linearly updates the target template every 10 frames.A variant of the decision-level fusion method won the VOT-RGBT2020 Challenge.Source: §5 · p. 11
- Results on VOT-RGBT2019, VOT-RGBT2020, and GTOT support DFAT’s effectiveness and robustness, while the VOT-RGBT2020 challenge outcome indicates state-of-the-art tracking performance.Source: §5 · p. 11