Explicit Attention-Enhanced Fusion for RGB-Thermal Perception Tasks

Mingjian Liang; Junjie Hu; Chenyu Bao; Hua Feng; Fuqin Deng; Tin Lun Lam

Source-linked AI summary

Explicit Attention-Enhanced Fusion for RGB-Thermal Perception Tasks

Mingjian Liang, Junjie Hu, Chenyu Bao, Hua Feng, Fuqin Deng, Tin Lun Lam

arXiv:2303.15710v1Mar 28, 2023cs.CV

TL;DR

RGB-thermal perception still needs more effective fusion because RGB performance degrades under noise, poor lighting, and weather, while existing attention-based methods may not extract sufficiently effective features. EAEF addresses this with case-specific explicit attention enhancement and achieves state-of-the-art performance across multiple perception tasks, including reported gains in mIoU and MAE.

Problem
RGB-based perception degrades under noise, poor lighting, and weather, while existing attention methods may not extract sufficiently effective features.
Method
EAEF explicitly handles cases where both, only one, or neither modality provides useful features, using branches for feature enhancement and compensation before fusion.
Results
The method achieves state-of-the-art performance across semantic segmentation, object detection, salient detection, and crowd counting, including 1.6% improvement in mIoU and 3.1% improvement in MAE against reported state-of-the-art baselines.
Takeaways & Limitations
EAEF provides a better fusion approach for RGB-thermal perception tasks by extracting complementary cross-modality features.

Abstract

from arXiv · show

Recently, RGB-Thermal based perception has shown significant advances. Thermal information provides useful clues when visual cameras suffer from poor lighting conditions, such as low light and fog. However, how to effectively fuse RGB images and thermal data remains an open challenge. Previous works involve naive fusion strategies such as merging them at the input, concatenating multi-modality features inside models, or applying attention to each data modality. These fusion strategies are straightforward yet insufficient. In this paper, we propose a novel fusion method named Explicit Attention-Enhanced Fusion (EAEF) that fully takes advantage of each type of data. Specifically, we consider the following cases: i) both RGB data and thermal data, ii) only one of the types of data, and iii) none of them generate discriminative features. EAEF uses one branch to enhance feature extraction for i) and iii) and the other branch to remedy insufficient representations for ii). The outputs of two branches are fused to form complementary features. As a result, the proposed fusion method outperforms state-of-the-art by 1.6\% in mIoU on semantic segmentation, 3.1\% in MAE on salient object detection, 2.3\% in mAP on object detection, and 8.1\% in MAE on crowd counting. The code is available at https://github.com/FreeformRobotics/EAEFNet.

I. INTRODUCTION

RGB-Thermal perception uses thermal data to complement RGB images when visual conditions degrade, but existing fusion strategies remain insufficient. EAEF explicitly handles three modality-feature cases and combines complementary branches within an encoder-decoder framework.

RGB-based perception degrades under noise, poor lighting, and weather, motivating thermal data as complementary information.
Existing RGB-Thermal methods merge inputs, combine encoder features, or apply attention implicitly, but these strategies are insufficient.
EAEF explicitly addresses cases where both, only one, or neither modality provides meaningful features.
One EAEF branch preserves meaningful representations and enhances extraction, while the other compensates for insufficient representations; their combination yields complementary features.
EAEF is integrated between RGB and thermal encoders in an encoder-decoder network to fuse multi-scale features for dense prediction tasks.
The method reports state-of-the-art performance on semantic segmentation, object detection, salient detection, and crowd counting.

II. RELATED WORK

Related RGB-Thermal work commonly uses addition, gating, or attention-based fusion across segmentation, salient detection, crowd counting, and object detection. EAEF targets the limitation that attention and feature generation are often performed implicitly.

RGB-T semantic segmentation: RGB-Thermal semantic segmentation methods range from element-wise addition to channel, spatial, and self-attention fusion.
RGB-T salient object detection: Salient object detection methods use channel, spatial, cross-modal, and pixel-level attention to adaptively weight features.
RGB-T crowd counting: Crowd-counting methods employ element-wise addition, gating, and attention mechanisms for feature fusion.
RGB-T object detection: RGB-T object detection methods have directly combined RGB and thermal data at the input without attention-based fusion.
Existing attention methods do not guarantee effective feature extraction because feature generation and attention extraction occur implicitly inside CNNs.

III. METHODOLOGY

EAEF fuses RGB and thermal features through two explicitly designed branches that address complementary modality conditions, then merges their enhanced outputs with spatial attention.

Framework: EAEF is inserted between separate RGB and thermal encoders in an encoder-decoder framework to fuse multi-scale features.All models use an RGB encoder, a thermal encoder, EAEF modules, and a decoder.
Branch design: The fusion decomposes into an Attention Interaction Branch and an Attention Complement Branch for different combinations of discriminative modality features.AIB handles cases where both or neither encoder captures useful features, while ACB addresses cases where only one does.
Attention Interaction Branch: AIB multiplies RGB and thermal attention weights element-wise, then applies the resulting correlated attention channel-wise to both modality features.The scaling factor c is designed so the resulting attention is at least as strong as either individual modality attention.
Attention Complement Branch: ACB uses complementary attention to enhance features when only one modality provides sufficiently discriminative information.When modality attention signs oppose each other, correlated attention tends to be small, motivating the complementary factor 1−σ(c∗(R⊗T)).
Feature interaction: Within AIB, non-linear feature interaction prevents direct aggregation from collapsing into an identical mapping.The interaction module concatenates modality features, applies multi-modality interaction, and splits the result back into RGB and thermal features.
Output fusion: The enhanced RGB and thermal features are merged by concatenation followed by spatial attention implemented with a 1×1 convolutional layer.This produces the final EAEF output features from the two branch outputs.

1) Datasets:

The evaluation uses MFNet and PST900 benchmarks for RGB-T semantic segmentation, with qualitative comparisons and attention-map visualizations reported on MFNet.

MFNet: MFNet is a popular RGB-T semantic-segmentation benchmark containing nine semantic categories.The supplied passages identify MFNet as a benchmark but do not provide its full dataset size in the selected evidence.
MFNet: MFNet results are presented through quantitative comparisons, qualitative examples, and attention-map visualizations.The qualitative and attention-map passages identify the figure content but do not specify additional dataset structure.
PST900: PST900 is another RGB-T semantic-segmentation benchmark with five semantic categories and 894 RGB-T image pairs.The supplied dataset passage describes its resolution and train/test split, but those details are not needed for this point.

2) Implementation Details and Evaluation Metrics:

The method is evaluated with segmentation-specific training details and two metrics whose values are averaged across categories.

Implementation: The dense-prediction implementation uses a cascaded decoder with SELayer and ASPP modifications.The decoder is described as being based on BBSNet and adjusted for nine-category recognition.
Implementation: Training uses stochastic gradient descent with an initial learning rate of 0.02, momentum 0.9, and weight decay 0.0005.
Evaluation metrics: Accuracy and Intersection over Union are used to quantify results, with mAcc and mIoU averaging across categories.The passages define mAcc and mIoU as category averages.

3) Results:

EAEFNet achieves strong RGB-T perception results across MFNet, PST900, and qualitative MFNet evaluations, including improvements over reported state-of-the-art baselines.

MFNet: 1.6% in mIoU and 0.4% in mAcc: EAEFNet improves over MFTNet on MFNet semantic segmentation.The comparison uses the ResNet-152 implementation against the current state-of-the-art MFTNet.
Qualitative results: EAEFNet performs better in both daytime and nighttime qualitative comparisons, with more superior results reported for nighttime images.The method is also reported to capture tiny objects more effectively in RGB and thermal images.
Attention visualization: EAEFNet generates better attention maps than the other compared methods on three MFNet examples.
PST900: 91.42 in mAcc and 85.56 in mIoU: EAEFNet achieves the best reported PST900 results and exceeds GMNet by 1.81% and 1.44%, respectively.

B. Object Detection

For M3FD object detection, EAEF is integrated into YOLOv5 and evaluated across RGB-T driving scenarios using paired RGB and thermal data.

Dataset: M3FD contains 4,200 RGB-T image pairs with 33,603 annotated labels across six object classes and four environmental scenarios.The scenarios are Daytime, Overcast, Night, and challenge.
Implementation: EAEF is integrated into YOLOv5 for object detection and trained with stochastic gradient descent.The implementation uses IoU loss and cross-entropy loss weighted 0.3 and 0.7, respectively.
Results: 2.3%: EAEFNet achieves 0.801 mAP, outperforming TarDAL on M3FD object detection.The passage also reports that thermal-only input performs worst overall, although it exceeds RGB in challenge scenarios.
Additional evaluation: The salient-object-detection evaluation uses VT821, VT1000, and VT5000 registered RGB-thermal image pairs.The selected passage describes the datasets and training/testing allocation but does not provide a result value.

2) Implementation Details and Evaluation Metric:

The evaluation uses task-specific RGB-T datasets and established backbones, training settings, augmentations, and metrics to assess the proposed method.

Implementation Details: The salient detection training uses 224 × 224 inputs, random flips, rotations, clipping, 20 epochs, batch size 8, Adam, and learning rate 0.001.These settings are reported alongside the semantic-segmentation-based framework.
Dataset: RGBT-CC contains 2,030 RGB-T pairs from public scenarios, with 138,389 annotated pedestrians across training, validation, and test splits.The images have 640 × 480 resolution, with 1,545, 300, and 1,200 pairs in the respective splits.
Implementation Details: Crowd counting uses a VGG16-based BL backbone with an MLP decoder composed of two 1 × 1 convolutions.The last EAEF module feeds feature maps into the decoder for final prediction.

3) Results:

Across RGB-T perception experiments, the method improves over baselines, with component ablations supporting contributions from both EAEF modules and conclusions reporting state-of-the-art performance across tasks.

Results: The method with pretraining outperforms DEFNet by 8% in RMSE on the RGBT-CC crowd-counting dataset.The version without pretraining outperforms TAFNet by 2.7% in RMSE.
Ablation Study: Both AIB and ACB improve the baseline, while their combination in EAEF achieves the best performance in the MFNet ablation study.The ablation isolates the contributions of the two EAEF components.
Overall Results: The paper reports state-of-the-art performance across semantic segmentation, object detection, salient detection, and crowd counting.The conclusion frames the result as a better RGB-thermal fusion approach for perception tasks.