Source-linked AI summary
Edge-aware Guidance Fusion Network for RGB Thermal Scene Parsing
Wujie Zhou, Shaohua Dong, Caie Xu, Yaguan Qian
TL;DR
RGB–thermal scene parsing methods can lose boundary detail, underuse high-level features, and inadequately fuse modalities. EGFNet addresses these issues with edge guidance, multimodal and semantic modules, and multitask supervision. Experiments on benchmark datasets and ablations report stronger performance and contributions from the main components.
Problem
RGB–thermal scene parsing remains limited by boundary-detail loss, inadequate high-level feature use, and simple multimodal fusion strategies.
Method
EGFNet uses prior edge maps, an MFM for complementary RGB–thermal fusion, GIM and SIM for high-level semantics, and multitask supervision.
Results
EGFNet outperforms other state-of-the-art methods on MFNet and shows superior visual results under varied challenging lighting conditions.
Takeaways & Limitations
Ablations report worse performance without edge information, MFM, GIM or SIM, or intermediate supervision, supporting the contribution of these components.
Abstract
from arXiv · showhide
RGB thermal scene parsing has recently attracted increasing research interest in the field of computer vision. However, most existing methods fail to perform good boundary extraction for prediction maps and cannot fully use high level features. In addition, these methods simply fuse the features from RGB and thermal modalities but are unable to obtain comprehensive fused features. To address these problems, we propose an edge-aware guidance fusion network (EGFNet) for RGB thermal scene parsing. First, we introduce a prior edge map generated using the RGB and thermal images to capture detailed information in the prediction map and then embed the prior edge information in the feature maps. To effectively fuse the RGB and thermal information, we propose a multimodal fusion module that guarantees adequate cross-modal fusion. Considering the importance of high level semantic information, we propose a global information module and a semantic information module to extract rich semantic information from the high-level features. For decoding, we use simple elementwise addition for cascaded feature fusion. Finally, to improve the parsing accuracy, we apply multitask deep supervision to the semantic and boundary maps. Extensive experiments were performed on benchmark datasets to demonstrate the effectiveness of the proposed EGFNet and its superior performance compared with state of the art methods. The code and results can be found at https://github.com/ShaohuaDong2021/EGFNet.
Proposed EGFNet
EGFNet combines prior edge guidance, multimodal fusion, and high-level feature processing in an encoder–decoder architecture for RGB–thermal scene parsing. Its modules extract complementary cross-modal, boundary, detailed, and semantic information before decoding.
- Architecture: EGFNet uses ResNet-152 encoders for RGB and thermal branches, reducing extracted feature channels to 64 with 1 × 1 convolution.The branch features are denoted Ri and Ti across five levels.
- Architecture: The encoder supplies fusion, boundary, and semantic features, while the decoder uses SFM to combine high-level semantic information with skip-connection features.The decoder later applies multitask deep supervision to semantic and boundary maps.
- Edge-aware guidance: Prior edge guidance applies Sobel detection to both modalities, fuses their edge information, and embeds the resulting map into boundary features.Elementwise multiplication embeds the prior edge information, while fusion with semantic predictions further improves the final semantic features.
- Multimodal fusion: The MFM combines RGB and thermal features through summation, multiplication, concatenation, and convolution to capture complementary cross-modal information.It produces fusion features at five levels rather than relying only on simple addition or concatenation.
- Feature processing: Residual learning and atrous spatial pyramid pooling extract deeper semantic and representative global-context features from the fused representations.The atrous convolutions use rates r = {1, 2, 3, 4}, and the resulting features are processed into detailed and semantic information.
GIM and SIM
GIM and SIM extract high-level semantic information, while SFM progressively combines it with multilevel features. Multitask supervision then trains boundary and semantic predictions, including edge-guided intermediate outputs.
- GIM and SIM: GIM and SIM capture high-level semantic information before SFM fuses cascaded multilevel cross-modal features.The design distinguishes low-level detailed information from high-level comprehensive semantic information.
- SFM: SFM upsamples high-level features to match lower-level feature sizes and combines them through elementwise summation in a coarse-to-fine decoder.The module aggregates multilevel features with high-level deep semantic information to obtain comprehensive features.
- Multitask deep supervision: Boundary supervision resizes boundary features to the edge-map size and embeds prior edge information to produce complete structures with sharper boundaries.The boundary maps B1, B2, and B3 are supervised using weighted cross-entropy loss.
- SGM: SGM upsamples intermediate semantic maps, concatenates them, and applies a 1 × 1 convolution to generate side-out semantic predictions.The semantic maps are upsampled by factors of 16 and 32 using bilinear interpolation.
- Multitask deep supervision: The prior edge map is also embedded into side-out and final semantic predictions, while the total loss combines boundary and semantic losses.Weighted cross-entropy uses separate weights for boundary and semantic loss calculations.
Experimental results
Experiments on MFNet and PST900, together with ablations, evaluate EGFNet against existing methods and isolate the contributions of its components. EGFNet outperforms comparison methods, while removing edge guidance, multimodal fusion, semantic modules, or deep supervision reduces performance.
- Comparative results: EGFNet provides superior segmentation under challenging daytime and nighttime lighting conditions on MFNet.
- Comparative results: Experiments on PST900 indicate that EGFNet has excellent applicability compared with CCNet, ACNet, EFFicient FCN, RTFNet, and PSTNet.
- Ablation studies: Removing prior edge information produces worse performance, demonstrating its importance for scene parsing.
- Ablation studies: Replacing the multimodal fusion module with simple addition lowers performance, supporting the module’s effectiveness.
- Ablation studies: Removing GIM and SIM or using only one supervision stage decreases performance, supporting high-level semantic extraction and multitask deep supervision.