Source-linked AI summary
FEANet: Feature-Enhanced Attention Network for RGB-Thermal Real-time Semantic Segmentation
Fuqin Deng, Hua Feng, Mingjian Liang, Hongmin Wang, Yong Yang, Yuan Gao, Junfeng Chen, Junjie Hu, Xiyue Guo, Tin Lun Lam
TL;DR
RGB-T segmentation must preserve useful spatial details while handling difficult lighting and modality-specific features. FEANet addresses this with two-stage feature extraction and fusion plus FEAM channel-spatial enhancement; experiments report stronger small-object segmentation and approximately 35 images/s inference on an RTX 2080 Ti.
Problem
Existing RGB-T segmentation methods face challenging lighting, cluttered scenes, and incomplete or incompatible extraction and fusion of RGB and thermal features.
Method
FEANet uses two-stage RGB-T feature extraction and fusion, with FEAM progressively enhancing multi-level features through channel and spatial attention.
Results
+7.1 % Acc and +0.2 % IoU for Guardrail, and +5.4 % Acc and +8.4 % IoU for Color Cone, compared to FuseSeg-161.
Takeaways & Limitations
FEANet improves small-target segmentation and produces sharper object boundaries while running at approximately 35 images/s on a single NVIDIA Geforce RTX 2080 Ti GPU.
Abstract
from arXiv · showhide
The RGB-Thermal (RGB-T) information for semantic segmentation has been extensively explored in recent years. However, most existing RGB-T semantic segmentation usually compromises spatial resolution to achieve real-time inference speed, which leads to poor performance. To better extract detail spatial information, we propose a two-stage Feature-Enhanced Attention Network (FEANet) for the RGB-T semantic segmentation task. Specifically, we introduce a Feature-Enhanced Attention Module (FEAM) to excavate and enhance multi-level features from both the channel and spatial views. Benefited from the proposed FEAM module, our FEANet can preserve the spatial information and shift more attention to high-resolution features from the fused RGB-T images. Extensive experiments on the urban scene dataset demonstrate that our FEANet outperforms other state-of-the-art (SOTA) RGB-T methods in terms of objective metrics and subjective visual comparison (+2.6% in global mAcc and +0.8% in global mIoU). For the 480 x 640 RGB-T test images, our FEANet can run with a real-time speed on an NVIDIA GeForce RTX 2080 Ti card.
I. INTRODUCTION
RGB-T segmentation supplements RGB imagery with thermal information for difficult lighting conditions, but existing methods still face feature-extraction and fusion challenges. FEANet addresses these issues with two-stage processing and FEAM-based feature enhancement.
- Motivation: RGB images can produce inaccurate predictions in similar-texture, cluttered, dim-light, or dark scenes, while thermal information supplements them.Thermal imagery is described as robust and effective for reducing ambiguity in challenging lighting conditions.
- Challenges: Existing RGB-T models can lose performance in cluttered backgrounds and varying illumination, motivating further advances for urban scenes.
- Challenges: High-level features capture semantic context, whereas low-level features provide micro details for reducing noise and refining segmentation boundaries.Direct multi-level extraction and merging can incompletely extract noisy low-level features, producing less-sharp boundaries.
- Challenges: Thermal images introduce low-quality noise, while common fusion strategies treat RGB and thermal information from the same perspective.RGB provides color and texture, whereas thermal maps provide spatial relations among objects.
- Proposed approach: FEANet uses two stages: FEAM progressively refines multi-level features, then corresponding RGB and thermal features are fused for segmentation.The proposed design targets object boundaries and small target objects in RGB-T urban-scene segmentation.
A. Semantic Segmentation
Semantic segmentation evolved from end-to-end convolutional models toward encoder-decoder architectures and real-time designs. These approaches balance resolution recovery, accuracy, and inference speed with varying trade-offs.
- Semantic Segmentation: Fully Convolutional Networks introduced end-to-end architectures that outperformed traditional hand-crafted feature extraction methods.
- Semantic Segmentation: SegNet used a pretrained VGG16 encoder with an up-sampling decoder, achieving state-of-the-art accuracy but low inference speed.
- Semantic Segmentation: ENet targeted real-time semantic segmentation and processed 480×640 RGB images efficiently, but underperformed SegNet on spectral-image datasets.
- Semantic Segmentation: BiSeNet combined spatial and semantic paths to preserve spatial information and obtain a sufficient receptive field, but had slow inference speed.
B. RGB-T Semantic Segmentation
RGB-T segmentation combines visible and thermal modalities to improve urban-scene understanding, while prior work explores separate processing and fusion. FEANet extends this direction with two encoder streams, FEAM enhancement, and decoder-based resolution recovery.
- RGB-T Semantic Segmentation: RGB-T urban-scene methods combine RGB and thermal images because RGB-only datasets and models are limited under challenging conditions.
- RGB-T Semantic Segmentation: Prior RGB-T fusion methods include separate RGB and spectral processing, while FEANet explicitly emphasizes feature enhancement before decoding.
- RGB-T Semantic Segmentation: FEANet uses two encoder streams and one decoder stream to extract and recover multi-level RGB-T features.The architecture is presented as an encoder-decoder design for RGB-T semantic segmentation.
- RGB-T Semantic Segmentation: The proposed FEAM enhances multi-level features and fuses RGB and thermal information in a complementary way.
A. Overall Architecture
FEANet extracts RGB and thermal features through two encoder streams, refines them with FEAM, fuses corresponding modalities, and restores resolution through a decoder. This design targets detail preservation and small-object boundaries.
- A. Overall Architecture: FEANet has two feature-extraction stages followed by resolution restoration through an output decoder.The architecture uses two encoder streams and one decoder stream.
- A. Overall Architecture: High-level features capture global context but lose object details, so upsampling them alone can blur predictions and boundaries.FEAM is designed to distinguish regions that are too small to detect easily.
- A. Overall Architecture: Stage 1 extracts and refines RGB and thermal maps with FEAM, while stage 2 aggregates corresponding maps through elementwise summation.The refined maps are then transmitted to the decoder.
- A. Overall Architecture: The two-stage feature-extraction strategy is intended to recover rich semantic information lost during intensive feature extraction.
B. Encoder-Feature Extracting
FEANet uses separate RGB and thermal encoder streams with FEAM modules to refine multi-level features. Sequential channel and spatial attention extracts foreground and small-target cues while preserving feature information.
- Encoder streams: Both RGB and thermal streams use five ResNet convolutional blocks, with a FEAM attached after every block.The first thermal convolution is modified for single-channel input.
- Feature enhancement: FEAM enhances compatibility between RGB and thermal features while preserving multi-level image information.It is inserted after each convolutional layer in both encoder streams.
- Attention mechanism: Channel attention emphasizes convolutional features and foreground cues, while spatial attention searches global areas for informative small-target regions.The two operations are applied sequentially within FEAM.
- Attention mechanism: FEAM is presented as the first attention mechanism in this work for extracting informative cues from RGB and thermal multi-level features.The paper reports effectiveness in RGB-T semantic segmentation experiments.
C. Decoder-Resolution Restoring
The decoder progressively restores feature resolution from multi-level encoder outputs to refine detailed segmentation maps. Its transposed blocks use normalization, convolution, upsampling, residual connections, and receptive-field enlargement.
- Resolution restoration: The decoder receives final RGB and thermal feature maps and restores predictions to the original image resolution.It is designed to leverage multi-level information for detail-pixel refinement.
- Transposed blocks: Decoder block A combines batch normalization, ReLU activation, convolution, and an element-wise shortcut connection.The shortcut links the block input with the output of the final batch-normalization layer.
- Transposed blocks: Decoder block B contains Conv 1 and two transposed-convolution layers, with an additional branch that enlarges the receptive field and a residual connection.The supplied configuration identifies these components as part of the residual-based transposed block.
- Evaluation setting: The MFNet dataset contains 1,569 RGB-thermal pairs from urban street scenes, split into 784 training, 392 validation, and remaining test pairs.It includes eight labeled object classes plus an unlabeled background class, with daytime and nighttime images.
B. Training Details
Training combines DiceLoss and SoftCrossEntropy under SGD optimization with cosine-annealed warm restarts. DiceLoss is defined over predicted and ground-truth binary segmentation volumes, while SoftCrossEntropy uses class indicators and predicted probabilities.
- Optimization: SGD trains the network with momentum 0.9, weight decay 0.0005, and an initial learning rate of 0.03.CosineAnnealingWarmRestarts gradually decreases the learning rate.
- Loss function: The training loss adds DiceLoss and SoftCrossEntropy with equal weights of 0.5.The combined objective is used for model training.
- DiceLoss: DiceLoss compares predicted binary segmentation volume values with corresponding ground-truth binary volume values across N voxels.The formulation is differentiable and yields a gradient.
- SoftCrossEntropy: SoftCrossEntropy uses n as the batch size; in this work, n = 5.The equation also uses binary class indicators and predicted probabilities normalized to a probability distribution.
C. Evaluation Metrics
The paper evaluates segmentation with Accuracy and Intersection-over-Union, reporting mean values across the eight labeled object classes.
- mAcc and mIoU average class-wise Accuracy and IoU values across all labeled classes.The dataset defines k = 8 hand-labeled object classes.
- IoU calculates the intersection of true labels and predictions separately for each class.
- The mAcc equation uses per-class correctly classified pixels and misclassification counts to compute mean Accuracy.
D. Results And Analysis
FEANet shows strong segmentation performance on small target objects and sharp boundaries, while maintaining real-time inference speed on a single GPU.
- +7.1% Acc and +0.2% IoU are reported for Guardrail versus FuseSeg-161.
- +5.4% Acc and +8.4% IoU are reported for Color Cone versus FuseSeg-161.
- FEANet produces sharper object boundaries, whereas other methods are disturbed by background regions.
- FuseSeg-161 performs best on Person and Bike, while RTFNet152 performs best on Car Stop.
- Approximately 35 images/s are achieved on a single NVIDIA Geforce RTX 2080 Ti GPU.
E. Ablation Study
The ablation study compares FEAM placements across RGB and thermal streams and reports that using FEAM in both streams generally performs best.
- FRTS, with FEAM in both streams, usually outperforms NFRS, NFTS, and NFRTS on RGB-T segmentation.The variants remove FEAM from the RGB stream, thermal stream, or both.
- FEAM applied at every layer provides a universal improvement in detection performance.
- FEAM in the thermal stream contributes more to the results than the other tested placements.
- FEANet is designed for urban-scene segmentation, with reported improvements on small target objects and sharp object boundaries.