PaperlayerView exact revision on arXiv

Source-linked AI summary

RGB-T Semantic Segmentation with Location, Activation, and Sharpening

Gongyang Li, Yike Wang, Zhi Liu, Xinpeng Zhang, Dan Zeng

arXiv:2210.14530v1cs.CV

TL;DR

RGB-T segmentation targets difficult scenes where RGB-only methods struggle and existing fusion approaches overlook modality or feature-level properties. LASNet uses feature fusion with level-specific modules for location, activation, and edge sharpening, and outperforms state-of-the-art methods on two public datasets. Ablations further show gains from the full module set and its multiplication-plus-summation integration.

  • Problem

    Existing RGB-T fusion methods can ignore RGB/TIR properties or process different feature levels with a unified operation.

  • Method

    LASNet applies CLM, CAM, and ESM to high-, middle-, and low-level cross-modal features for location, activation, and edge sharpening.

  • Results

    LASNet outperforms state-of-the-art methods on the MFNet and PSTNet datasets.

  • Takeaways & Limitations

    Ablations show the full LASNet improves mIoU over the baseline, while removing CLM correlation modeling lowers performance by 2.6%.

Abstract

from arXiv · show

Semantic segmentation is important for scene understanding. To address the scenes of adverse illumination conditions of natural images, thermal infrared (TIR) images are introduced. Most existing RGB-T semantic segmentation methods follow three cross-modal fusion paradigms, i.e. encoder fusion, decoder fusion, and feature fusion. Some methods, unfortunately, ignore the properties of RGB and TIR features or the properties of features at different levels. In this paper, we propose a novel feature fusion-based network for RGB-T semantic segmentation, named \emph{LASNet}, which follows three steps of location, activation, and sharpening. The highlight of LASNet is that we fully consider the characteristics of cross-modal features at different levels, and accordingly propose three specific modules for better segmentation. Concretely, we propose a Collaborative Location Module (CLM) for high-level semantic features, aiming to locate all potential objects. We propose a Complementary Activation Module for middle-level features, aiming to activate exact regions of different objects. We propose an Edge Sharpening Module (ESM) for low-level texture features, aiming to sharpen the edges of objects. Furthermore, in the training phase, we attach a location supervision and an edge supervision after CLM and ESM, respectively, and impose two semantic supervisions in the decoder part to facilitate network convergence. Experimental results on two public datasets demonstrate that the superiority of our LASNet over relevant state-of-the-art methods. The code and results of our method are available at https://github.com/MathLee/LASNet.

I. INTRODUCTION

RGB-T semantic segmentation supplements RGB images with TIR data for difficult scenes, but existing fusion strategies often overlook modality and feature-level characteristics. LASNet addresses this through level-specific location, activation, and sharpening modules, achieving superior results on MFNet and PSTNet.

  • TIR images complement RGB data because they are insensitive to illumination changes, helping address adverse illumination and fog or smoke occlusions.
  • Existing RGB-T methods use encoder, decoder, or feature fusion, but some apply simple or unified operations that ignore RGB, TIR, and feature-level properties.
  • LASNet explores cross-modal features at different levels through three ordered steps: object location, region activation, and edge sharpening.
  • CLM, CAM, and ESM respectively process high-, middle-, and low-level features for object location, region activation, and edge sharpening.
  • LASNet outperforms state-of-the-art methods on the MFNet and PSTNet datasets.

II. RELATED WORK

Related work covers unimodal RGB segmentation, multimodal segmentation, and RGB-T fusion paradigms. LASNet differs by tailoring CLM, CAM, and ESM to high-, middle-, and low-level cross-modal features.

  • A. RGB Semantic Segmentation: RGB semantic segmentation research developed multi-scale, contextual-dependency, and transformer-based techniques from CNN architectures.
  • B. Multi-modal Semantic Segmentation: Unimodal methods often struggle with low contrast, cluttered backgrounds, adverse illumination, and fog or smoke occlusions, motivating multimodal segmentation.
  • B. Multi-modal Semantic Segmentation: RGB-D methods use depth's geometric information through input concatenation, encoder fusion, or feature-level fusion.
  • 2) RGB-T Semantic Segmentation: RGB-T methods address adverse illumination and fog or smoke occlusions using encoder, decoder, and feature fusion paradigms.
  • 2) RGB-T Semantic Segmentation: Unlike methods that process levels indiscriminately or use simple high-level fusion, LASNet tailors CLM, CAM, and ESM to different feature levels.

A. Network Overview

LASNet uses parallel RGB and TIR ResNet-152 branches to produce five-level cross-modal features, then applies level-specific modules before decoder segmentation. CLM, CAM, and ESM target location, activation, and edges, respectively.

  • A. Network Overview: LASNet follows feature fusion with a feature extractor, three specific modules, and a decoder.
  • A. Network Overview: Two parallel ResNet-152 branches extract five-level features from RGB and TIR inputs.
  • A. Network Overview: CLM, CAM, and ESM are assigned to high-, middle-, and low-level features for object location, region activation, and edge sharpening.
  • A. Network Overview: The decoder combines outputs from the three modules to generate the segmentation result Ssem.

B. Collaborative Location Module

The Collaborative Location Module (CLM) models pixel-level correlations between cross-modal high-level RGB and TIR features to locate potential objects. It combines complementary correlation features and adds location supervision for more accurate localization.

  • B. Collaborative Location Module: CLM models pixel-level correlations between cross-modal high-level semantic features to collaboratively identify potential objects.The module uses co-attention to model spatial dependencies in RGB and TIR representations.
  • B. Collaborative Location Module: Element-wise multiplication extracts coexistence information and highlights RGB regions supported by evident TIR content in low-illumination scenes.This operation helps reveal objects that are difficult to distinguish in RGB features but visible in TIR features.
  • B. Collaborative Location Module: CLM combines cross-modal correlation features through element-wise multiplication, summation, and concatenation to obtain informative object-location features.Summation provides comprehensive information, while multiplication highlights common objects.
  • B. Collaborative Location Module: A location head after CLM receives location supervision to improve the accuracy of object localization.The supervision is imposed during training after the CLM output.

C. Complementary Activation Module

The Complementary Activation Module (CAM) processes middle-level cross-modal features to activate precise object regions at multiple scales. It combines multiplication-based complementary features with spatial and channel attention to produce robust discriminative representations.

  • C. Complementary Activation Module: CAM targets exact object regions in middle-level features after high-level CLM features locate potential objects.It operates across feature levels and scales to refine object regions and restore resolution.
  • C. Complementary Activation Module: CAM combines element-wise multiplication and summation of RGB and TIR features to mine common valuable content.Multiplication provides pithy complementary information, while summation supplies broader information without omission.
  • C. Complementary Activation Module: Spatial attention highlights target regions, while channel self-attention models channel-wise dependencies in the combined features.The two attention mechanisms enhance the robustness of discriminative feature representations.
  • C. Complementary Activation Module: CAM is applied to three cross-modal feature levels to mine multi-level complementary information and activate object regions at different scales.The formulation uses spatial attention, channel self-attention, and convolutional layers.

D. Edge Sharpening Module

The Edge Sharpening Module (ESM) uses low-level texture features to capture object details and sharpen object boundaries. It combines cross-modal features with multi-head dilated convolutions to perceive details across receptive fields, then applies edge supervision.

  • D. Edge Sharpening Module: ESM processes low-level features because their rich texture information describes object details and boundaries.This complements the high- and middle-level processing performed by CLM and CAM.
  • D. Edge Sharpening Module: Multi-scale features are aggregated by concatenation to generate the ESM output feature.The resulting feature is designed to perceive details with different receptive fields.
  • D. Edge Sharpening Module: ESM combines RGB and TIR features through element-wise multiplication and summation before applying multi-head dilated convolutions.The parallel dilated convolutions extract multi-scale detail information.
  • D. Edge Sharpening Module: An edge head after ESM receives edge supervision to improve the accuracy of object-edge information.A semantic head after decoder block D2 also receives semantic supervision to improve object-region feature representation.

E. Loss Function

LASNet uses four training supervisions: location, edge, and two semantic objectives. The total loss combines weighted binary cross-entropy terms with a Lovász loss for the final semantic prediction.

  • E. Loss Function: LASNet has location, edge, intermediate semantic, and final semantic supervisions during training.The location and edge objectives follow CLM and ESM, while semantic objectives supervise decoder outputs D2 and Ssem.
  • E. Loss Function: The total loss is Ltotal = 0.5Lloc + Leg + Lsem2 + Lsem.The location, edge, and intermediate semantic losses use weighted binary cross-entropy; the final semantic loss also includes Lovász loss.
  • E. Loss Function: Predicted location and edge maps are generated by LocHead and EdgeHead, while semantic predictions are generated by SemHead.The corresponding ground truths supervise the respective outputs.
  • E. Loss Function: Because datasets provide only semantic ground truth, LASNet algorithmically constructs binary location and edge ground truths.Background is assigned zero for location ground truth, while non-background object edges are assigned one for edge ground truth.

A. Experimental Protocol

LASNet is trained and evaluated on MFNet and PST900 using standard segmentation metrics and fixed dataset partitions, augmentation, and optimization settings.

  • Datasets: LASNet is trained and evaluated on the MFNet and PST900 RGB-T semantic segmentation datasets.MFNet contains daytime and nighttime image pairs, while PST900 contains aligned RGB and TIR pairs across five classes including background.
  • Experimental Setup: The experiments use the datasets’ stated training and testing partitions, with previous PST900 work corrected to 597 training pairs rather than 606.The note on dataset partitioning distinguishes the actual training-set size used by previous works.
  • Evaluation Metrics: Performance is evaluated with mean accuracy (mAcc) and mean intersection over union (mIoU).Both metrics average class-level performance, and higher values indicate better segmentation.
  • Implementation Details: Training retains the original image resolution, applies random flipping and cropping, and uses Ranger with batch size 4 and learning rate 5e−5.The feature extractor is initialized from pretrained ResNet-152 weights, while new convolutional layers receive separate initialization.

B. Comparison with State-of-the-arts

LASNet is compared with state-of-the-art RGB, RGB-D, and RGB-T methods on MFNet and PST900. It achieves strong quantitative and visual results across datasets, daytime scenes, and nighttime scenes.

  • MFNet Dataset: On MFNet, LASNet achieves the best overall mAcc and mIoU among the compared methods.It exceeds the second-best method by 2.7% on mAcc and by 0.1% on mIoU.
  • MFNet Dataset: On MFNet, LASNet obtains seven first-place and four second-place results across 18 reported metrics.Its car stop performance exceeds EGFNet by 8.1% on mAcc and 5.8% on mIoU.
  • Daytime and Nighttime: LASNet shows superiority in both MFNet daytime and nighttime scenes, especially under nighttime illumination conditions.The authors attribute this to the coordinated location, activation, and sharpening steps using TIR information.
  • Visual Comparisons: Visual comparisons show LASNet more accurately identifies, locates, and completes objects than representative methods in cluttered daytime cases.The authors associate localization and region activation with CLM and CAM, while complete object boundaries also benefit from CAM and ESM.
  • PST900 Dataset: On PST900, LASNet ranks first on mIoU, exceeding SA-Gate by 5.35%, and improves fire-extinguisher IoU by 9.00%.It ranks second on mAcc, 2.46% below EGFNet, while exceeding EGFNet by 5.89% on mIoU.

C. Ablation Studies

Ablation studies show that LASNet’s three modules, their internal feature-combination designs, and auxiliary supervision each contribute to segmentation performance.

  • Module Contributions: The full LASNet reaches 54.9% mIoU, compared with 48.3% for the element-wise-summation baseline.Adding ESM, CAM, or CLM individually improves the baseline, while pairwise combinations improve performance further.
  • ESM Components: Removing either multiplication or summation from ESM reduces performance to 53.3% or 53.2%, respectively.Replacing multi-head dilated convolutions with a single convolution also hurts multi-scale edge detection.
  • CAM Components: Removing CAM’s multiplication-based feature enhancement lowers performance, supporting its role in fine region activation.The study also evaluates spatial attention and channel self-attention as components of CAM.
  • CLM Components: Removing cross-modal correlation modeling from CLM causes a 2.6% performance drop, the largest degradation among the CLM variants.Removing multiplication or summation correlation combinations causes drops of 1.7% and 1.9%, respectively.
  • Feature Integration: The original M+S feature integration outperforms all five alternatives using concatenation or only one of multiplication and summation.The authors relate this to combining coexistence information from multiplication with comprehensive information from summation.
  • Auxiliary Supervision: Auxiliary location and edge supervision are evaluated through variants that remove both or retain only one supervision.The reported purpose is to improve segmentation accuracy without increasing inference-time computational cost.

V. CONCLUSION

LASNet addresses RGB-T semantic segmentation by processing features at different levels with dedicated modules for location, activation, and edge sharpening. Its ablations and comparisons support the effectiveness and superiority of this integrated design.

  • LASNet follows feature fusion and assigns CLM, CAM, and ESM to object location, region activation, and edge sharpening.The modules process RGB and TIR features at different levels and focus on complementary information from multiplication and summation.
  • CLM, CAM, and ESM are plug-and-play modules designed for level-specific processing of RGB-T features.
  • The modules extract complementary information from feature combinations of multiplication and summation rather than directly from RGB and TIR features.
  • Comprehensive ablation studies demonstrate the effectiveness of the three modules, while extensive comparisons demonstrate LASNet's superiority.
Loading 2210.14530v1…