PaperlayerView exact revision on arXiv

Source-linked AI summary

CM-UNet: Hybrid CNN-Mamba UNet for Remote Sensing Image Semantic Segmentation

Mushui Liu, Jun Dan, Ziqian Lu, Yunlong Yu, Yingming Li, Xi Li

arXiv:2405.10530v1cs.CV

TL;DR

Large remote-sensing images pose challenges for CNN- and Transformer-based segmentation methods in capturing long-range context efficiently. CM-UNet pairs a CNN encoder with a Mamba-based decoder, CSMamba blocks, and MSAA; experiments report superiority across three datasets, including a Potsdam mF1 of 93.05%.

  • Problem

    CNN- and Transformer-based approaches for large remote-sensing images are described as suboptimal for capturing long-range dependencies or computationally complex.

    Source: §I · p. 1, A. CSMamba Block · p. 2

  • Method

    CM-UNet pairs a CNN encoder with a Mamba-based decoder using attention-gated CSMamba blocks and MSAA to fuse multi-scale features.

    Source: §I · pp. 1–2, A. CSMamba Block · p. 2

  • Results

    CM-UNet is reported superior across three remote-sensing datasets; on ISPRS Potsdam, its mF1 is 93.05%.

    Source: 1) The ISPRS Potsdam dataset · p. 3, §IV · p. 5

  • Takeaways & Limitations

    The authors report that CM-UNet addresses target variation in large remote-sensing images and demonstrates superiority on three segmentation datasets.

    Source: §IV · p. 5

Abstract

from arXiv · show

Due to the large-scale image size and object variations, current CNN-based and Transformer-based approaches for remote sensing image semantic segmentation are suboptimal for capturing the long-range dependency or limited to the complex computational complexity. In this paper, we propose CM-UNet, comprising a CNN-based encoder for extracting local image features and a Mamba-based decoder for aggregating and integrating global information, facilitating efficient semantic segmentation of remote sensing images. Specifically, a CSMamba block is introduced to build the core segmentation decoder, which employs channel and spatial attention as the gate activation condition of the vanilla Mamba to enhance the feature interaction and global-local information fusion. Moreover, to further refine the output features from the CNN encoder, a Multi-Scale Attention Aggregation (MSAA) module is employed to merge the different scale features. By integrating the CSMamba block and MSAA module, CM-UNet effectively captures the long-range dependencies and multi-scale global contextual information of large-scale remote-sensing images. Experimental results obtained on three benchmarks indicate that the proposed CM-UNet outperforms existing methods in various performance metrics. The codes are available at https://github.com/XiaoBuL/CM-UNet.

I. INTRODUCTION

Large remote-sensing scenes challenge CNN- and Transformer-based UNet approaches to balance global context and computational efficiency. CM-UNet combines a CNN encoder with a Mamba-based decoder, CSMamba blocks, and MSAA to integrate local and global information.

I. INTRODUCTION

  • CNN- and Transformer-based UNet approaches can struggle to capture global context or incur high computational complexity on large remote-sensing scenes with significant target variation.The paper motivates more efficient architectures that capture comprehensive local-global information.Source: §I · p. 1
  • Prior remote-sensing Mamba methods either train Vision Mamba blocks from scratch or use pretrained blocks, and rarely integrate local and global information.The authors suggest this may limit their use of features from well-pretrained CNN models.Source: §I · p. 1
  • CM-UNet combines a CNN encoder with a CSMamba-based Mamba decoder to integrate local and global information for remote-sensing segmentation.The decoder uses channel and spatial attention, while MSAA assists the encoder skip connections and multi-output supervision progressively guides segmentation.Source: §I · pp. 1–2
  • Experiments on ISPRS Potsdam, ISPRS Vaihingen, and LoveDA are reported to show CM-UNet's superiority.The paper identifies these as three well-known, publicly available remote-sensing datasets.Source: §I · p. 2

II. METHODOLOGY

CM-UNet uses a ResNet encoder, an MSAA feature-fusion module, and a CSMamba decoder in place of vanilla UNet skip connections.

II. METHODOLOGY

  • The MSAA module fuses multi-level ResNet features and replaces vanilla UNet skip connections to support the decoder.The encoder extracts multi-level features, and assembled CSMamba blocks aggregate local features into a comprehensive semantic understanding.Source: §II · p. 2
  • A ResNet encoder extracts multi-level features for the CM-UNet architecture.These features are fused by MSAA before supporting the decoder.Source: §II · p. 2
  • The CSMamba decoder assembles blocks to aggregate local features into a comprehensive semantic understanding.It is one of the framework's three core components, alongside the CNN encoder and MSAA module.Source: §II · p. 2

A. CSMamba Block

The CSMamba block combines Mamba-based long-range modeling with channel and spatial attention for remote-sensing image features. Its 2D selective scan processes features in four directions before restoring their 2D structure.

A. CSMamba Block

  • The design responds to the need for global and local information processing while avoiding the square time complexity attributed to self-attention transformers on large images.The paper motivates Mamba through its long-range modeling with linear complexity.Source: A. CSMamba Block · p. 2
  • The block uses parallel branches for 2D-SSM processing and channel-spatial attention, then combines them by Hadamard product and projects the result back to the input channel count.The input feature has shape H × W × C, and the output has the same shape.Source: A. CSMamba Block · p. 2
  • The 2D-SSM flattens image features into sequences, scans in four directions to capture long-range dependencies, and merges the sequences to recover 2D structure.The scan directions are top-left to bottom-right, bottom-right to top-left, top-right to bottom-left, and bottom-left to top-right.Source: A. CSMamba Block · p. 3

B. Multi-Scale Attention Aggregation.

MSAA refines and fuses features from multiple ResNet encoder levels through spatial and channel attention paths before they reach subsequent network layers.

B. Multi-Scale Attention Aggregation.

  • MSAA concatenates outputs from ResNet stages F1, F2, and F3 with adjacent-stage features, then feeds the combined features into the module for refinement.The combined feature representation is denoted F-hat and has dimensions C1 × H × W.Source: B. Multi-Scale Attention Aggregation · p. 3
  • Spatial refinement projects channels, sums convolutions with multiple kernel sizes, then uses mean and max pooling and a 7 × 7 convolution to form spatial features.The spatial features are multiplied elementwise with the sigmoid-activated feature map.Source: B. Multi-Scale Attention Aggregation · p. 3
  • In parallel, global average pooling and 1 × 1 convolutions with ReLU generate channel attention, which is combined with the spatially refined map.The resulting features carry refined spatial and channel-wise information for subsequent network layers.Source: B. Multi-Scale Attention Aggregation · p. 3

C. Multi-Output Supervision.

CM-UNet applies intermediate supervision at each CSMamba decoder block, producing progressively generated segmentation maps. Each block’s features are mapped to class predictions, and training combines cross-entropy and Dice losses.

C. Multi-Output Supervision.

III. EXPERIMENTS

Experiments evaluate CM-UNet on ISPRS Potsdam, ISPRS Vaihingen, and LoveDA using mF1, mIoU, and OA. The setup specifies the dataset splits and training procedure, and lists segmentation methods used as comparison benchmarks.

A. Datasets

  • Evaluation uses Potsdam, Vaihingen, and LoveDA, with mF1, mIoU, and OA as metrics.Potsdam has 14 test images and 23 training images after excluding image 710; Vaihingen has 12 training patches and 4 test patches; LoveDA has 1,156 training and 677 test images.Source: A. Datasets · p. 3

B. Implementation details

  • Training uses one NVIDIA 3090, AdamW at a 6e-4 base learning rate, 100 epochs, batch size 16, and 512 × 512 random crops.The passage also specifies cosine learning-rate adjustment, augmentations, and flip-based test-time augmentation.Source: B. Implementation details · p. 3

C. Performance Comparison

  • Benchmarks include DeepLabV3+, DANet, ABCNet, BANet, CMTFNet, UNetformer, ESDINet, and Segmenter.The listed approaches use encoder architectures including R18, VMamba, and Swin-Base.Source: C. Performance Comparison · p. 3

1) The ISPRS Potsdam dataset:

On Potsdam and Vaihingen, CM-UNet reports strong segmentation metrics and qualitative comparisons with UNetFormer. The supplied results describe both benchmark scores and visual differences in delineation.

1) The ISPRS Potsdam dataset:

  • 93.05% mF1, 91.86% OA, and 87.21% mIoU put CM-UNet ahead of UNetformer by 0.25%, 0.56%, and 0.41%, respectively.On Potsdam, the reported mIoU gains over DANet and Segmenter are 6.91% and 6.51%, respectively.Source: 1) The ISPRS Potsdam dataset · p. 3, Table I · p. 4

2) The ISPRS Vaihingen Dataset:

3) The LoveDA Dataset:

On LoveDA, CM-UNet reports an mIoU of 52.17% and strong performance in several classes. Visual comparisons describe its delineation of land-cover features in complex urban and agricultural scenes.

3) The LoveDA Dataset:

D. Further Analysis

The analysis reports that CM-UNet balances lower FLOPs and parameter counts with a modest memory footprint while surpassing other models in mIoU. Ablations show that MSAA and multi-output supervision each improve segmentation, with their combination yielding the highest gains across metrics.

1) Effect of Model Architecture:

2) Model Complexity:

  • CM-UNet combines lower FLOPs and parameter counts with a relatively modest memory footprint, while surpassing other models in mIoU on Vaihingen.The complexity comparison used two 256 × 256 images on a single NVIDIA 3090 GPU.Source: 2) Model Complexity · pp. 4–5, Table V · p. 5

IV. CONCLUSION

The conclusion presents CM-UNet as an efficient framework for remote-sensing semantic segmentation, validated on three datasets. Its ResNet encoder and CSMamba decoder are complemented by MSAA and multi-output enhancement for multi-scale feature learning.

IV. CONCLUSION

  • CM-UNet was validated on three remote-sensing semantic-segmentation datasets, with experiments demonstrating the approach’s superiority.The framework uses a ResNet encoder, CSMamba decoder, MSAA module, and multi-output enhancement.Source: §IV · p. 5
Loading 2405.10530v1…