Source-linked AI summary
RS3Mamba: Visual State Space Model for Remote Sensing Images Semantic Segmentation
Xianping Ma, Xiaokang Zhang, Man-On Pun
TL;DR
Remote sensing segmentation requires both long-range modeling and computational efficiency, challenges for CNNs and Transformers. RS3Mamba combines a VSS auxiliary branch with a convolutional main branch and CCM fusion, and experiments on ISPRS Vaihingen and LoveDA Urban report improved performance over existing methods.
Problem
CNNs have insufficient long-range modeling, while Transformers face high computational complexity in remote sensing image semantic segmentation.
Method
RS3Mamba uses VSS blocks in an auxiliary branch to provide global information to a convolution-based main branch, with CCM enhancing and fusing their features.
Results
90.34% mF1 and 82.78% mIoU were achieved on ISPRS Vaihingen, increasing 0.49% and 0.81%, respectively, over UNetformer.
Takeaways & Limitations
Experiments on ISPRS Vaihingen and LoveDA Urban demonstrate the effectiveness and potential of RS3Mamba for remote sensing semantic segmentation.
Abstract
from arXiv · showhide
Semantic segmentation of remote sensing images is a fundamental task in geoscience research. However, there are some significant shortcomings for the widely used convolutional neural networks (CNNs) and Transformers. The former is limited by its insufficient long-range modeling capabilities, while the latter is hampered by its computational complexity. Recently, a novel visual state space (VSS) model represented by Mamba has emerged, capable of modeling long-range relationships with linear computability. In this work, we propose a novel dual-branch network named remote sensing images semantic segmentation Mamba (RS3Mamba) to incorporate this innovative technology into remote sensing tasks. Specifically, RS3Mamba utilizes VSS blocks to construct an auxiliary branch, providing additional global information to convolution-based main branch. Moreover, considering the distinct characteristics of the two branches, we introduce a collaborative completion module (CCM) to enhance and fuse features from the dual-encoder. Experimental results on two widely used datasets, ISPRS Vaihingen and LoveDA Urban, demonstrate the effectiveness and potential of the proposed RS3Mamba. To the best of our knowledge, this is the first vision Mamba specifically designed for remote sensing images semantic segmentation. The source code will be made available at https://github.com/sstary/SSRS.
I. INTRODUCTION
Remote sensing segmentation must handle complex scenes and scale variation, but CNNs struggle with long-range dependencies while Transformers incur high computational and memory costs. RS3Mamba addresses these issues with a VSS-assisted dual-branch design and CCM, evaluated on two public datasets.
- Semantic segmentation classifies every remote sensing image pixel into distinct categories for analyzing surface conditions.
- CNNs are limited by local receptive fields, whereas Transformers model long-range dependencies but have high computational complexity and memory requirements.
- RS3Mamba uses VSS blocks in an auxiliary branch to provide global information to a convolution-based main branch.
- The collaborative completion module fuses dual-branch features to address differences between global and local semantics.
- RS3Mamba is presented as the first exploration of VSS-based models for remote sensing image semantic segmentation.
- Experiments on ISPRS Vaihingen and LoveDA Urban report advantages over existing CNN- and Transformer-based methods.
II. METHODOLOGY
RS3Mamba combines a VSS auxiliary encoder, a residual main encoder with CCM fusion, and a decoder. Features are extracted and fused at four scales before decoding.
- RS3Mamba contains a VSS auxiliary encoder, a residual main encoder with CCMs, and a decoder.
- The two encoders generate corresponding multiscale features that are fused by CCM before entering the decoder.
A. Auxiliary encoder
The auxiliary encoder uses successive VSS stages to model long-range dependencies through four-directional selective scanning. Its independent feature flow supplies multiscale global representations for later fusion.
- The VSS block uses 2D-selective-scan as its core unit for modeling long-range dependencies.
- SS2D expands image patches into four directional sequences, processes them with SSMs, and merges the outputs into a 2D feature map.
- The auxiliary encoder has four stages, each pairing a patch operator with a VSS block.
- The first stage uses Patch Embedding, while the next three use Patch Merging before their VSS blocks.
- Because the auxiliary branch is independent of the main branch, it can be computed entirely beforehand.
B. Main encoder and CCM
The main encoder extracts local representations with ResNet18, while CCM enhances and fuses complementary global and local features at multiple scales before decoding.
- ResNet18 serves as the main encoder, using four residual blocks to learn local multiscale representations.
- Auxiliary features are incorporated into the main branch at each scale through CCM to compensate for limited global information extraction.
- CCM contains parallel global and local branches that enhance main-branch and auxiliary-branch features, respectively.
- Window-based multihead self-attention models long-range dependencies for main-branch features, while convolution adds local detail to VSS-derived features.
- The decoder receives fused features through skip connections and produces the final prediction map using cross-entropy training.
A. Datasets
The study evaluates RS3Mamba on ISPRS Vaihingen and LoveDA Urban, which differ in resolution, object categories, and label accuracy.
- ISPRS Vaihingen: ISPRS Vaihingen contains 16 very-high-resolution NIRRG orthophotos with five foreground classes and one background class.The images average 2500 × 2000 pixels with a 9 cm ground sampling distance; 12 patches are used for training and four for testing.
- LoveDA Urban: LoveDA Urban contains 1833 RGB images at 1024×1024 pixels covering seven landcover categories.The imagery has a 30 cm ground sampling distance.
- LoveDA Urban: LoveDA Urban is partitioned into 1156 training images and 677 testing images.
- Dataset comparison: Using both datasets tests RS3Mamba across differences in sampling resolution, ground-object categories, and label accuracy.
B. Experimental Setup
Experiments benchmark RS3Mamba against established supervised segmentation models and include qualitative comparisons on ISPRS Vaihingen.
- Training and baselines: UNetformer with a ResNet18 backbone serves as the baseline, alongside ABCNet, TransUNet, and CMTFNet as comparison methods.All models are trained in PyTorch on a single NVIDIA GeForce RTX 4090 GPU using SGD.
- Qualitative evaluation: The ISPRS Vaihingen qualitative comparison uses 512×512 inputs and shows NIRRG images, ground truth, and outputs from five models.Two samples are presented for each model, including RS3Mamba.
C. Performance Comparison
RS3Mamba improves segmentation performance over the baseline and existing methods on ISPRS Vaihingen and LoveDA Urban, with qualitative comparisons showing its predicted outputs.
- Qualitative evaluation: Qualitative comparisons on LoveDA Urban use 1024 × 1024 inputs and display NIRRG images, ground truth, and predictions from five models.Two samples are shown for each model.
- ISPRS Vaihingen: 90.34% mF1 and 82.78% mIoU are achieved on ISPRS Vaihingen, increasing 0.49% and 0.81% over UNetformer.RS3Mamba also improves selected class-level F1 and IoU values relative to CMTFNet and UNetformer.
- Qualitative evaluation: On ISPRS Vaihingen, RS3Mamba is described as producing smoother borders and fewer noise points than the compared methods.
- LoveDA Urban: On LoveDA Urban, RS3Mamba improves overall mF1 by 1.52% and mIoU by 1.81% over the baseline.The reported category-level gains include the largest improvements for agriculture.
- LoveDA Urban: LoveDA Urban results report F1/IoU values of 56.86%/39.72% for background, 73.35%/57.92% for road, 56.80%/39.67% for forest, and 50.72%/33.98% for agriculture.
D. Ablation Study
Ablation experiments examine the residual main branch, VSS auxiliary branch, and CCM, while complexity analysis compares RS3Mamba with other models.
- Ablation study: Four ablation experiments evaluate the VSS auxiliary branch and CCM within the RS3Mamba architecture.The components include the residual main branch, VSS auxiliary branch, and CCM for feature fusion.
- Ablation study: Replacing the residual main branch with the VSS auxiliary branch does not outperform the classic ResNet in the reported ablation setting.The passage attributes this to VSS-based backbones being in development and having insufficient pre-training.
- Ablation study: Simple element-wise summation in the dual-branch model is used as an ablation instead of CCM to assess the proposed feature-fusion design.
- Model complexity: Complexity analysis measures FLOPs, model parameters, and memory footprint, with lower values described as preferable.
- Model complexity: RS3Mamba is more complex than UNetformer but has lower computational complexity and model scale than TransUNet.The added complexity is attributed to the auxiliary branch and cross-branch feature-fusion module.
IV. CONCLUSION
RS3Mamba introduces a VSS-based auxiliary branch for remote sensing semantic segmentation, supplying global awareness alongside a convolutional main branch. Evaluations on two remote sensing datasets show it outperforms existing CNN- and Transformer-based methods.
- RS3Mamba introduces a VSS-based model for remote sensing image semantic segmentation.
- Its VSS auxiliary branch provides additional global awareness information with minimal linear computational complexity.
- The method bridges global and local branch features using a CAM module before element-wise fusion.
- Experiments on two distinct remote sensing datasets show RS3Mamba outperforms state-of-the-art CNN- and Transformer-based segmentation methods.