Source-linked AI summary
Samba: Semantic Segmentation of Remotely Sensed Images with State Space Model
Qinfeng Zhu, Yuanzhi Cai, Yuan Fang, Yihan Yang, Cheng Chen, Lei Fan, Anh Nguyen
TL;DR
High-resolution remotely sensed image segmentation challenges CNN receptive fields and ViT sequence handling. Samba addresses this with Mamba-based encoder blocks and a UperNet decoder, outperforming CNN- and ViT-based methods across three datasets, while limited local-information extraction remains a stated constraint.
Problem
High-resolution remote-sensing segmentation challenges CNNs through limited receptive fields and ViTs through long-sequence complexity and substantial data requirements.
Method
Samba combines Mamba-based Samba Blocks and MLPs in an encoder-decoder framework with UperNet as decoder for multi-level semantic extraction.
Results
Samba surpassed state-of-the-art CNN- and ViT-based methods on LoveDA, ISPRS Vaihingen, and ISPRS Potsdam; on LoveDA, it exceeded Segformer by 3.95% and ConvNeXt by 10.3% in mIoU.
Takeaways & Limitations
The study presents Samba as the first Mamba-based application for remotely sensed image segmentation and establishes a benchmark for Mamba-based techniques.
Takeaways & Limitations
Mamba-based methods have limited local-information extraction, motivating future combinations with CNNs and transfer learning under limited annotations.
Abstract
from arXiv · showhide
High-resolution remotely sensed images pose a challenge for commonly used semantic segmentation methods such as Convolutional Neural Network (CNN) and Vision Transformer (ViT). CNN-based methods struggle with handling such high-resolution images due to their limited receptive field, while ViT faces challenges in handling long sequences. Inspired by Mamba, which adopts a State Space Model (SSM) to efficiently capture global semantic information, we propose a semantic segmentation framework for high-resolution remotely sensed images, named Samba. Samba utilizes an encoder-decoder architecture, with Samba blocks serving as the encoder for efficient multi-level semantic information extraction, and UperNet functioning as the decoder. We evaluate Samba on the LoveDA, ISPRS Vaihingen, and ISPRS Potsdam datasets, comparing its performance against top-performing CNN and ViT methods. The results reveal that Samba achieved unparalleled performance on commonly used remote sensing datasets for semantic segmentation. Our proposed Samba demonstrates for the first time the effectiveness of SSM in semantic segmentation of remotely sensed images, setting a new benchmark in performance for Mamba-based techniques in this specific application. The source code and baseline implementations are available at https://github.com/zhuqinfeng1999/Samba.
I. INTRODUCTION
The introduction identifies limitations of CNNs and ViTs for high-resolution remotely sensed image segmentation and presents Samba as an SSM-based alternative. Samba combines Mamba with MLPs in an encoder and UperNet as decoder, achieving strong benchmark performance across three datasets.
- Motivation: CNNs struggle with high-resolution images because their limited receptive field restricts semantic context.Scaling can lose resolution, while dilated convolution may lose information through coarse feature subsampling.
- Motivation: ViTs provide global receptive fields but face exponentially increasing attention complexity as image resolution grows and require substantial training data.These constraints are particularly relevant when remotely sensed image datasets have scarce training data.
- Motivation: Mamba uses an SSM to capture global semantic information with linear complexity for long sequences.This motivates replacing ViT multi-head self-attention with Mamba in vision tasks.
- Approach: Samba combines Mamba and multiple MLPs into Samba blocks within an encoder-decoder framework using UperNet as decoder.The framework is designed to extract multi-level semantic information from remotely sensed images.
- Evaluation: Samba achieved unparalleled performance against top-performing CNN and ViT methods on LoveDA, ISPRS Vaihingen, and ISPRS Potsdam without pretrained parameters.The authors describe this as an effective application of SSMs and a benchmark for Mamba-based remote-sensing segmentation.
- Contributions: The study introduces Samba as the first Mamba-based architecture for remotely sensed image segmentation and establishes a benchmark through extensive comparisons.The authors also present the experiments as evidence of Mamba’s potential as a segmentation backbone.
II. METHODOLOGY
Samba uses a four-stage encoder with progressive downsampling, followed by a UperNet decoder that upsamples features into segmentation results.
- Encoder: Samba’s encoder organizes Samba Blocks into four stages for progressive spatial downsampling.The encoder begins with an H×W×3 input image and successively reduces feature dimensions.
- Decoder: UperNet processes the encoder features and incrementally upsamples them to produce segmentation results.This forms the decoder portion of Samba’s encoder-decoder architecture.
B. Samba Block
The Samba Block retains the ViT encoder’s general structure while replacing multi-head self-attention with a Mamba block for efficient high-resolution feature extraction.
- ViT-inspired design: The ViT encoder combines multi-head self-attention, residual connections, Layer Normalization, and an MLP-based FFN.These components capture information across representational subspaces and add nonlinear transformations.
- Samba Block: Samba Blocks replace ViT multi-head self-attention with Mamba to avoid quadratic computation on high-resolution image sequences.A Mamba Block and MLP are combined to strengthen representation and learning for complex data.
C. Mamba Block
The Mamba Block applies an input-dependent SSM to image features, discretizes its continuous dynamics for sequential processing, and adapts Mamba to visual inputs through projection and normalization components.
- Selective state modeling: Mamba makes SSM parameters input-dependent, allowing selective propagation or discarding of information at each token.This input-dependent behavior has motivated applications in computer vision.
- State-space formulation: The core SSM maps an input sequence through a latent state using evolution matrix A and projection parameters B and C.Its continuous formulation is expressed through ordinary differential equations for latent-state evolution and output generation.
- Architecture: The overall Samba architecture is presented as a dedicated encoder-decoder design for semantic segmentation.The cited architecture figure provides the high-level layout of the model.
- Discretization: The continuous state transition is discretized because deep-learning models use discrete rather than continuous transitions.Discrete outputs are obtained from sampled input values and the input time step.
- Visual adaptation: Samba adapts Mamba to images by linearly projecting flattened patches and combining convolutional layers, projection layers, and Layer Normalization.Unlike prior visual adaptations, it does not scan unfolded image patches in multiple directions and builds the block on H3.
A. Datasets and Metrics
The study evaluates Samba on three remote-sensing benchmarks and uses mIoU to assess segmentation accuracy through class-level true-positive, false-positive, and false-negative counts.
- Datasets: Samba is validated on the LoveDA, ISPRS Vaihingen, and ISPRS Potsdam benchmark datasets.The datasets differ in image resolution, channels, class counts, and train-validation-test arrangements.
- Datasets: LoveDA contains 2,522 training images, 1,669 validation images, and 1,796 test images across seven semantic classes.Its RGB imagery has 0.3-meter spatial resolution and 1024 × 1024 image resolution; validation data are used for evaluation.
- Datasets: ISPRS Vaihingen uses 9-centimeter imagery with six semantic classes and near-infrared, red, and green channels, excluding clutter from evaluation.The average image resolution is 2494×2064, with specified tiles used for testing and validation.
- Datasets: ISPRS Potsdam uses 5-centimeter RGB imagery with six semantic classes and 6000×6000 images, excluding clutter from evaluation.Testing and validation use specified image IDs, while remaining images are used for training.
- Metrics: Mean Intersection over Union (mIoU) assesses segmentation accuracy using true positives, false positives, and false negatives for each class.True positives are correctly identified class pixels, false positives are incorrect class assignments, and false negatives are missed class pixels.
B. Training Settings
The experiments compare Samba with established CNN- and ViT-based methods under standardized initialization and input-cropping conditions, with augmentation used to address limited training data.
- Comparison protocol: The evaluation compares Samba with CNN methods ConvNeXt, ResNet, Deeplab V3+, and PSPNet, plus ViT methods Swin-T and Segformer.All tested methods are uniformly initialized without pretrained parameters, and input images are cropped into 512×512 patches.
- Reported evaluation: Tables II, III, and IV report semantic-segmentation accuracy for LoveDA, ISPRS Vaihingen, and ISPRS Potsdam, respectively.The tables present category accuracy using IoU, with the highest scores highlighted in bold.
- Training procedure: Random resize, random crop, random flip, and photometric distortion augment the training data.The augmentation strategy is motivated by limited training data and concerns about deep-learning model generalization.
C. Results
Samba achieves the best reported performance across LoveDA, ISPRS Vaihingen, and ISPRS Potsdam, exceeding CNN-based methods and slightly surpassing ViT-based methods. Visual results further indicate stronger segmentation of large regions, although some comparisons emphasize local-detail errors.
- Overall results: Samba achieved the best performance on LoveDA, ISPRS Vaihingen, and ISPRS Potsdam, surpassing CNN-based methods and slightly exceeding ViT-based methods.The comparison uses top-performing CNN- and ViT-based methods as baselines.
- LoveDA: 3.95% mIoU was Samba’s LoveDA improvement over Segformer, while its improvement over ConvNeXt was 10.3% mIoU.Both comparisons use UperNet as the decoder.
- Qualitative results: Visual comparisons show Samba’s consistently enhanced segmentation performance across LoveDA, ISPRS Vaihingen, and ISPRS Potsdam.The reported visualizations compare Samba with the other evaluated models.
PARAMETERS BETWEEN SAMBA AND OTHER COMPARED
Samba produced more complete and accurate large-area segmentations than compared CNN and ViT methods, while using fewer computational resources in the reported comparison. Its main remaining weakness was reduced local-detail focus, which caused omissions of some small objects.
- Samba surpassed other methods in mIoU and IoU for most segmentation categories on the ISPRS Vaihingen and Potsdam datasets.These datasets differed substantially in training-data quantity, spatial resolution, and segmentation categories.
- Samba used lower FLOPs per patch and fewer parameters than Swin-T, ResNet50, and ConvNeXt when combined with UperNet.
- CNN methods showed fragmented, blurred, and incomplete large-object segmentation, while ViT methods produced smoother results but still had blurred boundaries and localized false positives.
- Samba achieved relatively complete segmentation with limited training data and performed well on LoveDA’s challenging Agricultural category.Compared methods showed false negatives on larger Vaihingen and Potsdam regions and false positives in large LoveDA agricultural areas.
- Samba produced more complete large-area segmentations and minimized false positives more effectively than the compared methods.The authors attribute this performance to SSM inductive capability for long sequences.
- Samba remained prone to omitting some small objects because of lower focus on local details, producing false-negative errors.The authors identify additional strategies for mitigating overgeneralized feature attribution as a future need.
V. CONCLUSION
Samba integrates Mamba into semantic segmentation for high-resolution remotely sensed images and was evaluated against state-of-the-art CNN and ViT methods. It surpassed those methods across three datasets, with particularly large LoveDA mIoU gains over Segformer and ConvNeXt.
- Samba introduces Mamba into high-resolution remotely sensed image segmentation for the first time.
- Samba surpassed state-of-the-art CNN-based and ViT-based methods on LoveDA, ISPRS Vaihingen, and ISPRS Potsdam.
- 3.95% and 10.3% were Samba’s LoveDA mIoU advantages over Segformer and ConvNeXt, respectively, with UperNet as decoder.