PaperlayerView exact revision on arXiv

Source-linked AI summary

RS-Mamba for Large Remote Sensing Image Dense Prediction

Sijie Zhao, Hao Chen, Xueliang Zhang, Pengfeng Xiao, Lei Bai, Wanli Ouyang

arXiv:2404.02668v2cs.CV

TL;DR

Large VHR remote sensing images require global, multidirectional context, but transformer quadratic complexity encourages patching that loses contextual information. RSM applies state space modeling with linear complexity and omnidirectional selective scanning to dense prediction, achieving state-of-the-art results across semantic segmentation and change detection tasks. The models are simple and can process large images without patch segmentation, although the authors report performance degradation at larger image sizes for RSM-CD on WHU-CD.

  • Problem

    Large VHR images contain large-scale, multidirectional contextual features, while transformer quadratic complexity makes large-image processing difficult and patching loses contextual information.

  • Method

    RSM applies state space modeling with linear complexity and omnidirectional selective scanning to process large VHR images for semantic segmentation and change detection.

  • Results

    RSM-SS and RSM-CD achieve state-of-the-art performance on semantic segmentation and change detection tasks across the reported datasets.

  • Takeaways & Limitations

    RSM can process large VHR images without patch segmentation while modeling global context across multiple directions using simple task-specific architectures.

  • Takeaways & Limitations

    RSM-CD on WHU-CD performs best at image size 1024, with performance decreasing as image size increases.

Abstract

from arXiv · show

Context modeling is critical for remote sensing image dense prediction tasks. Nowadays, the growing size of very-high-resolution (VHR) remote sensing images poses challenges in effectively modeling context. While transformer-based models possess global modeling capabilities, they encounter computational challenges when applied to large VHR images due to their quadratic complexity. The conventional practice of cropping large images into smaller patches results in a notable loss of contextual information. To address these issues, we propose the Remote Sensing Mamba (RSM) for dense prediction tasks in large VHR remote sensing images. RSM is specifically designed to capture the global context of remote sensing images with linear complexity, facilitating the effective processing of large VHR images. Considering that the land covers in remote sensing images are distributed in arbitrary spatial directions due to characteristics of remote sensing over-head imaging, the RSM incorporates an omnidirectional selective scan module to globally model the context of images in multiple directions, capturing large spatial features from various directions. Extensive experiments on semantic segmentation and change detection tasks across various land covers demonstrate the effectiveness of the proposed RSM. We designed simple yet effective models based on RSM, achieving state-of-the-art performance on dense prediction tasks in VHR remote sensing images without fancy training strategies. Leveraging the linear complexity and global modeling capabilities, RSM achieves better efficiency and accuracy than transformer-based models on large remote sensing images. Interestingly, we also demonstrated that our model generally performs better with a larger image size on dense prediction tasks. Our code is available at https://github.com/walking-shadow/Official_Remote_Sensing_Mamba.

I. INTRODUCTION

VHR remote sensing images contain rich, large-scale contextual features distributed across arbitrary directions, while transformer processing of large images is constrained by quadratic complexity and patch-based context loss. RSM addresses these challenges with linear-complexity global modeling and omnidirectional scanning for dense prediction.

  • Motivation: VHR images contain rich contextual information and large spatial features within and among objects, distributed across multiple directions.These properties make global context modeling and multidirectional feature extraction important for semantic segmentation and change detection.
  • Motivation: Transformer self-attention captures global context but its quadratic complexity necessitates dividing large VHR images into patches.Patch division loses spatial features and gives each patch limited contextual information relative to the original image.
  • Motivation: Vim and VMamba use directional selective scanning, but their horizontal or horizontal-vertical directions cannot adequately capture arbitrary directional features in VHR images.Remote sensing objects and dependencies can span large spatial scales and vary in direction because imagery is captured from a top-down perspective.
  • Proposed approach: RSM introduces SSM to VHR dense prediction, providing global modeling and linear complexity for directly processing large images without patch segmentation.This design preserves contextual information while handling large remote sensing images without self-attention operations.
  • Proposed approach: OSSM selectively scans forward and backward along horizontal, vertical, diagonal, and anti-diagonal axes to extract multidirectional spatial features.The module is designed to enhance global contextual modeling across multiple directions.
  • Contributions: Experiments on WHU, Massachusetts Road, LEVIR-CD, and WHU-CD report state-of-the-art performance for semantic segmentation and change detection.The authors use simple RSM-SS and RSM-CD architectures without sophisticated modules or training techniques.

II. RELATED WORKS

VHR dense prediction requires both local and global spatial information, but existing CNN, transformer, and hybrid approaches face complementary limitations on large images. RSM is proposed as a linear-complexity alternative that retains global context without the same patching constraint.

  • Dense Prediction of Very High Resolution Remote Sensing: Dense prediction in VHR remote sensing mainly includes semantic segmentation and change detection, where large images provide crucial contextual information.Experiments cited in the passage associate larger remote sensing images with superior performance.
  • CNN-based models: CNNs efficiently capture local spatial features but struggle to model comprehensive global features and dependencies across large spatial scales.Their limited global receptive fields constrain analysis of large objects and long-range relationships.
  • Transformer-based models: Transformers model global context through self-attention but incur quadratic complexity and substantial computational demands on large VHR images.This limitation motivates preprocessing large images into smaller segments.
  • Hybrid-based models: CNN-transformer hybrid models combine local and global features but still face self-attention complexity when processing large VHR images.Their computational burden can require image segmentation, which loses contextual information and limits dense prediction effectiveness.
  • Proposed solution: RSM uses linear complexity and global modeling to handle large VHR images with rich contextual information without the same patch-based constraint.The passage presents RSM as a response to the limitations of transformer and hybrid approaches.

B. State Space Models

Earlier SSM variants model long-range dependencies efficiently but are limited by directional processing, while image-oriented methods extend scanning to selected axes. RSM’s omnidirectional scan targets the multiple-direction spatial structure of VHR remote sensing images.

  • State Space Models: SSMs evolved to address long-range dependency modeling, with successive variants improving computational efficiency and structural flexibility.The described progression includes LSSL, S4, complex-diagonal variants, and multiple-input multiple-output configurations.
  • State Space Models: Earlier SSMs process unidirectional sequences and cannot directly handle image data lacking a specific direction.Selective scanning in multiple directions was introduced to extend SSM-based global image modeling.
  • Image-oriented SSMs: Vim scans horizontally while VMamba scans horizontally and vertically to establish global image connections with linear complexity.These methods were developed for image tasks such as classification and segmentation on natural images.
  • Remote sensing adaptation: Vim and VMamba are not ideally suited to VHR remote sensing because large spatial features may be distributed in multiple arbitrary directions.The OSSM addresses this setting by scanning in multiple directions to capture large spatial features.

A. Preliminaries: State Space Models

SSMs map input sequences to outputs through evolving hidden states, with continuous dynamics discretized for deep-learning computation. Mamba adds context-dependent parameter selection, while RSM applies multidirectional global modeling to remote sensing architectures.

  • SSM formulation: An SSM maps an input sequence x(t) to an output y(t) through a hidden state governed by linear dynamical equations.The system uses matrices A, B, C, and D to describe state evolution and input-output relationships.
  • SSM formulation: The state-transition matrix A governs hidden-state evolution, while B, C, and D connect inputs, states, and outputs.Continuous equations are discretized for computational tractability and alignment with data sampling rates.
  • Discretization: Discretization converts the continuous ODE into a discrete-time SSM using a zero-order-hold assumption on the input.The resulting representation uses discrete hidden states and a transition matrix Φ, with Γ derived from the continuous parameters.
  • Mamba: Mamba uses selective scanning to dynamically adjust system matrices B and D according to current and historical input context.These context-dependent adjustments allow the SSM to model complex temporal dynamics more effectively.
  • RSM architectures: RSM-SS and RSM-CD use omnidirectional selective scanning to globally model image context across multiple directions with linear complexity.The frameworks apply this design to semantic segmentation and change detection.

B. Overall Architecture

RSM provides task-specific dense prediction frameworks built around lightweight Omnidirectional State Space blocks. These blocks combine feature extraction, multidirectional contextual modeling, gating, and residual refinement for VHR remote sensing images.

  • Frameworks: RSM-SS uses a U-Net encoder-decoder, while RSM-CD uses an FC-Siam-Conc Siamese architecture for bitemporal change detection.Both frameworks use patch embedding and encoders composed of stages containing OSS blocks; RSM-CD shares encoder weights across the two temporal inputs.
  • OSS Block: The OSS block is a feature extraction unit whose OSSM core selectively scans images across multiple orientations for global contextual modeling.It is designed for both semantic segmentation and change detection in VHR remote sensing images.
  • OSS Block: Within the OSS block, normalization, linear projection, depth-wise convolution, OSSM processing, gating, and a residual connection form the feature-processing pipeline.The OSSM scans forward and backward along horizontal, vertical, diagonal, and anti-diagonal directions before the output is transformed and combined with the input.
  • Design Objective: The OSS block is lightweight and efficient, allowing more blocks to be stacked under a similar overall model-depth budget.This design targets a balance between computational efficiency and rich spatial-feature extraction.

D. Omnidirectional Selective Scan Module

The omnidirectional scan extends prior one- and two-dimensional scanning strategies to multiple image orientations. Experiments evaluate RSM across semantic segmentation and change detection datasets, including the listed dataset configurations.

  • Prior Scan Strategies: Vim scans horizontally in both directions, whereas VMamba scans horizontally and vertically in both directions.These directions are contrasted as prior selective-scan strategies for image data.
  • OSSM Structure: OSSM produces eight image-patch sequences by scanning horizontal, vertical, diagonal, and anti-diagonal directions together with their reverses.The sequences are stacked and independently processed by the S6 block for global modeling.
  • Evaluation: The experiments cover semantic segmentation on WHU and Massachusetts Roads and change detection on WHU-CD and LEVIR-CD.RSM-SS is evaluated on the first two datasets, while RSM-CD is evaluated on the latter two.
  • Evaluation: Table I provides a brief introduction to the experimental datasets.The supplied material identifies this table as the dataset overview.
  • Dataset Overview: The WHU building dataset subset used here contains 8,189 aerial images at 0.3-meter spatial resolution.It is divided into 4,736 training, 1,036 validation, and 2,416 test images.
  • Dataset Overview: The Massachusetts Roads dataset contains 1,171 1500×1500-pixel aerial photographs and is segmented into overlapping 1024 × 1024-pixel patches.The dataset includes urban, suburban, and rural environments.

2) Change Detection Datasets:

The change-detection datasets provide bitemporal VHR imagery for building-change analysis, using dataset-specific patching and split procedures. The evaluation compares RSM with CNN, transformer, and hybrid benchmarks.

  • WHU-CD: WHU-CD contains bitemporal VHR aerial images from 2012 and 2016 showing changes in building structures.Its nonoverlapping 1024 × 1024 pixel patches are split into training, validation, and test sets at a 7:1:2 ratio.
  • LEVIR-CD: LEVIR-CD contains 0.5 m/pixel bitemporal Google Earth images documenting building transformations over 5 to 14 years.The dataset focuses on changes such as construction and demolition and contains specialist-labeled binary change masks.
  • Benchmarking: The comparative experiments use benchmark methods evaluated on the same dataset splits and data.This establishes a shared data basis for the reported comparisons.
  • Benchmarking: Semantic-segmentation comparisons include CNN, transformer, and CNN-transformer hybrid models.The listed transformer baselines are Segformer and RoadFormer.
  • Benchmarking: Change-detection comparisons include CNN, transformer, and CNN-transformer hybrid models.ChangeFormer is the listed transformer baseline, alongside BIT, MTCNet, MSCANet, and AMTNet-50 hybrids.

1) Data Augmentation:

The experiments use straightforward augmentation and standard optimization settings, then ablate selective-scan variants to assess OSSM. OSSM outperforms the one- and two-direction alternatives on both evaluated tasks.

  • 1) Data Augmentation:: RSM-SS uses flipping and transposing, each with probability p=0.5, without elaborate training tricks.The augmentation strategy is described as straightforward.
  • 1) Data Augmentation:: RSM-CD adds bitemporal-image swapping at p=0.5 to flipping and transposing, which also use p=0.5.The swapping operation is specific to the change-detection framework.
  • Training Setup: Training uses AdamW with an initial learning rate of 0.001 and weight decay of 0.001 on a single RTX A100 GPU.The loss combines binary cross-entropy and Dice coefficient loss.
  • Evaluation Metrics: Performance is evaluated with precision, recall, F1-score, and intersection over union.Precision and recall capture different error types, while F1-score is their harmonic mean.
  • D. Ablation Study: The OSSM ablation compares SS1D, SS2D, and OSSM on the Massachusetts Roads and WHU-CD datasets.SS1D scans horizontally in both directions; SS2D adds vertical scanning and reverses.
  • D. Ablation Study: OSSM outperforms SS1D and SS2D in both semantic segmentation and change detection.The result is attributed to scanning multiple directions to capture large road and building features across orientations.
  • D. Ablation Study: Table II reports the OSSM ablation results for the Massachusetts Road and WHU-CD datasets, with best values bolded within each dataset.The table is organized around the three selective-scan variants.

E. Overall Comparison

RSM-SS and RSM-CD achieve the highest reported segmentation and change-detection scores across the evaluated datasets. Their results are linked to global, multidirectional context modeling for large VHR images.

  • Semantic Segmentation Task: RSM-SS achieves the highest IoU (0.6735) and F1-score (0.8049) on the Massachusetts Road dataset.
  • Semantic Segmentation Task: RSM-SS achieves the highest IoU (0.9081) and F1-score (0.9518) on the WHU dataset.
  • Semantic Segmentation Task: RSM-SS accurately segments roads and buildings in the Massachusetts Road and WHU test-set inference examples.
  • Change Detection Task: RSM-CD achieves the highest IoU (0.8496) and F1-score (0.9187) on the WHU-CD dataset, outperforming all other change-detection models.
  • Change Detection Task: RSM-CD achieves the highest IoU (0.8366) and F1-score (0.9110) on the LEVIR-CD dataset.

F. Impact of Image Size and Spatial Resolution

Experiments show that dense-prediction performance generally benefits from larger image inputs, while downsampling can remove important high-resolution spatial features. These findings underscore the joint importance of contextual information and spatial detail in VHR imagery.

  • Semantic Segmentation: Model performance improves with increasing image size across downsampling ratios in the Massachusetts Roads semantic-segmentation experiments.
  • Semantic Segmentation: Higher downsampling ratios perform worse for equal-sized images because downsampling removes spatial features of elongated roads.
  • Change Detection: WHU-CD change-detection performance initially increases with image size before declining at larger sizes.
  • Overall Implications: The experiments highlight the importance of contextual information and high-resolution spatial features for VHR dense prediction.

G. Handling Large Remote Sensing Images

RSM is designed to process large VHR remote sensing images with linear complexity while globally modeling context across multiple directions. Compared with CNN- and transformer-based alternatives, it maintains efficiency and achieves strong performance, but its simple models and data requirements limit scope.

  • Computational Comparison: RSM-CD has linear computational growth with image size, whereas ViT-S-CD has quadratic growth and cannot train on 1024- or 2048-pixel images.
  • Performance Comparison: RSM-CD achieves the best performance at image size 1024 with a similar parameter count and lower computational cost than the compared models.
  • Motivation: Transformer-based models struggle with large VHR images because quadratic complexity forces processing through small patches with limited context.
  • RSM Design: RSM combines linear complexity with global, multidirectional context modeling to process large VHR images without smaller-patch segmentation.
  • Limitations: RSM models are overly simplistic, performance declines at image size 2048, and dense prediction requires extensive training data.

VI. CONCLUSION

The paper proposes Remote Sensing Mamba for dense prediction in VHR remote sensing imagery. Its linear-complexity, multidirectional context modeling supports large-image processing and strong performance on semantic segmentation and change detection.

  • RSM processes large VHR images with linear complexity and models global context through selective scanning in multiple directions.
  • Experiments demonstrate superior RSM performance across semantic segmentation and change detection tasks involving different objects.
Loading 2404.02668v2…