PaperlayerView exact revision on arXiv

Source-linked AI summary

RSMamba: Remote Sensing Image Classification with State Space Model

Keyan Chen, Bowen Chen, Chenyang Liu, Wenyuan Li, Zhengxia Zou, Zhenwei Shi

arXiv:2403.19654v1cs.CV

TL;DR

Remote sensing classification remains challenging because scenes are complex and spatio-temporal resolutions vary. RSMamba uses an SSM-based Mamba architecture with dynamic multi-path activation, and experiments across three datasets report advantages over CNN- and Transformer-based methods.

  • Problem

    Remote sensing image classification must handle complex, diverse scenarios and variable spatio-temporal resolutions despite its importance for downstream interpretation.

  • Method

    RSMamba combines SSM-based Mamba global modeling with position-sensitive dynamic multi-path activation for remote sensing images.

  • Results

    Experiments across three remote sensing image classification datasets report that RSMamba outperforms or holds advantages over CNN- and Transformer-based methods.

  • Takeaways & Limitations

    RSMamba shows potential as a backbone network for next-generation visual foundation models.

Abstract

from arXiv · show

Remote sensing image classification forms the foundation of various understanding tasks, serving a crucial function in remote sensing image interpretation. The recent advancements of Convolutional Neural Networks (CNNs) and Transformers have markedly enhanced classification accuracy. Nonetheless, remote sensing scene classification remains a significant challenge, especially given the complexity and diversity of remote sensing scenarios and the variability of spatiotemporal resolutions. The capacity for whole-image understanding can provide more precise semantic cues for scene discrimination. In this paper, we introduce RSMamba, a novel architecture for remote sensing image classification. RSMamba is based on the State Space Model (SSM) and incorporates an efficient, hardware-aware design known as the Mamba. It integrates the advantages of both a global receptive field and linear modeling complexity. To overcome the limitation of the vanilla Mamba, which can only model causal sequences and is not adaptable to two-dimensional image data, we propose a dynamic multi-path activation mechanism to augment Mamba's capacity to model non-causal data. Notably, RSMamba maintains the inherent modeling mechanism of the vanilla Mamba, yet exhibits superior performance across multiple remote sensing image classification datasets. This indicates that RSMamba holds significant potential to function as the backbone of future visual foundation models. The code will be available at \url{https://github.com/KyanChen/RSMamba}.

I. INTRODUCTION

Remote sensing classification must handle complex, diverse scenes and variable spatio-temporal resolutions. RSMamba addresses this challenge with efficient global modeling and dynamic multi-path processing for image data.

  • Remote sensing image classification supports downstream interpretation tasks but remains difficult across complex scenarios and variable spatio-temporal resolutions.
  • CNNs extract hierarchical features, whereas attention networks model long-distance dependencies across image regions.
  • Transformer attention provides whole-image responses but incurs square complexity, creating efficiency and memory challenges as sequences or networks grow.
  • RSMamba is an efficient SSM-based global feature modeling method designed for remote sensing image classification.
  • RSMamba forms overlapping patch-token sequences with positional encoding, processes forward, reverse, and random paths using shared Mamba parameters, and activates them through linear mapping.
  • Its position-sensitive dynamic multi-path activation mechanism addresses vanilla Mamba’s causal, unidirectional, and spatially position-insensitive modeling limitations.
  • Experiments on three remote sensing classification datasets indicate advantages over CNN- and Transformer-based methods.

II. METHODOLOGY

The State Space Model maps input sequences to outputs through a hidden state and linear system parameters. RSMamba’s formulation discretizes this continuous process and expresses the resulting sequence operation as structured convolution.

  • A. Preliminaries: An SSM maps a continuous input signal x ∈ R^N to a response y ∈ R^N through a hidden state.
  • A. Preliminaries: The continuous SSM is formulated as a linear ordinary differential equation describing hidden-state dynamics.
  • A. Preliminaries: The state transition matrix A and projection matrices B and C connect the input, hidden state, and output.
  • A. Preliminaries: A and B are discretized with a zero-order hold using time-scale parameter ∆ so the continuous system can be integrated into deep learning methods.
  • A. Preliminaries: After discretization, the output is rewritten in convolutional form using a structured convolutional kernel K̄ ∈ R^L.

B. RSMamba

RSMamba converts images into position-aware sequences and models long-distance dependencies with dynamic multi-path Mamba blocks. Mean pooling then produces dense features for category prediction.

  • B. RSMamba: RSMamba maps 2-D images to 1-D patch sequences using convolution, flattening, and positional encoding.The input image is processed into patch embeddings before sequence modeling.
  • B. RSMamba: Dynamic multi-path activation Mamba blocks model long-distance dependencies without using a [CLS] token.The sequence is directly processed by multiple blocks rather than aggregated through a class token.
  • B. RSMamba: Mean pooling over the sequence produces dense features, followed by layer normalization and projection to class outputs.The projection maps latent dimension d to the number of classes.

C. Dynamic Multi-path Activation

The dynamic multi-path activation mechanism adapts causal Mamba to two-dimensional visual data by modeling forward, reverse, and randomly shuffled sequences. Shared Mamba parameters process the paths, whose outputs are reordered and selectively fused.

  • C. Dynamic Multi-path Activation: RSMamba duplicates each input sequence into forward, reverse, and random-shuffle paths to model spatially different token orders.The mechanism addresses vanilla Mamba’s unidirectional and position-insensitive modeling.
  • C. Dynamic Multi-path Activation: A shared-parameter Mamba mixer models token dependencies independently within the three paths.Using shared parameters preserves the structure of the vanilla Mamba block.
  • C. Dynamic Multi-path Activation: The path outputs are restored to forward order, concatenated, mean-pooled, and linearly projected into three activation weights.These operations form the gate that condenses information from the three sequence paths.
  • C. Dynamic Multi-path Activation: Softmax-based gating activates representations from the three information flows.The gate adaptively selects contributions from the different paths rather than simply averaging them.

D. Model Architecture

RSMamba is offered in base, large, and huge versions with distinct parameter settings. The architecture configurations specify block counts, dimensions, time-step rank, and SSM state size.

  • D. Model Architecture: RSMamba has three versions—base, large, and huge—with different parameter sizes.Their specific hyperparameters are listed in Table I.
  • D. Model Architecture: Table I reports the number of blocks, hidden size, intermediate size, time-step rank, and SSM state size for each version.The caption defines N, HS, IS, TSR, and SSMSS as the principal configuration fields.

A. Dataset Description

The evaluation uses three remote sensing scene-classification datasets with different category counts, image quantities, spatial resolutions, and training splits.

  • A. Dataset Description: Experiments use UC Merced, AID, and RESISC45, each containing distinct scene categories and image quantities.The datasets provide varied remote sensing evaluation settings.
  • A. Dataset Description: UC Merced contains 21 categories and 2,100 images at 0.3m resolution, with 70 images per category used for training.Each category contains 100 aerial images of 256 × 256 pixels.
  • A. Dataset Description: AID contains 30 categories and 10,000 images with spatial resolutions from 8m to 0.5m, using 50% of each category for training.Images are 600 × 600 pixels and category sizes range from 220 to 420.
  • A. Dataset Description: RESISC45 contains 31,500 images across 45 categories, with spatial resolutions from approximately 30m to 0.2m and 70% used for training.Each category contains 700 RGB images at 256 × 256 pixels.

B. Implementation Details

The implementation compares RSMamba with CNN- and Transformer-based classifiers across three remote sensing datasets and evaluates its components through ablation experiments.

  • Dataset and model comparisons: RSMamba is compared with ResNet, DeiT, ViT, and Swin Transformer across the UC Merced, AID, and RESISC45 datasets.These comparisons cover CNN- and Transformer-based image-classification methods.
  • Dataset and model comparisons: RSMamba’s performance is minimally impacted by training-data volume across datasets of varying sizes.The authors attribute this to relatively few parameters and reduced dependence on extensive inductive-bias data.
  • Dataset and model comparisons: Increasing RSMamba’s depth and width improves performance across all three datasets, but less than for ResNet and Transformer series.The base version already achieves high accuracy relative to other methods.
  • Training behavior: RSMamba’s performance does not rely on extensive data accumulation, although longer training can produce substantial additional gains.This contrasts with Transformer architectures, whose performance depends on inducing general features from large-scale training data.
  • Ablation setup: Ablation experiments evaluate the effects of individual components on the AID dataset using the base model unless otherwise specified.Associated hyperparameters remain unchanged unless explicitly stated.

1) Effect of Class Tokens:

The ablations examine class-token placement, mean pooling, and multi-path activation for aggregating sequence information in RSMamba.

  • Effect of Class Tokens: Class tokens at the head, tail, or both positions do not outperform the alternatives, whereas middle insertion substantially improves performance.The comparison is reported in the class-token and mean-pooling ablation.
  • Effect of Class Tokens: Mean pooling aggregates global sequence information to obtain dense semantic features for classification.RSMamba uses mean pooling instead of the ViT-style class-token design.
  • Effect of Multiple Scanning Paths: The vanilla Mamba’s causal-sequence formulation is difficult to apply to two-dimensional images without causal relationships.The proposed response is a multiple-scanning-path mechanism using forward, reverse, and random shuffling paths.
  • Effect of Multiple Scanning Paths: Increasing the number of scanning paths improves classification effectiveness, and gating provides advantages over feature averaging.The ablation uses average-pooled features for classification.
  • Effect of Multiple Scanning Paths: Without the multi-path scheme, a ViT-like class-token design causes a substantial performance decline.This observation is reported alongside the scanning-path ablation.

3) Effect of Positional Encoding:

RSMamba uses positional encoding and overlapping patches to preserve spatial information while modeling remote sensing images globally.

  • 3) Effect of Positional Encoding:: Removing positional encoding degrades performance, while Fourier and learnable encodings improve it.The comparison evaluates the presence, absence, and type of positional encoding.
  • 3) Effect of Positional Encoding:: RSMamba restores tokens from different paths to their original order, affecting the observed role of positional encoding.The passage notes that this restoration influences the impact of including positional information.
  • Patch division: Overlapping image patches let each token capture more exhaustive information and improve performance.The study compares patch overlap with no overlap.
  • Patch division: Increasing image size includes more details and correspondingly yields substantial performance gains.This result is reported in the patch-division analysis.
  • Model: RSMamba is introduced as a novel state space model for remote sensing image classification.The method is positioned as the paper’s central architecture.
Loading 2403.19654v1…