PaperlayerView exact revision on arXiv

Source-linked AI summary

SS-MAE: Spatial-Spectral Masked Auto-Encoder for Multi-Source Remote Sensing Image Classification

Junyan Lin, Feng Gao, Xiaocheng Shi, Junyu Dong, Qian Du

arXiv:2311.04442v1eess.IVcs.CV

TL;DR

Existing MIM methods largely emphasize spatial modeling, while spectral modeling and local or high-frequency information can be overlooked in multi-source remote-sensing classification. SS-MAE uses spatial and spectral reconstruction branches and lightweight CNNs for local features; experiments on three datasets report that it outperforms several state-of-the-art methods.

  • Problem

    Existing MIM methods for multi-source remote-sensing classification leave spectral-feature modeling underexplored, and Transformer-based extraction may miss local or high-frequency information.

    Source: §I · pp. 1–2

  • Method

    SS-MAE reconstructs masked spatial patches and spectral channels in separate branches, then uses lightweight CNNs alongside Transformer features for local-global modeling.

    Source: §I · p. 2

  • Results

    Experiments on three publicly available multi-source datasets report that SS-MAE outperforms several state-of-the-art methods.

    Source: §I · p. 2

  • Takeaways & Limitations

    The authors describe SS-MAE as the first MIM work for multi-source remote-sensing image classification.

    Source: §I · p. 2

Abstract

from arXiv · show

Masked image modeling (MIM) is a highly popular and effective self-supervised learning method for image understanding. Existing MIM-based methods mostly focus on spatial feature modeling, neglecting spectral feature modeling. Meanwhile, existing MIM-based methods use Transformer for feature extraction, some local or high-frequency information may get lost. To this end, we propose a spatial-spectral masked auto-encoder (SS-MAE) for HSI and LiDAR/SAR data joint classification. Specifically, SS-MAE consists of a spatial-wise branch and a spectral-wise branch. The spatial-wise branch masks random patches and reconstructs missing pixels, while the spectral-wise branch masks random spectral channels and reconstructs missing channels. Our SS-MAE fully exploits the spatial and spectral representations of the input data. Furthermore, to complement local features in the training stage, we add two lightweight CNNs for feature extraction. Both global and local features are taken into account for feature modeling. To demonstrate the effectiveness of the proposed SS-MAE, we conduct extensive experiments on three publicly available datasets. Extensive experiments on three multi-source datasets verify the superiority of our SS-MAE compared with several state-of-the-art baselines. The source codes are available at \url{https://github.com/summitgao/SS-MAE}.

I. INTRODUCTION

Multi-source remote-sensing classification can combine complementary HSI and LiDAR/SAR information, but supervised fusion requires substantial labeled data and existing MIM methods leave key spatial-spectral or local-global modeling challenges. SS-MAE combines spatial and spectral reconstruction with CNN-based local-feature extraction, and experiments on three benchmark datasets report performance above state-of-the-art methods.

  • Multi-source fusion can address cases where HSI alone struggles to distinguish same-material classes, while LiDAR height and SAR’s all-weather observation provide complementary information.The paper focuses on joint HSI and LiDAR/SAR classification.Source: §I · pp. 1–2
  • Supervised multi-source fusion commonly requires substantial labeled data, while MIM has rarely been considered for multi-source remote-sensing classification.MIM learns representations from unlabeled data by reconstructing masked image patches.Source: §I · p. 2
  • Existing MIM approaches either mask spatial patches and ignore spectral information or use band-by-band masking that may leak information; Transformers may also lose local or high-frequency features.The paper identifies effective spectral-feature integration and joint local-global modeling as challenges.Source: §I · p. 2
  • SS-MAE uses spatial and spectral branches to reconstruct masked pixels and channels, respectively, and adds lightweight CNNs to capture local features alongside global representations.The authors describe it as the first MIM work for multi-source remote-sensing image classification.Source: §I · p. 2
  • Experiments on three benchmark datasets report that SS-MAE outperforms state-of-the-art methods.The authors also state that they will release the code.Source: §I · p. 2

II. RELATED WORKS

The related work reviews supervised fusion, self-supervised learning, and Transformer-based approaches. It motivates SS-MAE as a reconstruction-based method for multi-source classification that combines spatial-spectral modeling with local feature extraction.

A. Supervised Multi-Source Remote Sensing Data Classification

  • Supervised multi-source classification uses decision-level or feature-level fusion, but these methods commonly depend on large labeled training sets.Self-supervised methods that need fewer training samples may address this limitation.Source: A. Supervised Multi-Source Remote… · pp. 2–3

B. Self-Supervised Learning

  • Self-supervised methods include contrastive and reconstruction-based approaches; SatMAE reconstructs masked multispectral imagery, while SS-MAE uses two branches for spatial and spectral feature modeling.The paper notes that SatMAE’s token sequences can become excessively long for high-dimensional spectral data, potentially consuming substantial computational resources.Source: B. Self-Supervised Learning · p. 3

C. Vision Transformers

  • Transformer-based methods learn long-range dependencies but can miss local or high-frequency information, motivating lightweight CNNs for local feature extraction.SS-MAE’s framework uses global and local features in its feature representation.Source: Figure 2 · p. 3, C. Vision Transformers · p. 4

III. METHODOLOGY

SS-MAE pretrains spatial-wise and spectral-wise branches to reconstruct masked inputs, then combines their global representations with local features from two inverted-residual-block CNNs for classification.

A. Proposed SS-MAE Framework

  • Separate spatial and spectral encoders and decoders reconstruct the masked inputs, and pretraining minimizes the sum of their mean-squared-error losses.The pseudocode applies both losses against the concatenated input.Source: A. Proposed SS-MAE Framework · p. 4
  • For paired HSI and LiDAR/SAR patches covering the same region, SS-MAE concatenates the inputs and applies separate spatial-wise and spectral-wise masks.The spatial branch masks pixels across all channels at selected locations, while the spectral branch masks channels across the patch.Source: A. Proposed SS-MAE Framework · p. 4
  • At training and inference, the two MAE encoder representations are combined with features from two inverted residual blocks and passed to a classifier.The IRBs extract high-frequency information-rich representations from the HSI and LiDAR/SAR inputs.Source: A. Proposed SS-MAE Framework · p. 4

B. Pre-training Stage

SS-MAE pre-trains on unlabeled data using spatial and spectral masking, reconstructing masked pixels and channels to learn feature representations.

B. Pre-training Stage

  • SS-MAE uses spatial-spectral masking for hyperspectral feature representation, extending masking beyond the spatial dimension.The pre-training goal is to learn representations from unlabeled data through mask reconstruction.Source: B. Pre-training Stage · p. 4
  • Spatial masking hides random pixels, while spectral masking hides entire bands and reconstructs their missing values.Each spectral masked unit spans one band across the full P × P image.Source: B. Pre-training Stage · pp. 4–5
  • Spectral masking is excluded for single-band LiDAR data, but is essential for joint HSI and multispectral or PolSAR classification.Source: B. Pre-training Stage · p. 5
  • The reconstruction objective combines spatial and spectral losses, computes loss only on masked pixels, and sets λ to 2.The two losses use mean squared error.Source: B. Pre-training Stage · p. 5

C. Dual-Attention Transformer

The dual-attention Transformer encodes visible spatial or spectral tokens and decodes them with mask tokens to reconstruct masked inputs.

C. Dual-Attention Transformer

  • The dual-attention Transformer uses visible and masked tokens as inputs to reconstruct the masked data.Visible tokens are retained for encoding, while the decoder receives both encoded visible tokens and mask tokens.Source: C. Dual-Attention Transformer · p. 5
  • Each spatial Transformer block applies spatial-wise attention followed by a feed-forward network, with multi-head attention splitting Q, K, and V across H heads.The feed-forward network expands the dimension by a factor of 4 and then reduces it by the same ratio.Source: C. Dual-Attention Transformer · p. 5
  • Spatial attention computes similarities among visible pixels, whereas spectral attention computes similarities among visible spectral bands.The spatial attention map has dimensions p × p, where p is the number of visible spatial tokens.Source: C. Dual-Attention Transformer · p. 5

D. Training and Optimization

Lightweight CNNs complement Transformer features by capturing local information, and their outputs are integrated with spatial and spectral features for classification.

D. Training and Optimization

  • Two lightweight CNNs capture local features to complement Transformer representations that may lose edges, textures, or other high-frequency information.The passage describes this loss as a potential weakness of Transformer feature extraction for hyperspectral data.Source: D. Training and Optimization · p. 6
  • The LiDAR/SAR inverted residual block uses projection, depth-wise convolution, and expansion projection layers, while the HSI block substitutes 3D convolution.The HSI block uses an expansion ratio of 2; the depth-wise convolution is described as extracting local features with negligible extra computational cost.Source: D. Training and Optimization · p. 6
  • Spatial-wise and spectral-wise Transformer features are stacked with inverted residual block features and integrated for classification.Source: D. Training and Optimization · p. 6

IV. EXPERIMENTAL RESULTS AND ANALYSIS

The evaluation uses three multi-source datasets: Berlin and Augsburg pair hyperspectral with SAR data, while Houston 2018 pairs hyperspectral with LiDAR data.

A. Dataset Description

  • The evaluation covers Berlin and Augsburg for HSI-SAR classification and Houston 2018 for HSI-LiDAR classification.The cited dataset description identifies the Houston pairing but does not provide its full dataset details here.Source: A. Dataset Description · p. 6

1) Berlin Dataset:

  • Berlin pairs simulated EnMAP hyperspectral data with Sentinel-1 dual-polarization SAR data, using nearest-neighbor interpolation to match spatial resolution.The HSI has 797 × 220 pixels and 244 bands; the SAR image has 1723 × 476 pixels.Source: 1) Berlin Dataset · pp. 6–7

2) Augsburg Dataset:

  • Augsburg combines HySpex hyperspectral data and Sentinel-1 SAR data, with both images at 30 m ground sample distance and 332 × 485 pixels.The HSI has 80 bands, and the SAR data has four features derived from polarization decomposition.Source: 2) Augsburg Dataset · p. 7

3) Houston 2018 Dataset:

The Houston 2018 experiments use HSI and LiDAR data, with OA, AA, and Kappa as evaluation metrics. Parameter tests identify settings that perform best on this dataset, including two Transformer blocks, 256-dimensional tokens, and four heads.

3) Houston 2018 Dataset:

  • Houston 2018 pairs 48-band HSI with three-band multispectral LiDAR, and the experiments use the training subset.The HSI covers 380–1050 nm; OA, AA, and Kappa are the evaluation metrics.Source: 3) Houston 2018 Dataset · p. 7

1) Block Number:

  • Two Transformer blocks give the highest Houston OA, 91.91%.The same two-block setting gives the highest OA on Berlin and Augsburg as well.Source: 1) Block Number · p. 7

2) Token Dimension:

3) Heads Number:

  • Four attention heads are best for Houston 2018; increasing head count does not significantly improve performance and can reduce accuracy.Berlin and Augsburg perform best with eight heads.Source: 3) Heads Number · pp. 7–8

1) Input Patch Size:

The input-data parameter tests identify a 7×7 patch as optimal across the three datasets, while the preferred number of HSI principal components varies by dataset.

1) Input Patch Size:

  • A 7×7 input patch is optimal for Berlin, Augsburg, and Houston 2018.The patch size determines each token’s spatial receptive field; larger patches increase information redundancy.Source: 1) Input Patch Size · pp. 8–9, Figure 7 · p. 9

2) Number of Principal Components:

  • Berlin and Augsburg perform best with 30 principal components, while Houston 2018 performs best with 20.The authors conclude that the datasets differ in spectral complexity and redundancy.Source: 2) Number of Principal Components · p. 9

3) Pre-Training Sample Number:

Across the three datasets, pretraining improves SS-MAE results, with the reported comparison varying by dataset. The experiments also show dataset-dependent preferred pretraining sample counts and masking ratios, while SS-MAE ranks second in model parameters and GFLOPs on Houston 2018.

3) Pre-Training Sample Number:

  • Optimal pretraining sample counts are 200000 for Berlin and 50000 for Augsburg and Houston 2018.Performance generally improves with more samples, but decreases slightly above 50000 on Augsburg and Houston; the optimal count depends on dataset diversity.Source: 3) Pre-Training Sample Number · p. 9

4) The Spatial / Channel Masking Ratio:

  • The optimal spatial and channel masking ratios are 0.3 for Berlin, 0.7 for Augsburg, and 0.5 for Houston 2018.Ratios that are too high may prevent comprehensive reconstruction and impair classification performance.Source: 4) The Spatial / Channel Masking Ratio · p. 9

C. Performance Comparison

D. Ablation Study

Ablations show that spatial and spectral attention together, supplemented by lightweight CNNs for local features, improve classification. Pretraining also consistently boosts SS-MAE in the Houston 2018 few-shot tests.

D. Ablation Study

  • Adding spatial attention to spectral attention further improves classification, and adding IRB raises OA by 1.08%, 1.36%, and 0.7% on Berlin, Augsburg, and Houston 2018.Using spectral attention alone exceeds spatial attention alone by 3.31, 3.88, and 4.08 percentage points, respectively.Source: D. Ablation Study · p. 12, Table X · p. 13
  • Pretraining consistently boosts SS-MAE performance when trained with 2, 4, 8, or 16 Houston 2018 samples.The study compares pretrained SS-MAE with SS-MAE without pretraining.Source: D. Ablation Study · p. 12, Table XI · p. 13

V. CONCLUSION

  • The conclusion reports that SS-MAE performs well even without pretraining and that its Transformer-CNN hybrid benefits cross-modal feature representation.The architecture combines spatial and spectral reconstruction with lightweight CNN feature extraction.Source: §V · p. 13
  • The authors plan to explore more diverse cross-modal fusion strategies and conduct further masked-image-modeling experiments for multisource remote-sensing classification.Source: §V · p. 13

Related papers

Loading 2311.04442v1…