PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatial-Spectral Feature Extraction via Deep ConvLSTM Neural Networks for Hyperspectral Image Classification

Wen-Shuai Hu, Heng-Chao Li, Lei Pan, Wei Li, Ran Tao, Qian Du

arXiv:1905.03577v2cs.CV

TL;DR

HSI classification needs feature extraction that preserves spatial structure while modeling spectral dependencies. This paper introduces SSCL2DNN and SSCL3DNN, two ConvLSTM-based models using band-wise patches or 3-D cubes. Across three HSI datasets, the proposed models report better classification performance than considered comparison models, including gains in overall accuracy.

  • Problem

    LSTM can model long-range spectral dependencies for HSI classification, but one-dimensional unfolding loses spatial information and existing approaches insufficiently fuse spatial-spectral features.

  • Method

    The paper develops SSCL2DNN using local patches as band-wise spectral sequences and SSCL3DNN using a 3-D ConvLSTM structure that directly processes local cubes.

  • Results

    The proposed models provide better classification performance than considered models across three HSI datasets, with SSCL3DNN improving OA over 3-D CNN by 0.51%, 2.12%, and 7.96%.

  • Takeaways & Limitations

    ConvLSTM-based spatial-spectral feature extraction can preserve hyperspectral structure while modeling spectral dependencies for HSI classification.

Abstract

from arXiv · show

In recent years, deep learning has presented a great advance in hyperspectral image (HSI) classification. Particularly, long short-term memory (LSTM), as a special deep learning structure, has shown great ability in modeling long-term dependencies in the time dimension of video or the spectral dimension of HSIs. However, the loss of spatial information makes it quite difficult to obtain the better performance. In order to address this problem, two novel deep models are proposed to extract more discriminative spatial-spectral features by exploiting the Convolutional LSTM (ConvLSTM). By taking the data patch in a local sliding window as the input of each memory cell band by band, the 2-D extended architecture of LSTM is considered for building the spatial-spectral ConvLSTM 2-D Neural Network (SSCL2DNN) to model long-range dependencies in the spectral domain. To better preserve the intrinsic structure information of the hyperspectral data, the spatial-spectral ConvLSTM 3-D Neural Network (SSCL3DNN) is proposed by extending LSTM to 3-D version for further improving the classification performance. The experiments, conducted on three commonly used HSI data sets, demonstrate that the proposed deep models have certain competitive advantages and can provide better classification performance than other state-of-the-art approaches.

I. INTRODUCTION

HSI classification has progressed from conventional spatial-spectral methods toward deep feature extraction, but standard LSTM-based approaches can lose spatial structure. The paper proposes SSCL2DNN and SSCL3DNN to jointly model spatial-spectral information with ConvLSTM.

  • HSIs are 3-D data used across environmental, agricultural, ecological, and geological applications.
  • HSI classification has used SVM, composite kernels, sparse representation, and joint sparse models to exploit spectral and spatial information.
  • Deep learning methods increasingly extract spatial-spectral features for HSI classification using architectures such as stacked autoencoders and 3-D CNNs.
  • LSTM models long-range spectral dependencies, but unfolding HSI inputs into one-dimensional form loses spatial structure information.
  • The paper proposes SSCL2DNN, which feeds local patches into memory cells band by band to jointly learn spatial-spectral features and spectral dependencies.
  • SSCL3DNN extends ConvLSTM to three dimensions and directly processes local patches as 3-D cubes to better preserve hyperspectral structure.

II. RELATED WORK

CNNs provide a core architecture for HSI feature extraction, but their sliding-window spatial processing and adjacent-layer transmission limit information interaction. These limitations motivate architectures that preserve richer spatial context and dependencies.

  • CNNs comprise convolutional, pooling, fully connected, and classification layers, with convolution filters defining 2-D or 3-D feature extraction.
  • In CNNs, convolutional processing transforms prior-layer outputs using filters, biases, and nonlinear activation functions into feature maps.
  • CNNs use sliding windows for spatial features and transmit data only between adjacent layers, limiting information interaction within each layer.

B. Long Short-Term Memory

LSTM uses gated memory cells to model long-term dependencies while addressing recurrent gradient problems, but conventional LSTM can lose spatial structure. ConvLSTM replaces recurrent transitions with convolution to retain spatial context.

  • LSTM addresses recurrent learning difficulties with memory cells and input, output, and forget gates that control information protection, transmission, and retention.
  • Applying LSTM to HSI classification can model long-range spectral dependencies, but independent spatial and spectral branches and one-dimensional inputs lose intrinsic structure.
  • ConvLSTM replaces LSTM input-to-state and state-to-state transitions with convolution while retaining the gated recurrent structure.
  • ConvLSTM’s multidimensional gate operations capture spatial context and model long-range dependencies in video time or HSI spectral domains.

III. SPATIAL-SPECTRAL CONVLSTM 2-D NEURAL NETWORK (SSCL2DNN)

SSCL2DNN addresses incomplete spatial-spectral fusion and spatial-structure loss by processing local HSI patches as spectral sequences with ConvLSTM2D. Its recurrent layers model long-range spectral dependencies while preserving spatial information for classification.

  • SSCL2DNN targets insufficient spatial-spectral fusion, spatial-structure loss in SSLSTMs, and underused correlations between spectral bands in Bi-CLSTM.
  • The proposed model combines ConvLSTM2D and pooling layers, with spectral components fed into memory cells one by one.
  • Each local s × s × K HSI patch is formed using the first K PCA components and decomposed into K 2-D components for sequential processing.
  • When K = 1 and τ = 1, SSCL2DNN reduces to SaCL2DNN, which uses spatial structure information more effectively than 2-D CNN.
  • ConvLSTM2D supports data transmission within layers and between layers, enabling joint spatial-spectral modeling through long-range spectral dependencies.

B. Loss Function and Optimization Method

SSCL2DNN produces final class predictions by transforming learned spatial-spectral features into vectors, mapping them through a fully connected layer, and optimizing cross-entropy with ADAM.

  • The K memory-cell outputs are retained as inputs to the next layer, while the final ConvLSTM2D output represents learned spatial-spectral features.
  • The feature vectors are input to a softmax function that predicts the conditional probability distribution across N HSI classes.
  • Cross entropy is used as the loss function, and the final classification results are optimized with the ADAM algorithm.

IV. SPATIAL-SPECTRAL CONVLSTM 3-D NEURAL NETWORK (SSCL3DNN)

SSCL3DNN extends ConvLSTM to preserve the intrinsic structure of hyperspectral data and learn more discriminative spatial-spectral features for improved classification performance.

  • ConvLSTM3D is developed to better preserve the intrinsic structure information of hyperspectral data.
  • The proposed SSCL3DNN architecture is presented in Fig. 2.
  • SSCL3DNN is constructed from ConvLSTM3D to yield more discriminative spatial-spectral features and further improve classification performance.

A. ConvLSTM3D

ConvLSTM3D extends ConvLSTM2D by applying 3-D convolution to 4-D tensor inputs, states, outputs, and gate units while retaining a similar gated calculation model.

  • ConvLSTM3D uses the whole 3-D data cube as the input of each memory cell rather than processing bands separately.
  • Its inputs, states, outputs, and gates are 4-D tensors whose final three dimensions represent spectral and two spatial dimensions.
  • The convolution filters in ConvLSTM3D are 3-D tensors, and ⊛ denotes the defined 3-D convolution operation.
  • ConvLSTM3D retains a calculation model similar to LSTM and ConvLSTM2D, but changes the convolution performed within each gate unit.
  • Fig. 3 illustrates the inner structure of ConvLSTM3D.

B. SSCL3DNN

SSCL3DNN extends SSCL2DNN with a 3-D ConvLSTM architecture that directly processes whole data cubes to preserve hyperspectral structure and learn spatial-spectral features.

  • B. SSCL3DNN: SSCL3DNN extends SSCL2DNN using ConvLSTM3D to process hyperspectral data in a 3-D structure.The model is designed to capture intrinsic HSI structure and improve classification performance.
  • B. SSCL3DNN: Unlike SSCL2DNN’s band-by-band processing, SSCL3DNN takes the whole data cube as each memory cell’s input.This directly preserves the spatial-spectral organization of the hyperspectral data.
  • B. SSCL3DNN: After cascaded ConvLSTM3D and pooling layers, extracted spatial-spectral features are fed into a classification layer.Convolution kernels can be 3 × 3 × 3 or 5×5×5, while pooling uses a 2×2×2 kernel.
  • B. SSCL3DNN: SSCL3DNN uses spatial-spectral features and per-pixel vector representations that differ from SSCL2DNN’s layer processing.The loss function and optimization method remain the same between the two models.
  • B. SSCL3DNN: Experiments compare SSCL3DNN with SVM, CNN, LSTM, and ConvLSTM alternatives using OA, AA, and Kappa across three HSI datasets.Each experiment is repeated 10 times to reduce bias from random training-sample selection.

1) Indian Pines:

The experiments use three HSI datasets with PCA-based preprocessing, tuned model parameters, and classification evaluations under specified training-sample settings.

  • 1) Indian Pines:: Indian Pines contains agricultural fields, 200 usable spectral bands, and 10,249 non-background pixels after preprocessing.The original image is 145×145 pixels, with 224 bands before removing unusable and null bands.
  • 1) Indian Pines:: Salinas Valley has 16 ground-truth classes and 204 preserved spectral bands after removing water-absorption and noise-affected bands.The dataset contains 512 lines and 217 columns and was collected by the 224-band AVIRIS sensor.
  • 1) Indian Pines:: University of Pavia contains 9 distinguishable classes, 103 spectral bands, and a 610 × 340-pixel spatial extent.It was acquired by the ROSIS sensor with 1.3 mpp spatial resolution.
  • 1) Indian Pines:: PCA reduces computational complexity: one principal component feeds several baselines, while SSCL2DNN and SSCL3DNN use the top K components.The proposed models search K values after PCA, with optimal SSCL3DNN K equal to 10 across the three datasets.
  • 1) Indian Pines:: The proposed models tune local-window size, PCA components, convolution-kernel size, and ConvLSTM feature-map count.The selected local window is fixed at 27×27 because it provides satisfactory performance while avoiding higher complexity and runtime.
  • 1) Indian Pines:: Classification results are reported for Indian Pines with 10% training samples and for Salinas Valley and University of Pavia with 1% training samples.Additional tables report settings and results for training with small samples.

C. Classification Performance

Under limited-label settings, SSCL2DNN and SSCL3DNN outperform the considered methods across three HSI datasets, with improved spatial-spectral modeling and clearer classification maps.

  • C. Classification Performance: 1% training samples are used for Salinas Valley and University of Pavia, while Indian Pines uses 10%; remaining samples are used for testing.The study emphasizes small-training-size evaluation because labeled HSI samples are costly and difficult to obtain.
  • C. Classification Performance: SSCL2DNN and SSCL3DNN provide better classification performance than the considered comparison models.The evaluation uses quantitative assessments under the stated experimental settings.
  • C. Classification Performance: 0.86%, 5.03%, and 4.80% OA improvements are reported for SaCL2DNN over 2-D CNN on Indian Pines, Salinas Valley, and University of Pavia.The comparison is presented as evidence that incorporating spatial context improves classification relative to 2-D CNN.
  • C. Classification Performance: 2.41%, 0.58%, and 6.25% OA gains are reported for SSCL2DNN over Bi-CLSTM on the three datasets.The paper attributes the design to alternating ConvLSTM2D and pooling layers, reducing features and model complexity while modeling spectral dependencies.
  • C. Classification Performance: 0.76%, 2.99%, and 5.63% OA gains are reported for SSCL3DNN over SSCL2DNN on Indian Pines, Salinas Valley, and University of Pavia.SSCL3DNN also improves OA over 3-D CNN by 0.51%, 2.12%, and 7.96%, respectively.
  • C. Classification Performance: Classification maps from the proposed models are closest to ground truth, with fewer misclassifications and better-recognized class boundaries.The reported improvements are especially noted for several highly correlated classes across the three datasets.

D. Sensitivity Comparison and Analysis under Small Samples

Under small training sets, the proposed models retain strong classification performance, with SSCL3DNN achieving the best reported OA gains over 3-D CNN across all three data sets.

  • 10 samples per class were randomly selected to construct smaller training sets for evaluating model sensitivity.
  • Even with small training sets, the proposed deep models showed better classification performance.
  • SSCL2DNN improved OA over SaCL2DNN by 3.12%, 15.43%, and 3.83% on the three HSI data sets, respectively.
  • SSCL3DNN improved OA over 3-D CNN by 2.07%, 4.96%, and 22.61% on the three data sets, respectively.
  • Across varying training-sample counts, SSCL3DNN provided the highest classification performance for all three HSI data sets.

VI. CONCLUSION

The paper proposes SSCL2DNN and SSCL3DNN to extract discriminative spatial-spectral features for HSI classification. SSCL3DNN is designed to preserve hyperspectral structure more effectively, and experiments show competitive advantages, especially with small training sets.

  • SSCL2DNN and SSCL3DNN are proposed as deep ConvLSTM networks for extracting effective spatial-spectral features in HSI classification.
  • SSCL2DNN treats local patches as spectral sequences and processes them band by band to model long-range spectral dependencies.
  • SSCL3DNN extends LSTM to three dimensions to better preserve the intrinsic structure of hyperspectral data.
  • Experiments on three widely used HSI data sets show competitive advantages over state-of-the-art approaches, especially with small training sizes.
Loading 1905.03577v2…