Source-linked AI summary
Bidirectional-Convolutional LSTM Based Spectral-Spatial Feature Learning for Hyperspectral Image Classification
Qingshan Liu, Feng Zhou, Renlong Hang, Xiaotong Yuan
TL;DR
HSI classification must address high-dimensional spectral data and the challenge of learning useful spectral-spatial features. The paper proposes Bi-CLSTM, combining bidirectional recurrent processing across spectral bands with convolutional spatial processing. Experiments on three HSIs report improved OA, AA, and κ compared with CNN and improved classification performance with more homogeneous classification-map regions.
Problem
HSI classification involves hundreds of spectral bands and limited training samples, creating a challenging spectral-spatial feature-learning problem.
Method
Bi-CLSTM treats the whole spectral channels as a sequence, applies bidirectional recurrent connections across them, and uses convolutional operators to learn spatial features.
Results
Bi-CLSTM improves OA, AA, and κ on three HSIs compared with CNN, while spatial information improves classification performance and produces more homogeneous classification-map regions.
Takeaways & Limitations
The proposed framework jointly learns spectral-spatial features for HSI classification rather than relying only on spectral information.
Abstract
from arXiv · showhide
This paper proposes a novel deep learning framework named bidirectional-convolutional long short term memory (Bi-CLSTM) network to automatically learn the spectral-spatial feature from hyperspectral images (HSIs). In the network, the issue of spectral feature extraction is considered as a sequence learning problem, and a recurrent connection operator across the spectral domain is used to address it. Meanwhile, inspired from the widely used convolutional neural network (CNN), a convolution operator across the spatial domain is incorporated into the network to extract the spatial feature. Besides, to sufficiently capture the spectral information, a bidirectional recurrent connection is proposed. In the classification phase, the learned features are concatenated into a vector and fed to a softmax classifier via a fully-connected operator. To validate the effectiveness of the proposed Bi-CLSTM framework, we compare it with several state-of-the-art methods, including the CNN framework, on three widely used HSIs. The obtained results show that Bi-CLSTM can improve the classification performance as compared to other methods.
I. INTRODUCTION
HSI classification is challenged by high-dimensional spectral data, limited training samples, and the need to combine spectral and spatial information without losing image structure. The paper proposes Bi-CLSTM to learn these features jointly through recurrent spectral processing and convolutional spatial processing.
- Challenges: HSIs contain hundreds of spectral bands, so limited training samples can produce dimensionality-related classification difficulties.The introduction identifies the curse of dimensionality and Hughes phenomenon as central issues.
- Challenges: Using spatial information is necessary because higher spatial resolution can increase within-class spectral variation and decrease between-class variation.The paper therefore motivates combining spatial and spectral information for classification.
- Existing approaches: Directly concatenating independently extracted spatial and spectral features creates a high-dimensional feature space.Alternative fusion strategies include decision-level combination and regularization, although MRF optimization can be time consuming on high-resolution data.
- Deep learning approaches: Flattening local image patches for fully connected models may destroy their two-dimensional structure and lose spatial information.CNN-based models preserve local image structure through local connections and weight sharing.
- Deep learning approaches: CNN models can learn spatial features but may fail to extract spectral features, while 3D CNNs increase parameters and risk overfitting with few training samples.These limitations motivate a model that handles both spectral sequences and spatial structure.
- Proposed approach: Bi-CLSTM treats spectral bands as an image sequence, uses LSTM connections across bands, replaces fully connected operators with convolutions, and adds bidirectional recurrence.Dropout and data augmentation are adopted during training to alleviate overfitting.
II. METHODOLOGY
Bi-CLSTM learns hyperspectral spectral-spatial features by processing spectral channels as sequences while applying convolution across spatial neighborhoods, with bidirectional recurrence to use both spectral directions.
- Bi-CLSTM framework: Bi-CLSTM represents a pixel neighborhood as a spectral sequence of image patches and extracts spectral and spatial features simultaneously.A sub-cube Xij is split across spectral channels into an l-length sequence, while recurrent and convolution operators process the inputs.
- CLSTM framework: CLSTM modifies LSTM by using convolutional operations within its recurrent memory-cell computations.The cell uses forget, input, candidate-memory, and output computations to update and expose information.
- Bidirectional spectral learning: Bi-CLSTM feeds image patches through CLSTM in forward and backward spectral orders to capture correlated spectral-channel information.The two directions produce spectral-spatial feature sequences that are concatenated before classification.
- Classification: The concatenated feature vector is passed to a softmax layer that estimates the probability of each class for the target pixel.The classification stage follows bidirectional feature extraction.
- Regularization and augmentation: Flipping and rotation augmentations increase the number of training samples eightfold, while dropout randomly suppresses neuron outputs during learning.The augmentation scheme includes rotations and horizontal or vertical flips; dropout excludes selected neurons from forward and back-propagation.
A. Datasets
The study evaluates Bi-CLSTM on three hyperspectral datasets—Indian Pines, Pavia University, and KSC—with different spatial, spectral, and land-cover characteristics.
- Dataset selection: The experiments use three widely used hyperspectral images to evaluate classification performance.The datasets are Indian Pines, Pavia University, and Kennedy Space Center.
- Pavia University: Pavia University was acquired by ROSIS over northern Italy and provides a scene with nine mutually exclusive land-cover classes.The image has 610×340 pixels and 1.3 m spatial resolution; each class has more than 1000 labeled pixels.
- Kennedy Space Center: KSC contains 176 retained AVIRIS spectral bands, a 512 × 614 image, and 18 m spatial resolution.Discrimination is difficult because certain vegetation types have similar spectral signatures, and the dataset contains 13 mutually exclusive land-cover classes.
B. Experimental Setup
The evaluation compares Bi-CLSTM with feature-extraction and classification baselines under fixed training splits, repeated runs, and multiple accuracy metrics.
- Compared methods: Bi-CLSTM is compared with PCA, LDA, NWFE, RLDE, MDA, CNN, and original pixels as a benchmark.The listed methods include several feature-extraction approaches and a CNN framework.
- Bi-CLSTM configuration: Bi-CLSTM uses two CLSTM layers in a bidirectional network, with convolution followed by max-pooling and 3 × 3 kernels with 32 filters.The CLSTM state is initialized to zeros, and the detailed configuration appears in Table IV.
- Training and testing: Indian Pines and KSC use 10% of each class for training, while Pavia University uses 3921 training pixels.The remaining pixels form the testing sets, and detailed counts are reported in Tables I–III.
- Ablation and sensitivity analyses: The experiments separately examine input-patch size, forward versus bidirectional CLSTM, and the effects of dropout and data augmentation on KSC OA.These analyses correspond to Tables V–VII.
- Evaluation protocol: All algorithms are repeated five times, with average performance reported using OA, AA, per-class accuracy, and the Kappa coefficient.OA measures correctly classified testing pixels, AA averages class accuracies, and κ corrects agreement for chance.
C. Parameter Selection
Parameter selection evaluates dropout, data augmentation, patch size, and bidirectional structure, identifying settings that improve Bi-CLSTM classification on KSC.
- Four influence factors are examined: dropout, data augmentation, network framework, and input image-patch size.
- 64 × 64 is selected as the optimal patch size because larger patches increase computation time while providing limited accuracy improvement.
- The bidirectional network outperforms an otherwise equivalently configured forward network on KSC.
- 94.41% to 99.13%: dropout significantly improves accuracy on KSC.
- 95.07% to 99.13%: data augmentation improves accuracy when the number of training samples is expanded eightfold.
D. Performance Comparison
Across Indian Pines, Pavia University Scene, and KSC, Bi-CLSTM achieves the strongest reported classification performance, while spatial information improves map homogeneity and classification accuracy.
- PCA performs worst on Indian Pines because it extracts spectral features without considering spatial features.
- MDA outperforms LDA-related spectral-only methods by extracting spatial and spectral features simultaneously, while CNN outperforms other methods in the cited comparison.
- Bi-CLSTM achieves the highest OA among the compared methods on Indian Pines and produces more homogeneous classification maps.
- On Pavia University Scene, MDA, CNN, and Bi-CLSTM outperform the other methods, but Bi-CLSTM's gains over MDA or CNN are less remarkable because their performance is already high.
- Bi-CLSTM achieves the highest OA, AA, and κ on the KSC dataset among the compared methods.
IV. CONCLUSION
The conclusion presents Bi-CLSTM as a framework that jointly learns spectral-spatial features through bidirectional spectral recurrence and spatial convolution. Experiments on three HSIs indicate improved classification performance and more homogeneous maps compared with CNN.
- Bi-CLSTM automatically learns spectral-spatial features from hyperspectral images.
- The network processes all spectral channels with bidirectional recurrent connections to explore spectral information.
- Convolution operators replace fully connected operators across the spatial domain to capture spatial information.
- Experiments compare Bi-CLSTM with several feature-extraction methods, including CNN, on three HSIs collected by AVIRIS and ROSIS instruments.
- Using spatial information improves classification performance and produces more homogeneous classification-map regions.
- Bi-CLSTM improves OA, AA, and κ compared with CNN, while the study also evaluates dropout, data augmentation, and patch size.