PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for Hyperspectral Image Classification: An Overview

Shutao Li, Weiwei Song, Leyuan Fang, Yushi Chen, Pedram Ghamisi, Jón Atli Benediktsson

arXiv:1910.12861v1eess.IVcs.CV

TL;DR

HSI classification is challenging because hyperspectral data are complex, nonlinear, and high-dimensional while labeled samples are limited. This survey systematically reviews deep-learning classification methods, groups them by extracted feature type, and examines limited-sample strategies. Experiments on real HSIs report strong performance for deep methods, with residual learning obtaining the highest improvement among the evaluated strategies.

  • Problem

    HSI classification is difficult because complex spectral data and large spatial variability must be handled with limited labeled samples relative to high dimensionality.

  • Method

    The survey reviews deep-learning HSI classification and organizes methods into spectral-feature, spatial-feature, and spectral-spatial-feature networks, while examining limited-sample strategies.

  • Results

    Deep learning-based methods generally achieved higher classification accuracies than traditional methods, with residual learning obtaining the highest improvement among the evaluated strategies.

  • Takeaways & Limitations

    The review provides a unified overview of deep HSI classification methods and guidelines for handling limited available samples.

Abstract

from arXiv · show

Hyperspectral image (HSI) classification has become a hot topic in the field of remote sensing. In general, the complex characteristics of hyperspectral data make the accurate classification of such data challenging for traditional machine learning methods. In addition, hyperspectral imaging often deals with an inherently nonlinear relation between the captured spectral information and the corresponding materials. In recent years, deep learning has been recognized as a powerful feature-extraction tool to effectively address nonlinear problems and widely used in a number of image processing tasks. Motivated by those successful applications, deep learning has also been introduced to classify HSIs and demonstrated good performance. This survey paper presents a systematic review of deep learning-based HSI classification literatures and compares several strategies for this topic. Specifically, we first summarize the main challenges of HSI classification which cannot be effectively overcome by traditional machine learning methods, and also introduce the advantages of deep learning to handle these problems. Then, we build a framework which divides the corresponding works into spectral-feature networks, spatial-feature networks, and spectral-spatial-feature networks to systematically review the recent achievements in deep learning-based HSI classification. In addition, considering the fact that available training samples in the remote sensing field are usually very limited and training deep networks require a large number of samples, we include some strategies to improve classification performance, which can provide some guidelines for future studies on this topic. Finally, several representative deep learning-based classification methods are conducted on real HSIs in our experiments.

I. INTRODUCTION

HSI classification is difficult because spectral signatures vary spatially while labeled samples are scarce relative to data dimensionality. This survey reviews deep-learning methods and strategies for limited-sample settings.

  • HSI classification assigns each pixel to a class using its spectral characteristics.
  • The main challenges are large spatial variability in spectral signatures and limited training samples compared with hyperspectral dimensionality.Variability can arise from illumination, environmental, atmospheric, and temporal changes; limited samples can produce ill-posed problems and reduce classifier generalization.
  • Hand-crafted and shallow descriptors depend on task-specific design and expert parameter choices, limiting their applicability and discriminative representation.
  • Deep learning can automatically learn complex, high-level hyperspectral features and has been introduced to HSI classification after successful computer-vision applications.
  • The survey organizes methods into spectral-feature, spatial-feature, and spectral-spatial-feature networks.
  • The paper also reviews strategies for limited available samples and compares representative methods on real HSIs.

A. SAEs

SAEs and DBNs construct hierarchical representations from layer-wise building blocks and can be connected to classifiers for HSI classification.

  • A. SAEs: An auto-encoder maps input x to hidden feature h through an encoder and maps h to reconstructed output y through a decoder.
  • A. SAEs: The auto-encoder minimizes Euclidean reconstruction error between x and y to approximate the input data.
  • A. SAEs: An SAE stacks multiple auto-encoder layers, attaching each layer’s output to the next layer’s input.
  • A. SAEs: An SAE can serve as a spectral classifier by treating each pixel vector as input and connecting the representation to logistic regression.
  • B. DBNs: An RBM is a two-layer model with visible and hidden units and serves as a building block for DBNs.
  • B. DBNs: Stacking RBMs forms a DBN that extracts deep hierarchical representations, with logistic regression added to use the DBN as a classifier.

C. CNNs

CNNs extract contextual spatial features through convolution and pooling, then use fully connected layers for deeper abstract representations. Their weight sharing reduces parameters, while pooling also reduces computation and feature-map size.

  • CNNs use local connections to extract contextual 2-D spatial features and weight sharing to significantly reduce network parameters.
  • Convolutional Layers: A convolutional layer applies k learnable filters to an input cube, producing multiple feature maps through convolution, bias addition, and activation.The input cube has spatial dimensions m×n and d channels; the j-th filter uses weight wj and bias bj.
  • A conventional CNN contains two convolutional-pooling layers followed by two fully connected layers.
  • Convolutional Layers: The activation function improves network nonlinearity, with ReLU commonly used because of fast convergence and robustness to gradient vanishing.
  • Pooling Layers: Pooling layers progressively reduce feature-map size, parameters, and computation while making extracted representations more abstract.

3) Fully Connected Layers:

Fully connected layers reshape preceding feature maps into vectors to extract deeper abstract features. The section also introduces recurrent networks for sequential data, including their long-term dependency limitation and relevance to hyperspectral spectra.

  • Fully Connected Layers: Fully connected layers flatten preceding feature maps into an n-dimensional vector and extract deeper, more abstract features.The cited example gives n=4096 for AlexNet.
  • Fully Connected Layers: The input, output, weight, and bias of a fully connected layer are represented by X′, Y′, W, and b, respectively.
  • RNNs: RNNs recognize patterns in sequences and dynamic temporal characteristics using a recurrent hidden state that depends on previous steps.
  • RNNs: The recurrent hidden state is used to predict the output at each time step through nonlinear transformations and coefficient matrices.
  • RNNs: Conventional RNN performance can degrade on long-term sequential data because of vanishing or exploding gradients, motivating LSTM and GRU architectures.
  • RNNs: RNNs suit HSI analysis because each hyperspectral pixel vector can be treated as an orderly sequence of spectral measurements.A cited method uses a modified gated recurrent unit and a new activation function for pixel-wise spectral classification.

E. GANs

GANs pair a generator with a discriminator trained adversarially: the generator creates realistic fake data while the discriminator distinguishes real from fake. HSI classification variants add class labels and optimize source and class-label objectives.

  • Generative models learn data-distribution parameters to generate samples, whereas discriminative models model label dependence for prediction.
  • GAN Architecture: A GAN contains a generator G and discriminator D trained adversarially, with G producing realistic fake inputs and D distinguishing real from fake.
  • GAN Objectives: The generator maps random noise z to fake data G(z), while the discriminator estimates whether x is a true sample.
  • GAN Objectives: The discriminator maximizes log(D(x)) and the generator minimizes log(1−D(G(z))) within the GAN minimax objective.
  • GAN Variants: Because the original discriminator only distinguishes real from fake, auxiliary classifier GANs modify it into a softmax classifier for multi-class label probabilities.
  • GAN-Based HSI Classification: In GAN-based HSI classification, G receives class labels c alongside noise z, while D processes labeled real samples and generated fake data.The objective combines log-likelihoods for the correct input source, Ls, and class labels, Lc.

III. DEEP NETWORKS-BASED HSI CLASSIFICATION

Deep learning-based HSI classification is organized by the spectral, spatial, or combined spectral-spatial features extracted by different networks. The reviewed approaches address nonlinear spectral structure, spatial context, and feature fusion through multiple network designs.

  • Deep learning automatically learns high-level features from complex hyperspectral data and can address large spatial variability in spectral signatures.
  • The survey divides HSI classification networks into spectral-feature, spatial-feature, and spectral-spatial-feature categories.
  • Spectral-Feature Networks: Spectral-feature networks target redundant, high-dimensional spectral bands that traditional linear extraction methods cannot fully model because hyperspectral properties are complex.
  • Spectral-Feature Networks: Spectral-feature methods include stacked autoencoders, deep belief networks, 1-D CNNs, 1-D GANs, and RNNs for extracting deep spectral representations.
  • Spectral-Spatial-Feature Networks: Spectral-spatial-feature networks are categorized by feature-fusion stage as preprocessing-based, integrated, or postprocessing-based networks.
  • Spatial-Feature Networks: Spatial-feature networks exploit neighborhood information with deep networks, then fuse learned spatial features with spectral features from other extraction techniques.

C. Spectral-Spatial-Feature Networks

Spectral-spatial-feature networks jointly represent spectral and spatial information through preprocessing, parallel feature extraction, or direct convolution on hyperspectral data. The section also introduces sample-limited training strategies, including augmentation and transfer learning.

  • C. Spectral-Spatial-Feature Networks: Spectral-spatial networks obtain joint features by fusing low-level spectral-spatial inputs, combining separate deep spectral and spatial features, or extracting them directly from original data.Direct extraction includes 2-D and 3-D CNNs, while other designs fuse features before or after separate deep networks.
  • C. Spectral-Spatial-Feature Networks: Preprocessing-based networks fuse spectral-spatial features before deep processing, then classify the resulting representations with simple classifiers such as SVM, ELM, or multinomial logistic regression.A common implementation flattens neighboring spatial regions and stacks them with spectral vectors for fully connected networks.
  • C. Spectral-Spatial-Feature Networks: Integrated networks directly extract joint spectral-spatial features from original hyperspectral data using 2-D or 3-D convolution.Three-dimensional convolution operates across spectral and spatial dimensions without relying on preprocessing or postprocessing techniques.
  • C. Spectral-Spatial-Feature Networks: Deep-network training is constrained by limited labeled samples, motivating strategies that improve HSI classification under sample scarcity.The survey includes data augmentation and other approaches to address the imbalance between network weights and available training samples.

1) Transformation-based sample generation:

Sample augmentation generates additional virtual training samples through transformations or mixtures of known samples, while transfer learning reuses information from trained source networks. These strategies target the limited labeled-data setting in HSI classification.

  • 1) Transformation-based sample generation:: Transformation-based augmentation creates virtual samples by applying operations such as rotation, flipping, or mirroring and adding weighted random Gaussian noise.The generated sample retains the same class as the known training sample.
  • 1) Transformation-based sample generation:: Mixture-based augmentation forms a virtual sample as a linear combination of two same-class training samples whose spectral characteristics are similar.The affinity between the samples is represented by αij.
  • 1) Transformation-based sample generation:: The survey characterizes transformation-based and mixture-based augmentation as simple yet effective ways to generate virtual samples.
  • 1) Transformation-based sample generation:: Transfer learning introduces information learned from source data into target data and can significantly decrease the demand for training samples.It can initialize network parameters from other trained deep networks, providing better classification performance than random initialization-based methods.

C. Unsupervised/Semi-supervised Feature Learning

Unsupervised and semisupervised feature learning seeks useful HSI representations from unlabeled data, while network-optimization methods improve feature extraction through architectural and training choices. Residual learning supports deeper networks, but optimization remains challenging.

  • C. Unsupervised/Semi-supervised Feature Learning: Unsupervised and semisupervised deep learning extracts useful HSI features from unlabeled data for classification.The associated framework distinguishes an unlabeled-data pathway from a semisupervised pathway that uses labeled data.
  • C. Unsupervised/Semi-supervised Feature Learning: Fully connected approaches can pre-train on unlabeled data by encoding samples into intermediate features and reconstructing them before fine-tuning with labeled data.
  • D. Network Optimization: Network optimization uses modules such as ReLU, batch normalization, and residual learning to improve representation or training behavior.Residual learning is used to build very deep networks, while ReLU followed by batch normalization can alleviate overfitting during training.
  • D. Network Optimization: Residual blocks apply two convolutions with trainable kernels and biases, using ReLU, and stacking them increases depth while making extracted features more discriminative.
  • D. Network Optimization: Network optimization remains challenging, and future HSI classifiers should account more fully for hyperspectral data characteristics.

V. EXPERIMENTS

The experiments compare deep learning and classical HSI classifiers across Houston, University of Pavia, and Salinas datasets, while examining learned representations and limited-sample strategies. The evaluated methods differ in whether they use spectral or spectral-spatial features.

  • V. EXPERIMENTS: The experiments assess deep learning advantages over traditional methods, compare recent approaches, visualize learned features and weights, and examine strategy effectiveness.
  • V. EXPERIMENTS: The evaluation uses Houston, University of Pavia, and Salinas scenes with different spatial resolutions, spectral bands, and class counts.Houston has 144 bands and 15 classes; University of Pavia has 103 bands and 9 classes; Salinas has 204 bands and 16 classes.
  • V. EXPERIMENTS: The reviewed deep methods include 3D-CNN, Gabor-CNN, CNN-PPF, S-CNN, 3D-GAN, and DFFN, alongside SVM, EMP, JSR, and EPF baselines.
  • V. EXPERIMENTS: SVM and CNN-PPF use only spectral features, whereas the other listed methods use spectral-spatial features.

C. Classification Results

Across three real HSI datasets, deep learning methods generally outperform traditional approaches in classification accuracy and visual map quality. The experiments also examine learned features and show how convolutional layers build increasingly complex representations.

  • All deep learning methods achieved overall accuracies above 80% on the Houston dataset, exceeding traditional SVM, EMP, JSR, and EPF methods.
  • DFFN delivered the best Houston classification accuracies for OA, AA, and Kappa among the compared methods.DFFN combines residual learning with feature fusion in a deep CNN framework.
  • CNN-PPF, which uses only spectral information, performed worse than spectral-spatial methods such as 3D-CNN, Gabor-CNN, S-CNN, and DFFN.
  • Deep learning methods showed advantages over traditional methods in both visual classification maps and quantitative results across the evaluated datasets.S-CNN performed best among the compared SVM-based classifiers on three hyperspectral datasets.
  • Deep Feature Visualization: Earlier convolutional layers extracted texture and edge features, which deeper layers composed into more complex high-level features automatically.The feature visualizations covered learned weights and representations after convolution, ReLU, pooling, and three convolutional layers.

E. Effectiveness Analysis of Strategies for Limited Samples

The experiments evaluate data augmentation, transfer learning, and residual learning under limited-label conditions. Each strategy improved CNN overall accuracy over the original CNN, with residual learning providing the strongest improvement.

  • CNN-DA, CNN-TL, and CNN-RL produced higher OA values than CNN-Original across all tested training-sample settings.The Salinas experiments used 5, 10, 15, 20, 25, and 30 labeled samples per class.
  • Nearly 1% improvement was obtained by CNN-TL over CNN-Original with 5 training samples per class.
  • CNN-RL performed better than the other methods in most training situations.Residual learning was incorporated into the CNN to optimize the network.
  • Residual learning achieved the highest improvement among the evaluated strategies.The authors state that this result may provide guidelines for future study.
Loading 1910.12861v1…