PaperlayerView exact revision on arXiv

Source-linked AI summary

Feature Extraction for Hyperspectral Imagery: The Evolution from Shallow to Deep (Overview and Toolbox)

Behnood Rasti, Danfeng Hong, Renlong Hang, Pedram Ghamisi, Xudong Kang, Jocelyn Chanussot, Jon Atli Benediktsson

arXiv:2003.02822v4cs.CVcs.LGeess.IV

TL;DR

Hyperspectral imagery combines rich spectral information with high dimensionality, limited training samples, and challenges for robust classification. This paper surveys shallow supervised and unsupervised methods alongside deep feature extraction, compares 15 advanced techniques, and shares supporting code. Across three datasets, deep CNN- and PCNN-based methods can outperform shallow methods when sufficient training data are available, while unsupervised shallow methods generally outperform supervised shallow methods under limited samples.

  • Problem

    High dimensionality and limited labeled samples make robust statistical estimation and accurate hyperspectral classification difficult.

  • Method

    The paper provides a technical overview of shallow unsupervised, shallow supervised, and deep feature extraction techniques, compares advanced methods, and releases code and libraries.

  • Results

    Across three datasets, deep CNN- and PCNN-based feature extraction can outperform shallow methods when sufficient training data are available, while unsupervised shallow methods outperform supervised shallow methods.

  • Takeaways & Limitations

    Feature-extraction performance depends on the availability of training data, with unsupervised methods advantageous when labeled samples are limited.

Abstract

from arXiv · show

Hyperspectral images provide detailed spectral information through hundreds of (narrow) spectral channels (also known as dimensionality or bands) with continuous spectral information that can accurately classify diverse materials of interest. The increased dimensionality of such data makes it possible to significantly improve data information content but provides a challenge to the conventional techniques (the so-called curse of dimensionality) for accurate analysis of hyperspectral images. Feature extraction, as a vibrant field of research in the hyperspectral community, evolved through decades of research to address this issue and extract informative features suitable for data representation and classification. The advances in feature extraction have been inspired by two fields of research, including the popularization of image and signal processing as well as machine (deep) learning, leading to two types of feature extraction approaches named shallow and deep techniques. This article outlines the advances in feature extraction approaches for hyperspectral imagery by providing a technical overview of the state-of-the-art techniques, providing useful entry points for researchers at different levels, including students, researchers, and senior researchers, willing to explore novel investigations on this challenging topic. In more detail, this paper provides a bird's eye view over shallow (both supervised and unsupervised) and deep feature extraction approaches specifically dedicated to the topic of hyperspectral feature extraction and its application on hyperspectral image classification. Additionally, this paper compares 15 advanced techniques with an emphasis on their methodological foundations in terms of classification accuracies. Furthermore, the codes and libraries are shared at https://github.com/BehnoodRasti/HyFTech-Hyperspectral-Shallow-Deep-Feature-Extraction-Toolbox.

I. INTRODUCTION

Hyperspectral imagery offers rich spectral information but its high dimensionality, limited training data, and redundant or spatially incomplete features challenge conventional analysis. Feature extraction addresses these issues through dimensionality reduction and increasingly automated spectral-spatial representations spanning shallow and deep approaches.

  • Motivation: Hyperspectral sensors capture hundreds of narrow, contiguous spectral channels spanning visible, short-wave infrared, and other infrared regions.This spectral richness supports discrimination between classes with slightly different spectral characteristics.
  • Motivation: Limited training samples and high dimensionality make robust statistical estimation and supervised classification difficult.Required sample sizes increase with dimensionality, reaching exponential growth for nonparametric classifiers.
  • Feature Extraction: Feature extraction transforms input data linearly or nonlinearly into informative lower-dimensional representations and can reduce storage, transmission, redundancy, and computation.Spatial feature extraction may instead model contextual information and increase the number of features.
  • Motivation: Beyond an unknown number of bands, classification accuracy can decrease even as class separability increases, an effect known as the curse of dimensionality or Hughes phenomenon.This challenges the assumption that using the full spectral dimensionality is always best.
  • Machine and Deep Learning: Machine learning commonly uses hand-crafted feature extraction before classification, whereas deep learning integrates automatic feature extraction and classification in an end-to-end framework.Original spectral features can be redundant and omit dependencies among adjacent pixels.
  • Evolution and Scope: Hyperspectral feature extraction has evolved from unsupervised to supervised, spectral or spatial to spectral-spatial, manual to automatic, and shallow to deep approaches.The paper organizes this evolution into shallow unsupervised, shallow supervised, and deep feature extraction lines.

II. DATASETS AND NOTATIONS

The paper evaluates feature extraction on three hyperspectral datasets: Indian Pines 2010, Houston University 2013, and Houston University 2018. It also defines notation for observed data, training samples, labels, feature dimensions, and matrix operations.

  • 1) Indian Pines 2010:: Indian Pines 2010 uses a 445×750-pixel subset with 360 spectral bands, 2 m spatial resolution, and 16 land-cover classes.The dataset includes separate training and test samples for experiments.
  • 2) Houston University 2013:: Houston University 2013 contains 349 × 1905 pixels, 144 spectral bands spanning 0.38–1.05 µm, 2.5 m spatial resolution, and 15 classes.The study uses the corresponding training and test samples shown for the dataset.
  • 3) Houston University 2018:: Houston University 2018 covers 601 × 2384 pixels, 48 bands from 380–1050 nm, 1 m ground sampling distance, and 20 land-cover classes.The experiments use the training portion distributed for the 2018 data fusion contest.
  • Notations: The notation represents the observed HSI as X ∈ R^p×n, with p spectral bands, n pixels, d feature dimensions, and Xm containing m training samples.ym stores class labels, while I, X̂, and the Frobenius norm denote the identity matrix, an estimate of X, and matrix norm, respectively.

1) Conventional Data Projection/Transformation Techniques:

Conventional unsupervised projection and transformation methods reduce hyperspectral data into lower-dimensional subspaces, while reconstruction-based methods incorporate priors such as sparsity, smoothness, and spatial variation.

  • Projection and transformation: Conventional techniques linearly project hyperspectral data X into a lower-dimensional feature space using a transformation matrix or subspace bases V.The projected data are denoted by Z.
  • Projection and transformation: PCA projects data onto covariance-matrix eigenvectors to capture maximum signal variance, while MNF maximizes signal-to-noise ratio.MNF is also called noise-adjusted principal components, and its optimization uses the noise covariance matrix.
  • Projection and transformation: ICA assumes a linear mixture of non-Gaussian independent sources and estimates the source signals and mixing matrix simultaneously.This formulation makes ICA a blind source-separation technique for hyperspectral analysis.
  • Nonlinear extensions: Kernel MNF, KICA, and KPCA extend corresponding methods to nonlinear data by projecting samples into a kernel-defined feature space.KICA and KPCA have been used for hyperspectral change detection and classification.
  • Band clustering and merging: Band clustering and merging groups highly correlated spectral bands, then selects or fuses representative bands or features from each group.These methods are computationally cheap, although they can neglect spectral information unless additional filtering or processing is applied.
  • Low-rank reconstruction-based techniques: Low-rank reconstruction methods find an orthogonal subspace through constrained optimization and use data priors to extract features robust to noise.They are often more computationally expensive than conventional projection and band-clustering groups.
  • Low-rank reconstruction-based techniques: WSRRR imposes sparsity on wavelet coefficients, whereas OTVCA uses total variation to model spatially piece-wise-smooth hyperspectral features.SSLRA jointly extracts sparse features S and smooth features F using sparsity and total-variation penalties.

4) Graph Embedding and/or Manifold Learning Techniques:

Graph embedding and manifold-learning methods capture local or global geometric structure in hyperspectral data by constructing neighborhoods, assigning weights, and embedding samples into lower dimensions.

  • Manifold learning: Manifold-learning methods address hyperspectral nonlinearity by modeling the local geometric structure of neighboring pixels in feature space.The Swiss-roll example illustrates why Euclidean distance can misrepresent similarity before manifold unfolding.
  • Graph embedding: Graph embedding methods generally select neighboring pixels, determine weights, and construct a lower-dimensional embedding.These steps provide a common framework for several nonlinear feature-extraction techniques.
  • ISOMAP: ISOMAP constructs a neighborhood graph, computes shortest-path distances, and uses them to obtain a nonlinear embedding based on geodesic distances.It is characterized as a global geometric nonlinear feature-extraction method.
  • Local geometry: LLE reconstruction weights are invariant to rotations, rescalings, and translations of each data point and its neighbors.This invariance characterizes intrinsic neighborhood geometry and supports transformation into a lower-dimensional space.
  • Graph embedding: ISOMAP, LE, and LPP can be formulated through graph embedding, with their main differences determined by the choices of similarity matrix W and normalization or penalty matrix B.LE and LPP use a Gaussian function with standard deviation σ to define similarity.
  • Supervision: These techniques are treated as supervised when applied only to training samples and when ground-reference information is used to construct the edge matrix W.Applying them to the full dataset without labels corresponds to the unsupervised setting described earlier.

B. Supervised Feature Extraction Techniques

Supervised feature extraction uses label information to learn class-separable low-dimensional representations, with subspace-learning methods emphasized over band-selection strategies.

  • Supervised feature extraction: Supervised feature extraction can produce more class-separable features than unsupervised methods because it uses label information.The paper categorizes supervised approaches into subspace-learning and band-selection streams.
  • Subspace learning: Subspace-learning methods learn low-dimensional representations by imposing supervised rules based on class labels.Representative methods include LDA, MDA, and DBFE.
  • Subspace learning and band selection: Band-selection methods screen for representative and informative spectral bands, whereas subspace learning directly learns a reduced representation.The paper focuses primarily on subspace-learning techniques because of their stronger learning ability compared with band-selection strategies.
  • Supervised subspace learning: The reviewed subspace-learning techniques are organized into discriminant-analysis feature extraction and regression-induced representation learning.Their similarities, differences, advantages, and disadvantages are compared in the supervised-feature-extraction overview.

1) Discriminant Analysis Feature Extraction (DAFE):

DAFE methods learn projections or transformations that use label information to produce class-separable features in a lower-dimensional space. The section covers LDA variants, graph-based embeddings, kernelized methods, and spectral-discriminant processing.

  • DAFE overview: DAFE estimates a projection P into a d-dimensional subspace and obtains features by projecting the samples as Z = P^T X.The resulting vectors are collected as z_i = P^T x_i.
  • LDA and Its Variants: LDA maximizes between-class scatter relative to within-class scatter, with its projection obtained through generalized eigenvalue decomposition.The optimization uses the constraint P^T S_w P = I and yields a generalized eigenproblem.
  • LDA and Its Variants: Regularized LDA adds an l2-norm term to the within-class scatter matrix to improve stability under high-dimensional noise and small sample sizes.The regularized matrix is S_w^reg = S_w + γI, while the solution remains obtainable with GED.
  • LDA and Its Variants: LFDA weights within-class and between-class scatter using local neighborhood relations, modeling data as nonlinear manifolds rather than a homogeneous Gaussian space.This local weighting is intended to excavate underlying local structure.
  • Spectral feature discrimination: Spectral feature discriminant analysis first maximizes between-spectral scatter and then applies LDA to extract class-separable feature representations.The spectral representation is transformed with P_f before the resulting features enter the next-step LDA.
  • GDA and Its Extensions: Graph-based discriminant methods derive features from affinity matrices whose construction determines the extracted graph-embedding representation.The section discusses LDA-, LPP/LE-, LLE-, sparse-, and collaborative-representation-guided affinities.

2) Regression-induced Representation Learning (RIRL):

RIRL models feature extraction by directly connecting training samples with their labels through regression, rather than indirectly using graph or affinity matrices. The section places this family alongside least-squares mutual-information methods and joint progressive learning.

  • RIRL overview: RIRL bridges training samples and labels through regression instead of representing label information indirectly with graphs or affinity matrices.This provides a regression-based perspective on feature-extraction behavior.
  • Least-Squares Dimension Reduction: SIR assumes that data and labels are conditionally independent given features in the estimated subspace, expressed as (X ⊥ Y) | Z.LSDR searches for projections by maximizing squared-loss mutual information under this framework.
  • Least-Squares Quadratic Mutual Information: LSQMI replaces squared-loss mutual information with quadratic mutual information to reduce sensitivity to outliers.Its optimization follows the LSDR form with QMI substituted for SMI.
  • Least-Squares QMI Derivative: LSQMID estimates the derivative of QMI rather than QMI itself, providing a more computationally effective method with demonstrated accuracy and efficiency.The method addresses difficulty in accurately computing the QMI estimator’s derivative.
  • Joint & Progressive Learning Strategy: JPlay uses a multi-layered regression framework with layer-wise auto-reconstruction to address spectral variability from complex noise and atmospheric effects.The paper describes this strategy as breaking through the performance bottleneck of traditional linear methods.
  • Joint & Progressive Learning Strategy: Joint feature extraction and classification can indicate which features are positive for the classification task in JL-based strategies.The model uses a soft constraint to relax orthogonality.

IV. DEEP FEATURE EXTRACTION TECHNIQUES

Deep feature extraction addresses limitations of shallow methods by learning high-level features automatically from raw hyperspectral data in a hierarchical fashion. These learned features are described as more discriminative, abstract, and robust than shallow features.

  • Motivation: Shallow feature extraction often depends on careful engineering and expert domain knowledge, limiting its applications.The paper contrasts this requirement with automated deep representation learning.
  • Deep feature extraction: Deep learning techniques automatically learn high-level features from raw data hierarchically.The section frames this as the principal distinction from shallow feature extraction.
  • Deep feature extraction: Deep features are characterized as more discriminative, abstract, and robust than shallow features, supporting their use in hyperspectral-image feature extraction.The paper notes their broad adoption in recent hyperspectral studies.

A. AEs

Autoencoders learn hidden representations by reconstructing their inputs, while stacked variants use layer-wise pretraining for hyperspectral feature extraction. The section also introduces CNN components and their spectral, spatial, and joint-feature applications.

  • A. AEs: An autoencoder contains an encoder that maps input x to hidden representation h and a decoder that reconstructs the input as x̂.Training minimizes the residual between x and x̂.
  • A. AEs: After training, the decoder is removed and the hidden layer is used as the feature representation of the input.Multiple autoencoders can be stacked to extract deeper features.
  • A. AEs: Stacked autoencoders can pretrain each component with labeled and unlabeled samples before fine-tuning, providing better initialization than random parameters.The paper reports that fine-tuning only a few layers can acquire satisfactory discriminant features and alleviate overfitting.
  • A. AEs: Because SAE inputs are vectors, local hyperspectral cubes are reshaped into vectors before being fed into the model for spatial-feature extraction.The cited approaches use local cubes from principal components or incorporate Gabor and extended morphological features.
  • A. AEs: Graph and Fisher-based regularization can be added to SAEs to preserve neighborhood similarity or bring same-category hidden features closer together.These additions embed prior structural or discriminant information into the hidden layers.
  • B. Convolutional Neural Networks (CNNs): CNNs combine convolutional, pooling, and fully connected layers to extract features, reduce feature-map size, and learn high-level outputs.Convolution uses filters and biases, while pooling produces more robust features.
  • B. Convolutional Neural Networks (CNNs): CNNs can extract spectral or spatial features depending on network inputs, while 3-D convolutions process spectral-spatial information jointly.Weight sharing and local connections make CNN training more efficient and effective than traditional fully connected networks.
  • B. Convolutional Neural Networks (CNNs): CNN design must address small training sets with high-dimensional spectra and the need for spectral-spatial joint features that can improve classification over single-domain features.The paper lists dropout, weight decay, dimensionality reduction, and 3-D convolution among relevant strategies.

C. Recurrent Neural Networks (RNNs)

RNNs model sequential hyperspectral information through recurrent connections, with spectral bands commonly treated as sequences for feature extraction. The section also situates RNNs within broader spectral-spatial integration strategies and the comparative evaluation toolbox.

  • RNN fundamentals: RNNs use recurrent connections to model contextual relationships across sequence positions and can represent the final hidden state as a sequence feature.The hidden state at each step depends on the current input and preceding hidden state.
  • Spectral sequence construction: A common hyperspectral strategy feeds each pixel’s spectral bands sequentially into an RNN and uses the last hidden-layer output as the extracted spectral feature.The bands are processed from the first to the last for each pixel.
  • Sequence design: Long spectral sequences increase training difficulty because gradients may vanish or explode, motivating LSTM-based handling and division into shorter subsequences.Indian Pines can produce sequences of length 200 when bands are used directly.
  • Sequence design: Grouping adjacent bands into subsequences and using another RNN to fuse their features addresses long-sequence construction while exploiting complementarity among nonadjacent bands.This strategy extracts features from shorter band groups before fusion.
  • Spectral-spatial integration: Unlike AEs and RNNs, CNNs process image patches and cubes directly, motivating integrated networks that combine spectral and spatial feature extraction.Examples include parallel SAE-CNN frameworks and convolutional operators embedded in AE- or RNN-like architectures.
  • Comparative evaluation: The comparison evaluates four unsupervised, four supervised, and five deep feature-extraction techniques using a unified subspace-dimension setting.The compared methods include PCA, MSTV, OTVCA, LPP, LDA, CGDA, LSDR, JPlay, SAE, RNN, CNN, CAE, and CRNN.

2) Supervised FE:

The supervised-feature-extraction evaluation uses cross-validation for parameter selection and compares representative shallow and deep methods across three hyperspectral datasets. Implementation choices standardize feature dimensionality and classification settings where applicable.

  • Parameter selection: Cross-validation on the training set is used to select parameters for supervised feature-extraction algorithms.The strategy applies because labeled samples are available in supervised feature extraction.
  • Shallow supervised methods: The supervised methods include parameter-free LDA, regularized CGDA, Gaussian and regularized LSDR, and multi-parameter JPlay.CGDA, LSDR, and JPlay parameters are selected from prescribed ranges using cross-validation.
  • Deep methods: The deep methods use distinct input structures: SAE and RNN process pixel spectra, while CNN, CAE, and CRNN process 16 × 16 × p cubes.RNN uses two GRU recurrent layers, whereas SAE uses three hidden layers.
  • Deep methods: PCNN first reduces spectral dimensionality with PCA to k dimensions, where k is the number of classes, before CNN processing.Its CNN input cube is 16 × 16 × k.
  • Datasets and metrics: The experiments apply feature extraction to Indian Pines 2010, Houston 2013, and Houston 2018 and report class accuracies, AA, OA, and κ.The results are presented in Tables V, VI, and VII.

1) Unsupervised FE:

The comparisons show that feature-extraction performance depends on the method category, dataset, and use of spatial information. OTVCA, JPlay, and spectral-spatial deep models are highlighted for different strengths and trade-offs.

  • Unsupervised FE: PCA is weaker than the other compared techniques but still improves classification over applying RF directly to spectral bands.The paper attributes PCA’s disadvantages to noise sensitivity and neglect of spatial information.
  • Unsupervised FE: LPP performs best among unsupervised methods on Indian Pines, whereas OTVCA performs best among them on the Houston datasets.OTVCA’s improvements are described as particularly considerable for Houston 2013.
  • Unsupervised FE: OTVCA generally exceeds other unsupervised methods, combining noise-aware low-rank reconstruction with spatial total-variation regularization.The resulting features are described as piece-wise smooth and having high SNR.
  • Supervised FE: LDA generally outperforms direct RF classification on spectral signatures, indicating the value of incorporating supervised information into feature extraction.CGDA is inferior overall to LDA but offers automated sample-similarity computation and potentially more stable features in complex scenes.
  • Supervised FE: JPlay outperforms the other supervised feature-extraction methods, while its multilayered parameter updates incur high computational cost despite ADMM acceleration.Its features are described as semantically meaningful and robust because of self-reconstruction constraints.
  • Deep FE: Spectral-spatial models generally outperform spectral-only SAE and RNN models on AA, OA, and Kappa by jointly using spectral and spatial information.The comparison includes CNN, PCNN, CAE, and CRNN as spectral-spatial models.
  • Deep FE: PCNN outperforms CNN on all three datasets, with substantial improvements on Houston 2013 and 2018 because PCA removes redundant spectral information and reduces trainable parameters.The reduced parameter count makes PCNN easier to learn under the same conditions.

4) Shallow UFE Versus Shallow SFE:

Across the evaluated datasets, shallow unsupervised feature extraction generally outperformed shallow supervised approaches, while deep methods—especially CNN-based variants—were competitive or superior under favorable conditions. Classification maps were smoother with feature extraction, although spectral-spatial methods could become over-smoothed.

  • Shallow UFE Versus Shallow SFE: Across all three datasets, unsupervised feature extraction achieved higher classification accuracies than supervised feature extraction.The paper attributes this advantage partly to using all hyperspectral-image samples and extracting spatial-spectral information, whereas supervised performance depends more heavily on obtaining ground samples.
  • Shallow FE versus Deep FE: CNN-based feature extraction was consistent across datasets, while shallow methods slightly outperformed deep methods on Indian Pines 2010 and Houston 2013.This comparison qualifies the aggregate advantage of shallow methods by highlighting CNN’s cross-dataset consistency.
  • Shallow FE versus Deep FE: PCNN achieved the best performance on Houston 2018 and the second-highest accuracies on Indian Pines 2010 and Houston 2013.PCNN applies PCA before CNN; the paper reports that CNN-based methods gained at least 10% over shallow methods on Houston 2018.
  • Classification Maps: Classification maps from feature-extraction methods were generally smoother than spectral-classifier maps, which tended to contain sparse mislabeled pixels.Spectral-spatial methods such as OTVCA, CNN, and PCNN could instead become over-smoothed and create false structures, especially on Indian Pines 2010 and Houston 2018.
  • Performance with respect to the Number of Training Samples: With increasing training samples, JPlay improved over the spectral classifier but remained below unsupervised and deep feature-extraction methods.The paper links JPlay’s lower performance to limited spatial-context modeling and weaker nonlinear representation than deep methods.
  • Performance with respect to the Number of Training Samples: OTVCA and CNN performed similarly in the training-sample experiment, while PCA before CNN produced very high overall accuracies across cases.Randomly selecting training samples across class regions improved random-forest performance on OTVCA features, and the authors report an advantage for the PCA reduction stage before deep learning.
Loading 2003.02822v4…