PaperlayerView exact revision on arXiv

Source-linked AI summary

Hyperspectral Image Classification-Traditional to Deep Models: A Survey for Future Prospects

Muhammad Ahmad, Sidrah Shabbir, Swalpa Kumar Roy, Danfeng Hong, Xin Wu, Jing Yao, Adil Mehmood Khan, Manuel Mazzara, Salvatore Distefano, Jocelyn Chanussot

arXiv:2101.06116v3eess.IVcs.CV

TL;DR

HSIC must address nonlinear, high-dimensional spectral-spatial data while labeled samples are costly and scarce. This survey systematically reviews DL frameworks for spectral, spatial, and joint spectral-spatial learning, alongside strategies for improving generalization. It concludes that limited labeled data, computational complexity, and the need for stronger joint feature exploitation remain central directions for future work.

  • Problem

    HSIC is difficult because HSI has complex high-dimensional spectral-spatial structure, while deep models require large amounts of costly labeled training data.

  • Method

    The survey categorizes state-of-the-art DL-based HSIC frameworks by spectral, spatial, and joint spectral-spatial feature learning and reviews strategies addressing limited training data.

  • Results

    The survey reports that reviewed DL approaches have achieved strong HSIC performance across supervised, semi-supervised, and unsupervised learning schemes, with some methods remaining stable when labeled samples are limited.

  • Takeaways & Limitations

    Future HSIC research should improve generalization with limited data, exploit spectral-spatial features jointly, and develop computationally efficient architectures for time-critical applications.

  • Takeaways & Limitations

    Deep neural network-based HSIC remains substantially constrained by the limited availability of labeled HSI data, and significant improvements are still needed to use such data efficiently.

Abstract

from arXiv · show

Hyperspectral Imaging (HSI) has been extensively utilized in many real-life applications because it benefits from the detailed spectral information contained in each pixel. Notably, the complex characteristics i.e., the nonlinear relation among the captured spectral information and the corresponding object of HSI data make accurate classification challenging for traditional methods. In the last few years, Deep Learning (DL) has been substantiated as a powerful feature extractor that effectively addresses the nonlinear problems that appeared in a number of computer vision tasks. This prompts the deployment of DL for HSI classification (HSIC) which revealed good performance. This survey enlists a systematic overview of DL for HSIC and compared state-of-the-art strategies on the said topic. Primarily, we will encapsulate the main challenges of traditional machine learning for HSIC and then we will acquaint the superiority of DL to address these problems. This survey breakdown the state-of-the-art DL frameworks into spectral features, spatial features, and together spatial-spectral features to systematically analyze the achievements (future research directions as well) of these frameworks for HSIC. Moreover, we will consider the fact that DL requires a large number of labeled training examples whereas acquiring such a number for HSIC is challenging in terms of time and cost. Therefore, this survey discusses some strategies to improve the generalization performance of DL strategies which can provide some future guidelines.

I. INTRODUCTION

HSI provides detailed spectral information for diverse applications, but its high-dimensional spectral-spatial structure challenges traditional classification. The survey reviews DL-based HSIC methods and organizes them by spectral, spatial, and joint spectral-spatial feature learning.

  • HSI foundations: HSI samples hundreds of narrow, contiguous bands across broad electromagnetic regions, providing detailed information for many environmental, agricultural, medical, and industrial applications.Applications also include military, space, air, and underwater contexts.
  • Challenges for traditional methods: Traditional image-analysis techniques cannot be directly applied to HSI because its high-dimensional spectral and spatial data have unique statistical and geometrical properties.The survey notes that hypercubes concentrate toward corners while hyperspheres concentrate in outside shells.
  • Survey scope: The survey reviews DL-based HSIC techniques, categorizing them into individual spectral or spatial feature learning and joint spectral-spatial feature learning.It also identifies directions for improving model generalization and robustness under limited reliable training samples.
  • Traditional machine learning: Hand-crafted features rely on engineering skills and domain expertise to encode characteristics such as shape, texture, color, spectral, and spatial details.Their optimal balance between robustness and discriminability can vary across datasets.
  • Traditional machine learning: Color histograms lack spatial contextual information, making objects with similar colors but different distributions difficult to distinguish.HOG captures edge and local-shape details, whereas SIFT provides local features but increases computational cost because of mathematical complexity.

B. Hyperspectral Data Characteristics and DL Challenges

HSI classification is constrained by high-dimensional, noisy, mixed spectral-spatial data and limited labeled samples. DL introduces additional training, interpretability, computational, and optimization challenges despite available techniques that mitigate some of them.

  • HSI-specific challenges: HSI contains hundreds of contiguous narrow spectral channels, high spectral resolution, low spatial resolution, and limited training data, making processing resource-intensive.Rich spectral information benefits classification but requires substantial time and resources to compute.
  • HSI-specific challenges: High dimensionality increases parameters and can cause the curse of dimensionality, while insufficient or unreliable labeled data can cause overfitting and Hughes phenomena.HSI labeling is time-consuming and expensive because it generally requires human experts or real-time investigation.
  • HSI-specific challenges: Reflectance variability, instrumental noise, redundant bands, and spectral mixing introduce additional artifacts that increase classification and computational difficulty.Low-to-average spatial resolution causes pixels to cover broad surface regions, contributing to spectral mixing.
  • DL challenges: DNN training requires difficult parameter optimization because convergence is not guaranteed, although advances in CNN optimizers make convergence easier.Examples include SGD, SGDM, RMSProp, Adam, AdamW, diffGrad, RAdam, GC, and AngularGrad.
  • DL challenges: Supervised DNNs require substantial training data, and their black-box behavior can hinder interpretability and optimization design.The high-dimensional nature of HSI combined with few labeled samples increases the tendency to overfit.
  • DL challenges: DNNs impose high memory-bandwidth, computational, and storage demands, while increasing depth can cause exploding or vanishing gradients that degrade convergence.Parallel, distributed, and high-performance-computing approaches can help process large data volumes.

III. HSI REPRESENTATION

HSI represents data as a cube containing spectral and spatial information, enabling spectral-only, spatial-only, or joint spectral-spatial representations for classification.

  • An HSI cube contains 1D spectral and 2D spatial details, with B spectral bands and N×M spatial dimensions.
  • Spectral Representation: Spectral representations isolate each pixel as a spectral vector, often after dimensionality reduction to reduce redundancy and improve class separability.
  • Spatial Representation: Spatial representations arrange pixels from each band as matrices and incorporate neighboring pixels because spatially adjacent pixels are likely to share classes.
  • Spatial Representation: Gabor filters, LBP, and GLCM provide handcrafted texture or spatial representations, while DNNs can extract spatial information from image patches.
  • Spectral-Spatial Representation: Spectral-spatial representations process spectral features together with spatial context by concatenating details or preserving the 3D cube structure.

IV. LEARNING STRATEGIES

HSIC uses supervised, unsupervised, and semi-supervised learning strategies, supported by configurable DNN layers whose choices affect feature processing and computational behavior.

  • Learning Strategies: Supervised learning trains models with labeled inputs and outputs, iteratively updating parameters to predict labels for new data.
  • Learning Strategies: Unsupervised learning discovers statistical structure without explicit labels, using methods such as PCA and k-means clustering.
  • Learning Strategies: Semi-supervised learning uses a small labeled set together with unlabeled data, thereby utilizing more available data for HSIC.
  • DNN Frameworks: DNN frameworks surveyed from 2017 onward support flexible combinations of configurable layer types for HSIC.
  • DNN Layers: Fully connected layers support class-probability prediction but introduce many parameters, increasing computational overhead and sensitivity to overfitting.
  • DNN Layers: Convolutional and activation layers respectively transform inputs through learned filters and nonlinear functions, enabling feature extraction beyond linear representations.

D. Pooling or sub-sampling layers

Pooling layers reduce feature-map size and support invariance and overfitting control, while CNN pipelines combine convolution, activation, pooling, and classification stages for HSIC.

  • Pooling or sub-sampling layers: Pooling reduces an input volume to a single value, providing invariance to small distortions while lowering data size, parameters, and computation.
  • Pooling or sub-sampling layers: Max-pooling, average-pooling, sum-pooling, and wavelet-pooling are described as down-sampling operations.
  • CNN Architecture: CNN feature-extraction networks hierarchically stack convolution, activation, and pooling layers to transform input data and reduce feature-map resolution.
  • CNN Architecture: The classification stage uses fully connected layers and Softmax to estimate class probabilities, although global average pooling can replace fully connected layers.
  • HSIC CNN Frameworks: HSIC CNN frameworks are categorized as spectral, spatial, or spectral-spatial according to the information processed.
  • Spectral CNN Frameworks: Spectral CNNs process 1D spectral information, with dimensionality reduction optionally determining the number of input bands.
  • Spectral CNN Frameworks: Reported spectral approaches include CNN-recurrent hybrids, PCA preprocessing, Gabor-kernel fusion, and averaged spectra for reducing pixel-wise noise effects.
  • Spatial CNN Frameworks: Spatial CNNs reduce spectral dimensionality and use 2D inputs, while Gabor filtering and deformable sampling address spatial detail and overfitting.

C. Spectral-Spatial CNN frameworks for HSIC

Spectral-spatial HSIC frameworks combine spectral and spatial information through CNN branches, fusion mechanisms, adaptive features, lightweight designs, and graph-based models.

  • Spectral-Spatial CNN frameworks: Spectral-spatial methods integrate neighborhood information with spectral features, including pixel-pair representations and residual 3D convolutions.
  • Spectral-Spatial CNN frameworks: Adaptive weighting and adjustable receptive fields are used to represent varying spatial context and make convolutional kernels more flexible.
  • Spectral-Spatial CNN frameworks: Lightweight CNNs address overfitting and gradient vanishing associated with deeper networks and limited labeled HSI samples.
  • Spectral-Spatial CNN frameworks: MorphCNN combines spectral and spatial morphological blocks in a dual-path architecture.
  • Spectral-Spatial CNN frameworks: Two-stage and multiscale frameworks learn joint features using CNNs, stacked autoencoders, fully convolutional networks, weighted fusion, or logistic regression.
  • Spectral-Spatial CNN frameworks: Dual-channel and two-branch networks separately extract spectral and spatial features before combining them for classification, sometimes with data augmentation or band selection.
  • Spectral-Spatial CNN frameworks: Hybrid models use CNNs with metric learning, virtual RGB images, multiscale extraction, attention, or Markov random fields to fuse or refine spectral-spatial representations.
  • GCN Frameworks: GCNs model relations among spectral signatures, but graph construction can be computationally expensive for large hyperspectral scenes.

E. Future directions for CNN-based HSIC

CNN-based HSIC has achieved strong classification performance, but future work must jointly preserve spectral and spatial information while reducing computational cost and dependence on labeled data.

  • CNN-based HSIC often uses dimensionality reduction, which can discard useful spectral information while improving spectral-spatial representation.
  • Preserving spectral information increases computational burden and slows training, motivating FPGA and GPU parallelization for faster models.
  • Deeper CNNs require more labeled data, so integrating unsupervised or semisupervised learning remains an important research direction.
  • Autoencoder-based HSIC: Autoencoder-based HSIC should more fully exploit joint spatial-spectral information and address high intra-class variability and inter-class similarity.

A. Future directions for DBN-based HSIC

DBN-based HSIC learns features through unsupervised layer-wise RBM pretraining, but future work should expand joint spectral-spatial modeling and stabilize training.

  • Future directions: DBN research should prioritize robust joint spectral-spatial methods and regularization that addresses dead or potentially over-tolerant neurons.
  • RNN architectures: RNNs process HSI spectral bands as time steps, enabling sequence modeling but creating differing complexity trade-offs among Vanilla, LSTM, and GRU variants.
  • Hybrid architectures: CNN-RNN hybrids combine convolutional spatial extraction with recurrent spectral or fusion modeling for spectral-spatial HSIC.

A. Future directions for RNN-based HSIC

RNN-based HSIC exploits spectral sequences and has been extended with spatial and hybrid models, but long sequences increase overfitting and computation; limited labels require data-efficient strategies.

  • Treating each spectral-band pixel as a sequential point can make RNN input sequences excessively long and increase overfitting risk.
  • Large spectral sequences increase computational time, motivating parallel processing, spectral-band grouping, and use of entire spectral signatures.
  • Data augmentation: Data augmentation creates additional training samples without labeling costs and can reduce overfitting caused by limited training data.
  • Data augmentation: A data augmentation-based CNN achieved a 10% increase in HSIC accuracy compared with a PCA-based CNN model.
  • Semi-supervised learning: Semi-supervised HSIC expands training through labeled and unlabeled data using co-training, self-training, GANs, graph-based models, and related strategies.
  • Generative models: GAN-based approaches generate spectral, spatial, or spectral-spatial samples to address scarce training examples and class imbalance.

D. Transfer Learning

Transfer learning addresses insufficient HSIC training data by adapting knowledge from related source domains, while active learning selectively labels valuable unlabeled samples.

  • Transfer learning: Transfer learning transfers knowledge from a relevant source domain to a target task, making it suitable when labeled training data are insufficient or unavailable.
  • Transfer learning: HSIC transfer methods commonly retain generic lower-layer features, initialize higher target-specific layers, and fine-tune using limited target labels.
  • Heterogeneous transfer learning: Heterogeneous transfer learning addresses source-target differences such as varying HSI dimensions caused by different sensors.
  • Active learning: Active learning iteratively selects valuable samples from an unlabeled pool for oracle labeling and adds them to the training set.
  • Active learning: Pool-based selection should incorporate sample diversity and consider heterogeneity, model performance, and representativeness.
  • Selection strategies: Uncertainty sampling queries uncertain instances, query-by-committee selects classifier disagreements, and expected model change targets samples causing large gradient changes.

2) Performance-based Selection:

Performance-based active learning selects samples according to estimated error or variance, while related HSIC work combines these criteria with representative, spatial, spectral, deep, transfer, and batch-mode strategies.

  • Performance-based Selection:: Expected error reduction selects samples estimated to reduce label uncertainty and error.Expected variance reduction instead queries instances associated with the lowest model variance.
  • Performance-based Selection:: Representative sampling favors dense input regions and avoids outliers or unrepresentative samples.Density-weighted methods such as information density implement this principle.
  • Performance-based Selection:: HSIC active-learning frameworks exploit feature spaces, spectral-spatial features, neighborhoods, superpixels, and attribute profiles to select informative samples.These approaches incorporate spatial context into query functions or uncertainty estimates.
  • Performance-based Selection:: Batch-mode active learning selects multiple samples per iteration, making diversity important to avoid redundant queries.A cited method combines diversity, uncertainty, and cluster-assumption measures.
  • Performance-based Selection:: Active learning has also been combined with multiview learning, deep networks, transfer learning, and domain adaptation for HSIC.Examples include Bayesian CNNs, autoencoders, DBNs, stacked sparse autoencoders, and multiple-kernel frameworks.
  • Performance-based Selection:: Fuzziness-based active learning selects samples near estimated class boundaries to improve classifier generalization.The selection uses fuzziness-based distances and favors greater fuzziness with smaller boundary distance.
  • Performance-based Selection:: Fair comparisons require identical sample counts and geographical locations across training, validation, testing, and cross-validation settings.Randomly changing samples can make results differ across methods, while overlap between training and testing can bias accuracy upward.

A. Experimental Datasets

The experiments use five benchmark hyperspectral datasets spanning vegetation, wetland, urban, and mixed land-cover scenes, with disjoint train/test samples standardized across methods.

  • A. Experimental Datasets: Indian Pines contains 224 spectral bands, 16 vegetation classes, and a 145 × 145-pixel scene at 20 meters per pixel.The dataset covers wavelengths from 400 to 2500 nm after removing 24 null or corrupted bands.
  • A. Experimental Datasets: Kennedy Space Center contains 176 spectral bands, 5202 labeled samples, and 13 upland and wetland classes in a 512 × 614-pixel image.Low signal-to-noise ratio bands were removed, and disjoint training/test maps are provided.
  • A. Experimental Datasets: University of Pavia contains 103 spectral bands, nine urban land-cover classes, and a 610 × 340-pixel scene at 2.5 meters per pixel.The dataset was acquired by the ROSIS sensor over the University of Pavia campus.
  • A. Experimental Datasets: Houston comprises 340 × 1905 pixels, 144 spectral bands, 15 land-cover classes, and 2.5-meter spatial resolution.Its wavelength range is 0.38 to 1.05 µm, with disjoint training/test maps presented.
  • A. Experimental Datasets: University of Trento contains 63 spectral bands, six vegetation classes, and a 600 × 166-pixel scene at 1 meter per pixel.The data were collected with the AISA Eagle sensor and divided into disjoint training and test samples.
  • A. Experimental Datasets: All competing methods use the same train/test sample percentages and geographical locations, with dataset summaries and disjoint sample counts reported in Tables I and II.The experimental tables compare results on these standardized splits.

B. Experimental Results on Disjoint Train/Test Samples

Using disjoint train/test samples and 10-fold cross-validation, the survey compares traditional, recurrent, convolutional, and MorphCNN approaches across benchmark HSI scenes.

  • B. Experimental Results on Disjoint Train/Test Samples: The experiments use disjoint training and test samples and 10-fold cross-validation to compute overall, average, and kappa accuracy.Results are reported in Tables III–VI and Figures 20–23.
  • B. Experimental Results on Disjoint Train/Test Samples: Comparative methods mostly misclassify Meadows and Bare Soil in Pavia University because their spatial structures are similar.The survey also notes lower overall accuracy for Grapes Untrained than for other classes.
  • B. Experimental Results on Disjoint Train/Test Samples: Higher numbers of labeled training samples can produce better accuracies for all competing methods.The survey therefore relates training-set size to classification accuracy.
  • B. Experimental Results on Disjoint Train/Test Samples: Works and outperform other comparative methods especially with fewer labeled training samples and show stable results across sample sizes.The survey concludes that these methods address limited training-sample availability to some extent.
  • B. Experimental Results on Disjoint Train/Test Samples: Autoencoder-based models perform worse than CNN-based methods under limited labeled training data.The survey attributes this partly to task representations that are not sufficiently driven by classification objectives and to autoencoder training parameters.
  • B. Experimental Results on Disjoint Train/Test Samples: Active or semi-supervised learning selects important samples, while few-shot, transfer, and domain-adaptation approaches exploit sample relationships, cross-image similarity, or added sample diversity.These strategies are discussed as ways to address limited labeled data and improve robustness.

C. Experiments with Convolutional Feature Extractors

The experiments compare traditional and gradient-centralized convolutional feature extractors for hyperspectral image classification across benchmark datasets. G2C-Conv3D generally performs best, while Trento is an exception where 3D convolution performs slightly better.

  • Experimental focus: The experiments evaluate convolutional feature extraction rather than overall model performance.The comparison is designed to assess the convolutional process itself using representative state-of-the-art approaches.
  • Compared extractors: The compared baselines include Conv2D, Conv3D, G2C-Conv2D, and G2C-Conv3D.Conv2D and Conv3D provide traditional convolutional baselines, while G2C variants use generalized gradient-centralized convolution.
  • Datasets and evaluation: The extractors are evaluated on five hyperspectral datasets: IP, PU, Trento, UH, and KSC.Results use disjoint train-test datasets, and Table VIII emphasizes higher accuracies.
  • Results: G2C-Conv3D outperforms Conv2D and Conv3D, followed by G2C-Conv2D, across the evaluated datasets.The reported ranking is consistent except on Trento, where the 3D convolutional process performs slightly better than Conv2D and G2C-Conv2D.
  • Implications and future directions: G2C-Conv3D is described as simple to implement and readily pluggable into existing CNNs to improve robustness and classification performance.The survey also identifies limited labeled data and computational complexity as continuing HSIC challenges motivating future work.
Loading 2101.06116v3…