Deep Learning for Classification of Hyperspectral Data: A Comparative Review

Nicolas Audebert; Bertrand Saux; Sébastien Lefèvre

Source-linked AI summary

Deep Learning for Classification of Hyperspectral Data: A Comparative Review

Nicolas Audebert, Bertrand Saux, Sébastien Lefèvre

arXiv:1904.10674v1Apr 24, 2019cs.LGcs.CVcs.NEeess.IV

TL;DR

Hyperspectral classification combines fine spectral resolution with low spatial resolution, while traditional methods require difficult feature engineering. The article reviews machine-learning and deep-learning approaches, compares network families experimentally, and releases a toolbox; its evidence indicates that spatial-spectral CNNs can improve accuracy on higher-resolution datasets, although annotated data remain scarce.

Problem
Fine spectral resolution, low spatial resolution, and the feature engineering required by shallow methods make hyperspectral classification challenging.
Method
The article reviews standard and deep-learning classification methods, experimentally compares network architectures, and provides the DeepHyperX PyTorch toolbox for benchmarking them on public datasets.
Results
3D spatial-spectral CNNs significantly improved accuracy over 1D CNNs by 3% on Pavia University and 2% on DFC2018, while performance decreased on low-resolution Indian Pines.
Takeaways & Limitations
2D and 3D convolutional networks can combine spatial and spectral information and achieve state-of-the-art performance without expert sensor or physics knowledge.
Takeaways & Limitations
The main challenge is the scarce availability of massively annotated datasets for hyperspectral learning.

Abstract

from arXiv · show

In recent years, deep learning techniques revolutionized the way remote sensing data are processed. Classification of hyperspectral data is no exception to the rule, but has intrinsic specificities which make application of deep learning less straightforward than with other optical data. This article presents a state of the art of previous machine learning approaches, reviews the various deep learning approaches currently proposed for hyperspectral classification, and identifies the problems and difficulties which arise to implement deep neural networks for this task. In particular, the issues of spatial and spectral resolution, data volume, and transfer of models from multimedia images to hyperspectral data are addressed. Additionally, a comparative study of various families of network architectures is provided and a software toolbox is publicly released to allow experimenting with these methods. 1 This article is intended for both data scientists with interest in hyperspectral data and remote sensing experts eager to apply deep learning techniques to their own dataset.

I. INTRODUCTION

Hyperspectral data offer fine spectral characterization but pose challenges for deep learning because spatial resolution, annotation, and model-transfer assumptions differ from standard optical imagery. The article reviews these issues, existing approaches, and emerging research directions.

Deep-learning models developed for three-channel computer-vision images do not transfer straightforwardly to hyperspectral cubes because spectral resolution prevails over spatial neighborhood.Hyperspectral data have high spectral resolution but commonly low spatial resolution.
Low spatial resolution limits training samples and complicates annotation because small objects mix with neighboring materials, making data volume and labeling central challenges.The paper identifies these issues as the main challenges for deep learning in hyperspectral image processing.
The article bridges data scientists and hyperspectral remote-sensing experts by reviewing standard and deep-learning methods, comparing network families experimentally, and emphasizing emerging research axes.It also summarizes hyperspectral-imaging principles and public reference datasets.
Hyperspectral sensors capture spectra across several hundred channels, while reflectance is preferred for classification because it characterizes material properties independently of atmospheric conditions.Atmospheric correction methods convert intensity images into reflectance images by accounting for atmospheric and acquisition effects.

1) Pavia:

The paper introduces public hyperspectral datasets spanning urban, agricultural, and mixed scenes, while emphasizing their limited sample sizes and dataset-specific constraints. Pavia University, Houston DFC2018, and Indian Pines differ substantially in spatial and spectral resolution.

1) Pavia:: Pavia uses ROSIS at 1.3 m GSD, with 103 bands and 610 × 340 pixels for Pavia University, and nine annotated urban, water, and vegetation classes.Pavia Center contains 102 bands and 1096 × 715 pixels; annotations cover 50% of the whole surface.
1) Pavia:: Pavia is a major labeled reference dataset, but preprocessing may be needed to remove pixels without spectral information, and its ground truth contains some errors.
2) DFC2018:: DFC2018 covers Houston at 1 m GSD across 48 contiguous bands and 20 classes, combining urban categories, vegetation types, and specific materials.The dataset spans 380–1,050 nm and accompanies very-high-resolution imagery and multispectral LiDAR.
3) Indian Pines:: Indian Pines is a 20 m/px AVIRIS scene with 224 bands, 145 × 145 pixels, and 16 labeled agricultural and vegetative classes, including rare classes.Water-absorption bands are usually removed before processing.
4) Dataset summary:: Public hyperspectral datasets contain few samples relative to standard optical imagery, and sensor diversity and postprocessing prevent simultaneous training across datasets.DASE provides access to Indian Pines, Pavia, and DFC2018 and supports online evaluation and leaderboard comparison.

III. HYPERSPECTRAL IMAGE ANALYSIS ISSUES AND STANDARD APPROACHES

Hyperspectral classification commonly requires preprocessing to correct, normalize, or reduce spectral data before applying standard statistical models. These steps address noise, scale, dimensionality, and classifier-specific assumptions.

Atmospheric and geometric corrections, band selection, and normalization are common preprocessing operations for hyperspectral images.Normalization affects how classifiers separate spectral features.
Removing water-absorption, low-signal-to-noise, and saturated bands reduces spectral noise and helps address the curse of dimensionality.The paper links this preprocessing to improved classifier robustness.
Zero-mean and unit-variance normalization supports common classifier assumptions, while spectral-angle normalization gives each spectrum unit Euclidean norm.The latter is associated with spectral-angle similarity measures such as SAM.
Moment normalization can be applied per band for classifiers such as SVMs or globally across the image, with bandwise processing potentially squashing spectral dynamics.
Mapping values to [0, 1] mainly supports numerical optimization, but bandwise scaling can remove relative amplitudes and compress spectral dynamics.

B. Spectral classification

Spectral classification treats each hyperspectral pixel as a one-dimensional spectral signature, matching fine spectral resolution and low spatial resolution. Traditional pipelines reduce dimensionality or engineer features before statistical classification.

B. Spectral classification: A pixel can be modeled as a one-dimensional spectral signature because hyperspectral data have fine spectral resolution but low spatial resolution.The paper focuses on machine-learning approaches requiring little expert processing.
1) Unmixing:: Low spatial resolution often makes a pixel a mixture of several materials rather than a pure material spectrum.Under a planar-surface assumption, the observed spectrum can be represented as a linear combination of endmember spectra.
1) Unmixing:: Spectral unmixing estimates material-specific abundance maps by decomposing mixed pixel spectra into reference endmembers and their proportional contributions.
2) Dimension reduction:: Dimension-reduction methods extract discriminative information from redundant, correlated spectral bands to simplify the representation space and learning stage.Principal component analysis is cited as one feature-extraction approach.
3) Statistical classification:: After feature processing, hyperspectral classification can use standard models including decision trees, random forests, support vector machines, and manifold-learning methods.

C. Spatial-spectral classification

Spatial-spectral classification incorporates neighborhood structure alongside spectral information because neighboring pixels often share meaningful structural relationships. The reviewed approaches use postprocessing, preprocessing, or jointly learned features, while deep learning reduces manual feature engineering.

Spatial context improves model robustness and efficiency by exploiting structural dependencies among neighboring pixels.Examples include polygon-shaped buildings and fractal-like vegetation patterns.
Spatial regularization can be applied after spectral classification using supervised MRF or CRF models.
Unsupervised preprocessing methods segment images and aggregate spectrumwise features to enforce local consistency.Hierarchical segmentations can derive local morphological features such as morphological attribute profiles.
Joint spatial-spectral methods learn combined features directly over fixed or adaptive local neighborhoods using specialized statistical models.
Deep learning addresses a central limitation of shallow methods by automatically learning task-tailored representations instead of relying on manually engineered features.

B. Spectral classification

Spectral classification progresses from fully connected and one-dimensional models toward convolutional approaches that incorporate spatial information. Earlier methods often use hand-designed or unsupervised dimensionality reduction, leaving end-to-end representation learning incomplete.

Deep fully connected networks and 1D CNNs extend standard spectral classifiers by learning more discriminative representations from individual spectra.
Auto-encoders perform learned dimensionality reduction and compression, potentially retaining information more efficiently than PCA.Cascades of auto-encoders were proposed for denoising and dimensionality reduction.
Standard CNN transfer is difficult because filter parameters scale with input channels, reaching 5×5×n×100 for 100-band hyperspectral images versus 5×5×n×3 for RGB.
One approach projects hyperspectral data into a 3-channel tensor with PCA before applying a standard 2D CNN for classification.
Another approach uses separate 1D and 2D CNNs to extract spectral and spatial features, concatenating them before classification.
Earlier spectral-spatial pipelines did not fully exploit end-to-end representation learning because they relied on ad hoc dimension splitting or unsupervised reduction.

2) 2D+1D CNN:

2D+1D CNNs alternate or combine spatial and spectral processing to reduce hyperspectral dimensionality while learning both types of representations. Related unsupervised and fully convolutional designs extend this strategy to efficient pixelwise prediction.

End-to-end CNNs alternate spatial and spectral convolutions to reduce feature-map size while preserving both information types.
A 3D-patch CNN alternates 1×1×n spectral reduction with k×k×1 spatial processing before fully connected classification.This alternation learns spectral and spatial representations in successive stages.
An alternative performs spatial-spectral convolutions in the first layer for supervised spectral reduction, followed by conventional 2D CNN layers.
Residual 2D CNNs can learn low-dimensional representations of hyperspectral pixels together with their neighborhoods in an unsupervised setting.
The FCN in applies multiscale spatial-spectral filtering, 1D convolutions, and nonlinearities to classify all pixels in an input patch.Its fully convolutional design improves sliding-window inference efficiency because it predicts several pixels rather than only the central one.

3) 3D CNN:

3D CNNs process hyperspectral cubes across spatial and spectral dimensions simultaneously, combining pattern-recognition strategies with fewer parameters and layers. The review also emphasizes architecture selection, optimization, regularization, and validation practices for reliable experimentation.

3D CNN: 3D CNNs produced higher classification performance than 2D counterparts in the cited comparison.
3D CNN: 3D convolutions combine spatial and spectral recognition in one filter, enabling more complex reflectance-pattern modeling with fewer parameters and layers.
Toolbox: DeepHyperX provides a PyTorch-based toolbox for benchmarking SVMs and multiple CNN families on public hyperspectral datasets.Supported architectures include 1D, 2D, and 3D CNNs, with tunable hyperparameters and several datasets.
Best practices: Model choice should reflect spatial correlations, with 3D CNNs suited to exploiting correlations across all three hyperspectral dimensions.
Best practices: Larger models require more training samples, while large 3D kernels are slower and fully convolutional designs reduce parameters and improve prediction efficiency.
Best practices: Initialization, dropout, batch normalization, data augmentation, separate validation data, and checkpointing support optimization and reduce overfitting risks.Hyperparameters should be tuned on validation data rather than the test set to avoid optimistic results.

3) Experiments:

The experiments reimplement and compare multiple hyperspectral classification architectures under standardized, spatially disjoint evaluation, addressing inconsistencies in prior experimental setups. Results show that reported performance depends strongly on the train/test split and dataset resolution.

Experimental setup: Prior studies vary in dataset splits, class subsets, training-set sizes, and validation procedures, preventing principled comparison of architectures.These differences include random whole-image sampling, omitted rare classes, and tuning directly on test data.
Experimental setup: Randomly sampled neighboring pixels can make test data too similar to training data and allow CNN receptive fields to include test samples.The resulting overlap can cause overfitting and bias evaluation toward overly optimistic generalization.
Experimental setup: The study uses IEEE GRSS DASE train/test splits from significantly disjoint image regions to prevent test pixels from entering 3D CNN training patches.The toolbox reimplements published models as closely as possible and compares them under this common protocol.
Compared models: The comparison includes 1D CNN and RNN models, 2D+1D CNN, 3D CNN, an SVM, and a fully connected 1D neural-network baseline.The 2D+1D and 3D CNN models use 5 × 5 pixel patches, class-imbalance weighting, and spatial symmetry augmentation.
Results: Reimplementations generally approach original reported scores, but strongly disjoint testing produces substantially lower results than the literature reports.The RNN reimplementation is an exception, underperforming its original score.
Results: 3D spatial-spectral CNNs improve accuracy by 3% on Pavia University and 2% on DFC2018 versus 1D CNNs, while spatial information reduces accuracy on Indian Pines.The authors relate the contrast to Indian Pines’ 20 m/px resolution, where pixels mix ground materials and neighboring information adds less discrimination.

E. Perspectives

The paper identifies fully three-dimensional networks, unsupervised or semi-supervised learning, and data synthesis as promising directions for addressing hyperspectral classification’s spatial, annotation, and dataset limitations.

Fully 3D networks: Fully 3D end-to-end networks could reduce dimensions in both spatial and spectral domains while extracting meaningful local features.The paper contrasts this direction with directly transferring 2D CNNs, which mainly rely on spatial context.
Reference datasets: Larger and more complex reference datasets are needed because existing standard approaches already perform satisfactorily on many benchmarks.This makes it difficult to distinguish among newly proposed approaches.
Unsupervised learning: Unsupervised and semi-supervised models could address the sparse annotations available for hyperspectral data.The paper gives 3D spatial-spectral auto-encoders as an example for learning representations independently of acquisition context.
Data synthesis: Generative models can synthesize hyperspectral scenes for data augmentation and potentially estimate costly image transformations.Examples include atmospheric correction, transfer-function estimation between sensors, and image denoising.
Data synthesis: The benefit of adding generative-model data for augmentation does not yet appear substantial.The paper nevertheless identifies broader transformation-estimation uses as potentially valuable.

V. CONCLUSION

Deep learning is already effective for hyperspectral classification, with 2D and 3D convolutional networks combining spatial and spectral information and often achieving state-of-the-art performance. The main remaining obstacle is the scarcity of massively annotated datasets, motivating unsupervised learning and synthetic data augmentation.

Conclusion: 2D and 3D convolutional networks combine spatial and spectral information and show state-of-the-art performance in early published studies.These results are often obtained without expert knowledge of the physics or sensor.
Conclusion: Massively annotated datasets remain scarce, limiting the available data for statistical approaches.The conclusion identifies data volume as key to the success of statistical methods.
Conclusion: Unsupervised approaches and data augmentation by synthesis are proposed to help overcome annotation scarcity and unlock deep learning’s potential in hyperspectral imaging.The paper presents these as promising paths rather than established solutions.

APPENDIX

The appendix provides biographical information about the paper’s authors, describing their research positions and expertise in computer vision, machine learning, remote sensing, and Earth observation.

Author biographies: Nicolas Audebert is a research scientist whose work focuses on computer vision, deep learning, remote sensing, and Earth Observation.His doctoral research concerned deep neural networks for scene understanding of airborne and satellite optical images.
Author biographies: Bertrand Le Saux is an ONERA research scientist working on data-driven visual understanding, remote sensing, robotics, and 3D vision.He also develops machine learning and deep learning methods for these areas.
Author biographies: Sébastien Lefèvre is a computer science professor leading the OBELIX team on image analysis and machine learning for remote sensing and Earth Observation.His research is conducted within IRISA.