PaperlayerView exact revision on arXiv

Source-linked AI summary

Hyperspectral Classification Based on Lightweight 3-D-CNN With Transfer Learning

Haokui Zhang, Ying Li, Yenan Jiang, Peng Wang, Qiang Shen, Chunhua Shen

arXiv:2012.03439v1cs.CV

TL;DR

Limited labeled HSI samples make deep 3-D CNN classification prone to overfitting despite the value of spectral–spatial features. The paper introduces lightweight 3-D-LWNet with cross-sensor and cross-modal transfer learning, and reports improved classification performance over conventional 3-D-CNN methods.

  • Problem

    HSI classification involves complex 3-D data but relatively few labeled samples, limiting the use of deep CNNs and increasing overfitting risk.

  • Method

    The paper proposes a deep lightweight 3-D-LWNet and combines it with cross-sensor and cross-modal transfer learning.

  • Results

    3-D-LWNet uses fewer parameters while achieving higher classification accuracy than conventional 3-D-CNNs, and transfer learning improves results with small training sets.

  • Takeaways & Limitations

    Cross-sensor transfer can use HSI data from different sensors, while cross-modal transfer can use 2-D RGB datasets for HSI classification.

Abstract

from arXiv · show

Recently, hyperspectral image (HSI) classification approaches based on deep learning (DL) models have been proposed and shown promising performance. However, because of very limited available training samples and massive model parameters, DL methods may suffer from overfitting. In this paper, we propose an end-to-end 3-D lightweight convolutional neural network (CNN) (abbreviated as 3-D-LWNet) for limited samples-based HSI classification. Compared with conventional 3-D-CNN models, the proposed 3-D-LWNet has a deeper network structure, less parameters, and lower computation cost, resulting in better classification performance. To further alleviate the small sample problem, we also propose two transfer learning strategies: 1) cross-sensor strategy, in which we pretrain a 3-D model in the source HSI data sets containing a greater number of labeled samples and then transfer it to the target HSI data sets and 2) cross-modal strategy, in which we pretrain a 3-D model in the 2-D RGB image data sets containing a large number of samples and then transfer it to the target HSI data sets. In contrast to previous approaches, we do not impose restrictions over the source data sets, in which they do not have to be collected by the same sensors as the target data sets. Experiments on three public HSI data sets captured by different sensors demonstrate that our model achieves competitive performance for HSI classification compared to several state-of-the-art methods

I. INTRODUCTION

HSI classification combines rich spectral–spatial information with limited labeled data, making deep models vulnerable to overfitting. The paper proposes a deeper, lightweight 3-D-LWNet with cross-sensor and cross-modal transfer learning.

  • HSIs provide abundant spectral and spatial information, but their higher dimensionality complicates efficient and accurate feature extraction.
  • Spectral-only methods miss spatial variability because different objects can share spectra and identical objects can vary spectrally across locations.
  • Limited HSI training samples make very deep CNNs difficult to apply and increase the risk of overfitting.
  • 3-D-LWNet uses tens of 3-D convolution layers while requiring fewer parameters than conventional 3-D-CNNs.
  • The framework combines 3-D-LWNet with cross-sensor and cross-modal transfer learning to address limited training samples.

II. RELATED WORK

Prior HSI classification work spans SAE, DBN, CNN, and RNN models, with CNNs increasingly favored for spectral–spatial feature learning and classification performance.

  • SAE and DBN approaches require spatial information to be flattened into one-dimensional vectors, limiting efficient spatial feature extraction.
  • CNN use in HSI classification has grown rapidly, and CNN performance is generally better than that of SAE and DBN methods.
  • HSI classification CNNs are categorized as 1-D, 2-D, or 3-D according to the dimensions convolved by their kernels.
  • RNNs model HSIs as ordered spectral sequences, but remain less common than SAE-, DBN-, and CNN-based approaches.
  • Modern CNN architectures trend toward greater depth and lower computation cost, as illustrated by ResNet, DenseNet, and MobileNet.

1) GoogleNet:

The paper situates its lightweight design among efficient CNN architectures and motivates transfer learning because deep models often require many labeled samples.

  • 1) GoogleNet:: GoogleNet uses multiple inception modules, each containing four different convolution paths.
  • ResNet uses shortcut connections and bottleneck structures to support extreme depth while reducing time complexity.
  • DenseNet connects each layer to every other layer in a feedforward fashion within dense blocks.
  • MobileNet uses depthwise separable convolutions and pointwise convolutions to reduce computation and combine channel features.
  • Transfer learning is useful when training data are insufficient, especially for deep CNNs with massive parameter counts.
  • A common strategy pretrains a model on a large labeled dataset before transferring it to a target dataset for fine-tuning.

III. PROPOSED METHOD

The proposed framework extracts centered spectral–spatial cubes, processes them with a 3-D-LWNet, and produces classification scores through a three-part network.

  • HSI samples are S × S × L cubes extracted from neighborhood windows centered on target pixels, with center-pixel labels.
  • The framework consists of sample extraction, 3-D-LWNet processing, and classification-result generation.
  • 3-D-LWNet processes extracted samples through three parts to obtain classification scores.
  • 1) 3-D-LWNet:: The first network part applies 3-D convolution, batch normalization with ReLU, and 3-D max pooling to input batches.
  • 1) 3-D-LWNet:: The second part passes pooled features through eight LW units arranged in four groups with progressively wider channel outputs.
  • 1) 3-D-LWNet:: Adaptive average pooling in the final part adjusts feature size to a fixed value before classification.

2) Training Strategy:

The training strategy combines a lightweight 3-D CNN design with regularized optimization to reduce overfitting and computation for limited-sample HSI classification.

  • Optimization: Training minimizes negative log-likelihood plus L2 regularization, with weight decay empirically set to 1e-5.The regularization term restricts the sum of squared network parameters to help avoid overfitting.
  • Lightweight network design: 3-D-LWNet uses pointwise and depthwise convolutions, batch normalization, ReLU activations, and shortcut connections to reduce parameters.Its lightweight unit contains two pointwise convolutions surrounding a 3-D depthwise convolution with a 3×3×3 kernel.
  • Shortcut connections: For downsampling with doubled channels, the shortcut uses average pooling and pointwise convolution to match spatial size and channel dimensionality.The average pooling uses stride 2 and kernel size 2.
  • Shortcut connections: When input and output size and channel count are unchanged, the shortcut is left unchanged and combined with the transformed features by elementwise addition.The design uses two shortcut cases for different output configurations.

C. Transfer Learning

The paper uses transfer learning to address limited HSI labels through cross-sensor and cross-modal pretraining followed by target-dataset fine-tuning.

  • Fine-Tuning Strategy: The system includes two pretraining parts and one fine-tuning part for adapting the pretrained model to the target HSI dataset.The transfer-learning flow is organized into cross-sensor or cross-modal pretraining followed by target adaptation.
  • Cross-Sensor Strategy: Cross-sensor transfer pretrains 3-D-LWNet on an HSI dataset with more labeled samples, then transfers it to a target HSI dataset for fine-tuning.The source and target datasets are not required to come from the same sensor.
  • Cross-Sensor Strategy: The method assumes knowledge learned from HSI datasets with different spectral configurations can be transferred.This supports applying cross-sensor transfer without requiring identical sensor configurations.
  • Cross-Modal Strategy: Cross-modal transfer pretrains 3-D-LWNet on natural RGB image datasets and transfers it to target HSIs for fine-tuning.The strategy leverages RGB datasets containing many more labeled samples than HSI datasets.

A. Data Description and Experiment Design

Experiments evaluate 3-D-LWNet and transfer learning across multiple HSI and RGB datasets, using controlled splits, candidate architectures, and convergence-based training settings.

  • Experiment Design: The experiments compare 3-D-LWNet with 3-D-CNN-LR and two-CNN on Pavia University, Indian Pines, and KSC.Salinas and Pavia Center serve as source HSI datasets, while CIFAR-10 and CIFAR-100 provide RGB pretraining data.
  • Data Description: The HSI datasets span different sensors, spatial resolutions, spectral bands, and class counts.Pavia datasets use ROSIS, Indian Pines and Salinas use AVIRIS, and KSC was acquired by the AVIRIS instrument.
  • Data Description: CIFAR-10 and CIFAR-100 each contain 60 thousand 32×32 color images for RGB-to-HSI transfer experiments.CIFAR-10 has 10 classes, whereas CIFAR-100 has 100 classes.
  • Experiment Design: Without transfer learning, each target HSI dataset is split into training, validation, and test subsets.The transfer-learning experiments instead use randomly selected class-specific test samples and the remaining samples for training.
  • 3-D-ResNet: Ten 3-D ResNet candidates are evaluated with fivefold cross-validation, and the model with the highest average validation accuracy becomes the baseline.The candidates are trained with the same settings across the three target HSI datasets.
  • Training Configuration: Training uses SGD with momentum, weight decay of 1e-5, batch size 20, and a learning-rate reduction after validation loss saturates around 40–50 epochs.The learning rate is divided by 10 at epoch 50.
  • 3-D-ResNet: Bottleneck 3-D ResNets generally outperform corresponding non-bottleneck models, while accuracy peaks and then drops as network depth continues increasing.ResNet-20 is selected as the unified baseline because its overall accuracy is 95.35%, only 0.16 below ResNet-26 on KSC.

C. 3-D-LWNet

The 3-D-LWNet replaces conventional 3-D-ResNet residual units with lightweight units, reducing parameters while maintaining or improving classification performance across HSI datasets.

  • C. 3-D-LWNet: 3-D-LWNet achieves the best performance across OA, AA, and kappa coefficient among the compared models.The evaluation compares 3-D-ResNet-20 and 3-D-LWNet with 3-D-CNN-LR under the same train-test settings.
  • C. 3-D-LWNet: 3-D-LWNet replaces 3-D-ResNet residual units with LW units, reducing the number of parameters involved.The model contains 20 learnable layers while using depthwise convolution to reduce parameter count.
  • C. 3-D-LWNet: 1.31% improvement on Indian Pines and 1.91% on KSC are achieved by 3-D-LWNet over 3-D-CNN-LR.On Pavia University, the OA of 3-D-ResNet and 3-D-LWNet is lower than 3-D-CNN-LR, while the averaged accuracy gap is only 0.14%.
  • C. 3-D-LWNet: 763008 parameters in 3-D-LWNet represent more than 98.3% savings compared with 44892160 parameters in 3-D-CNN-LR for Indian Pines.The comparison illustrates that the deeper lightweight network can use substantially fewer parameters than the shallower baseline.

D. Transfer Learning Between Different HSI Data Sets

Transfer learning from different HSI datasets improves classification with few target samples, and the source dataset need not use the same sensor as the target.

  • D. Transfer Learning Between Different HSI Data Sets: 3-D-LWNet supports transfer learning by using pretrained models from source HSI datasets to initialize target-dataset classification.Pavia Center and Salinas are used for pretraining, while Pavia University, Indian Pines, and KSC are target datasets.
  • D. Transfer Learning Between Different HSI Data Sets: Cross-sensor transfer is permitted because source and target HSI datasets do not have to be captured by the same sensor.This differs from previous work that restricted pretraining data to the target sensor.
  • D. Transfer Learning Between Different HSI Data Sets: 4.17% OA improvement with Salinas pretraining and 2.76% with Pavia Center pretraining occur at 25 samples per class.At 50 samples per class, the improvements decrease to 2.68% and 1.36%, respectively.
  • D. Transfer Learning Between Different HSI Data Sets: Salinas pretraining often outperforms Pavia Center pretraining, despite Pavia Center containing more labeled samples.The paper suggests that Salinas may provide more diverse information because it contains more object classes.

E. Influence of the Number of Object Classes

Transfer-learning accuracy generally increases as the number of object classes in the source pretraining subset increases, whereas very small subsets can harm performance.

  • E. Influence of the Number of Object Classes: Accuracy of transfer learning grows with the number of classes used in the pretraining subset.The experiments pretrain on Salinas subsets containing 4, 8, 12, or 16 classes before fine-tuning on three target HSI datasets.
  • E. Influence of the Number of Object Classes: 87.78% accuracy from a four-class subset is lower than 88.37% without that reduced-subset pretraining on Pavia University.The four-class subset generally harms resulting classification performance.

F. Transfer Learning Between RGB Image Data Set and HSI Data Sets

RGB-image pretraining transfers effectively to HSI classification, with larger and more diverse source datasets producing stronger improvements, especially under limited target samples.

  • F. Transfer Learning Between RGB Image Data Set and HSI Data Sets: RGB-to-HSI transfer learning improves classification performance on all three target HSI datasets.The experiments pretrain 3-D-LWNet on CIFAR-10 or CIFAR-100 before fine-tuning on target HSI data.
  • F. Transfer Learning Between RGB Image Data Set and HSI Data Sets: 8.58% OA improvement from CIFAR-100 and 6.77% from CIFAR-10 occur on Pavia University with 25 samples per class.At 50 samples per class, the improvements decrease to 2.75% and 1.71%, respectively.
  • F. Transfer Learning Between RGB Image Data Set and HSI Data Sets: CIFAR-100 usually achieves higher classification accuracy than CIFAR-10 during transfer to HSI datasets.The paper relates this pattern to the influence of source-dataset diversity and number of object classes.

V. CONCLUSION

The paper addresses limited HSI training samples with 3-D-LWNet and two transfer learning strategies. It reports a deeper, lighter network with higher classification accuracy and identifies CNN architecture optimization as future work.

  • 3-D-LWNet addresses limited training samples for spectral–spatial HSI classification.
  • 3-D-LWNet is deeper than conventional 3-D-CNN while using fewer parameters and achieving higher classification accuracy.
  • Cross-sensor and cross-modal transfer learning strategies are integrated with 3-D-LWNet to further improve classification performance.Cross-sensor transfer uses source HSI data and target HSI data, while cross-modal transfer connects 2-D RGB data with 3-D HSI data.
  • CNN architecture optimization through intelligent algorithms remains an important future work.
Loading 2012.03439v1…