Source-linked AI summary
HybridSN: Exploring 3D-2D CNN Feature Hierarchy for Hyperspectral Image Classification
Swalpa Kumar Roy, Gopal Krishna, Shiv Ram Dubey, Bidyut B. Chaudhuri
TL;DR
HSI classification requires models that capture both spectral and spatial information, while 2D-CNN and 3D-CNN approaches have complementary limitations. HybridSN combines spectral-spatial 3D convolutions with a spatial 2D convolution, and experiments across three benchmark datasets report superior performance, computational efficiency relative to 3D-CNN, and strong results with small training data.
Problem
HSI classification depends on both spectral and spatial information, but 2D-CNNs cannot adequately handle spectral information and 3D-CNNs have increased computational complexity.
Method
HybridSN applies PCA-reduced HSI patches to three 3D convolutions followed by one 2D convolution and three fully connected layers.
Results
HybridSN outperforms the compared methods across Indian Pines, University of Pavia, and Salinas Scene while maintaining minimum standard deviation.
Takeaways & Limitations
The hybrid model combines complementary spatio-spectral and spectral information, is computationally efficient relative to 3D-CNN, and performs well with small training data.
Abstract
from arXiv · showhide
Hyperspectral image (HSI) classification is widely used for the analysis of remotely sensed images. Hyperspectral imagery includes varying bands of images. Convolutional Neural Network (CNN) is one of the most frequently used deep learning based methods for visual data processing. The use of CNN for HSI classification is also visible in recent works. These approaches are mostly based on 2D CNN. Whereas, the HSI classification performance is highly dependent on both spatial and spectral information. Very few methods have utilized the 3D CNN because of increased computational complexity. This letter proposes a Hybrid Spectral Convolutional Neural Network (HybridSN) for HSI classification. Basically, the HybridSN is a spectral-spatial 3D-CNN followed by spatial 2D-CNN. The 3D-CNN facilitates the joint spatial-spectral feature representation from a stack of spectral bands. The 2D-CNN on top of the 3D-CNN further learns more abstract level spatial representation. Moreover, the use of hybrid CNNs reduces the complexity of the model compared to 3D-CNN alone. To test the performance of this hybrid approach, very rigorous HSI classification experiments are performed over Indian Pines, Pavia University and Salinas Scene remote sensing datasets. The results are compared with the state-of-the-art hand-crafted as well as end-to-end deep learning based methods. A very satisfactory performance is obtained using the proposed HybridSN for HSI classification. The source code can be found at \url{https://github.com/gokriznastic/HybridSN}.
I. INTRODUCTION
HSI classification must account for both spectral and spatial information in volumetric data. HybridSN is motivated by combining 3D and 2D convolutions to address limitations of using either alone.
- Motivation: HSI contains multiple image bands whose spectral and spatial correlations provide useful scene information but increase analysis difficulty.The literature addresses HSI classification through hand-designed or learning-based feature extraction.
- Motivation: 2D-CNNs may miss channel relationships, while 3D-CNNs impose greater computational complexity and can perform worse on classes with similar textures across bands.These shortcomings motivate a hybrid CNN design for HSI data.
- Proposed direction: HybridSN assembles 3D-CNN and 2D-CNN layers to use spectral and spatial feature maps together for HSI classification.The model is introduced as a hybrid-CNN approach to overcome the cited limitations of single-dimensional convolutional designs.
II. PROPOSED HYBRIDSN MODEL
HybridSN preprocesses HSI cubes with PCA and applies a hierarchy of 3D and 2D convolutions to learn spectral-spatial representations. Its architecture uses three 3D convolutions, one 2D convolution, and fully connected layers.
- Input preparation: PCA reduces spectral bands from D to B while preserving the input cube’s M×N spatial dimensions.The reduced cube is represented as X ∈ R^M×N×B.
- Architecture configuration: The architecture uses a 25×25 window, and its final dense layer is dataset-dependent through the number of output classes.For Indian Pines, HybridSN has 5,122,176 trainable weight parameters and is trained with Adam and softmax loss.
- Input preparation: The model creates overlapping S×S×B 3D patches from the PCA-reduced cube, with each patch labeled by its centered pixel.The number of patches is (M−S+1)×(N−S+1).
- Network architecture: The 3D convolution kernels operate over multiple contiguous spectral bands to capture spectral information in the feature maps.The spectral kernel extent is represented by 2η+1, while the remaining parameters follow the 2D formulation.
- Network architecture: HybridSN comprises three 3D convolutions, one 2D convolution, and three fully connected layers trained for HSI classification.The 3D kernels jointly process spatial and spectral information, while the 2D stage learns spatial feature maps.
- Network architecture: The 2D convolution is applied once before flattening to discriminate spatial information within different spectral bands without substantial spectral-information loss.The reported 2D kernel uses 64 kernels with a 3×3 spatial dimension and 576 input feature maps.
A. Dataset Description and Training Details
Experiments use three public hyperspectral datasets with standardized spatial-spectral patches and report classification maps and convergence behavior for Indian Pines.
- Datasets: The experiments use Indian Pines, University of Pavia, and Salinas Scene hyperspectral datasets.Indian Pines contains 16 vegetation classes after removing 24 water-absorption bands.
- Evaluation outputs: Indian Pines classification maps compare SVM, 2D-CNN, 3D-CNN, M3D-CNN, SSRN, and HybridSN predictions against false-color imagery and ground truth.The maps are shown in Fig. 2.
- Evaluation outputs: Confusion matrices are provided for Indian Pines, University of Pavia, and Salinas Scene using the proposed method.The first, second, and third matrices correspond to these datasets, respectively.
- Evaluation outputs: Accuracy and loss convergence are plotted against epochs for the Indian Pines dataset.The convergence curves are shown in Fig. 4.
B. Classification Results
HybridSN outperforms the compared methods across the evaluated datasets while remaining more efficient than 3D-CNN and retaining strong performance with less training data.
- Overall results: HybridSN outperforms all compared methods on each dataset for Overall Accuracy, Average Accuracy, and Kappa while maintaining the minimum standard deviation.The comparisons use 30% training data and 70% testing data, with SVM, 2D-CNN, 3D-CNN, M3D-CNN, and SSRN as baselines.
- Overall results: HybridSN combines hierarchical spectral-spatial 3D-CNN features with spatial 2D-CNN features that are described as complementary.The results are presented in Table II.
- Dataset-specific behavior: 3D-CNN performs worse than 2D-CNN on the Salinas Scene dataset, probably because Grapes-untrained and Vinyard-untrained have similar textures across most spectral bands.The paper attributes this difference to increased spectral-band redundancy.
- Qualitative results: HybridSN and SSRN produce better classification maps than the other methods, with HybridSN better than SSRN in small segments.These qualitative comparisons are reported for an example Indian Pines image.
- Efficiency and data sensitivity: HybridSN is more efficient than 3D-CNN and remains superior in almost all cases when training uses only 10% of the samples.With less training data, the performance of each model decreases slightly.
IV. CONCLUSION
The paper concludes that HybridSN combines 3D and 2D convolutional representations for hyperspectral image classification and performs well across benchmark datasets and limited-training settings.
- Conclusion: HybridSN combines complementary spatio-spectral and spatial information through 3D and 2D convolutions, respectively.The hybrid model is evaluated on three benchmark datasets against recent state-of-the-art methods.
- Conclusion: HybridSN is computationally efficient compared with 3D-CNN and shows superior performance with small training data.These conclusions are reported after experiments on three benchmark datasets.