Going Deeper with Contextual CNN for Hyperspectral Image Classification

Hyungtae Lee; Heesung Kwon

Source-linked AI summary

Going Deeper with Contextual CNN for Hyperspectral Image Classification

Hyungtae Lee, Heesung Kwon

arXiv:1604.03519v3Apr 12, 2016cs.CVcs.LG

TL;DR

HSI classification needs deeper models that can jointly exploit spectral and spatial information despite limited training data. The paper proposes a residual-learning, fully convolutional CNN with a multi-scale filter bank, and reports enhanced performance on three benchmark datasets over current state-of-the-art approaches.

Problem
Existing CNN-based HSI methods do not fully exploit spectral and spatial information together, while limited training data constrain deeper and wider networks.
Method
The proposed end-to-end network combines a multi-scale filter bank, residual learning, and a fully convolutional network to jointly process spatio-spectral information.
Results
Enhanced classification performance is reported over current state-of-the-art approaches on the Indian Pines, Salinas, and University of Pavia datasets.
Takeaways & Limitations
Joint spatio-spectral exploitation and deeper training are identified as the main sources of the proposed network’s improved classification performance across three benchmark datasets.

Abstract

from arXiv · show

In this paper, we describe a novel deep convolutional neural network (CNN) that is deeper and wider than other existing deep networks for hyperspectral image classification. Unlike current state-of-the-art approaches in CNN-based hyperspectral image classification, the proposed network, called contextual deep CNN, can optimally explore local contextual interactions by jointly exploiting local spatio-spectral relationships of neighboring individual pixel vectors. The joint exploitation of the spatio-spectral information is achieved by a multi-scale convolutional filter bank used as an initial component of the proposed CNN pipeline. The initial spatial and spectral feature maps obtained from the multi-scale filter bank are then combined together to form a joint spatio-spectral feature map. The joint feature map representing rich spectral and spatial properties of the hyperspectral image is then fed through a fully convolutional network that eventually predicts the corresponding label of each pixel vector. The proposed approach is tested on three benchmark datasets: the Indian Pines dataset, the Salinas dataset and the University of Pavia dataset. Performance comparison shows enhanced classification performance of the proposed approach over the current state-of-the-art on the three datasets.

I. INTRODUCTION

The paper addresses limited HSI training data and incomplete joint use of spectral and spatial information by proposing a deeper, wider CNN with residual learning and fully convolutional processing.

Motivation: Limited large-scale HSI datasets hinder training deep, parameter-rich CNNs and motivate approaches that can exploit deeper and wider networks.Existing methods often reduce spectral dimensionality or use fewer layers and nodes, sacrificing performance.
Architecture: The end-to-end fully convolutional network combines a multi-scale filter bank and residual learning without additional pre- or post-processing.Its fully convolutional design accepts hyperspectral images of arbitrary size and avoids pooling layers that change output size.
Evaluation: Performance comparisons on Indian Pines, University of Pavia, and Salinas show enhanced classification performance over current state-of-the-art CNN approaches.The three benchmark datasets have different image sizes, and the network is evaluated across all three.
Contribution: The proposed network uses residual learning to support increased depth and width despite limited training samples.The paper identifies residual learning as a response to sub-optimality associated with limited training data.
Contribution: The architecture jointly optimizes spectral and spatial information rather than processing these feature types separately.The motivation is that existing CNN-based approaches fall short of fully exploiting spectral and spatial information together.

II. RELATED WORKS

Prior HSI classification research includes kernel and deep-learning methods, but the paper targets deeper and wider CNNs under limited training data.

Kernel methods: Kernel methods are suited to limited training samples because they can learn complex decision boundaries with relatively few parameters.They project data into a high-dimensional reproducing kernel Hilbert space.
Deep learning: Deep learning approaches typically require datasets whose size is proportional to network parameters to avoid overfitting.This requirement constrains the use of large networks when HSI data are limited.
Deep HSI methods: Earlier deep HSI methods separately acquire spectral and spatial vectors before jointly using them for feature learning and classification.One described approach combines a spectral pixel vector with a spatial vector derived from PCA over the spectral dimension.
Positioning: The paper’s approach attempts to build a much deeper and wider network using relatively small amounts of training samples.Its expected benefit is enhanced performance over relatively shallow and narrow networks after effective optimization.

III. THE CONTEXTUAL DEEP CONVOLUTIONAL NEURAL NETWORK

The paper introduces CNN components that extract nonlinear features through convolutional layers and classify inputs probabilistically with softmax.

CNN foundations: Deep CNNs use multiple neuron layers to extract progressively different levels of nonlinear features.Nonlinearity is produced by applying activation functions to outputs of local convolutional filters.
CNN foundations: AlexNet provides a reference architecture with five convolutional layers and three fully connected layers.Its fully connected layers use linear weights to connect input and output vectors.
Convolutional layers: Convolutional layers apply local filters to extract nonlinear features from input vectors.The filters are smaller than the fully connected weight matrix, and convolution denotes the operator used.
CNN operations: LRN normalizes activations across adjacent filters, while max pooling downsamples feature maps for dimensionality reduction.These are among the nonlinear components used in the referenced CNN design.
CNN operations: ReLU keeps positive activations, dropout reduces overfitting, and softmax supports probabilistic multiclass classification including HSI classification.The paper specifies a dropout threshold of 0.5 and describes softmax through class-specific functions.

B. Architecture of the Proposed Network

The proposed architecture is a nine-layer fully convolutional network that combines convolutional processing, residual learning, and convolutionalized classification layers for HSI data.

Architecture: The network begins with a multi-scale filter bank, continues through two residual-learning blocks, and ends with three convolutional classification layers.The final layers function similarly to AlexNet’s fully connected layers for classification using local features.
Convolutionalized model: A 1 × 1 × d convolution with l filters can replace a fully connected layer using the same d × l weights.This convolutionalized model enables FCNs to perform pixel classification.
Architecture: The architecture preserves blob height and width while changing only depth, with no dimensionality reduction during FCN processing.The first two convolutional outputs are normalized by LRN.
Depth and training data: The proposed network contains nine layers, making it deeper than other CNNs for HSI classification trained on the same datasets.Its depth remains smaller than image-classification networks such as ResNet because HSI training sets are much smaller.
Depth and training data: Data augmentation is used alongside a parameter-to-training-data comparison to support the proposed deeper architecture.The paper reports that the proposed networks have parameter-to-data ratios at least approximately eight times larger than image-classification CNNs.

C. Multi-scale Filter Bank

The multi-scale filter bank jointly captures spectral and spatial correlations by applying filters of three receptive-field sizes and merging their feature maps.

Filter bank design: The first layer applies 1 × 1 × B, 3 × 3 × B, and 5 × 5 × B filters to the input hyperspectral image.B denotes the number of spectral bands.
Filter bank design: The 1 × 1 × B filters address spectral correlations, while the 3 × 3 × B and 5 × 5 × B filters exploit local spatial correlations.Their outputs are combined into a joint spatio-spectral feature map for subsequent convolutional layers.
Feature-map alignment: Zero padding adjusts the three feature-map sizes so they can be combined into a joint feature map.The resulting dimensions before further adjustment are (H +4, W +4), (H +2, W +2), and (H, W).
Computational challenge: Large spatial filters with many spectral bands and feature-map merging increase computational complexity and network size.With few training samples, the enlarged network faces overfitting and divergence concerns, motivating an additional strategy.
Design rationale: The filter bank is conceptually similar to an inception module and is intended to exploit diverse local image structures.The paper connects this design with deeper networks and state-of-the-art image-classification performance reported for inception modules.

D. Residual Learning

Residual learning lets the proposed network learn deep layers relative to their inputs, improving training efficiency while preserving computational constraints for small-sample HSI classification.

Residual learning: The proposed network uses two residual-learning modules to improve training efficiency in a deep network.The paper states that residual learning helps optimize deep networks with relatively few training samples.
Residual learning: Residual learning represents the output as the input plus a learned residual mapping through convolutional filters.The mapping uses shortcut connections around two convolutional layers.
Residual learning: Multi-scale filtering and residual learning increase network depth and width while keeping the computational budget constrained.
Training-data augmentation: Training samples are augmented by mirroring 5×5 neighborhoods across horizontal, vertical, and diagonal axes.

E. Learning the Proposed Network

The network is trained end-to-end on sampled hyperspectral pixels and their 5×5 neighborhoods, then evaluated against established baselines across three benchmark datasets.

Training setup: Each training sample consists of a sampled pixel and its surrounding 5×5 neighborhood, with mirroring used to quadruple the training data.The network has approximately 1000K parameters learned from several hundred training pixels per material category.
Datasets: The study evaluates Indian Pines, Salinas, and University of Pavia hyperspectral datasets with differing spatial and spectral dimensions.Indian Pines uses eight classes, while Salinas and University of Pavia use 16 and nine classes, respectively.
Baselines: The proposed nine-convolutional-layer network is compared with a shallower CNN, an RBF-kernel SVM, D-DBN, and additional neural networks on Indian Pines.
Evaluation protocol: For fair comparison, 200 samples are randomly selected from each class for training, with the remaining samples used for testing.The selected classes and train/test counts are reported in dataset-specific tables.
Evaluation: Classification performance is compared across the proposed network and baselines on all three datasets.The comparison is summarized in Table V, including results across train/test partitions.

B. HSI Classification

The proposed network outperforms the evaluated baselines across Indian Pines, Salinas, and University of Pavia, with gains reported in mean classification performance.

Classification performance: The proposed network improves over every baseline on all three benchmark datasets.
Classification performance: 2.58%, 2.47%, and 2.86% are the mean-performance gains over the best baseline on Indian Pines, Salinas, and University of Pavia, respectively.The classification maps shown for the datasets come from one arbitrary train/test partition among 20.
Classification performance: The paper attributes the performance enhancement mainly to deeper and wider networks and joint exploitation of spatio-spectral information.Residual learning is also reported to improve performance by optimizing training efficiency with relatively few samples.

C. Finding the Optimal Depth and Width of the Network

Experiments identify dataset-dependent network widths and a shared optimal depth, while multi-scale filtering improves performance but overly large spatial filters cause boundary spillover.

Network width: 128 filters are optimal for Indian Pines and University of Pavia, whereas Salinas achieves its best performance with 192 filters.The paper relates Salinas’s larger training set and class count to the need for more weights.
Network width: Adding filters beyond each dataset’s optimal width reduces performance and increases computational cost.
Network depth: Two residual-learning modules achieve the best performance across all three datasets, while three modules may overfit and degrade performance.
Selected configuration: The selected networks use two residual-learning modules with width 128 for Indian Pines and University of Pavia, and width 192 for Salinas.
Multi-scale filter bank: The multi-scale filter bank significantly outperforms a 1×1-only network by 39.94%, 44.45%, and 30.35% in mean performance on the three datasets.The reported gains are for Indian Pines, Salinas, and University of Pavia, respectively.
Multi-scale filter bank: Including a 7×7 spatial filter degrades performance through spillover near class boundaries, motivating the 1×1, 3×3, and 5×5 configuration.

E. Effectiveness of Residual Learning

Residual learning improves optimization of the proposed network under sparse training data, while classification accuracy increases with larger training sets.

Residual learning: The network without residual learning failed to converge, while replacing the first residual module with two convolutional layers produced sub-optimal performance.Both compared networks used nine convolutional layers.
Residual learning: Figure 9 compares training loss and classification accuracy across three datasets for the proposed and modified networks over training iterations.The proposed architecture is labeled “w/ residual learning,” and the modified architecture “w/o residual learning.”
Training dataset size: Indian Pines was not evaluated with 800 examples per class because several classes contain fewer than 800 samples.Examples include Grass-pasture with 483, Hay-windrowed with 478, and Soybean-clean with 593 samples.
Training dataset size: Classification accuracy monotonically increases as the number of training examples per class rises.The evaluated training sizes are 50, 100, 200, 400, or 800 examples per class.
Training dataset size: With only 50 or 100 examples per class, the proposed network achieves higher accuracy than multiple kernel learning-based HSI classification.The comparison is reported in Table XII.

G. False Positives Analysis

False positives are concentrated near class boundaries, while the proposed architecture combines depth, residual learning, and multi-scale spatio-spectral processing to support classification.

Class-specific errors: For Indian Pines, accuracy falls below 95% in only corn-notill and soybean-mintill among the eight classes.These two classes have substantially more samples than the others, and limited training data may impair representation of their overall spectral characteristics.
Boundary effects: False positives are more frequent near class boundaries across all three datasets.Pixels at the boundary are labeled zero, pixels one pixel away are labeled one, and the remaining pixels are labeled ≥2.
Boundary effects: Boundary-near false positives are identified as a major factor degrading the proposed network’s performance.Pixels farther than one pixel from boundaries are less prone to misclassification because they are less affected by spillover.
Architecture: The proposed nine-layer network uses residual learning and a 3×3, 5×5, and 1×1 filter bank to jointly exploit spatial and spectral correlations.The 3×3 and 5×5 filters address local spatial correlations, while the 1×1 filter addresses spectral correlations.
Architecture: Two residual learning modules outperform one, while the proposed network improves classification performance over state-of-the-art CNN architectures on all three benchmark datasets.The number of residual modules, network depth, and width are selected by cross-validation.