PaperlayerView exact revision on arXiv

Source-linked AI summary

Hyperspectral Image Classification with Attention Aided CNNs

Renlong Hang, Zhu Li, Qingshan Liu, Pedram Ghamisi, Shuvra S. Bhattacharyya

arXiv:2005.11977v2eess.IVcs.CV

TL;DR

Hyperspectral CNN classification must exploit spectral and spatial information despite varying discriminative abilities across bands and positions. The paper proposes parallel spectral and spatial attention sub-networks with adaptive output fusion, and reports superior classification performance across three datasets compared with CNN-related models.

  • Problem

    Different spectral bands and spatial positions have unequal discriminative abilities, while relying only on spectral features can cause misclassification in hyperspectral image classification.

  • Method

    A two-branch attention-aided CNN uses spectral attention for discriminative channels, spatial attention for semantic positions, and adaptively weighted summation to fuse branch outputs.

  • Results

    The integrated model further improves over its separate branches and achieves the best OA, AA, Kappa, and F1 scores among compared state-of-the-art CNN-related models.

  • Takeaways & Limitations

    Attention modules improve both spectral and spatial sub-networks, while their integration combines complementary information with a balance between classification performance and computation time.

Abstract

from arXiv · show

Convolutional neural networks (CNNs) have been widely used for hyperspectral image classification. As a common process, small cubes are firstly cropped from the hyperspectral image and then fed into CNNs to extract spectral and spatial features. It is well known that different spectral bands and spatial positions in the cubes have different discriminative abilities. If fully explored, this prior information will help improve the learning capacity of CNNs. Along this direction, we propose an attention aided CNN model for spectral-spatial classification of hyperspectral images. Specifically, a spectral attention sub-network and a spatial attention sub-network are proposed for spectral and spatial classification, respectively. Both of them are based on the traditional CNN model, and incorporate attention modules to aid networks focus on more discriminative channels or positions. In the final classification phase, the spectral classification result and the spatial classification result are combined together via an adaptively weighted summation method. To evaluate the effectiveness of the proposed model, we conduct experiments on three standard hyperspectral datasets. The experimental results show that the proposed model can achieve superior performance compared to several state-of-the-art CNN-related models.

I. INTRODUCTION

Hyperspectral classification benefits from jointly exploiting spectral and spatial information, but CNNs can better use these cues by attending to discriminative bands and positions. The paper therefore proposes an attention-aided spectral-spatial CNN with two branches and adaptive fusion.

  • Motivation: Hyperspectral classification assigns predefined categories to pixels using spectral signatures captured across multiple bands.Spectral information provides an important feature representation for each pixel.
  • Motivation: Spectral variability within classes and spectral similarity across classes can produce misclassified pixels and “salt and pepper” noise.Joint spectral-spatial features can alleviate this issue.
  • Related Work: Spectral-spatial CNNs jointly extract spectral and spatial features, often outperforming spectral CNNs, although 3-D convolutions can require substantial time or parameters.Two-branch networks are explored to reduce these costs.
  • Motivation: Different spectral bands and spatial positions have different discriminative abilities, motivating attention mechanisms that refine CNN representations.The paper applies attention to hyperspectral spectral-spatial CNNs.
  • Contribution: Lightweight attention modules are designed for limited training samples using small convolutional operators and output layers that guide module learning.Global average pooling reduces spatial effects in the spectral module, while a 1 × 1 convolution reduces spatial-module channels to one.
  • Contribution: The proposed two-branch network uses spectral and spatial attention modules, then fuses their classification results through adaptive weighted summation.The modules focus on discriminative channels or spatial positions while suppressing unnecessary ones.

A. Framework of the Proposed Model

The proposed framework processes a pixel-centered hyperspectral cube through parallel spectral and spatial attention sub-networks. Their class-probability outputs are combined using adaptively learned weights.

  • Framework: A small cube centered on each pixel is fed simultaneously into spectral and spatial attention sub-networks.The two branches produce separate classification outputs.
  • Adaptive Fusion: The final output is O = α × Ospe + β × Ospa subject to α + β = 1.Ospe and Ospa are the spectral and spatial branch outputs, respectively.
  • Adaptive Fusion: The weighting parameters α and β are learned adaptively during whole-network optimization, and each element of O is a category probability.This fusion combines the two branches at the classification stage.

B. The Structure of CNNs

Both attention sub-networks use a shallow CNN backbone suited to limited hyperspectral training data. Convolution, normalization, nonlinear activation, and pooling progressively transform the input cube.

  • CNN Backbone: Each spectral and spatial sub-network uses three sequential convolutional layers because hyperspectral datasets provide limited training samples.The layers are followed by batch normalization and ReLU activation.
  • CNN Backbone: Max-pooling before the second and third convolutions reduces data variance and computational complexity.This design keeps the backbone shallow while progressively processing features.
  • CNN Backbone: The convolutional kernels are 3 × 3, with channel counts increasing from 32 to 64 to 128 across the three layers.These settings define the backbone’s layer widths and receptive-field operations.
  • Convolution: A convolutional feature map is formed by convolving preceding feature maps with learned kernels, adding biases, and applying ReLU.Padding preserves the spatial size between successive feature maps.
  • Convolution: The convolution operator learns spatial features, while summation aggregates spectral features across input or preceding feature maps.This separates the roles of spatial filtering and spectral aggregation in the CNN.

C. Attention Modules

The model uses complementary spectral and spatial attention modules to recalibrate intermediate feature maps. Spectral attention weights channels, while spatial attention weights semantic positions before later processing and output prediction.

  • Spectral Attention: The spectral attention module uses inter-channel relationships to focus the spectral sub-network on discriminative channels.It first squeezes spatial dimensions with global average pooling, then generates a channel attention map using two 1-D convolutions.
  • Spatial Attention: Spatial attention recalibrates feature maps so the spatial sub-network attends more to semantic positions.The refined map is subsequently processed by the next convolutional layer and output branch.
  • Spectral Attention: The spectral attention map is applied by element-wise multiplication to the original feature map, with attention values copied across spatial positions.The refined map feeds the next convolutional layer and an output branch.
  • Spectral Attention: The spectral module’s output branch provides supervised information and regularization for learning discriminative refined feature maps.The branch contains global max-pooling and a fully connected layer.
  • Spatial Attention: The spatial attention module aggregates channels with a 1 × 1 convolution, then uses two 2-D convolutions to produce a spatial attention map.It exploits inter-spatial relationships in intermediate feature maps.
  • Module Organization: Fig. 2 contrasts the spectral attention module in (a) with the spatial attention module in (b).The distinction is channel-oriented versus position-oriented attention.

D. Network Training

The proposed network is trained in two stages: independent sub-network pre-training followed by joint fine-tuning after adding the weighted summation layer.

  • D. Network Training: Training first pre-trains the two sub-networks independently, then fine-tunes the complete network after adding the weighted summation layer.
  • D. Network Training: The loss sums weighted losses from attention-module outputs across training pixels using cross-entropy.N denotes the total number of training pixels, and L(·, ·) denotes the loss function.
  • D. Network Training: Deeper convolutional layers receive larger empirical loss weights: γ1 = 0.01, γ2 = 0.1, and γ3 = 1.The weights reflect the expectation that deeper layers capture more discriminative features.
  • D. Network Training: During fine-tuning, the combined output is recalculated and the updated loss is optimized using gradient descent.
  • D. Network Training: Table I reports Houston 2013 pixel distributions, including the proportion of training pixels among all available pixels.

A. Data Description and Experimental Setup

Experiments use three hyperspectral datasets and evaluate model components and CNN-related baselines under a PyTorch implementation on specified hardware.

  • A. Data Description and Experimental Setup: The experiments use Houston 2013, Houston 2018, and HyRANK hyperspectral datasets.Houston 2013 has spatial size 349 × 1905, 144 spectral bands, and 15 land-cover classes.
  • A. Data Description and Experimental Setup: Houston 2013 visualization includes a false-color image, training-data visualization, and test-data visualization.
  • A. Data Description and Experimental Setup: The Houston 2018 and HyRANK datasets have pixel-distribution tables accompanying their experimental descriptions.
  • A. Data Description and Experimental Setup: Houston 2018 visualization includes a false-color image, training-data visualization, and test-data visualization.
  • A. Data Description and Experimental Setup: HyRANK visualization shows false-color images and available pixel maps for the Dioni and Loukia data.
  • A. Data Description and Experimental Setup: Experiments assess attention-module effects and compare the proposed model with state-of-the-art CNN-related models.Models are implemented in PyTorch using a computer with 32GB RAM and a GTX TITAN X graphics card.

B. Model Analysis

The model analysis evaluates attention placement, branch integration, and adaptive fusion parameters. Attention improves classification over a plain CNN, combining spectral and spatial branches yields further gains, and performance depends on where and how many modules are used.

  • Effects of different components: SpeAtt and SpaAtt outperform Plain on Houston 2013, Houston 2018, and HyRANK, while SSAtt improves further by combining both branches.The comparison uses OA, AA, and Kappa; the authors attribute SSAtt’s gain to complementary information from the two branches.
  • Effects of different components: SpeAtt and SpaAtt achieve similar performance in most cases, and each attention branch performs better than the no-attention baseline.Plain contains three convolutional layers and an output layer without attention modules.
  • Effects of attention numbers: For SpeAtt, placing attention in the second convolutional layer produces the highest OA because it contains more discriminative features than the first layer while retaining more improvement space than the third.The first layer is less discriminative, whereas the third layer has a smaller improvement space than the second.
  • Effects of attention numbers: SpeAtt2 achieves higher OAs than corresponding SpeAtt1 models, and SpeAtt3 outperforms SpeAtt2 as more convolutional layers receive attention.The results support progressively refining additional layers in the spectral attention sub-network.
  • Effects of attention numbers: SpaAtt3 performs best because every convolutional layer is refined by spatial attention, while SpaAtt2 exceeds corresponding SpaAtt1 configurations.The reported comparison covers different numbers and placements of spatial attention modules.
  • Analysis on α and β: The optimal α and β values vary across datasets, so the authors favor adaptive optimization over empirically fixing them.The variation is associated with differences in spatial and spectral resolutions; Houston 2013 and HyRANK contain more than 100 spectral bands.

C. Model Comparison

SSAtt outperforms the compared CNN-related models across three hyperspectral datasets while reducing computation time relative to SSRN. Qualitative maps are more reasonable, though cube-based inputs can misclassify object boundaries.

  • Quantitative comparisons: SSAtt improves OA, AA, Kappa, and F1 scores over SSRN on both Houston 2018 and HyRANK.These results are reported as validating the effectiveness of the proposed model.
  • Qualitative comparisons: SSAtt produces more reasonable classification maps than the comparison models on Houston 2013, Houston 2018, and HyRANK.The maps are compared qualitatively with the ground-truth maps.
  • Qualitative comparisons: Boundary pixels between objects are easily misclassified because CNN inputs are cubes around each pixel, causing over-smoothed classification maps.The limitation is observed across the CNN-related models' qualitative maps.
  • Computation time: 3DCNN training exceeds 10,000 seconds and MSDNSA training takes about 5,000 seconds on Houston 2013, whereas other models take hundreds of seconds.The higher costs are attributed to 3-D convolutional operations in spectral and spatial domains.
  • Computation time: SSAtt takes less time to train and test than SSRN on all three datasets.SSRN is described as offering the best balance among the seven compared models before comparison with SSAtt.

IV. CONCLUSIONS

The paper proposes an attention-aided CNN with separate spectral and spatial attention sub-networks for hyperspectral image classification. Experiments report improved performance for the individual attention sub-networks and their integrated model, with SSAtt achieving the best reported classification scores and a favorable computation-time balance.

  • Conclusions: The paper proposes an attention-aided CNN method for hyperspectral image classification.The method is designed for spectral-spatial classification.
  • Conclusions: Spectral and spatial attention modules are incorporated into separate sub-networks to focus on discriminative information in their respective domains.The integrated method fuses the complementary information from the two sub-networks.
  • Conclusions: Both attention sub-networks outperform the original CNN, and their integrated model further improves performance.The conclusion attributes the improvement to the aid of attention modules.
  • Conclusions: SSAtt achieves the best OA, AA, Kappa, and F1 scores among the compared state-of-the-art models while balancing classification performance and computation time.This is the paper's reported conclusion from the comparative experiments.
Loading 2005.11977v2…