PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Spatial-Spectral Prior for Super-Resolution of Hyperspectral Imagery

Junjun Jiang, He Sun, Xianming Liu, Jiayi Ma

arXiv:2005.08752v2eess.IVcs.CVcs.LGcs.MM

TL;DR

Single hyperspectral image super-resolution remains challenging because spatial and spectral resolution trade off, while existing priors insufficiently capture hyperspectral structure. SSPSR combines a spatial-spectral prior network with group convolution and progressive upsampling, and outperforms competing methods across reported datasets and objective measures.

  • Problem

    Single hyperspectral image super-resolution is challenging because higher spectral resolution can reduce spatial resolution, while manually designed priors insufficiently exploit spatial self-similarity and spectral correlation.

  • Method

    SSPSR uses a spatial-spectral prior network with spatial-spectral blocks, plus shared-parameter group convolution and progressive upsampling for high-dimensional, data-limited hyperspectral reconstruction.

  • Results

    SSPSR significantly outperforms competing methods across reported datasets; its average PSNR exceeds the second-best method by more than 0.30 dB on Chikusei and by 0.3/0.2 dB at ×4/×8 on Pavia Centre.

  • Takeaways & Limitations

    The reported results indicate that SSPSR can recover hyperspectral images with improved objective performance and preserved main structural information compared with competing approaches.

Abstract

from arXiv · show

Recently, single gray/RGB image super-resolution reconstruction task has been extensively studied and made significant progress by leveraging the advanced machine learning techniques based on deep convolutional neural networks (DCNNs). However, there has been limited technical development focusing on single hyperspectral image super-resolution due to the high-dimensional and complex spectral patterns in hyperspectral image. In this paper, we make a step forward by investigating how to adapt state-of-the-art residual learning based single gray/RGB image super-resolution approaches for computationally efficient single hyperspectral image super-resolution, referred as SSPSR. Specifically, we introduce a spatial-spectral prior network (SSPN) to fully exploit the spatial information and the correlation between the spectra of the hyperspectral data. Considering that the hyperspectral training samples are scarce and the spectral dimension of hyperspectral image data is very high, it is nontrivial to train a stable and effective deep network. Therefore, a group convolution (with shared network parameters) and progressive upsampling framework is proposed. This will not only alleviate the difficulty in feature extraction due to high-dimension of the hyperspectral data, but also make the training process more stable. To exploit the spatial and spectral prior, we design a spatial-spectral block (SSB), which consists of a spatial residual module and a spectral attention residual module. Experimental results on some hyperspectral images demonstrate that the proposed SSPSR method enhances the details of the recovered high-resolution hyperspectral images, and outperforms state-of-the-arts. The source code is available at \url{https://github.com/junjun-jiang/SSPSR

I. INTRODUCTION

Hyperspectral super-resolution is challenging because high spectral dimensionality, limited training data, and spatial–spectral tradeoffs complicate reliable reconstruction. SSPSR addresses these constraints with grouped, progressively upsampled processing and a spatial-spectral prior network.

  • Motivation: Hyperspectral imaging combines spatial imaging with spectral detection across the electromagnetic spectrum.Each pixel is dispersed into dozens or hundreds of narrow spectral bands.
  • Motivation: High spectral resolution can reduce spatial resolution when imaging conditions must preserve signal-to-noise ratio.The imaging process therefore faces an inherent spatial–spectral resolution tradeoff.
  • Research gap: Single hyperspectral super-resolution has received limited attention because spectral complexity and absent auxiliary information make the problem difficult.Earlier approaches used sparse, dictionary-learning, or low-rank priors to exploit spectral correlations.
  • Research gap: Limited hyperspectral samples and high-dimensional spectral bands make computationally efficient, effective deep-network training nontrivial.Different cameras can also produce datasets with different spectral-band counts.
  • Proposed approach: SSPSR uses shared-parameter group convolution and progressive upsampling to reduce model size and stabilize training under small-data conditions.Its spatial-spectral prior network cascades spatial-spectral blocks containing spatial residual and spectral attention residual modules.
  • Proposed approach: The network divides the low-resolution hyperspectral input into overlapping spectral groups, processes them in branches, then concatenates their upsampled features for global extraction.Branch parameter sharing avoids heavy computational cost and simplifies optimization.

II. RELATED WORK

Related work spans fusion-based and single hyperspectral super-resolution, alongside deep methods for gray/RGB super-resolution. Single hyperspectral approaches remain constrained by limited data and the difficulty of exploiting spatial and spectral correlations.

  • Fusion-based hyperspectral super-resolution: Hyperspectral super-resolution is commonly divided into fusion-based methods using auxiliary imagery and single-image methods without auxiliary information.Fusion methods typically combine low-resolution hyperspectral data with higher-resolution multispectral or panchromatic data.
  • Single hyperspectral super-resolution: Single hyperspectral super-resolution methods include acquisition-model reconstruction, low-rank and group-sparse constraints, sparse representations, and dictionary learning.These methods address settings without a co-registered auxiliary image.
  • Single hyperspectral super-resolution: Earlier single-image methods often require complex, time-consuming test-time optimization, while manually designed priors may inadequately characterize hyperspectral data.The relevant data properties include spatial information and correlation among spectral bands.
  • Single gray/RGB image super-resolution: Deep convolutional networks achieved strong results in single gray/RGB image super-resolution by learning restoration mappings from paired original and corrupted images.This line of work includes SRCNN, VDSR, and DRCN.

III. THE PROPOSED SSPSR METHOD

SSPSR predicts a high-resolution hyperspectral image from a low-resolution input using grouped branch networks and a global network. The architecture extracts features, progressively upsamples them, and reconstructs the output.

  • SSPSR maps a low-resolution hyperspectral image ILR to a high-resolution output ISR through an end-to-end reconstruction network.
  • 1) The Branch Network:: The input is divided into spectral groups, with neighboring groups allowed to overlap, reducing each branch’s feature dimensionality while retaining neighboring-band correlations.
  • 1) The Branch Network:: Each branch applies convolutional shallow-feature extraction, SSPN-based deep extraction, upsampling, and a reconstruction layer before global processing.
  • 1) The Branch Network:: PixelShuffle performs branch upsampling, while an additional convolution reduces feature channels to the spectral number of the input group.
  • 1) The Branch Network:: The spatial-spectral block combines a spatial residual module with a spectral attention residual module.
  • 1) The Branch Network:: The branch and global networks share the same overall structure, supporting progressive super-resolution reconstruction before the final output stage.

2) The Global Network:

The global network integrates features from all spectral branches, processes their combined representation, and progressively upsamples it to the target image size. Bicubic-upsampled input features provide a residual-learning pathway.

  • 2) The Global Network:: Features from all branch networks are concatenated in spectral order, with overlapping bands integrated by averaging their feature values.
  • 2) The Global Network:: A convolution extracts shallow features from the concatenated branch representation before the global SSPN extracts global spatial-spectral features.
  • 2) The Global Network:: The global upsampling module enlarges the extracted features to the target size as part of the progressive reconstruction strategy.
  • 2) The Global Network:: The reconstruction combines upscaled spatial-spectral features with features extracted from the bicubic-upsampled low-resolution input for residual learning.

B. Spatial-Spectral Prior Network (SSPN)

SSPN is designed to exploit spatial structure and spectral dependencies in hyperspectral images. It cascades spatial-spectral blocks and uses residual connections plus spectral attention to improve feature representation.

  • Hyperspectral restoration requires priors that capture spatial non-local self-similarity and high correlation across spectra, which manually designed constraints may inadequately represent.
  • SSPN cascades R spatial-spectral blocks, each combining spatial residual processing with spectral attention residual processing.
  • A long skip connection passes low-frequency features directly to the end, allowing the residual body to focus on high-frequency information and supporting stable training.
  • The spatial residual module uses standard residual blocks with 3×3 convolutions to extract spatial features.
  • Spectral processing uses learnable 1×1-filter combinations to embed dependencies among neighboring spectral bands.
  • Channel attention globally pools feature maps and uses gated nonlinear interactions to produce channel scaling coefficients that reweight spectral features.

C. Loss Function

The loss design combines l1 reconstruction with spatial-spectral total variation to preserve credible spatial and spectral information. Experiments report that l1 is more suitable than alternative losses, with SSTV providing a slight improvement.

  • l2 and l1 losses are preferred over perceptual and adversarial losses because the latter may restore details absent from the original remote-sensing images.
  • The spatial-spectral total variation constraint jointly imposes spatial and spectral smoothness on the reconstruction.
  • The final objective is a weighted sum of l1 loss and SSTV, with α set to 1e-3 in the experiments.
  • On four Chikusei testing images at upsampling factor 4, l1 loss is reported as more suitable because it penalizes small errors and maintains better convergence.
  • Adding the SSTV constraint produces slightly better results than using l1 loss alone.

D. Implementation Details

The implementation uses grouped spectral processing, shared parameters, and specified training and architectural settings for SSPSR.

  • Training settings: SSPSR models are trained for scale factors 4 and 8 using PyTorch, ADAM, an initial learning rate of 1e-4, and batch size 32.The learning rate decays by a factor of 10 at 30 epochs, and stable performance is reached after about 40 epochs.
  • Spectral grouping: The default grouping configuration uses 8 spectral bands per group with an overlap of 2 between neighboring groups.A fallback strategy selects the last 8 bands when the final group contains fewer bands.
  • Architecture: The network uses three spatial-spectral blocks, 3×3 convolutions, 1×1 spectral residual kernels, and zero-padding to preserve feature-map size.Most convolutional layers use 256 filters, except the channel-downscaling reconstruction layer.
  • Evaluation setup: The study evaluates variants through an ablation table covering alternative SSPSR components and six picture-quality indices.The reported implementation abbreviations include grouping strategy, progressive upsampling, parameter sharing, and spectral attention.

IV. EXPERIMENTS AND RESULTS

Experiments evaluate SSPSR against established methods on three public hyperspectral datasets using six quantitative picture-quality indices.

  • Datasets and comparisons: The evaluation uses Chikusei, Pavia Center, and CAVE datasets and compares SSPSR with eight deep single gray/RGB super-resolution methods.The comparison methods include VDSR, EDSR, RCAN, and SAN, alongside additional approaches listed in the experimental section.
  • Evaluation metrics: Performance is measured with CC, SAM, RMSE, ERGAS, PSNR, and SSIM.For hyperspectral PSNR and SSIM, the reported values are means across all spectral bands.

A. Ablation Studies

Ablations examine grouping, overlap, progressive upsampling, parameter sharing, and spectral attention, showing trade-offs between reconstruction quality and computational cost.

  • Grouping Strategy: Grouping improves SSPSR over the no-grouping variant by +0.17 dB at ×4 and +0.45 dB at ×8.The grouping strategy also produces considerable gains on other objective indicators.
  • Grouping Strategy: Band-wise grouping performs better than treating all spectra as one group, but its many branches substantially increase computational overhead.The comparison links more branches with more calculations, while the proposed grouped configuration balances performance and cost.
  • Grouping Strategy: Increasing overlap from 0 to 6 improves performance but also increases the model’s calculation amount, while parameter sharing keeps parameter counts unchanged.The selected setting is p = 8 and o = 2 to balance parameters, FLOPs, and objective results.
  • Visual comparisons: For a Chikusei composite at d = 4, SSPSR is compared visually and by PSNR and SSIM against the ground truth and eight competing methods.The figure uses bands 70-100-36 as RGB and reports reconstructed composite-image metrics.
  • Progressive Upsampling: Progressive upsampling improves all six indices over its ablated variant, with especially important benefits at the larger ×8 scale.The framework decomposes the super-resolution task into easier stages.
  • Parameter Sharing: Parameter sharing greatly reduces computational complexity while maintaining representation ability and outperforming unshared parameters across all six indices at d = 4 and d = 8.Shared branches can use training samples from different groups with one set of network parameters.
  • Spectral Attention: Spectral attention yields larger relative gains for SAM than PSNR: 2.2% versus 0.43% at d = 4, and 11% versus 1.3% at the reported second scale.The ablation associates spectral attention more strongly with spectral-feature representation than spatial reconstruction.

B. Results on Chikusei Dataset

On Chikusei, SSPSR outperforms competing methods quantitatively and recovers both fine-grained textures and coarse-grained structures more effectively at ×4 and ×8 upsampling.

  • The Chikusei data come from a 128-band Headwall Hyperspec-VNIR-C acquisition covering an urban area in Ibaraki, Japan.The dataset was captured on 29 July 2014.
  • SSPSR significantly outperforms the competing algorithms across all objective indexes, with average PSNR more than 0.30 dB above the second-best method.The comparison averages performance over four testing images.
  • GDRRN achieves the second-best SAM results, consistent with its group strategy for exploiting spectral information.
  • RCAN and SAN slightly outperform EDSR, which the authors associate with their channel-attention mechanisms for capturing spectral features.
  • SSPSR better recovers finer-grained textures and coarser-grained structures than other methods at upsampling factors d = 4 and d = 8.The visual comparisons use reconstructed composite images from the Chikusei test image.

C. Results on Pavia Centre Dataset

On Pavia Centre, SSPSR achieves the strongest average objective performance and maintains structural information more effectively than competing methods at ×4 and ×8 upsampling.

  • The Pavia Centre scene contains 102 spectral bands and a meaningful region of 1096×715 pixels after regions without information are removed.
  • SSPSR’s average PSNR is 0.3 dB higher at ×4 and 0.2 dB higher at ×8 than the second-best method.The averages cover six PQIs over four testing images.
  • SSPSR significantly outperforms competing methods on almost all objective evaluation indexes.
  • EDSR, 3DCNN, and GDRRN produce very blurry results, whereas RCAN and SAN appear to introduce noise in the visual comparisons.
  • SSPSR maintains the main structural information and its error maps show no obvious image-contour information.

D. Results on CAVE dataset

On CAVE natural-scene images, SSPSR outperforms the competing methods across reported evaluations and generally provides the strongest spectral-band reconstruction fidelity.

  • The CAVE database contains 32 everyday-object scenes with 31 spectral bands spanning 400nm to 700nm at 10nm intervals.
  • SSPSR outperforms all competing methods on the CAVE dataset across the reported evaluation results.The comparisons use six PQIs and different upsampling factors.
  • At d = 4/8, SSPSR’s average PSNR advantage is 0.3/0.4 dB and its SSIM advantage is 0.002/0.012 over the second-best method.
  • SSPSR performs best in most spectral-band cases, while 3DCNN and GDRRN show unstable performance across different bands.
  • SSPSR and RCAN achieve the best reconstruction fidelity for details such as checkerboard edges and the contours of dog ears and apples.
Loading 2005.08752v2…