PaperlayerView exact revision on arXiv

Source-linked AI summary

Spectral Response Function Guided Deep Optimization-driven Network for Spectral Super-resolution

Jiang He, Jie Li, Qiangqiang Yuan, Huanfeng Shen, Liangpei Zhang

arXiv:2011.09701v2eess.IVcs.CV

TL;DR

HSIs offer rich spectral information but often lack spatial detail, while high-resolution MSIs have too few bands to represent full spectral characteristics. This paper unfolds a variational model into an optimization-driven, SRF-guided CNN with spatial-spectral priors. HSRnet reports simultaneous spatial and spectral performance gains, and improved remote-sensing classification results.

Abstract

from arXiv · show

Hyperspectral images are crucial for many research works. Spectral super-resolution (SSR) is a method used to obtain high spatial resolution (HR) hyperspectral images from HR multispectral images. Traditional SSR methods include model-driven algorithms and deep learning. By unfolding a variational method, this paper proposes an optimization-driven convolutional neural network (CNN) with a deep spatial-spectral prior, resulting in physically interpretable networks. Unlike the fully data-driven CNN, auxiliary spectral response function (SRF) is utilized to guide CNNs to group the bands with spectral relevance. In addition, the channel attention module (CAM) and reformulated spectral angle mapper loss function are applied to achieve an effective reconstruction model. Finally, experiments on two types of datasets, including natural and remote sensing images, demonstrate the spectral enhancement effect of the proposed method. And the classification results on the remote sensing dataset also verified the validity of the information enhanced by the proposed method.

I. INTRODUCTION

Hyperspectral images provide rich spectral information but often lack spatial detail, while high-resolution multispectral images have too few bands to fully represent spectral characteristics. The paper proposes an end-to-end optimization-driven CNN that uses spatial-spectral priors and SRF-guided band grouping for spectral super-resolution.

I. INTRODUCTION

  • HSIs often lack fine spatial resolution, while HR MSIs capture spatial detail with too few spectral bands to fully represent objects.Earlier HSI-only reconstruction methods show poor spatial enhancement when the LR-to-HR ratio is large.Source: §I · p. 1
  • HR MSI-HSI fusion can improve spatial detail, but matching images may be unavailable, and multisensor registration and preprocessing are difficult.These difficulties can affect algorithm accuracy and performance.Source: §I · pp. 1–2
  • Deep learning methods can recover spectral information, but are described as data-driven black boxes, while prior CNN-regularized optimization separates learning and optimization into two stages.The paper positions its end-to-end optimization-driven CNN as combining data-driven learning with an optimization algorithm.Source: §I · p. 2

II. PROPOSED METHOD

  • HSRnet shows superior performance in both spatial and spectral evaluation, combining spatial fidelity with spectral preservation.The introduction identifies an end-to-end optimization-driven CNN with a deep spatial-spectral prior and SRF-guided grouping as the proposed approach.Source: §I · p. 2, §II · p. 2
  • The proposed network uses channel attention to learn parameters, SRF to group spectrally relevant bands, and spatial-spectral layers with a reformulated joint loss.The paper describes experiments across five natural and remote-sensing datasets.Source: §I · p. 2

A. Model Formulation

The model formulates spectral super-resolution as an underdetermined reconstruction problem constrained by spectral degradation and an HSI prior. Variable splitting and gradient-based updates are unfolded into a CNN that repeatedly combines data fidelity and learned spatial-spectral information.

A. Model Formulation

  • The degradation model relates the observed MSI to the HSI through a transformation matrix linked to the sensor SRF, making reconstruction from lower-dimensional data underdetermined.The matrix is described as approximately estimable by methods including Hysure and RWL1-SF.Source: A. Model Formulation · pp. 2–3
  • The optimization objective combines data fidelity to the degradation model with an HSI-prior regularizer, then uses variable splitting to separate the terms.The split objective introduces an auxiliary H and a penalty parameter μ that can vary across iterations.Source: A. Model Formulation · p. 3
  • A gradient-descent update for X combines the transformed current estimate, the observed MSI, and the auxiliary variable H.The update uses the degradation matrix Φ and step size ε.Source: A. Model Formulation · p. 3
  • The HSI prior constrains spatial features and spectral properties, and a spatial-spectral network learns this nonlinear prior to update intermediate reconstructions.The unfolded CNN replaces the original alternating updates with a unified end-to-end update whose layers include transformation and initial-restoration blocks.Source: A. Model Formulation · p. 3

B. SRF-Guided Initial Restoration

The initial restoration network uses SRF information to group spectrally related bands and reconstruct an initial HSI estimate. Its grouped operators are intended to avoid distortion from mixing channels with excessive spectral differences.

B. SRF-Guided Initial Restoration

  • SRF-guided initial restoration groups bands by spectral radiation characteristics and reconstructs the initial SSR result with different operators.The paper presents this as an auxiliary physical operation in the initial restoration network.Source: B. SRF-Guided Initial Restoration · p. 4
  • A two-layer CNN extracts spectral features, then feeds them to SRF-guided convolutional layers grouped according to spectral relevance.The feature input is formed from spectral gradients of the RGB/MS image with dimension W×H×(2c−1).Source: B. SRF-Guided Initial Restoration · p. 4
  • Grouping channels with high spectral relevance under shared convolution operators is intended to avoid reconstruction distortion from irrelevant spectral information.For CAVE, the described ranges are blue-only, blue-and-green, and green-and-red coverage.Source: B. SRF-Guided Initial Restoration · p. 4

C. Deep Spatial-Spectral Prior

HSRnet models the HSI prior with a spatial-spectral network, then unfolds iterative optimization into a multistage network. The design combines spatial and spectral feature extraction with repeated updates to the intermediate reconstruction.

C. Deep Spatial-Spectral Prior

  • The spatial-spectral network uses serial spatial- and spectral-feature subnetworks to model the HSI prior.The paper says this prior implicitly constrains the solution space for SSR.Source: C. Deep Spatial-Spectral Prior · p. 4
  • A 3 × 3 convolution extracts spatial features, followed by feature selection and a 1 × 1 convolution for pixelwise spectral fine-tuning.A skip connection adds the spatial-network input to its output.Source: C. Deep Spatial-Spectral Prior · p. 4

D. Optimization Stages in HSRnet

  • Unfolding the optimization yields multiple network stages that perform the iterative update in a deep-learning manner.Each update combines a transformed intermediate estimate, the initial restoration, and the spatial-spectral prior.Source: D. Optimization Stages in HSRnet · p. 5
  • The initial restoration network maps the input RGB/MS image to an initial HSI estimate before iterative optimization.The optimization is trained in a feed-forward manner to learn the HSI prior and match the spectral degradation model.Source: D. Optimization Stages in HSRnet · p. 5

E. Attention-Based Parametric Self-learning

Channel attention adapts optimization parameters across iterations, while a reformulated spectral-angle loss supports joint spatial and spectral reconstruction. The loss is expressed in tensor form to allow parallel GPU computation.

E. Attention-Based Parametric Self-learning

  • Channel-attention blocks adapt the step size and balance parameter at each iteration, emphasizing bands needing more optimization through inter-channel relationships.The attention block uses pooling, convolutional layers, and sigmoid-activated channel weights.Source: E. Attention-Based Parametric… · p. 5
  • Introducing channel attention lets HSRnet learn iteration-specific parameter vectors instead of fixed values.The paper links this to adaptive weighting during spectral optimization.Source: E. Attention-Based Parametric… · p. 5

F. Fast Joint Spatial-Spectral Loss

III. EXPERIMENTAL RESULTS

The evaluation compares HSRnet with dictionary-based and deep-learning SSR methods, using spatial- and spectral-fidelity metrics. Training uses nine optimization stages and Adam.

1) Comparison Methods

  • The comparison includes Arad, A+, DenseUnet, CanNet, HSCNN+, and sRCNN, spanning dictionary and deep-learning SSR methods.The authors describe these as current state-of-the-art methods and reproduce A+ and Arad using Wu et al.'s program.Source: 1) Comparison Methods · p. 6

2) Quantitative Metrics

  • CC, PSNR, and SSIM measure spatial fidelity, while SAM measures spectral fidelity, with smaller SAM indicating better preservation.The spatial metrics are computed per channel and averaged across spectral bands.Source: 2) Quantitative Metrics · p. 6

3) Implementation Detail

  • The selected configuration uses nine optimization stages, a 0.001 learning rate, and Adam for training.The loss trade-off parameter is 0.0001; training used PyTorch on Windows 10 with one Nvidia RTX 2080 GPU and 16 GB RAM.Source: 3) Implementation Detail · p. 6

4) Experimental Dataset

Experiments use CAVE and remote-sensing datasets, including a simulated Sen2OHS dataset. The evaluation also includes three additional simulated remote-sensing datasets with different spatial and spectral characteristics.

4) Experimental Dataset

a) CAVE dataset

  • CAVE contains 32 scenes of 512 × 512 images with 31 HSI bands spanning 400–700 nm, alongside corresponding RGB images.Twenty-six scenes train the model using 128 × 128 patches, with flipping and rotation for eightfold augmentation; six scenes are held out for testing.Source: a) CAVE dataset · p. 6

b) Remote Sensing dataset

  • Sen2OHS uses four Chinese Orbita hyperspectral-satellite images, with HSIs sampled to 32 bands across 400–1000 nm.Sentinel-2 multispectral images are simulated from OHS HSIs; 6,000 OHS images train the model, and test images come from Xiongan New Area.Source: b) Remote Sensing dataset · pp. 6–7
  • Three further simulated datasets—Xiongan, Washington DC Mall, and Chikusei—test restoration across differing channel scales and spatial resolutions.Their spectral channels are downsampled to match OHS and Sentinel-2 using Hysure.Source: b) Remote Sensing dataset · p. 7, Figure 10 · p. 7

B. Results on CAVE Dataset

On CAVE, HSRnet shows strong spatial and spectral performance, including substantial gains over dictionary-learning methods. The supplied remote-sensing passages also describe visual comparisons on CAVE’s “Lemon Slices” image.

1) Quantitative and visual results

  • 63.57% average SAM reduction and 22.94% average PSNR increase versus dictionary-learning methods show HSRnet’s gains in spectral and spatial evaluation.The section also reports HSRnet as superior in spatial and spectral evaluation simultaneously across four image-quality indexes.Source: 1) Quantitative and visual results · p. 7
  • HSCNN+ and sRCNN show good spatial fidelity but worse spectral evaluation than HSRnet.The absolute-difference images compare Arad, A+, DenseUnet, CanNet, HSCNN+, sRCNN, and HSRnet across six wavelengths.Source: 1) Quantitative and visual results · p. 7, Figure 11 · p. 8

C. Results on Remote Sensing Dataset

  • The CAVE “Lemon Slices” reflectance comparisons show spectra at real and fake lemon-slice locations.The accompanying difference figure displays reconstructed-image errors at 450, 500, 550, 600, 650, and 700 nm.Source: Figure 11 · p. 8, Figure 12 · p. 9

a) Sen2OHS Dataset

On Sen2OHS, HSRnet improves reported spatial metrics over Arad and A+, while showing advantages in both spatial fidelity and spectral preservation against deep-learning methods. Visual analysis also identifies bands where limited or unmatched spectral information constrains reconstruction.

a) Sen2OHS Dataset

  • 16.18% higher average CC, 23.63% higher average PSNR, and 48.85% higher average SSIM versus Arad and A+ are reported for HSRnet.The reported SAM improvement over Arad and A+ is beyond 33.75%.Source: a) Sen2OHS Dataset · p. 9
  • Compared with four deep-learning methods, HSRnet shows an advantage in both spatial fidelity and spectral preservation.sRCNN has a tiny advantage over HSRnet, attributed to spectra-by-spectra band optimization with huge computation.Source: a) Sen2OHS Dataset · p. 9
  • HSRnet shows lower error and less detail loss across the “Urban” image, while its visual results are more balanced across bands than sRCNN’s.The comparison covers bands 5, 10, 15, 20, 25, and 30; deep-learning methods struggle with buildings of varying geometry, while dictionary methods struggle with regular farmlands of diverse brightness.Source: a) Sen2OHS Dataset · pp. 9–10, Figure 13 · p. 10
  • All methods perform poorly on bands 9–21, where Sentinel-2’s spectral response functions do not cover several OHS bands.Bands 9–12, 16–20, and 30–32 of OHS are not covered by Sentinel-2 SRF; with SRF guidance, HSRnet shows lower average errors as fluctuations rather than a surge when related spectral information is deficient.Source: a) Sen2OHS Dataset · pp. 9–10, Figure 14 · p. 11

b) HR Simulation Datasets

Across four remote-sensing scales, HSRnet is reported to perform strongly in spectral fidelity, and on real Sentinel-2 data its reconstructed HSI improves classification results. The evaluation includes HR simulation datasets and a 10 m real-data classification comparison.

b) HR Simulation Datasets

  • The HR simulations use Xiongan, Washington DC Mall, and Chikusei to compare methods on datasets with finer spatial details.The experiments report that spectral super-resolution is easier on these datasets, with purer spectral information in HR training samples.Source: b) HR Simulation Datasets · p. 11
  • Across remote-sensing datasets at four scales, HSRnet shows stability and superiority over other deep-learning algorithms in spectral fidelity.On HR simulation datasets, HSRnet also performs better than sRCNN with lower computational complexity.Source: b) HR Simulation Datasets · p. 11

2) Classification results on real data

  • The real-data comparison uses a 512 × 512 image south of Nantes, France, with HSI displayed using bands 27, 13, and 8.The real Sentinel-2 image is shown with bands 8, 4, and 3, with vegetation displayed in red.Source: 2) Classification results on real data · p. 11
  • On real Sentinel-2 data, classification results improve in OA and Kappa, supporting the reported reliability of the reconstructed spectral information.The classification uses 16 classes and SVM; reconstructed-HSI results show less discrete objects and help combine adjacent similar objects.Source: 2) Classification results on real data · p. 11

D. Discussion

The discussion evaluates HSRnet’s optimization stages, channel attention, SRF guidance, and spatial-spectral loss through ablations. The reported findings attribute improvements in reconstruction measures to these design components.

1) Ablation Study

  • Optimization stages alone substantially improve spatial and spectral fidelity over Resnet, and show a slight advantage over DenseUnet.The ablation uses a 19-layer Resnet baseline and identifies optimization stages, CAM, SRF, and SAM Loss as the strategies being assessed.Source: 1) Ablation Study · p. 11
  • Channel-attention-based parametric self-learning improves spatial fidelity and spectral enhancement by adaptively learning parameters for different iterations and bands.The comparison is between HSRnet without SRF and HSRnet without CAM.Source: 1) Ablation Study · p. 11
  • SRF guidance yields small spatial improvements but substantially improves spectral maintenance; guided reconstruction has higher PSNR on both CAVE and remote-sensing datasets.The PSNR comparison is shown in Fig. 16 for CAVE and Sen2OHS.Source: 1) Ablation Study · p. 12, Figure 16 · p. 12
  • Adding SAMLoss improves SAM and spatial-fidelity metrics, with the paper reporting mutual reinforcement between spatial fidelity and spectral preservation.Source: 1) Ablation Study · p. 13

2) Computational Speed Analysis

HSRnet combines optimization-driven physical structure with deep spatial-spectral learning, achieving strong SSR performance with acceptable computational cost while retaining spatial detail. The paper also identifies spectral coverage between input and output bands as an important factor, especially for Sen2OHS.

2) Computational Speed Analysis

  • DenseUnet trains an epoch faster but converges at 200 epochs; HSRnet takes longer per epoch yet converges earlier and uses less total training time.The computational comparison covers parameter count, FLOPs, and training and test time; Fig. 17 shows validation loss.Source: 2) Computational Speed Analysis · pp. 13–14, Table VII · p. 13, Figure 17 · p. 13
  • DenseUnet has approximately twice HSRnet’s parameters and the fewest FLOPs, but its downsampling blurs spatial details that HSRnet preserves.The visual comparison includes CAVE and Sen2OHS results; HSRnet shows mild spatial degradation on Sen2OHS, where OHS-A resolution is slightly coarser than Sentinel-2.Source: 2) Computational Speed Analysis · pp. 13–14, Figure 18 · p. 13
  • HSRnet achieves the best SSR performance with acceptable parameter numbers and computation complexity, while retaining more spatial detail with acceptable test time.It also converges early, resulting in less total training time despite longer training per iteration.Source: 2) Computational Speed Analysis · p. 14

IV. CONCLUSION

  • HSRnet uses SRF-guided spectral band grouping and channel attention to learn weights, embedding physical logic in an end-to-end optimization-driven CNN.The network reconstructs different spectral ranges using spectrally similar bands and a deep spatial-spectral prior.Source: §IV · p. 14
  • Spectral coverage between input and output data affects model performance, especially on Sen2OHS; using differently resolved MS bands to achieve complete coverage is future work.Source: §IV · p. 14

Related papers

Loading 2011.09701v2…