PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning A 3D-CNN and Transformer Prior for Hyperspectral Image Super-Resolution

Qing Ma, Junjun Jiang, Xianming Liu, Jiayi Ma

arXiv:2111.13923v1eess.IVcs.CV

TL;DR

HSISR seeks to recover spatial detail by fusing a low-resolution HSI with a high-resolution MSI, while hand-crafted and CNN priors have stated limitations. 3DT-Net unfolds proximal-gradient optimization and combines Transformer spatial modeling with 3D-CNN spatio-spectral modeling. The paper reports state-of-the-art quantitative and visual results on two simulated datasets and one real dataset.

  • Problem

    HSISR uses priors to constrain an ill-posed reconstruction problem, but hand-crafted priors may not capture HSI properties and CNNs have limited spatial or spectral modeling.

    Source: §1 · pp. 1–2, §3.1 · p. 3

  • Method

    3DT-Net unfolds proximal-gradient iterations, using Transformer layers to learn spatial priors and 3D convolutions to model spatio-spectral correlations.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 5

  • Results

    Evaluations on two public simulated datasets and one real dataset report state-of-the-art quantitative results and visual quality for 3DT-Net.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The proposed prior-modeling method is described as general for low-level restoration and suitable for high-dimensional data such as hyperspectral images, video, and light-field.

    Source: §5 · p. 8

    3DT-Net is a little more computationally taxing than convolutional residual-block-based networks.

    Source: §5 · p. 8

Abstract

from arXiv · show

To solve the ill-posed problem of hyperspectral image super-resolution (HSISR), an usually method is to use the prior information of the hyperspectral images (HSIs) as a regularization term to constrain the objective function. Model-based methods using hand-crafted priors cannot fully characterize the properties of HSIs. Learning-based methods usually use a convolutional neural network (CNN) to learn the implicit priors of HSIs. However, the learning ability of CNN is limited, it only considers the spatial characteristics of the HSIs and ignores the spectral characteristics, and convolution is not effective for long-range dependency modeling. There is still a lot of room for improvement. In this paper, we propose a novel HSISR method that uses Transformer instead of CNN to learn the prior of HSIs. Specifically, we first use the proximal gradient algorithm to solve the HSISR model, and then use an unfolding network to simulate the iterative solution processes. The self-attention layer of Transformer makes it have the ability of spatial global interaction. In addition, we add 3D-CNN behind the Transformer layers to better explore the spatio-spectral correlation of HSIs. Both quantitative and visual results on two widely used HSI datasets and the real-world dataset demonstrate that the proposed method achieves a considerable gain compared to all the mainstream algorithms including the most competitive conventional methods and the recently proposed deep learning-based methods.

1. Introduction

HSISR combines a low-resolution HSI with a high-resolution MSI to address the HSI’s limited spatial resolution, but hand-crafted priors do not capture all HSI characteristics. 3DT-Net unfolds proximal-gradient iterations and combines Transformer spatial modeling with 3D-CNN spatio-spectral modeling.

1. Introduction

  • HS/MS fusion combines a low-resolution HSI and a high-resolution MSI to produce an image with high spatial and spectral resolution.The HSI’s spatial resolution is usually coarser than RGB imagery, limiting its application value.Source: §1 · p. 1
  • Hand-crafted priors can fail to reflect all HSI characteristics, restricting optimization-based methods’ performance and generalization.Their design also requires substantial expertise.Source: §1 · p. 1
  • Earlier learning-based methods use data-driven priors and specialized attention modules, while unfolding methods connect the observation model with learned priors.Examples include spatial, channel, and joint attention, as well as ResNet- and U-net-based prior modules.Source: §1 · pp. 1–2
  • 3DT-Net unfolds proximal-gradient optimization into a multistage network, using Swin Transformer layers for spatial priors and 3D convolutions for spatio-spectral correlations.The model uses observation models for LR-HSIs and HR-MSIs.Source: §1 · p. 2

2. Related Work

Related work covers manually selected priors and learned priors for HSISR, alongside Transformer approaches for image processing. The proposed direction builds on data-driven prior modeling and Transformer methods.

2.1. Prior Modeling in HSISR

  • Traditional HSISR priors include total variation, sparse coding, spectral unmixing, and self-similarity, but recovery depends on prior suitability and manually tuned balance parameters.Examples include sparse regularization based on scene dictionaries and Bayesian sparse coding.Source: §2.1 · p. 2
  • Deep-learning HS/MS fusion methods learn data-driven priors, including CNN-based methods and ResNet or U-net denoising priors.Some approaches combine an initialized HR-HSI with a learned prior, while others use a CNN denoiser to regularize fusion.Source: §2.1 · p. 2

2.2. Vision Transformer

  • Swin Transformer uses shifted windows in a hierarchical design, supporting high-resolution image processing and long-range dependency modeling.Earlier image-transformer approaches faced computational burdens; patch-based processing also faced high-resolution limitations and quadratic complexity with image size.Source: §2.2 · pp. 2–3

3. Proposed Method

The method formulates HSISR as an optimization problem with a learned data-driven prior, then unfolds proximal-gradient iterations into a network. Its prior network combines Swin Transformer layers with 3D convolutions to model spatial and spectral information.

3.1. Model Formulation

  • The model fuses LR-HSI and HR-MSI through an optimization objective whose regularization term represents prior knowledge, learned here as a data-driven implicit prior.The target is an image with both high spatial and high spectral resolution.Source: §3.1 · p. 3

3.2. Optimization-based Unfolding Network

  • The network unfolds proximal-gradient updates into K stages, shares parameters across stages, and learns the step size η rather than selecting it by hand.It initializes the estimate by bicubic upsampling the LR-HSI and densely connects feature maps between prior modules.Source: §3.2 · p. 4

3.3. 3D-CNN and Transformer Prior Network

  • The prior network combines Swin Transformer layers for long-range modeling with 3D convolutions for spectral priors and spatio-spectral correlation.Figure 2 illustrates the 3DT module architecture.Source: §3.3 · p. 5, Figure 2 · p. 5

3.4. Training Details

  • The network is trained with an L1 loss and Adam (β1 = 0.9, β2 = 0.999, ϵ = 10−8) for 250000 iterations, using a batch size of 8.The learning rate is initialized at 0.0001.Source: §3.4 · p. 5

4. Experiments and Analyses

The experiments evaluate 3DT-Net on simulated CAVE and Harvard data and real WV-2 data, using four image-quality indices and visual comparisons. The reported results include benchmark gains, ablation findings, and qualitative reconstructions.

4. Experiments and Analyses

  • Evaluation compares four traditional and four deep-learning methods using PSNR, SAM, ERGAS, and SSIM; higher PSNR and SSIM and lower SAM and ERGAS indicate better results.The experiments include the CAVE and Harvard simulated datasets and WV-2 real-data experiments.Source: §4 · pp. 5–6, §4.1 · p. 6

4.1. Training Datasets

  • The simulated benchmarks use 32 CAVE images and 50 Harvard images, with 12 and 20 samples reserved for testing, respectively.Both datasets contain 31 visible-spectrum bands; training patches are 64 × 64 × 31 HR-HSIs, paired with 64 × 64 × 3 HR-MSIs and 8 × 8 × 31 LR-HSIs.Source: §4.1 · p. 6

4.2. Ablation Study

  • As unfolded stage count increases, PSNR and SSIM rise but decline when K exceeds 3, so the experiments use K = 3.The authors report that additional iterations may make the network too complex to train to an optimal solution.Source: §4.2 · p. 6, Table 1 · p. 6
  • Replacing the U-net prior with the Transformer prior raises PSNR by 1.05 dB with half the parameters, and adding 3D convolution raises PSNR by another 0.17 dB.The authors attribute the latter gain to better extraction of spatio-spectral correlation priors.Source: Table 2 · p. 6, §4.2 · p. 7

4.3. Experiments with Simulated Data

  • 3DT-Net’s average PSNR is more than 1.0 dB above MoG-DCN on CAVE and 0.4 dB above it on Harvard.The comparison covers four PQIs on each simulated dataset; Tables 3 and 4 report the quantitative comparisons.Source: §4.3 · p. 7, Table 3 · p. 7, Table 4 · p. 7
  • For CAVE and Harvard reconstructions, 3DT-Net is closest to ground truth on CAVE and best at recovering details with the smallest reconstruction error on Harvard.The comparison describes images from CAVE and Harvard reconstructed at 700 nm.Source: §4.3 · pp. 7–8, Figure 3 · p. 7, Figure 4 · p. 8

4.4. Experiments with Real Data

  • On WV-2, visual inspection shows 3DT-Net has a better visual effect than MHF-net and DBIN, whose reconstructions show blur and artifacts.The real-data experiment uses a WV-2 test portion; its ground-truth HR-HSI is unavailable.Source: §4.1 · p. 6, §4.4 · p. 8

5. Conclusions

The paper presents a proximal-gradient-based deep unfolding network, 3DT-Net, using Transformer layers for spatial priors and 3D convolutional layers for spatio-spectral correlation priors. It reports state-of-the-art quantitative results and visual quality on two public simulated datasets and one real dataset, while noting higher computational cost than convolutional residual-block networks.

5. Conclusions

  • 3DT-Net achieves state-of-the-art quantitative results and visual quality on two public simulated datasets and one real dataset.The conclusion reports both quantitative performance and visual quality across all three datasets.Source: §5 · p. 8
  • 3DT-Net uses Transformer layers to exploit spatial priors and 3D convolutional layers to exploit spatio-spectral correlation priors within a deep unfolding network based on the proximal gradient algorithm.The paper says Transformer has long-range modeling capabilities and 3D convolution operates across spatial and channel dimensions.Source: §5 · p. 8
  • The prior modeling method is described as general for low-level image restoration tasks and seamlessly integrable with Plug-and-Play methods, but it is more computationally taxing than convolutional residual-block networks.The authors identify efficient implementations as future work.Source: §5 · p. 8

Related papers

Loading 2111.13923v1…