Source-linked AI summary
Deep Hyperspectral Prior: Denoising, Inpainting, Super-Resolution
Oleksii Sidorov, Jon Yngve Hardeberg
TL;DR
Hyperspectral restoration lacks the large exemplar datasets that enable many CNN methods. This paper extends the deep-prior approach to HSI by optimizing CNN parameters without training data, and reports performance comparable to trained CNNs while outperforming single-image methods.
Problem
Hyperspectral datasets are often too small for accurate learning-based restoration methods, sometimes containing only one image.
Method
The method optimizes a CNN that maps fixed noise to the target image, using the network’s intrinsic prior instead of a learned or manually designed image prior.
Results
The proposed algorithm performs comparably to trained CNNs while outperforming single-image algorithms across the evaluated HSI restoration tasks.
Takeaways & Limitations
Training-free optimization makes the restoration approach applicable when task-specific HSI training data are unavailable.
Abstract
from arXiv · showhide
Deep learning algorithms have demonstrated state-of-the-art performance in various tasks of image restoration. This was made possible through the ability of CNNs to learn from large exemplar sets. However, the latter becomes an issue for hyperspectral image processing where datasets commonly consist of just a few images. In this work, we propose a new approach to denoising, inpainting, and super-resolution of hyperspectral image data using intrinsic properties of a CNN without any training. The performance of the given algorithm is shown to be comparable to the performance of trained networks, while its application is not restricted by the availability of training data. This work is an extension of original "deep prior" algorithm to HSI domain and 3D-convolutional networks.
1. Introduction
The paper addresses hyperspectral restoration when training data are scarce by using a CNN’s intrinsic structure as an image prior. It proposes a training-free HSI restoration algorithm and evaluates a 3D-convolutional implementation alongside related applications.
- Deep prior formulation: Image restoration tasks such as inpainting, noise removal, and super-resolution can be formulated as energy minimization with a task metric and image-prior regularization.The prior may be manually chosen or learned from data in conventional methods.
- Deep prior formulation: The deep-prior approach finds the image prior in CNN parameter space, removing the explicit regularization term and optimizing network weights using only a corrupted image.A CNN maps fixed noise input to the restored image during optimization.
- Motivation: Hyperspectral restoration is important because acquisition complexity increases corruption while available datasets are often limited to fewer than one hundred images or even one image.This constrains the use of accurate learning-based methods for particular HSI tasks.
- Contributions: The paper proposes an efficient hyperspectral image-restoration algorithm based on the deep-prior theory.The stated target is restoration of a single hyperspectral image without relying on a task-specific training dataset.
- Contributions: The work designs a 3D-convolutional implementation and evaluates the algorithm for HSI restoration against other methods.The 3D convolutions are presented as containing low-level image information usable as a prior, while the evaluation covers the proposed HSI application.
- Contributions: The authors make the source code publicly accessible and ready to use out of the box.
2. Related Works
Prior HSI restoration methods commonly process spectral bands separately or use spatial-spectral, optimization-based, generative, and tensor approaches. Existing methods also face training-data, corruption-setting, or auxiliary-image constraints across inpainting and super-resolution.
- Denoising: Band-wise HSI denoising can use standard filters, total variation, block-matching, or CNN methods, but ignoring spectral data may introduce spectral distortions and artifacts.This motivated methods based on spatial-spectral features, including wavelet shrinkage and low-rank tensor or matrix recovery.
- Inpainting: HSI inpainting methods include band-wise patch-similarity and variational approaches, GAN-based reconstruction, and HSI-specific regularizers.GAN-based methods can fill large holes but rely on large training datasets.
- Inpainting: FastHyIn achieves state-of-the-art HSI inpainting accuracy but uses information from intact bands and therefore cannot handle corruption across all bands.
- Super-resolution: Most HSI super-resolution methods fuse a hyperspectral image with an easier-to-obtain high-resolution multispectral image.Single-image super-resolution instead includes spectral mixture, low-rank tensor, network, energy-minimization, and transfer-learning approaches.
3. Methodology
The method optimizes a fully convolutional encoder-decoder’s parameters to reconstruct hyperspectral images from fixed noise, adapting the energy function for denoising, inpainting, and super-resolution. Both 2D and 3D architectures are explored, with 3D convolutions preserving volumetric spectral structure.
- Deep prior framework: The network maps fixed noise z to an image through a fully convolutional encoder-decoder, with parameters optimized by minimizing a task-dependent energy function.The parameters are initialized randomly and optimized using methods such as gradient descent.
- Task-specific objectives: For denoising, early stopping exploits faster reconstruction of natural-looking signal than random noise.This produces blind denoising by interrupting optimization before noise is recovered.
- Task-specific objectives: Inpainting modifies the energy term with a binary mask m and a Hadamard product so missing regions can be reconstructed from intact data.The mask distinguishes observed and corrupted regions in the reconstruction objective.
- Task-specific objectives: Super-resolution uses a downsampling operator d(x, α) so the predicted high-resolution image matches the observed low-resolution image after downsampling.The operator maps spatial dimensions from αN × αN × C to N × N × C.
- Network architectures: The experiments use 2D and 3D convolutional networks; 3D convolutions preserve volumetric structure and permit smaller filters such as 3 × 3 × 3 throughout the network.Unlike 2D processing, 3D volumes can be downsampled and upsampled along the spectral dimension as well.
- Implementation details: The network input is uniform noise shaped like the processed hyperspectral image, with optional Gaussian perturbation, LeakyReLU activations, convolutional-stride downsampling, interpolation upsampling, and ADAM optimization.Upsampling is nearest or bilinear in 2D, and trilinear in 3D.
4. Experimental setup
The evaluation covers hyperspectral denoising, inpainting, and single-image super-resolution on three datasets with task-specific corruptions or degradations. Results are compared against classical, HSI-specific, learning-based, and trained CNN baselines.
- Denoising: Denoising uses the HYDICE DC Mall image with 191 channels, cropped to 200 × 200 pixels and corrupted by Gaussian noise with σ = 100.Comparisons include HSSNR, LRTA, BM4D, LRMR, and HSID-CNN.
- Inpainting: Inpainting uses the 145 × 145 × 200 Indian Pines hyperspectral dataset with corrupted strips applied across all bands.The method is compared with Mumford-Shah, fourth-order total variation, and FastHyIn.
- Super-resolution: Super-resolution uses a 150 × 150 patch from Pavia Center with 102 spectral bands, spatially downsampled by a factor of 2.Baselines include nearest and bilinear upsampling, band-wise or grouped SRCNN variants, and 3D-FCNN.
5. Results and discussion
The proposed Deep HS prior outperformed single-image algorithms and matched trained CNNs without prior dataset training. Its 3D version also showed that convolutional architectures can contain an intrinsic image prior.
- The proposed method outperformed all single-image algorithms while achieving performance comparable to trained CNNs without prior dataset training.
- Figure 3 evaluates hyperspectral super-resolution on Pavia Center using rescale factor 2 and band 25 visualization.
- Figure 4 evaluates hyperspectral inpainting on AVIRIS Indian Pines at band 150, with corruption applied across all bands except one 75%-affected case.
- 3D convolutions demonstrated comparable performance and contained the image prior within the network’s intrinsic parameters.
6. Conclusions
The paper develops an effective single-hyperspectral-image restoration algorithm by treating the CNN itself as an image prior rather than relying on learned or manually designed priors.
- The proposed algorithm restores a single hyperspectral image using a CNN’s intrinsic image prior instead of training data or manually designed priors.