Source-linked AI summary
Hierarchical Regression Network for Spectral Reconstruction from RGB Images
Yuzhi Zhao, Lai-Man Po, Qiong Yan, Wei Liu, Tingyu Lin
TL;DR
RGB-to-hyperspectral reconstruction is an ill-posed inverse problem, while existing encoder-decoder approaches can lose pixel information and may not explicitly use contextual information. The paper proposes a four-level HRNet with lossless PixelShuffle-based inter-level interaction, residual dense and global blocks, and an eight-setting ensemble. HRNet wins the NTIRE 2020 real-world track and ranks third on the clean-image track.
Problem
RGB-to-hyperspectral reconstruction is ill-posed because RGB images contain less information than hyperspectral images, while prior encoder-decoder methods can lose pixel information.
Method
HRNet combines four-level hierarchical regression, PixelUnShuffle and PixelShuffle sampling, residual dense and global blocks, and an eight-setting ensemble.
Results
HRNet wins track 2 for real-world images and ranks 3rd on track 1 for clean images in the NTIRE 2020 challenge.
Takeaways & Limitations
The proposed architecture and ensemble strategy provide a competitive approach for spectral reconstruction on both real-world and clean RGB-image tracks.
Abstract
from arXiv · showhide
Capturing visual image with a hyperspectral camera has been successfully applied to many areas due to its narrow-band imaging technology. Hyperspectral reconstruction from RGB images denotes a reverse process of hyperspectral imaging by discovering an inverse response function. Current works mainly map RGB images directly to corresponding spectrum but do not consider context information explicitly. Moreover, the use of encoder-decoder pair in current algorithms leads to loss of information. To address these problems, we propose a 4-level Hierarchical Regression Network (HRNet) with PixelShuffle layer as inter-level interaction. Furthermore, we adopt a residual dense block to remove artifacts of real world RGB images and a residual global block to build attention mechanism for enlarging perceptive field. We evaluate proposed HRNet with other architectures and techniques by participating in NTIRE 2020 Challenge on Spectral Reconstruction from RGB Images. The HRNet is the winning method of track 2 - real world images and ranks 3rd on track 1 - clean images. Please visit the project web page https://github.com/zhaoyuzhi/Hierarchical-Regression-Network-for-Spectral-Reconstruction-from-RGB-Images to try our codes and pre-trained models.
1. Introduction
The paper frames RGB-to-hyperspectral reconstruction as an ill-posed inverse problem constrained by limited RGB information, noise, and information loss in encoder-decoder methods. It proposes HRNet with lossless hierarchical sampling, artifact-removing and context-modeling blocks, and ensemble settings, achieving leading NTIRE 2020 results.
- Traditional spectrometers are expensive, complex, slow, and poorly suited to portable platforms or high-speed moving scenes.
- Spectral reconstruction is ill-posed because one RGB image can correspond to many reasonable hyperspectral images, requiring a learned high-quality mapping.
- Previous auto-encoder methods risk losing original pixel information through repeated convolutional downsampling at low spatial resolution.
- Residual dense and residual global blocks target artifacts, noise, and remote pixel correlations by enlarging the perceptive field and improving generation quality.
- HRNet uses PixelUnShuffle and PixelShuffle for lossless downsampling and upsampling across hierarchical feature levels.
- HRNet wins track 2 for real-world images and ranks 3rd on track 1 for clean images in the NTIRE 2020 hyperspectral reconstruction challenge.
2. Related work
Prior work spans specialized hyperspectral scanners, RGB-to-hyperspectral reconstruction using sparse coding or deep learning, diverse datasets, and CNN-based low-level vision frameworks. These approaches reflect a shift toward learned reconstruction because conventional acquisition is costly, time-consuming, and less portable.
- Conventional hyperspectral acquisition uses spatial or spectral scanning devices such as pushbroom, whiskbroom, and band-sequential scanners.
- RGB-to-hyperspectral reconstruction learns mappings from paired RGB-HS datasets to provide a more convenient alternative to non-portable or time-consuming acquisition.
- Earlier reconstruction methods used data sparsity and sparse dictionaries of hyperspectral signatures with corresponding RGB projections.
- Researchers developed datasets with varying band counts, wavelength ranges, scene types, and image quantities to support hyperspectral reconstruction.
- CNNs have improved hyperspectral reconstruction alongside residual and dense blocks developed for convergence and feature extraction in low-level vision.
- NTIRE dataset visualizations include clean RGB images, real-world RGB images, and hyperspectral channels at 400, 410, 420, 500, 600, and 700 nm.
3. Methodology
The method trains HRNet on paired hyperspectral and RGB data, using a four-level architecture with lossless pixel rearrangement and specialized feature blocks. It also evaluates ensemble settings by selecting the best MRAE checkpoint from eight training methods.
- Dataset: The NTIRE 2020 dataset contains 450 RGB-HS training pairs, 31 spectral bands from 400–700 nm, and 482 × 512 spatial resolution.Clean RGB images use a known response function, whereas real-world RGB images include an unknown response function, blind noise, and demosaicking.
- HRNet architecture: HRNet uses four levels, with PixelUnShuffle for parameter-free downsampling and learnable PixelShuffle for upsampling and inter-level connections.PixelShuffle reshapes feature maps without interpolation, while PixelUnShuffle preserves all input pixels across levels.
- HRNet architecture: Each level integrates features across scales, reduces artifacts with residual dense blocks, and extracts global features with residual global blocks.The top level uses the most blocks, while the bottom level adds a 1 × 1 convolution to weight channels for tone mapping.
- Implementation details: Training uses L1 loss with local RGB and spectral patches cropped from the same spatial regions.The network uses LeakyReLU activations, no normalization, reflect padding, 10,000 epochs, Adam optimization, and 256 × 256 random crops.
- Ensemble strategy: The ensemble combines eight training methods, varying block order, batch size, crop size, and baseline retraining, then averages their best-MRAE checkpoints.MRAE is recorded every 1,000 epochs to select the checkpoint used from each method.
4. Experiment
Experiments evaluate HRNet against alternative architectures, ablations, compressed variants, and ensemble settings on NTIRE 2020 validation and test tracks. HRNet achieves strong reconstruction results, with its residual blocks, PixelShuffle design, and ensemble strategy contributing to performance.
- Evaluation metrics: MRAE measures pixel-wise disparity between generated and ground-truth spectral images, while RMSE measures their error across 31 bands.BPMRAE evaluates colorimetric accuracy after back-projecting spectra through a fixed camera response function.
- Ablation study: Removing either the residual dense block or residual global block produces the largest MRAE decrease among ablation settings, showing their combination is significant for spectral reconstruction.The ablation compares these blocks with plain convolution layers having similar FLOPs.
- Model compression: Compressed HRNet variants trade pixel fidelity for substantially smaller computational and storage requirements.The baseline uses 182.347 Gb MACs, 31.705 Mb parameters, and 123.879 Mb weights; users can choose MRAE = 0.042328 or Weights = 2.410 Mb.
- Testing results: HRNet ranks 3rd on track 1 clean images and 1st on track 2 real-world images in the NTIRE 2020 challenge.Testing results are summarized separately for the two tracks.
5. Conclusion
The paper presents a 4-level HRNet for generating spectra from RGB images, combining specialized feature blocks, PixelShuffle connections, and ensemble prediction. It outperforms U-Net and U-ResNet on the NTIRE 2020 dataset and achieves leading challenge rankings.
- The proposed 4-level HRNet automatically generates spectrum from RGB images.
- Residual dense and residual global blocks extract features at each level, while PixelShuffle connects levels.
- An 8-setting ensemble strategy further enhances the quality of predicted spectral images.
- The HRNet outperforms U-Net and U-ResNet on the NTIRE 2020 hyperspectral reconstruction dataset.
- The method wins track 2 for real-world images and ranks 3rd on track 1 for clean images.