PaperlayerView exact revision on arXiv

Source-linked AI summary

A survey on computational spectral reconstruction methods from RGB to hyperspectral imaging

Jingang Zhang, Runmu Su, Wenqi Ren, Qiang Fu, Felix Heide, Yunfeng Nie

arXiv:2106.15944v2eess.IVcs.CV

TL;DR

Hyperspectral acquisition provides rich spatial and spectral information but relies on expensive, complicated devices, motivating reconstruction from widely available RGB images. This survey reviews the image-formation foundations, datasets, metrics, and more than 25 prior-based and data-driven methods, finding that most deep-learning approaches outperform prior-based methods in reconstruction accuracy and quality despite lower speed.

  • Problem

    RGB-to-HSI reconstruction addresses the ill-posed inverse problem of recovering hyperspectral information from only three RGB channels, while suitable HSI datasets and evaluation remain limited.

  • Method

    The survey systematically reviews prior-based and data-driven reconstruction methods, image-formation models, HSI datasets, and performance metrics.

  • Results

    Most data-driven deep-learning methods achieve better reconstruction accuracy and quality than prior-based methods, while generally operating at lower speeds.

  • Takeaways & Limitations

    Dataset size and resolution affect reconstruction accuracy: prior-based methods may suit small datasets, whereas data-driven methods are favored when datasets are large.

  • Takeaways & Limitations

    Existing methods commonly depend on scarce supervised RGB-HSI pairs, and conventional metrics can average away large spectral deviations in narrow-band objects.

Abstract

from arXiv · show

Hyperspectral imaging enables versatile applications due to its competence in capturing abundant spatial and spectral information, which are crucial for identifying substances. However, the devices for acquiring hyperspectral images are expensive and complicated. Therefore, many alternative spectral imaging methods have been proposed by directly reconstructing the hyperspectral information from lower-cost, more available RGB images. We present a thorough investigation of these state-of-the-art spectral reconstruction methods from the widespread RGB images. A systematic study and comparison of more than 25 methods has revealed that most of the data-driven deep learning methods are superior to prior-based methods in terms of reconstruction accuracy and quality despite lower speeds. This comprehensive review can serve as a fruitful reference source for peer researchers, thus further inspiring future development directions in related domains.

Fundamentals

The paper models RGB formation from hyperspectral reflectance and illumination, then frames reconstruction as inverting this model from three known channels. It also introduces datasets and metrics for evaluating reconstructed HSIs.

  • HSIs contain many narrow spectral bands, whereas RGB images use three wide channels; spectral reflectance helps identify substances.
  • Under the Lambertian model, RGB channels integrate the product of spectral reflectance, illumination, and each channel’s spectral response function.
  • Spectral reconstruction inverts the forward image-formation model to recover the multichannel HSI when only the captured three-channel RGB image is known.
  • Five commonly used HSI datasets are small or medium-sized and differ in data amount, spatial resolution, spectral channels, and scene diversity.
  • MRAE and RMSE measure numerical reconstruction errors, while SAM measures pixel-level spectral similarity; these metrics can overlook narrow-band deviations averaged across wider scenes.

Algorithm survey

The survey classifies RGB-to-HSI reconstruction methods into prior-based and data-driven families, with prior knowledge representing image attributes and network models organized into several architectures.

  • The survey divides spectral reconstruction algorithms into prior-based and data-driven methods according to their underlying principles.
  • Data-driven methods are grouped by network model, including Linear CNN, U-Net, and GAN architectures.
  • The survey presents the overall taxonomy of analyzed algorithms in Figure 2 and an overview of prior-based methods in Table 2.
  • The reconstruction target is an HSI represented across spatial dimensions and spectral bands, with RGB channels forming the input representation.
  • Prior knowledge represents inherent image attributes and characteristics, including sparsity, spatial structure similarity, and spectral correlation.

Dictionary learning.

Dictionary-learning methods represent hyperspectral pixels with basis spectra and coefficients, then learn RGB-to-spectral mappings using sparsity and related priors.

  • Dictionary learning stores basis spectra and obtains coefficients to represent hyperspectral data sparsely before projecting the dictionary into RGB space through the SRF.
  • Sparse coding finds overcomplete basis spectra and sparse coefficients, and it is fast when the dataset is small.
  • SR A+ learns a local RGB-to-hyperspectral mapping from neighboring anchor points, improving speed and accuracy over global mapping while remaining pixel-wise.
  • Multiple non-negative sparse dictionaries cluster spectral reflections and independently learn compact dictionaries for different substance groups.
  • Sparse dictionaries that ignore local linearity can reduce reconstruction accuracy and produce metamerism, motivating locally linear embedding sparse dictionaries.

Local linear embedding sparse dictionary.

The surveyed prior-based methods progressively incorporate spatial context, low-dimensional manifolds, Gaussian-process spectral models, and patch similarity to improve reconstruction representations.

  • Hand-crafted local linearity and texture assumptions can make dictionaries compact and efficient while improving reconstructed HSI quality and reducing metamerism.
  • Spatially constrained dictionary learning adds spatial context to pixel-wise sparse representation, preserving image structures and physical connections.
  • Manifold learning represents hyperspectral data with low-dimensional manifolds and uses this prior to establish a reconstruction model.
  • The manifold method reduces the three-to-many mapping problem to three-to-three using isometric dimensionality reduction, constraining the solution space.
  • Gaussian processes model smooth basis spectra, while clustered similar patches introduce spatial similarity and spectral correlation to accelerate reconstruction.

SR Gaussian process.

Data-driven spectral reconstruction methods use varied neural architectures to recover hyperspectral information from RGB or other spectrally undersampled images. The surveyed designs range from simple linear CNNs to basis-function, GAN, and other complex models, with trade-offs involving priors, complexity, and overfitting.

  • Deep learning methods are grouped by distinctive network architectures, including linear CNN, U-Net, GAN, dense, residual, attention, and multi-branch designs.
  • HSCNN restores hyperspectral information with a VDSR-derived CNN trained using mean square error, but requires a known spectral response function.
  • SR2D/3DNet uses 2D and 3D convolutions to model spatial information separately or jointly with inter-channel relations, with the joint design slightly improving performance.
  • Modified architectures and loss functions increase network complexity, while training on small datasets can make the network prone to overfitting.
  • SRBFWU-Net learns 10 basis functions and pixelwise weights, replacing direct 3-to-many mapping with a continuously learned basis representation and adding unsupervised learning modules.
  • GAN-based reconstruction combines a generator that produces hyperspectral images with a discriminator that distinguishes reconstructions from ground truth.

SRCGAN.

Dense, residual, and multi-scale architectures improve feature propagation and representation for spectral reconstruction, while their design choices introduce accuracy or computational trade-offs. The surveyed models use dense connectivity, multi-scale processing, spectral variants, and residual learning in different combinations.

  • Dense network: Dense networks connect all layers to improve feature propagation, alleviate vanishing gradients, and promote training stability.
  • Dense network: SRTiramisuNet combines dense connectivity with multi-scale processing, using 64 × 64 patches and Euclidean loss for efficient, robust training.Max-pooling downsampling may reduce reconstruction accuracy.
  • Residual network: HSCNN+ introduces spectral upsampling, residual blocks, and dense blocks across three variants to improve reconstruction accuracy and alleviate vanishing gradients.
  • Residual network: Residual networks use skip connections and deeper blocks to preserve low-frequency information while producing more detailed reconstructions.
  • Residual network: SREfficientNet combines residual learning with staged feature extraction and reconstruction, and uses substantially less memory, computing resources, and running time than other recent CNN methods.

SREfficientNet.

SREfficientNet uses residual learning to improve feature expression while reducing resource demands, whereas its enhanced version estimates camera spectral response functions for unknown-camera settings. Attention-based alternatives further emphasize selected features but may require camera sensitivity priors.

  • SREfficientNet: SREfficientNet combines local and global residuals to expand the receptive field and strengthen feature expression.
  • SREfficientNet+: SREfficientNet+ adds an estimation block that supplies estimated spectral response functions, supporting training on datasets from unknown or varied cameras.
  • Attention network: Attention networks selectively weight features because spatial and spectral information are not always equally important for reconstruction.
  • Attention network: SRAWAN combines camera spectral sensitivity priors, residual attention blocks, and non-local spatial context to improve spectral reconstruction accuracy.
  • Attention network: SRAWAN’s accuracy-oriented design remains limited in practical cases because it still requires camera sensitivity as a prior.

SRRPAN.

Pixel attention and multi-branch architectures improve feature selection and representation by modeling pixelwise weights or combining complementary processing paths. These designs aim to capture richer spatial and contextual information, while increased complexity can affect performance and resources.

  • SRRPAN: SRRPAN adaptively rescales pixelwise weights across input feature maps using residual attention groups and pixel attention modules.
  • Multi-branch network: Multi-branch networks fuse diverse features from multiple context scales and provide multi-path signal flow for information exchange during training.
  • Multi-branch network: SRLWRDANet combines densely connected and multi-scale parallel subnets, sharing coordinated convolutional features and boundary information.
  • Evaluation: Table 4 compares representative methods using RMSE, MRAE, and SAM on the BGU-HS and ARAD-HS datasets, with the top two results highlighted.
  • Multi-branch network: SRPFMNet uses multiple branches to address the need for different receptive-field sizes and mapping functions across hyperspectral pixels.

Comparison and analysis

Data-driven methods generally outperform prior-based methods in spatial and spectral reconstruction, with attention-based models achieving especially strong results. The survey also identifies dataset scarcity, unknown camera parameters, limited spectral bands, and comparison trade-offs as important constraints.

  • Quantitative comparison: SRAWAN achieves the best trained-method performance on RMSE, MARE, and SAM across the BGU-HS and ARAD-HS comparisons.RMSE and MARE measure spatial reconstruction errors, while SAM measures spectral reconstruction error.
  • Quantitative comparison: Sparse coding shows much larger spatial and spectral errors than the other compared methods, whereas SRAWAN has negligible spatial errors and the smallest spectral-curve differences.The spatial comparison uses a 640 nm residual heat map, while the spectral comparison evaluates curves at two selected spatial points.
  • Method comparison: Prior-based methods rely on handcrafted priors and often lose high-frequency information, while data-driven methods learn hidden RGB-to-HSI mappings from image data.Prior-based methods may also degrade when the spectral response function is estimated rather than known.
  • Method comparison: CNN variants improve reconstruction through larger receptive fields, residual or dense connections, and attention mechanisms; SRAWAN and SRHRNet perform best among the surveyed network types.Attention mechanisms and multi-branch designs improve performance by increasing network complexity.
  • Evaluation considerations: Fair comparison among deep-learning methods remains difficult because accuracy, generalization, time cost, consistency, and network complexity all influence performance.The survey therefore cautions against identifying a universal winner using reconstruction metrics alone.
  • Future directions: Future progress depends on spectral-aware losses, richer spatial-context modeling, larger or augmented datasets, unsupervised learning, and compatibility with unknown cameras and hundreds of spectral bands.The survey notes that most methods use supervised RGB-HSI pairs, while current methods often perform poorly with unknown camera parameters or extreme spectral upsampling.

Conclusion

The review categorizes RGB-to-hyperspectral reconstruction methods, compares representative deep-learning approaches, and identifies dataset size as an important consideration when choosing between prior-based and data-driven methods. It also highlights accuracy, architecture, learning paradigms, and dataset expansion as priorities for future work.

  • The review categorizes spectral reconstruction algorithms into prior-based methods and data-driven deep-learning approaches.
  • Dataset size affects method choice: prior-based methods may suit small datasets, whereas deep-learning methods are preferred for larger datasets.The review associates small datasets with overfitting risks and larger datasets with better RGB-to-HSI mapping by data-driven methods.
  • Prior-based methods rely heavily on handcrafted priors and known spectral response functions, while data-driven methods tend to map RGB images better when datasets are large.
  • The authors trained and compared selected deep-learning methods on two typical datasets to guide architectural choices.
  • Future work should improve spectral reconstruction accuracy through spectral losses, advanced architectures, unsupervised learning, and larger HSI datasets.The review specifically mentions attention, multi-branch, hierarchical, and Transformer architectures, alongside dataset expansion.

Data availability

The study’s generated and analyzed datasets are available in the Spectral Reconstruction repository. The paper records receipt on 14 June 2022 and acceptance on 6 July 2022.

  • The datasets generated and analyzed in the study are available in the Spectral Reconstruction repository.The passage provides the repository link: https://github.com/Intelligent-Imaging-Center/Spectral-Reconstruction.
  • The manuscript was received on 14 June 2022 and accepted on 6 July 2022.

Additional information

The paper provides supplementary material online and identifies correspondence information, reprint permissions, publisher neutrality, and the Creative Commons licensing terms. These notices describe how additional materials, correspondence, reuse, and licensing are handled.

  • Supplementary Information is available with the online version of the article.The supplementary material is linked through the article DOI.
  • Correspondence and requests for materials should be addressed to Y.N.
  • Reprints and permissions information is available through Nature.
  • Springer Nature states that it remains neutral regarding jurisdictional claims in published maps and institutional affiliations.
  • The article is licensed under Creative Commons Attribution 4.0, permitting reuse with attribution, a license link, and disclosure of changes.Third-party material is covered by the license unless otherwise indicated.
Loading 2106.15944v2…