Source-linked AI summary
Deep Gaussian Scale Mixture Prior for Spectral Compressive Imaging
Tao Huang, Weisheng Dong, Xin Yuan, Jinjian Wu, Guangming Shi
TL;DR
CASSI reconstruction must recover a 3D hyperspectral image from a 2D compressive measurement, while hand-crafted priors are limited and heuristic network design is difficult. The paper formulates reconstruction as MAP estimation with a learned GSM prior, using DCNNs to learn scale priors and local means with end-to-end joint optimization. Experiments on synthetic and real datasets show that the method outperforms existing state-of-the-art algorithms.
Problem
CASSI reconstruction relies on hand-crafted priors with limited representation capability, while designing effective deep networks heuristically is nontrivial.
Method
The method uses MAP estimation with GSM models, learning scale priors and spatial-spectral local means through DCNNs with end-to-end joint optimization.
Results
Extensive experiments on synthetic and real datasets show that the proposed method outperforms existing state-of-the-art HSI reconstruction methods.
Takeaways & Limitations
The paper provides an interpretable HSI reconstruction method that learns GSM priors rather than manually designing them.
Abstract
from arXiv · showhide
In coded aperture snapshot spectral imaging (CASSI) system, the real-world hyperspectral image (HSI) can be reconstructed from the captured compressive image in a snapshot. Model-based HSI reconstruction methods employed hand-crafted priors to solve the reconstruction problem, but most of which achieved limited success due to the poor representation capability of these hand-crafted priors. Deep learning based methods learning the mappings between the compressive images and the HSIs directly achieved much better results. Yet, it is nontrivial to design a powerful deep network heuristically for achieving satisfied results. In this paper, we propose a novel HSI reconstruction method based on the Maximum a Posterior (MAP) estimation framework using learned Gaussian Scale Mixture (GSM) prior. Different from existing GSM models using hand-crafted scale priors (e.g., the Jeffrey's prior), we propose to learn the scale prior through a deep convolutional neural network (DCNN). Furthermore, we also propose to estimate the local means of the GSM models by the DCNN. All the parameters of the MAP estimation algorithm and the DCNN parameters are jointly optimized through end-to-end training. Extensive experimental results on both synthetic and real datasets demonstrate that the proposed method outperforms existing state-of-the-art methods. The code is available at https://see.xidian.edu.cn/faculty/wsdong/Projects/DGSM-SCI.htm.
1. Introduction
CASSI enables snapshot capture of hyperspectral data, but reconstruction remains difficult because hand-crafted priors are limited and deep networks are challenging to design. The paper proposes an interpretable MAP reconstruction method with learned GSM priors and jointly optimized parameters.
- CASSI captures 3D hyperspectral images in a single 2D compressive measurement using a mask and disperser.The reconstruction algorithm recovers the 3D HSI from that measurement.
- Hand-crafted-prior methods require manual parameter tuning and achieve limited reconstruction quality and speed.
- The proposed method formulates HSI reconstruction as MAP estimation with an interpretable learned GSM prior.
- The scale prior is learned by a DCNN instead of using a hand-crafted prior such as Jeffrey’s prior.
- Local GSM means are weighted averages of spatial-spectral neighboring pixels, with similarity weights estimated by a DCNN.
- Extensive experiments on synthetic and real datasets show that the proposed method outperforms existing state-of-the-art HSI reconstruction methods.
2. Related Work
Prior CASSI reconstruction methods use hand-crafted regularizers, while deep networks learn nonlinear mappings and GSM models provide a probabilistic signal representation. This paper learns both GSM scale priors and local means with DCNNs for HSI reconstruction.
- Model-based CASSI methods use sparse, dictionary, total-variation, and nonlocal regularizers, but require manual parameter selection and are time-consuming.
- Deep neural networks formulate HSI reconstruction as nonlinear mapping and have achieved better results than model-based methods.
- GSM models have been applied to image denoising, sparse coding, and video foreground estimation as probabilistic signal models.
- For HSI reconstruction, the paper learns both the GSM scale prior and local means with DCNNs, jointly training all parameters end-to-end.
3. The CASSI Observation Model
CASSI encodes a 3D spectral cube into a 2D measurement by mask modulation, wavelength-dependent dispersion, and spectral compression. The resulting forward model represents the measurement as a linear operator applied to the vectorized HSI, with additive noise.
- CASSI first modulates the 3D spectral data cube with a physical mask, then disperses wavelengths and records their mixture on a 2D sensor.
- Each wavelength channel of the modulated image is shifted spatially by a wavelength-dependent distance before measurement.
- The imaging sensor compresses the shifted spectral data into a 2D measurement that mixes information from all wavelengths.
- In vectorized form, x and y represent the HSI and measurement, while A is the CASSI measurement matrix.
- The forward model includes measurement noise and maps the vectorized hyperspectral image through the measurement operator.
4. The Proposed Method
The proposed method formulates HSI reconstruction as MAP estimation with learned GSM priors, using DCNNs to estimate scale-related weights and local means. An end-to-end staged network learns the reconstruction operators and spatially adaptive components jointly.
- GSM-based MAP formulation: The method models each HSI pixel with a nonzero-mean Gaussian whose scale prior is learned rather than manually specified.The GSM formulation jointly estimates the image and scale variables instead of relying on an analytically fixed scale prior.
- GSM-based MAP formulation: Local GSM means are estimated from weighted spatial-spectral neighbors using spatially variant 3D filters.The filters adapt to local HSI edges and textures, while factorization reduces filter coefficients from N · q^3 to 3 · N · q.
- Deep GSM optimization: A DCNN replaces the manually designed proximal operation for estimating w and jointly optimizes the x- and w-subproblems.The network uses T stages corresponding to iterative updates, with S(·) estimating w from the current reconstruction.
- Network architecture: A lightweight U-Net and weight generator estimate w, while a filter generator produces spatially variant filters for updating the local means.The proposed architecture includes separate modules for learned measurement operations, regularization weights, and GSM means.
- Network architecture: The staged network learns both the measurement operator A and its transpose A⊤ with sub-networks instead of explicitly constructing them.This enables training on small patches and reduces memory consumption and computational complexity for large masks and measurements.
- Network architecture: The learned weights vary spatially and align with image edges and textures in visualizations from the fourth stage.The visualization compares an RGB image with weight maps associated with four spectral bands.
5. Simulation Results
Simulation experiments on CAVE and KAIST compare the proposed reconstruction method with model-based and deep learning baselines across standard and multiple-mask settings. Ablations examine filter size, stage count, and dense connections.
- Experimental Setup: Experiments use CAVE and KAIST hyperspectral datasets, with simulated measurements generated from a real 256×256 mask.Training uses randomly extracted 96×96×28 HSI patches and produces 96×150 two-dimensional measurements.
- Comparison with State-of-the-Art Methods: The proposed method outperforms model-based and deep learning baselines, exceeding TSA-Net by 1.17dB average PSNR and 0.0227 average SSIM.It also improves average PSNR over HSSP and DNU by 2.28dB and 1.89dB, respectively.
- Multiple Mask Results: With five masks, one jointly trained proposed model still outperforms DNU and TSA-Net models trained separately for each mask.The experiment uses compound training and testing datasets generated from five masks.
- Ablation Study: Larger filters improve reconstruction quality until the improvement flattens after q = 7, while increasing the stage number T improves performance.The implementation sets q = 7 and T = 4 as a trade-off between reconstruction performance and computational complexity.
- Ablation Study: Dense connections increase PSNR from 30.52dB to 32.63dB and SSIM from 0.8802 to 0.9166.The comparison evaluates the proposed network without and with dense connections.
6. Real Data Results
The proposed method is evaluated on real SD-CASSI measurements containing 28 wavelengths, with training adapted to the real system and shot noise injected during simulation.
- Real Data Setup: Real-data experiments use SD-CASSI measurements with 28 wavelengths from 450nm to 650nm and 54-pixel column dispersion.The captured measurements have spatial size 660 × 714.
- Real Data Setup: The proposed network is retrained on CAVE and KAIST scenes while 11bit shot noise is injected to simulate real measurements.
7. Conclusions
The paper concludes with an interpretable CASSI reconstruction method that learns GSM priors and spatial-spectral dependencies through DCNNs. Experiments on synthetic and real datasets show superior performance, with potential use beyond spectral compressive imaging.
- Method: The method formulates HSI reconstruction as MAP estimation with GSM models and learns the GSM scale prior using a DCNN.
- Method: GSM local means are computed from weighted spatial-spectral neighboring pixels, with the filter coefficients estimated by a DCNN.
- Conclusion: Extensive experiments on synthetic and real datasets show that the proposed method outperforms existing state-of-the-art algorithms.
- Scope: The network is described as applicable to video snapshot compressive imaging systems in addition to CASSI and similar spectral systems.