Source-linked AI summary
HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imaging
Xiaowan Hu, Yuanhao Cai, Jing Lin, Haoqian Wang, Xin Yuan, Yulun Zhang, Radu Timofte, Luc Van Gool
TL;DR
HSI reconstruction from compressive measurements remains limited by reduced internal resolution in self-attention networks and frequency discrepancies after spatial-spectral optimization. HDNet combines high-resolution spatial-spectral attention with dynamically weighted frequency-domain supervision, achieving state-of-the-art quantitative and visual performance. Its complementary pixel- and frequency-level learning targets both fine spatial detail and difficult frequency components.
Problem
Existing HSI reconstruction methods sacrifice internal resolution for attention efficiency and can retain visual frequency-domain discrepancies despite spatial-spectral optimization.
Method
HDNet combines HR spatial-spectral attention and efficient feature fusion with DFT-based dynamic frequency-domain supervision that emphasizes difficult frequencies.
Results
HDNet achieves state-of-the-art performance in quantitative evaluation and visual comparison, with extensive experiments supporting its superiority.
Takeaways & Limitations
Complementary high-resolution pixel-level attention and frequency-level refinement improve HSI reconstruction quality across spatial and frequency representations.
Abstract
from arXiv · showhide
The rapid development of deep learning provides a better solution for the end-to-end reconstruction of hyperspectral image (HSI). However, existing learning-based methods have two major defects. Firstly, networks with self-attention usually sacrifice internal resolution to balance model performance against complexity, losing fine-grained high-resolution (HR) features. Secondly, even if the optimization focusing on spatial-spectral domain learning (SDL) converges to the ideal solution, there is still a significant visual difference between the reconstructed HSI and the truth. Therefore, we propose a high-resolution dual-domain learning network (HDNet) for HSI reconstruction. On the one hand, the proposed HR spatial-spectral attention module with its efficient feature fusion provides continuous and fine pixel-level features. On the other hand, frequency domain learning (FDL) is introduced for HSI reconstruction to narrow the frequency domain discrepancy. Dynamic FDL supervision forces the model to reconstruct fine-grained frequencies and compensate for excessive smoothing and distortion caused by pixel-level losses. The HR pixel-level attention and frequency-level refinement in our HDNet mutually promote HSI perceptual quality. Extensive quantitative and qualitative evaluation experiments show that our method achieves SOTA performance on simulated and real HSI datasets. Code and models will be released at https://github.com/caiyuanhao1998/MST
1. Introduction
CASSI enables real-time capture of 3D hyperspectral cubes but requires reliable recovery from compressed 2D measurements. HDNet addresses lost high-resolution features and frequency discrepancies through complementary spatial-spectral and frequency-domain learning.
- Motivation: CASSI captures 3D HSI cubes in real time by compressing them into 2D measurements, but reconstruction must recover the underlying 3D signal.Traditional hand-crafted priors have poor generalization, motivating learning-based reconstruction.
- Limitations of Existing Methods: Self-attention reconstruction methods often reduce feature resolution to control the computational cost of nonlocal attention maps.This sacrifices fine-grained high-resolution features.
- Limitations of Existing Methods: SDL-focused methods can leave an obvious frequency-domain discrepancy because CNNs preferentially fit easy low-frequency components.Reported artifacts include lost high frequencies, checkerboard patterns, and deviation toward limited frequency areas.
- Proposed Method: HDNet combines HR spectral attention, HR spatial attention, and efficient feature fusion with adaptive frequency-domain supervision.The HR attention preserves internal resolution, while FDL uses DFT-based frequency distances and dynamically emphasizes difficult frequencies.
- Contributions: Dynamic frequency-level supervision narrows the frequency-domain discrepancy and adaptively forces reconstruction of high and hard frequencies.This is presented as the first use of such supervision for improving HSI perceptual quality.
- Results: HDNet achieves state-of-the-art performance in quantitative evaluation and visual comparison, supported by extensive experiments.The introduction states that experiments prove the superiority of HDNet.
2. Related Work
Related work spans model-based priors, deep CNN reconstruction, self-attention for spatial and spectral features, and frequency-spectrum analysis. These studies motivate addressing both resolution limitations and frequency-domain gaps in HSI reconstruction.
- HSI Reconstruction: Model-based HSI reconstruction methods use hand-crafted priors but suffer from poor generalization ability.Deep CNN methods were introduced as a response to this limitation.
- HSI Reconstruction: Deep reconstruction methods include unfolding with pretrained denoisers, self-attention for spatial features, and deep Gaussian scale mixture priors.These approaches represent different strategies for learning HSI restoration mappings.
- Attention-Based Methods: Attention-based HSI restoration models separately or jointly capture spatial, spectral, and feature-correlation information.Examples include λ-Net feature autocorrelation, bidirectional spectral correlation, and TSA-Net spatial-spectral attention.
- Frequency-Domain Methods: Frequency-spectrum analysis links deep-learning low-frequency bias and periodic spectral patterns with frequency-domain gaps and spatial artifacts.Prior work therefore explores reducing visual differences by narrowing frequency-domain discrepancies.
- Frequency-Domain Methods: HDNet’s architecture combines spatial-spectral domain learning with frequency-domain learning and dynamically weighted frequency supervision.Its FDL uses a 2D DFT and assigns weights according to frequency distance.
3. The Proposed Method
HDNet combines high-resolution spatial-spectral attention with dynamic frequency-domain supervision for HSI reconstruction. Its design preserves fine internal features, fuses them efficiently, and emphasizes difficult frequencies during optimization.
- Overall Architecture: HDNet uses ResNet as a lightweight baseline with a plug-and-play SDL module and FDL-based loss optimization.The SDL block is inserted among stacked residual blocks, while FDL adaptively constrains frequency distance.
- Overall Architecture: CASSI measurements are shifted back and mask-modulated before a 3 × 3 convolution extracts shallow features.The mask modulates the HSI before dispersion, and the measurement is processed as the network input.
- Spatial-Spectral Domain Learning: SDL combines HR spectral attention, HR spatial attention, and efficient feature fusion without feature downsampling or channel narrowing.Feature reshaping and matrix multiplication maintain high internal resolution, while grouped split-and-merge supports efficient fusion.
- Spatial-Spectral Domain Learning: HR spectral attention recalibrates input features through channel-wise weights while retaining half-channel spectral resolution.The attention uses 1×1 convolutions, reshaping, matrix multiplication, and element-wise channel recalibration.
- Spatial-Spectral Domain Learning: HR spatial attention preserves full spatial resolution, and EFF groups fused features for dynamic interaction using depthwise-separable convolutions.The grouped fusion replaces standard convolution to reduce computational cost while extracting high-resolution features.
- Frequency Domain Learning: FDL applies 2D DFT spectra per channel, measures frequency distance between prediction and truth, and dynamically weights harder-to-synthesize frequencies.Structural textures and edges correspond to high-frequency signals, while the weighting factor varies linearly with frequency distance.
4. Experiments
Experiments evaluate HDNet on simulated and real HSI reconstruction, comparing quantitative performance, visual quality, attention resolution, frequency supervision, and patch-based frequency refinement. HDNet achieves strong reconstruction results while retaining high-resolution details and improving frequency alignment.
- Experimental Settings: Experiments use simulated CAVE and KAIST datasets, with 28 interpolated wavelengths from 450nm to 650nm and a shared 256×256 mask.Real-data experiments use 28 channels, 450nm–650nm wavelengths, a real mask, and injected 11-bit shot noise; ground truth is unavailable for qualitative comparison.
- Comparison with Other Methods: HDNet significantly outperforms competing methods on KAIST across 10 scenes, surpassing DGSMP by 1.71 dB average PSNR and 0.0406 average SSIM.It also exceeds HSSP and DNU by 3.99 dB and 3.60 dB, and TwIST and DeSCI by 11.22 dB and 9.98 dB, respectively.
- Model Analysis: HDNet achieves the best trade-off between performance and parameters with 2.37M parameters, fewer than one-eighteenth of TSA-Net while maintaining the same model complexity.The comparison evaluates channel resolution, spatial resolution, parameter count, model complexity, and performance across attention networks.
- Attention Feature Visualization: High-resolution spatial-spectral attention retains more textures, edges, high-frequency information, and complete HSI structure than lower-resolution TSA attention.Feature visualizations show that TSA can lose textures and edges or focus on background regions, whereas HSA preserves continuous high-resolution attention.
- FDL Ablation: Frequency-level supervision improves frequency reconstruction: without FDL, spectra show ringing artifacts, while FDL yields lower LFD and frequency statistics closer to truth.Dynamic weighting emphasizes frequencies that are harder to synthesize, addressing amplitude and phase distortion and preserving fine-grained high-frequency information.
- Patch-based Frequency Spectrum: Patch subdivision improves frequency-gap reduction as patches become smaller, but reconstruction worsens when p > 3; HDNet therefore uses p = 3.The p = 4 visualization indicates that overly small patches provide limited spectrum representation and biased supervision.
5. Conclusion
HDNet combines high-resolution spatial-spectral learning with frequency-domain learning for HSI reconstruction from compressive measurements. Experiments show superior pixel-level and frequency-level reconstruction results.
- HDNet reconstructs HSIs from compressive measurements using spatial-spectral domain learning and frequency domain learning.
- Its HR spatial-spectral attention and efficient feature fusion produce fine-grained pixel-level predictions while preserving high internal resolution.
- Dynamically adjusted frequency-level supervision narrows the frequency discrepancy between reconstructed HSI and the truth.
- Extensive visual and quantitative experiments demonstrate superior pixel-level and frequency-level HSI reconstruction results.