Source-linked AI summary
Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging
Yuanhao Cai, Jing Lin, Haoqian Wang, Xin Yuan, Henghui Ding, Yulun Zhang, Radu Timofte, Luc Van Gool
TL;DR
HSI reconstruction methods for CASSI lack degradation-aware iteration control and have limited long-range modeling when mainly CNN-based. The paper proposes DAUF with an HST denoiser to form DAUHST, which outperforms state-of-the-art methods while using cheaper computational and memory costs. Its ablations report improvements from both components individually and jointly.
Problem
Existing deep unfolding methods neither estimate CASSI degradation and ill-posedness for iterative adjustment nor adequately capture non-local self-similarity and long-range dependencies.
Method
DAUF estimates parameters from the compressed measurement and physical mask to control each iteration, while HST jointly extracts local contents and models non-local dependencies.
Results
DAUHST outperforms state-of-the-art methods while requiring cheaper computational and memory costs.
Takeaways & Limitations
DAUHST provides a Transformer-based deep unfolding approach for HSI reconstruction that combines degradation-aware iterative learning with local and non-local denoising.
Takeaways & Limitations
Model-based alternatives require manual parameter tweaking and have limited representation capacity and generalization ability.
Abstract
from arXiv · showhide
In coded aperture snapshot spectral compressive imaging (CASSI) systems, hyperspectral image (HSI) reconstruction methods are employed to recover the spatial-spectral signal from a compressed measurement. Among these algorithms, deep unfolding methods demonstrate promising performance but suffer from two issues. Firstly, they do not estimate the degradation patterns and ill-posedness degree from the highly related CASSI to guide the iterative learning. Secondly, they are mainly CNN-based, showing limitations in capturing long-range dependencies. In this paper, we propose a principled Degradation-Aware Unfolding Framework (DAUF) that estimates parameters from the compressed image and physical mask, and then uses these parameters to control each iteration. Moreover, we customize a novel Half-Shuffle Transformer (HST) that simultaneously captures local contents and non-local dependencies. By plugging HST into DAUF, we establish the first Transformer-based deep unfolding method, Degradation-Aware Unfolding Half-Shuffle Transformer (DAUHST), for HSI reconstruction. Experiments show that DAUHST significantly surpasses state-of-the-art methods while requiring cheaper computational and memory costs. Code and models will be released at https://github.com/caiyuanhao1998/MST
1 Introduction
HSI reconstruction with CASSI seeks to recover detailed spatial-spectral information from compressed measurements, but existing deep unfolding methods do not adapt iterations to CASSI degradation or capture long-range dependencies. DAUHST addresses these gaps with degradation-aware unfolding and a Half-Shuffle Transformer, achieving stronger reported performance with lower computational and memory costs.
- Motivation: Traditional HSI systems scan scenes along spectral or spatial dimensions, requiring long acquisition times and failing to capture dynamic objects.Snapshot compressive imaging systems were introduced to address this acquisition challenge.
- Limitations of Existing Methods: Existing deep unfolding methods do not estimate CASSI degradation patterns or ill-posedness to adjust linear projection and denoising in each iteration.Their iterative learning is highly related to the CASSI system, but this relationship is not explicitly used for iteration control.
- Limitations of Existing Methods: CNN-based deep unfolding methods have limitations in capturing non-local self-similarity and long-range dependencies important for HSI reconstruction.Transformers offer non-local interaction modeling but introduce computational and receptive-field challenges.
- Proposed Method: DAUF is a MAP-based unfolding framework that estimates informative parameters from compressed measurements and the physical mask to control iterative learning.The parameters capture CASSI degradation and ill-posedness cues, adaptively scaling linear projection and providing denoising-network noise information.
- Proposed Method: HST combines local-content extraction with non-local dependency modeling, and plugging it into DAUF establishes DAUHST as a Transformer-based deep unfolding method.The paper identifies DAUHST as the first Transformer-based deep unfolding method for HSI restoration.
- Results: DAUHST outperforms state-of-the-art methods by a large margin while requiring cheaper computational and memory costs.The paper also reports more visually pleasant results in real HSI reconstruction.
2 Proposed Method
DAUF unfolds CASSI reconstruction with iteration-specific, degradation-aware parameters, while DAUHST supplies a Transformer denoiser that combines local and non-local modeling at lower cost than global attention.
- Degradation-Aware Unfolding Framework: HQS separates reconstruction into alternating linear projection and denoising subproblems, with µ controlling convergence and output across iterations.DAUF replaces manual tuning with iteration-specific parameters µk and τ-related denoising controls.
- Degradation-Aware Unfolding Framework: The linear projection is made efficient by exploiting the fat sensing matrix and diagonal structure of ΦΦT for element-wise updates.This avoids directly inverting the large matrix ΦTΦ + µI.
- Degradation-Aware Unfolding Framework: DAUF estimates parameters from the compressed measurement and sensing matrix to adapt each iteration to CASSI degradation patterns and ill-posedness.Its estimator feeds parameters into the linear projection and denoising stages.
- Half-Shuffle Transformer: DAUHST uses HST as the denoiser within DAUF, addressing CNN limitations while making self-attention computationally cheaper than global MSA.The design targets non-local self-similarity and long-range dependencies important for HSI reconstruction.
- Half-Shuffle Transformer: HST uses a three-level U-shaped architecture whose HSAB units combine layer normalization, HS-MSA, and FFN components.The denoiser predicts a residual image that is added to xk to produce zk.
- Half-Shuffle Transformer: HS-MSA splits channels between local and non-local branches, capturing local contents and long-range dependencies through windowed attention and token shuffling.The non-local branch establishes inter-window correlations without globally sampling all tokens.
3 Experiment
Experiments evaluate DAUHST on simulated and real HSI reconstruction, benchmark it against state-of-the-art methods, and analyze its components and estimated parameters. Results show strong reconstruction quality, computational efficiency, and benefits from DAUF and HS-MSA.
- Quantitative Comparisons with State-of-the-Art Methods: DAUHST-9stg achieves 38.36 dB PSNR and 0.967 SSIM on 10 simulation scenes, outperforming BIRNAT and MST-L by 0.78 and 3.18 dB.The comparison includes 16 state-of-the-art methods under the same testing settings.
- Quantitative Comparisons with State-of-the-Art Methods: DAUHST-2stg outperforms MST-L by 1.16 dB while using 68.9% of its Params and 65.5% of its FLOPS.Against CNN-based methods, DAUHST-3stg also surpasses HDNet, TSA-Net, and λ-Net by 2.24, 5.75, and 8.68 dB with lower costs.
- Qualitative Comparisons with State-of-the-Art Methods: DAUHST-9stg produces more detailed contents, cleaner textures, fewer artifacts, and spectral curves with the highest correlation and coincidence with reference curves.Compared methods can produce over-smooth structures, chromatic artifacts, or blotchy textures.
- Qualitative Comparisons with State-of-the-Art Methods: On real Scene 1 data with four spectra, only DAUHST-3stg clearly reconstructs the picked flower at all wavelengths.The real-data comparison includes nine state-of-the-art methods and training with simulated 11-bit shot noise.
- Ablation Study: Jointly applying DAUF and HS-MSA improves the baseline by 4.16 dB, compared with 2.32 dB from DAUF alone and 2.44 dB from HS-MSA alone.The baseline-1 model removes both HS-MSA and DAUF from DAUHST-3stg.
- Ablation Study: HS-MSA improves baseline-2 by 1.26 dB, exceeding G-MSA, SW-MSA, and S-MSA by 0.42, 0.30, and 0.23 dB.Its advantage is attributed to jointly capturing local contents and non-local dependencies.
- Ablation Study: DAUF surpasses DNU, ADMM, and GAP by 2.59, 1.69, and 1.63 dB while adding only 0.05M Params and 0.94G FLOPS.The comparison replaces each framework’s single-stage network with HST and uses a three-stage architecture.
- Ablation Study: Jointly estimating α and β improves baseline-3 by 0.72 dB, while estimating α alone and feeding β into the denoiser contribute 0.45 and 0.34 dB.α adaptively scales linear projection, whereas β provides noise-level information to the denoising network as iterations proceed.
4 Conclusion
The paper addresses missing degradation-aware control and limited long-range modeling in deep unfolding for HSI reconstruction. DAUF estimates CASSI-related parameters for iterative control, while HST combines local and non-local modeling; DAUHST achieves strong performance with lower computational and memory costs.
- Conclusion: DAUF estimates parameters from the compressed measurement and physical mask to scale linear projection and provide denoising noise-level information.The framework is formulated using maximum a posteriori theory.
- Conclusion: HST jointly extracts local contents and models non-local dependencies, addressing limitations of CNN-based unfolding methods.Plugging HST into DAUF yields DAUHST, described as the first Transformer-based unfolding method for HSI reconstruction.
- Conclusion: DAUHST outperforms state-of-the-art methods by a large margin while requiring much cheaper memory and computational costs.The conclusion reports this pattern from comprehensive experiments without specifying a single aggregate metric.