PaperlayerView exact revision on arXiv

Source-linked AI summary

Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging

Yuanhao Cai, Jing Lin, Haoqian Wang, Xin Yuan, Henghui Ding, Yulun Zhang, Radu Timofte, Luc Van Gool

arXiv:2205.10102v3eess.IVcs.CV

TL;DR

HSI reconstruction methods for CASSI lack degradation-aware iteration control and have limited long-range modeling when mainly CNN-based. The paper proposes DAUF with an HST denoiser to form DAUHST, which outperforms state-of-the-art methods while using cheaper computational and memory costs. Its ablations report improvements from both components individually and jointly.

  • Problem

    Existing deep unfolding methods neither estimate CASSI degradation and ill-posedness for iterative adjustment nor adequately capture non-local self-similarity and long-range dependencies.

    Source: §1 · p. 2

  • Method

    DAUF estimates parameters from the compressed measurement and physical mask to control each iteration, while HST jointly extracts local contents and models non-local dependencies.

    Source: §1 · p. 2, §4 · p. 9

  • Results

    DAUHST outperforms state-of-the-art methods while requiring cheaper computational and memory costs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    DAUHST provides a Transformer-based deep unfolding approach for HSI reconstruction that combines degradation-aware iterative learning with local and non-local denoising.

    Source: §1 · p. 2, §4 · p. 9

    Model-based alternatives require manual parameter tweaking and have limited representation capacity and generalization ability.

    Source: §1 · p. 2

Abstract

from arXiv · show

In coded aperture snapshot spectral compressive imaging (CASSI) systems, hyperspectral image (HSI) reconstruction methods are employed to recover the spatial-spectral signal from a compressed measurement. Among these algorithms, deep unfolding methods demonstrate promising performance but suffer from two issues. Firstly, they do not estimate the degradation patterns and ill-posedness degree from the highly related CASSI to guide the iterative learning. Secondly, they are mainly CNN-based, showing limitations in capturing long-range dependencies. In this paper, we propose a principled Degradation-Aware Unfolding Framework (DAUF) that estimates parameters from the compressed image and physical mask, and then uses these parameters to control each iteration. Moreover, we customize a novel Half-Shuffle Transformer (HST) that simultaneously captures local contents and non-local dependencies. By plugging HST into DAUF, we establish the first Transformer-based deep unfolding method, Degradation-Aware Unfolding Half-Shuffle Transformer (DAUHST), for HSI reconstruction. Experiments show that DAUHST significantly surpasses state-of-the-art methods while requiring cheaper computational and memory costs. Code and models will be released at https://github.com/caiyuanhao1998/MST

1 Introduction

HSI reconstruction with CASSI seeks to recover detailed spatial-spectral information from compressed measurements, but existing deep unfolding methods do not adapt iterations to CASSI degradation or capture long-range dependencies. DAUHST addresses these gaps with degradation-aware unfolding and a Half-Shuffle Transformer, achieving stronger reported performance with lower computational and memory costs.

Motivation

  • Traditional HSI systems scan scenes along spectral or spatial dimensions, requiring long acquisition times and failing to capture dynamic objects.Snapshot compressive imaging systems were introduced to address this acquisition challenge.Source: §1 · p. 1

Limitations of Existing Methods

  • Existing deep unfolding methods do not estimate CASSI degradation patterns or ill-posedness to adjust linear projection and denoising in each iteration.Their iterative learning is highly related to the CASSI system, but this relationship is not explicitly used for iteration control.Source: §1 · p. 2
  • CNN-based deep unfolding methods have limitations in capturing non-local self-similarity and long-range dependencies important for HSI reconstruction.Transformers offer non-local interaction modeling but introduce computational and receptive-field challenges.Source: §1 · p. 2

Proposed Method

  • DAUF is a MAP-based unfolding framework that estimates informative parameters from compressed measurements and the physical mask to control iterative learning.The parameters capture CASSI degradation and ill-posedness cues, adaptively scaling linear projection and providing denoising-network noise information.Source: §1 · p. 2, Figure 2 · p. 3
  • HST combines local-content extraction with non-local dependency modeling, and plugging it into DAUF establishes DAUHST as a Transformer-based deep unfolding method.The paper identifies DAUHST as the first Transformer-based deep unfolding method for HSI restoration.Source: §1 · p. 2

Results

  • DAUHST outperforms state-of-the-art methods by a large margin while requiring cheaper computational and memory costs.The paper also reports more visually pleasant results in real HSI reconstruction.Source: §1 · p. 2

2 Proposed Method

DAUF unfolds CASSI reconstruction with iteration-specific, degradation-aware parameters, while DAUHST supplies a Transformer denoiser that combines local and non-local modeling at lower cost than global attention.

Degradation-Aware Unfolding Framework

  • HQS separates reconstruction into alternating linear projection and denoising subproblems, with µ controlling convergence and output across iterations.DAUF replaces manual tuning with iteration-specific parameters µk and τ-related denoising controls.Source: §2.2 · pp. 3–4
  • The linear projection is made efficient by exploiting the fat sensing matrix and diagonal structure of ΦΦT for element-wise updates.This avoids directly inverting the large matrix ΦTΦ + µI.Source: §2.2 · pp. 3–4
  • DAUF estimates parameters from the compressed measurement and sensing matrix to adapt each iteration to CASSI degradation patterns and ill-posedness.Its estimator feeds parameters into the linear projection and denoising stages.Source: §2.2 · p. 4

Half-Shuffle Transformer

  • DAUHST uses HST as the denoiser within DAUF, addressing CNN limitations while making self-attention computationally cheaper than global MSA.The design targets non-local self-similarity and long-range dependencies important for HSI reconstruction.Source: §2.3 · pp. 4–6
  • HST uses a three-level U-shaped architecture whose HSAB units combine layer normalization, HS-MSA, and FFN components.The denoiser predicts a residual image that is added to xk to produce zk.Source: §2.3 · pp. 4–5, Figure 3 · p. 5
  • HS-MSA splits channels between local and non-local branches, capturing local contents and long-range dependencies through windowed attention and token shuffling.The non-local branch establishes inter-window correlations without globally sampling all tokens.Source: §2.3 · pp. 5–6

3 Experiment

Experiments evaluate DAUHST on simulated and real HSI reconstruction, benchmark it against state-of-the-art methods, and analyze its components and estimated parameters. Results show strong reconstruction quality, computational efficiency, and benefits from DAUF and HS-MSA.

Quantitative Comparisons with State-of-the-Art Methods

  • DAUHST-9stg achieves 38.36 dB PSNR and 0.967 SSIM on 10 simulation scenes, outperforming BIRNAT and MST-L by 0.78 and 3.18 dB.The comparison includes 16 state-of-the-art methods under the same testing settings.Source: §3.2 · p. 7
  • DAUHST-2stg outperforms MST-L by 1.16 dB while using 68.9% of its Params and 65.5% of its FLOPS.Against CNN-based methods, DAUHST-3stg also surpasses HDNet, TSA-Net, and λ-Net by 2.24, 5.75, and 8.68 dB with lower costs.Source: §3.2 · p. 7

Qualitative Comparisons with State-of-the-Art Methods

  • DAUHST-9stg produces more detailed contents, cleaner textures, fewer artifacts, and spectral curves with the highest correlation and coincidence with reference curves.Compared methods can produce over-smooth structures, chromatic artifacts, or blotchy textures.Source: §3.3 · pp. 7–8
  • On real Scene 1 data with four spectra, only DAUHST-3stg clearly reconstructs the picked flower at all wavelengths.The real-data comparison includes nine state-of-the-art methods and training with simulated 11-bit shot noise.Source: §3.3 · p. 8, Figure 5 · p. 8

Ablation Study

  • Jointly applying DAUF and HS-MSA improves the baseline by 4.16 dB, compared with 2.32 dB from DAUF alone and 2.44 dB from HS-MSA alone.The baseline-1 model removes both HS-MSA and DAUF from DAUHST-3stg.Source: §3.4 · p. 8
  • HS-MSA improves baseline-2 by 1.26 dB, exceeding G-MSA, SW-MSA, and S-MSA by 0.42, 0.30, and 0.23 dB.Its advantage is attributed to jointly capturing local contents and non-local dependencies.Source: §3.4 · p. 8
  • DAUF surpasses DNU, ADMM, and GAP by 2.59, 1.69, and 1.63 dB while adding only 0.05M Params and 0.94G FLOPS.The comparison replaces each framework’s single-stage network with HST and uses a three-stage architecture.Source: §3.4 · p. 8
  • Jointly estimating α and β improves baseline-3 by 0.72 dB, while estimating α alone and feeding β into the denoiser contribute 0.45 and 0.34 dB.α adaptively scales linear projection, whereas β provides noise-level information to the denoising network as iterations proceed.Source: §3.4 · p. 9

4 Conclusion

The paper addresses missing degradation-aware control and limited long-range modeling in deep unfolding for HSI reconstruction. DAUF estimates CASSI-related parameters for iterative control, while HST combines local and non-local modeling; DAUHST achieves strong performance with lower computational and memory costs.

Conclusion

  • DAUF estimates parameters from the compressed measurement and physical mask to scale linear projection and provide denoising noise-level information.The framework is formulated using maximum a posteriori theory.Source: §4 · p. 9
  • HST jointly extracts local contents and models non-local dependencies, addressing limitations of CNN-based unfolding methods.Plugging HST into DAUF yields DAUHST, described as the first Transformer-based unfolding method for HSI reconstruction.Source: §4 · p. 9
  • DAUHST outperforms state-of-the-art methods by a large margin while requiring much cheaper memory and computational costs.The conclusion reports this pattern from comprehensive experiments without specifying a single aggregate metric.Source: §4 · p. 9

Related papers

Loading 2205.10102v3…