PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatial-Spectral Transformer for Hyperspectral Image Denoising

Miaoyu Li, Ying Fu, Yulun Zhang

arXiv:2211.14090v1cs.CVeess.IV

TL;DR

HSI sensing noise can harm downstream applications, while CNN-based denoisers face a trade-off between efficiency and modeling non-local HSI characteristics. The paper proposes a Spatial-Spectral Transformer with non-local spatial and global spectral self-attention. It reports better quantitative and visual results than state-of-the-art HSI denoising methods.

  • Problem

    HSI denoising is needed because sensing noise can harm downstream applications, while CNN-based methods face a trade-off between computational efficiency and modeling non-local HSI characteristics.

    Source: Introduction · p. 1

  • Method

    The Spatial-Spectral Transformer combines shifted-window non-local spatial self-attention with global spectral self-attention to exploit spatial similarity and band correlations.

    Source: Introduction · p. 1, Spatial-Spectral Transformer · p. 3

  • Results

    The method outperforms state-of-the-art HSI denoising methods in objective metrics and subjective visual quality across various noise degradations.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    Experiments verify the method’s superiority over state-of-the-art methods under various noise degradations, quantitatively and visually.

    Source: Conclusion · p. 7

    The authors identify existing CNN-based methods as underestimating spectral similarities and using fixed, stereotyped filters that lack adaptability.

    Source: HSI Denoising · p. 2

Abstract

from arXiv · show

Hyperspectral image (HSI) denoising is a crucial preprocessing procedure for the subsequent HSI applications. Unfortunately, though witnessing the development of deep learning in HSI denoising area, existing convolution-based methods face the trade-off between computational efficiency and capability to model non-local characteristics of HSI. In this paper, we propose a Spatial-Spectral Transformer (SST) to alleviate this problem. To fully explore intrinsic similarity characteristics in both spatial dimension and spectral dimension, we conduct non-local spatial self-attention and global spectral self-attention with Transformer architecture. The window-based spatial self-attention focuses on the spatial similarity beyond the neighboring region. While, spectral self-attention exploits the long-range dependencies between highly correlative bands. Experimental results show that our proposed method outperforms the state-of-the-art HSI denoising methods in quantitative quality and visual results.

Introduction

Noise from HSI sensing can harm downstream applications, making denoising an important preprocessing step. The paper proposes a Spatial-Spectral Transformer that combines spatial and spectral self-attention, and reports better quantitative and visual results than state-of-the-art methods.

Introduction

  • HSI sensing noise negatively influences downstream applications, so denoising is a critical preprocessing procedure.The passage attributes corruption and noise to limited light, photon effects, and atmospheric interference.Source: Introduction · p. 1
  • CNN-based HSI denoising faces a trade-off between computational efficiency and modeling non-local similarity, while fixed filters do not fully exploit target-image characteristics.Source: Introduction · p. 1
  • Existing Transformer-based approaches focus on spatial similarity but neglect HSI spectral correlations, which the paper says can negatively affect denoising.Source: Introduction · p. 1
  • The Spatial-Spectral Transformer combines shifted-window spatial self-attention for non-local spatial features with spectral self-attention for global band correlations.Its output is passed through an MLP and skip connection for smooth convergence.Source: Introduction · pp. 1–2
  • The proposed method outperforms state-of-the-art HSI denoising methods in objective metrics and subjective visual quality.The reported comparison covers various noise degradations.Source: Introduction · p. 2

Related Work

The related-work discussion reviews model-based and learning-based HSI denoising alongside vision Transformers. It identifies limitations in spectral-feature use and adaptability as motivation for the paper’s approach.

Related Work

  • Model-based HSI denoising uses hand-crafted spatial and spectral priors, while learning-based methods use deep CNNs to learn priors from large datasets.Model-based methods generally formulate denoising as a complex iterative problem that takes a long time to optimize.Source: HSI Denoising · p. 2

HSI Denoising

  • CNN-based HSI denoisers underestimate spectral similarities, and fixed convolution filters lack adaptability to noisy HSI characteristics.Source: HSI Denoising · p. 2

Vision Transformers

  • Vision Transformers mainly exploit spatial similarity and lack exploration of spectral correlation, potentially limiting their use of HSI features.Source: Vision Transformers · p. 2

Method

The method combines non-local spatial attention with global spectral attention to exploit HSI similarities across both dimensions. Its shifted-window spatial operation captures information beyond neighboring pixels, while spectral attention models correlations across bands.

Motivation and Formulation

  • The additive-noise formulation treats the observed HSI as the clean HSI corrupted by noise, with height, width, and spectral resolution defining its dimensions.The denoising goal is to restore the clean HSI from the noisy observation.Source: Motivation and Formulation · p. 2
  • SST combines shifted-window non-local spatial attention and global spectral attention to exploit spatial similarity and spectral correlation.The method applies spectral attention after spatial attention; shifted windows also support interactions between local windows.Source: Motivation and Formulation · p. 3, Spatial-Spectral Transformer · pp. 3–4

Spatial-Spectral Transformer

  • Each SSTL applies SSMA followed by an MLP with residual connections and LayerNorm; the SSMA output is the input to the SSTL.The SSMA module contains NLSA and GSA, and its output contains more spectral detail without losing critical spatial information.Source: Spatial-Spectral Transformer · pp. 3–4
  • Shifted-window NLSA captures spatial dependencies beyond neighboring pixels with linear complexity, avoiding the quadratic spatial-size cost of global spatial attention.The window shift enables interactions between local windows.Source: Spatial-Spectral Transformer · pp. 3–4
  • The network extracts shallow features with a 3×3 convolution, processes them through T RSSB layers, then uses convolutional skip connections and a global input skip connection to reconstruct the output.Each RSSB contains six SSTLs and ends with a 3×3 convolution.Source: Spatial-Spectral Transformer · p. 4

Experiments

Experiments compare SST on simulated ICVL noise patterns and real Urban data, and examine attention components and spatial-window size through ablations.

Experiments

Simulated Experiments

  • On ICVL, SST significantly outperforms compared methods under known-variance Gaussian noise, gains more than 0.7 dB PSNR at σ=70, and leads across five complex-noise cases.The experiments also report stronger low-noise results; the visual comparison at σ=50 favors SST's restored spatial and spectral quality.Source: Simulated Experiments · pp. 5–6, Table 1 · p. 5, Figure 3 · p. 5
  • In feature-map comparisons, GSA outputs contain more detailed information and structural texture than NLSA outputs.The visualization includes grayscale feature maps from different Transformer stages.Source: Simulated Experiments · p. 6, Figure 5 · p. 6
  • At comparable computation cost, SST achieves better results than the compared models, and its module is reported as more efficient than NLSA-NLSA and GSA-GSA alternatives.Table 3 compares model complexity; HSID-CNN GFLOPs account for its band-by-band processing.Source: Simulated Experiments · p. 7, Table 3 · p. 7

Real Data Experiments

  • On Urban real data, SST restores image texture well, while NGMeet oversmooths and QRNN3D and T3SC leave vertical stripes.Because Urban has no clean image, the real-data comparison is visual only.Source: Real Data Experiments · p. 7, Figure 6 · p. 7

Ablation Study

  • Removing either NLSA or GSA lowers PSNR by 0.5 dB; the combined module performs best at close computation cost, and the selected window size is M=8.Larger windows raise performance but also computation cost; GSA-GSA is worst among the last four compared methods.Source: Ablation Study · p. 7, Table 4 · p. 7, Table 5 · p. 7

Conclusion

The conclusion presents SST as an HSI denoiser combining spatial non-local and spectral global self-attention, and reports quantitative and visual superiority over state-of-the-art methods under varied noise degradations.

  • SST combines spatial non-local self-attention for coarse features with spectral self-attention to enrich representations with detail.Source: Conclusion · p. 7
  • The authors report that SST outperforms state-of-the-art methods quantitatively and visually under various noise degradations.Source: Conclusion · p. 7
  • The authors identify noise estimation as guidance for future work to boost Transformer denoising performance.Source: Conclusion · p. 7
Loading 2211.14090v1…