PaperlayerView exact revision on arXiv

Source-linked AI summary

Spectral Enhanced Rectangle Transformer for Hyperspectral Image Denoising

Miaoyu Li, Ji Liu, Ying Fu, Yulun Zhang, Dejing Dou

arXiv:2304.00844v1cs.CVeess.IV

TL;DR

HSI denoising is important, but existing methods have limitations in capturing non-local self-similarity and jointly modeling spatial and spectral correlations. SERT combines rectangle self-attention with spectral enhancement to address these gaps. The authors report effectiveness on synthetic and real noisy HSIs in objective metrics and visual quality.

  • Problem

    Existing HSI denoising methods have limitations in capturing non-local self-similarity and jointly considering spatial and spectral correlations, despite denoising's importance to HSI applications.

    Source: §1 · pp. 1–2

  • Method

    SERT combines multi-shape rectangle self-attention for non-local spatial similarity with spectral enhancement that uses global spectral memory to extract low-rank vectors and suppress noise.

    Source: §1 · p. 2

  • Results

    The authors report that SERT significantly outperforms state-of-the-art methods on simulated data and real noisy HSIs, with quantitative and qualitative experiments supporting its effectiveness.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    SERT jointly uses spatial non-local similarity and global spectral low-rank properties for HSI denoising, and the authors report effectiveness in objective metrics and visual quality.

    Source: §1 · p. 2, §5 · p. 8

    The stated degradation model treats noise as additive random noise, while realistic HSI degradation may include Gaussian, stripe, deadline, impulse, or mixed noise.

    Source: §3 · p. 3

Abstract

from arXiv · show

Denoising is a crucial step for hyperspectral image (HSI) applications. Though witnessing the great power of deep learning, existing HSI denoising methods suffer from limitations in capturing the non-local self-similarity. Transformers have shown potential in capturing long-range dependencies, but few attempts have been made with specifically designed Transformer to model the spatial and spectral correlation in HSIs. In this paper, we address these issues by proposing a spectral enhanced rectangle Transformer, driving it to explore the non-local spatial similarity and global spectral low-rank property of HSIs. For the former, we exploit the rectangle self-attention horizontally and vertically to capture the non-local similarity in the spatial domain. For the latter, we design a spectral enhancement module that is capable of extracting global underlying low-rank property of spatial-spectral cubes to suppress noise, while enabling the interactions among non-overlapping spatial rectangles. Extensive experiments have been conducted on both synthetic noisy HSIs and real noisy HSIs, showing the effectiveness of our proposed method in terms of both objective metric and subjective visual quality. The code is available at https://github.com/MyuLi/SERT.

1. Introduction

HSI denoising matters because noise degrades both visual quality and downstream applications. Existing approaches have difficulty modeling spatial non-local similarity and global spectral low-rank structure together; SERT is designed to address these gaps.

Motivation

  • HSI noise degrades visual quality and can negatively affect downstream applications, making denoising a fundamental step in HSI analysis and processing.HSIs offer spectral information that can distinguish materials in greater detail than RGB images.Source: §1 · p. 1

Existing limitations

  • Model-based methods use handcrafted spatial and spectral priors, but depend on prior accuracy and are generally time-consuming with limited generalization across scenarios.Common priors include total variation, non-local similarity, low-rank structure, and sparsity.Source: §1 · p. 1
  • Deep HSI denoising methods often use CNNs whose local filter responses separate noise and signal within a limited receptive field.Transformers can model long-range dependencies, but efficient non-local spatial attention remains challenging for large HSI feature maps.Source: §1 · p. 1
  • Existing HSI methods use low-rank characteristics through matrix factorization on a single image, while global low-rank structure across large datasets is hardly considered.The paper argues spatial non-local similarity and low-rank spectral statistics should be jointly used for denoising.Source: §1 · p. 2

Contributions

  • SERT jointly targets non-local spatial similarity and global spectral low-rank properties, using multi-shape rectangle self-attention and a spectral enhancement module with memory blocks.The spectral enhancement module extracts informative low-rank vectors from HSI cube patches to suppress noise.Source: §1 · p. 2

2. Related Works

Prior HSI denoising work includes model-based methods, deep networks, and Transformer approaches, but the paper identifies a need to model spatial and spectral similarity together. SERT is introduced as a task-oriented Transformer for that purpose.

2.1. Hyperspectral Image Denoising

  • Model-based HSI denoising commonly formulates noise removal as iterative optimization with handcrafted priors, including spatial regularization and low-rank methods.Prior work also includes adaptive spatial-spectral dictionaries and methods integrating spatial non-local similarity with global spectral low-rank structure.Source: §2.1 · p. 2
  • Deep HSI denoising methods include convolutional, recurrent, and model-guided networks, while convolution-based approaches are described as having limited receptive fields and fixed feature extraction.The paper distinguishes its Transformer approach as a way to model inner spatial and spectral similarity.Source: §2.1 · p. 2

2.2. Vision Transformer

  • Vision Transformers use varied attention designs to balance spatial receptive field and computation, but existing approaches are described as inefficient for HSI non-local similarity or as rarely combining spatial and spectral domains.The paper contrasts its task-oriented SERT with methods that directly transfer existing Transformer blocks to HSI tasks.Source: §2.2 · p. 2
  • SERT is framed around HSI spatial non-local similarity and global low-rank properties, with rectangle self-attention and spectral enhancement as its two components.The figure caption identifies RA and SE as the spatial rectangle self-attention and spectral enhancement modules.Source: §2.2 · p. 3, Figure 1 · p. 3

3. Spectral Enhanced Rectangle Transformer

SERT combines spatial rectangle self-attention with spectral enhancement to denoise HSIs. Its attention uses horizontal and vertical rectangles, while spectral enhancement uses low-rank vectors and memory to capture spectral statistics and connect spatial rectangles.

3. Spectral Enhanced Rectangle Transformer

  • Each Transformer block adds rectangle self-attention and spectral enhancement outputs to form comprehensive embeddings for noise removal.A Residual Transformer Layer contains six Transformer blocks.Source: §3 · p. 3

3.1. Spatial Rectangle Self-Attention

  • Rectangle attention targets non-local spatial similarity that convolutional methods limit to local information, using rectangles of different shapes to explore spatial regions and scales.The paper motivates this design by noting that distant pixels are less similar to a center pixel than neighboring pixels.Source: §3.1 · p. 3, Figure 2 · p. 4
  • The spatial attention splits features into two spectral parts and applies horizontal and vertical rectangle multi-head self-attention separately.The branches focus on different regions; spectral shuffle exchanges their information and enlarges the module's receptive field.Source: §3.1 · pp. 3–4

3.2. Spectral Enhancement

  • Spectral enhancement partitions features into cube patches, downsamples each patch spatially, and projects its aggregated spectral vector into a rank-K subspace.Patch size matches the long side of the rectangle, and shifting in the spatial domain connects adjacent cube patches.Source: §3.2 · p. 4
  • A learned global memory bank stores low-rank spectral vectors, whose most relevant statistics guide rescaling of the input cube patch to enhance spatial-spectral correlation and suppress noise.The module uses patch-level spectral statistics because neighboring pixels tend to share similar spectral statistics.Source: §3.2 · pp. 4–5

4. Experiments

Experiments span synthetic and real noisy HSIs, Transformer comparisons, and model analysis; SERT leads the reported denoising comparisons, while component studies examine accuracy and design choices.

4.1. Experiments on Synthetic Data

  • Synthetic evaluation uses ICVL, with i.i.d. Gaussian noise levels 10–70 and complex-noise models trained on mixtures before testing each noise case.ICVL is split into training, validation, and test images, with test crops used to limit computation for traditional methods.Source: §4.1 · pp. 5–6
  • SERT beats other deep-learning methods by at least 0.9 dB across tested noise cases, with visual comparisons showing fewer artifacts and better texture recovery.Synthetic tests include Gaussian noise and five complex-noise cases; models are trained on mixed noise for complex-noise evaluation.Source: §4.1 · pp. 5–6, Figure 3 · p. 6

4.2. Experiments on Real Noisy Data

  • On Realistic data, SERT exceeds competing HSI denoisers by almost 0.5 dB, while visual comparisons report cleaner Urban outputs and the most promising Realistic-scene result.Urban has no clean HSI and uses APEX pretraining; the Realistic dataset provides paired noisy and clean HSIs.Source: §4.2 · pp. 6–7, Figure 4 · p. 6, Table 3 · p. 7

4.3. Comparison with other Transformers

  • SERT achieves the best results among four Transformer methods on ICVL, supporting the suitability of its joint spatial-spectral block for HSI denoising.The comparison includes RGB-image Transformers and TRQ3D, an HSI denoising Transformer.Source: §4.3 · p. 7, Table 4 · p. 7

4.4. Model Analysis

  • Spectral enhancement improves denoising performance by 0.42 dB; spectral shuffle adds a slight gain, and the memory unit adds 0.18 dB in PSNR.The full model is reported to have competitive computation cost and inference time alongside better performance.Source: §4.4 · pp. 7–8, Table 6 · p. 8, Table 5 · p. 8
  • Non-local SE performs best, global SE slightly decreases performance, and road patches 7–9 yield similar low-rank vectors; longer rectangles are not necessarily better.The non-local design projects neighboring rectangles into one vector, while the rectangle-length comparison varies lengths across three Transformer layers.Source: §4.4 · p. 8, Figure 6 · p. 8, Figure 7 · p. 8

5. Conclusion

The spectral enhanced rectangle Transformer combines spatial non-local similarity modeling with spectral low-rank modeling for HSI denoising. Experiments on synthetic and real noisy HSIs show it outperforms competing methods.

5. Conclusion

  • The method significantly outperforms competing methods on synthetic and real noisy HSIs in quantitative and qualitative experiments.Source: §5 · p. 8
  • Multi-shape rectangle self-attention captures spatial non-local similarity with computation efficiency.Source: §5 · p. 8
  • A spectral enhancement module with a learnable memory unit explores global spectral low-rank properties while enabling cross-rectangle interactions, preserving informative characteristics and suppressing noise.Source: §5 · p. 8
  • The authors plan to extend the method to other HSI restoration tasks.Source: §5 · p. 8

Related papers

Loading 2304.00844v1…