PaperlayerView exact revision on arXiv

Source-linked AI summary

3D Quasi-Recurrent Neural Network for Hyperspectral Image Denoising

Kaixuan Wei, Ying Fu, Hua Huang

arXiv:2003.04547v1cs.CV

TL;DR

HSI denoising must handle acquisition noise while modeling structural spatio-spectral and global spectral correlations across inputs with varying band counts. QRNN3D combines 3D convolution, dynamically gated quasi-recurrent pooling, and alternating directionality to model these dependencies. Experiments report better denoising performance and faster speed across Gaussian and complex noise settings.

  • Problem

    Existing HSI denoising approaches trade model capability against spectral flexibility or insufficiently exploit global correlation along the spectrum.

  • Method

    QRNN3D uses 3D convolution for structural spatio-spectral correlation, quasi-recurrent pooling for global spectral correlation, and alternating directionality to remove causal dependency without extra computation.

  • Results

    QRNN3D outperforms leading-edge methods across benchmark datasets and varied Gaussian and complex noise settings, with better performance and faster speed.

  • Takeaways & Limitations

    The model can process HSIs with arbitrary numbers of bands and can be directly applied to remotely sensed imagery after pretraining on ICVL.

Abstract

from arXiv · show

In this paper, we propose an alternating directional 3D quasi-recurrent neural network for hyperspectral image (HSI) denoising, which can effectively embed the domain knowledge -- structural spatio-spectral correlation and global correlation along spectrum. Specifically, 3D convolution is utilized to extract structural spatio-spectral correlation in an HSI, while a quasi-recurrent pooling function is employed to capture the global correlation along spectrum. Moreover, alternating directional structure is introduced to eliminate the causal dependency with no additional computation cost. The proposed model is capable of modeling spatio-spectral dependency while preserving the flexibility towards HSIs with arbitrary number of bands. Extensive experiments on HSI denoising demonstrate significant improvement over state-of-the-arts under various noise settings, in terms of both restoration accuracy and computation time. Our code is available at https://github.com/Vandermode/QRNN3D.

I. INTRODUCTION

HSI denoising must address diverse acquisition noise while preserving structural spatio-spectral and global spectral correlations. QRNN3D combines 3D learning with quasi-recurrent pooling and alternating directionality to improve capability, flexibility, and denoising performance.

  • HSIs contain many wavebands but are degraded by Gaussian, stripe, deadline, and impulse noise during acquisition.These degradations negatively affect subsequent HSI processing, making denoising an essential preprocessing step.
  • Handcrafted low-rank tensor methods model HSI correlations effectively, but depend on how well their priors match intrinsic HSI characteristics.
  • Learned 2D convolutional approaches trade spectral flexibility against global-correlation modeling, requiring retraining or yielding lower performance.
  • QRU3D combines 3D convolution for local spatio-spectral features with dynamically gated quasi-recurrent pooling for global correlation along the spectrum.The pooling recurrently merges features across the whole spectrum, with weights computed from input features.
  • QRNN3D remains agnostic to spectral dimension and outperforms leading-edge methods across benchmark datasets and varied noise settings.
  • QRNN3D uses alternating directional structure to remove unidirectional causal dependency without additional computation cost.

B. Deep Learning for Image Denoising

Deep learning methods extend image-denoising architectures to HSI and sequence modeling, but spectral correlation and variable-length inputs remain central design concerns. QRNN3D addresses these concerns through elementwise recurrent processing and alternating-direction architecture.

  • Deep CNN encoder-decoder architectures have improved Gaussian denoising while reducing computation time in Gray/RGB image restoration.
  • Existing Gray/RGB denoising networks can extend to HSI, but do not specifically incorporate HSI domain knowledge.
  • Image-sequence models such as BRCN and ConvLSTM use weight-sharing convolutional recurrence to model temporal dependency at patch level.
  • Patch-based recurrent operations cannot efficiently capture spectral correlation, whereas QRNN3D uses elementwise recurrence that scales to HSIs with many bands.
  • The QRNN3D architecture uses symmetric QRU3D encoder-decoder layers, skip connections, and alternating directionality with bidirectional boundary layers.

III. THE PROPOSED METHOD

QRNN3D is a residual encoder-decoder for HSI denoising that combines 3D convolutions with quasi-recurrent processing. Its 3D design models spatial and spectral structure while accommodating inputs with arbitrary band counts.

  • Problem: HSI denoising is formulated as recovering the clean image X from noisy observation Y under additive noise.The noise term can represent Gaussian, sparse, or mixed noise.
  • Network architecture: QRNN3D uses six symmetric QRU3D pairs in an encoder-decoder network, totaling twelve layers.The encoder downsamples with stride-2 convolutions, while the decoder upsamples with stride-1/2 layers.
  • QRU3D: Each QRU3D combines a 3D convolutional subcomponent with quasi-recurrent pooling without enforcing a fixed number of spectral bands.This makes QRNN3D applicable to HSIs with arbitrary spectral dimensions.
  • 3D convolutional subcomponent: The 3D convolutional subcomponent applies separated filter banks to produce a candidate tensor Z and neural forget gate F.Z uses tanh activation, while F uses sigmoid activation; both preserve spatial and spectral dimensions.
  • 3D convolutional subcomponent: 3D convolution extracts structural spatio-spectral correlation by operating across both spatial and spectral dimensions.Spatial filtering can mimic patch-based operations, while spectral filtering models local spectral continuity.

Quasi-Recurrent Pooling.

Quasi-recurrent pooling merges spectral states using input-dependent forget gates, enabling global spectral correlation with efficient parallel computation. Alternating forward and backward directions removes one-way spectral dependence without the near-doubling cost of bidirectional layers.

  • Quasi-Recurrent Pooling: Quasi-recurrent pooling addresses the local nature of 3D convolution by explicitly exploiting global correlation along the spectrum.It combines pooling with a dynamic gating mechanism.
  • Quasi-Recurrent Pooling: The QRU3D splits candidate tensor Z and forget gate F along the spectrum before applying quasi-recurrent pooling to the resulting sequences.The output is formed by concatenating the hidden states across spectral bands.
  • Quasi-Recurrent Pooling: Each hidden state combines the previous state and current candidate according to the current forget gate: h_b = f_b ⊙ h_b−1 + (1 − f_b) ⊙ z_b.The gate balances current candidate information against previous spectral memory.
  • Quasi-Recurrent Pooling: Independent neural gates and elementwise recurrent operations make QRU3D highly parallel and scalable to HSIs with many bands.The forget gate depends on contiguous input bands rather than the previous hidden state used in typical RNNs.
  • Alternating Directional Structure: Alternating directional structure stacks forward and backward QRU3D layers in alternating directions to propagate global context across the spectrum.Unlike bidirectional layers, it keeps whole-spectrum dependency modeling with almost no additional computation cost.

IV. EXPERIMENTAL RESULTS

The experiments use simulated noise on ICVL and assess robustness across remotely sensed hyperspectral datasets with different spectral resolutions. Noise settings include Gaussian, stripe, deadline, impulse, and mixture corruptions.

  • Datasets: The ICVL dataset contains 201 images at 1392 × 1300 spatial resolution over 31 spectral bands.The data split uses 100 images for training, 5 for validation, and the remainder for testing.
  • Noise settings: The evaluation considers Gaussian, impulse, dead-pixel or line, and stripe noise as common forms of real-world HSI corruption.The paper defines five complex noise cases for experiments.
  • Noise settings: Case 1 uses non-i.i.d. Gaussian noise with zero-mean intensities randomly selected from 10 to 70 across all bands.Cases 2–4 combine this Gaussian noise with stripe, deadline, or impulse noise, while Case 5 mixes at least one noise type from Cases 1–4.

Competing Methods.

The paper compares QRNN3D with traditional and deep-learning baselines under Gaussian and complex noise, using incremental training from easier Gaussian denoising to harder complex-noise removal.

  • Baseline selection: Traditional baselines are selected according to their noise assumptions, while deep-learning methods are compared across Gaussian and complex noise settings.Gaussian comparisons include BM4D, TDL, ITSReg, and LLRT; complex-noise comparisons include LRMR, LRTV, NMoG, and TDTV.
  • Deep-learning competitors: The deep-learning comparison includes HSID-CNN and MemNet, with MemNet modified to accept 31 spectral bands and therefore requiring a fixed spectral dimension.Competing pretrained models are fine-tuned or retrained using the paper’s training strategy.
  • Training policy: Incremental training proceeds from fixed-level Gaussian denoising, through unknown Gaussian noise, to complex noise, reusing pretrained parameters between stages.The stages use σ = 50, σ uniformly sampled from 30 to 70, and complex noise from Cases 1–4; models from epochs 50 and 100 serve the Gaussian and complex tasks.
  • Training policy: QRNN3D uses two models, one for Gaussian denoising and one for complex-noise removal, rather than independently training networks for every noise type.The complex-noise model handles cases 1–5 simultaneously at testing time.

Quantitative Metrics.

The experiments evaluate restoration quality with spatial and spectral metrics, showing strong Gaussian and complex-noise performance for QRNN3D while using a single model across varied corruption levels or cases.

  • Evaluation metrics: PSNR, SSIM, and SAM measure spatial and spectral restoration quality, with larger PSNR and SSIM and smaller SAM indicating better performance.PSNR and SSIM are spatial-based indices, whereas SAM is spectral-based.
  • Gaussian noise case: At σ = 50 on ICVL, QRNN3D removes Gaussian noise while preserving HSI structure, whereas BM4D and TDL introduce visible artifacts.The model trained at stage 2 is applied to all corruption levels rather than using separate networks for each intensity.
  • Gaussian noise case: QRNN3D achieves better performance in most qualitative and quantitative assessments against the competing methods in the Gaussian-noise experiments.The passage characterizes this as confirming the method’s high-fidelity restoration.
  • Complex noise case: QRNN3D significantly outperforms other methods across five complex-noise cases, including a mixture-noise case absent from training.A single stage-3 model handles Cases 1–5 simultaneously at evaluation time.
  • Complex noise case: Across spectral bands, QRNN3D obtains obviously higher PSNR values than the compared methods in the complex-noise experiments.The comparison is presented through the per-band PSNR curves in Figure 8.

C. Experiments on Remotely Sensed Images

Experiments on remotely sensed datasets test QRNN3D’s flexibility across spectral dimensions and its transfer from ICVL to Pavia University, while also examining mixture-noise restoration.

  • Pavia University: On Pavia University mixture noise, the model trained from scratch on Pavia Centre performs undesirably, scoring 29.64 versus TDTV’s 30.06.This comparison is reported for Ours-S.
  • Pavia University: The ICVL-pretrained Ours-P model is directly applied to Pavia University despite its distinct spectral curve and performs better than all compared methods.The result tests transfer across datasets with different spectral characteristics.
  • Model flexibility: MemNet results are unavailable for Pavia University because its learned model cannot handle data with a different number of bands.HSID-CNN uses a pretrained ICVL model in this comparison.
  • Pavia University: Fine-tuning the ICVL-pretrained model on small Pavia Centre samples significantly boosts performance, and Ours-F combines cleaner results than the scratch and pretrained variants.The visual comparison reports residual Gaussian-like noise in Ours-S and stripes in Ours-P.

Real-world Noisy Data.

The paper analyzes QRNN3D’s real-world denoising, training strategy, and component trade-offs, including its behavior on unknown noise without corresponding ground truth.

  • Real-world noisy data: On real-world Indian Pines and Urban HSIs without ground truth, QRNN3D produces sharper and clearer results than competing methods under unknown noise.Atmospheric effects and water absorption severely degrade these images, while Gaussian methods cannot accurately estimate the clean image from non-Gaussian noise.
  • Component analysis: The analysis examines 3D convolution, quasi-recurrent pooling, and alternating-directional structure as the main QRNN3D components.These components are studied to clarify their functionality in HSI modeling and domain-knowledge embedding.
  • Training analysis: Incremental training reuses prior learned parameters and significantly stabilizes and accelerates optimization compared with training from scratch.Figure 12 compares average training loss and validation PSNR for the two training strategies in complex-noise removal.
  • Component analysis: Ablation experiments on ICVL Gaussian denoising evaluate QRNN3D components using PSNR, running time, and parameter count.The study targets the trade-off between denoising performance and computational burden.

Subcomponents Investigation.

The investigation shows that QRU3D’s 3D convolution and quasi-recurrent pooling are both important for HSI modeling, while alternating directionality provides bidirectional spectral dependency with lower memory than full bidirectionality. Hyperparameters were selected using a compact grid search, while the authors note that further tuning could improve performance.

  • 3D convolution: Removing 3D convolution causes a 1.6 dB performance loss, indicating that structural spectral correlation is important for QRU3D capacity.The QRU2D variant replaces 3D convolution with 2D convolution.
  • 3D convolution: QRU3D outperforms the parameter-matched WQRU2D with less computation cost, suggesting greater efficiency than the wider 2D approach.The comparison targets HSI modeling under comparable parameter counts.
  • Quasi-recurrent pooling: Removing quasi-recurrent pooling degrades performance by 3.4 dB, showing that the C3D variant lacks an effective mechanism for modeling GCS.C3D removes quasi-recurrent pooling and its associated neural gates.
  • Quasi-recurrent pooling: QRU3D reaches 40.23 dB versus 40.00 dB for WC3D, while using approximately 50% of its parameters and 80% of its running time.The wider WC3D model has four times as many parameters as C3D.
  • Directional structures: Alternating directional and bidirectional architectures achieve similar performance, 40.26 versus 40.23, while bidirectionality requires a larger memory footprint than alternating directionality.Both architectures significantly exceed the unidirectional version.
  • Hyperparameter selection: The network depth and width were selected through a small Gaussian-denoising grid search to balance PSNR, running time, and parameter count.The search used ICVL HSI data with noise level σ = 50.
  • Scope: The proposed building block is intended to be compact and reusable in network topologies beyond the encoder-decoder architecture used in this paper.The authors did not pursue exhaustive configuration searches, and further parameter tuning may improve performance.

D. Visualizing GCS Knowledge

The paper visualizes global correlation along the spectrum by measuring each input band’s contribution to each output band. The analysis shows broad spectral influence, including bands that are strongly correlated with nearly the entire spectrum.

  • Visualization method: The captured GCS for each QRU3D layer can be calculated through a single inference pass after unfolding Equation (3).The procedure uses Equation (5) to quantify band effects.
  • GCS definition: GCS_ij measures band i’s contribution to output h_j under a Frobenius-norm measure.The contribution is computed from Φ_j(z_i) divided element-wise by h_j.
  • Visualization results: Figure 13 shows that each band’s output can be highly affected by the whole spectrum in a randomly selected HSI.The figure visualizes captured GCS in a bidirectional QRU3D layer.
  • Visualization results: Bands 15 through 17 are deeply correlated with almost all bands in the analyzed HSI.Figure 13 summarizes relative-band statistics across testing images on ICVL.
  • Conclusion: The conclusion reports that visualized GCS results experimentally support effective exploitation of global spectral correlation by QRU3D.The authors suggest investigating QRU3D for other image-sequence modeling tasks.
Loading 2003.04547v1…