PaperlayerView exact revision on arXiv

Source-linked AI summary

SpectralDiff: A Generative Framework for Hyperspectral Image Classification with Diffusion Models

Ning Chen, Jun Yue, Leyuan Fang, Shaobo Xia

arXiv:2304.05961v2cs.CV

TL;DR

HSI classifiers must handle high-dimensional, redundant data while capturing spectral-spatial distributions and relationships across samples. SpectralDiff uses spectral-spatial diffusion and attention-based classification, and experiments on three public datasets show better performance than state-of-the-art methods.

  • Problem

    Existing HSI classification methods have limited ability to handle high-dimensional, highly redundant, complex data and capture spectral-spatial distributions and relationships between samples.

    Source: §I · pp. 1–2

  • Method

    SpectralDiff uses forward and reverse spectral-spatial diffusion to extract contextual features and construct sample relationships, then classifies pixels with an attention-based module.

    Source: §I · p. 2

  • Results

    Experiments on three public HSI datasets show that SpectralDiff outperforms state-of-the-art techniques.

    Source: §I · p. 2, §V · p. 13

  • Takeaways & Limitations

    SpectralDiff adaptively constructs sample relationships and captures HSI data distributions and contextual information without prior graph-structure or neighborhood knowledge.

    Source: §I · p. 2, §V · p. 13

Abstract

from arXiv · show

Hyperspectral Image (HSI) classification is an important issue in remote sensing field with extensive applications in earth science. In recent years, a large number of deep learning-based HSI classification methods have been proposed. However, existing methods have limited ability to handle high-dimensional, highly redundant, and complex data, making it challenging to capture the spectral-spatial distributions of data and relationships between samples. To address this issue, we propose a generative framework for HSI classification with diffusion models (SpectralDiff) that effectively mines the distribution information of high-dimensional and highly redundant data by iteratively denoising and explicitly constructing the data generation process, thus better reflecting the relationships between samples. The framework consists of a spectral-spatial diffusion module, and an attention-based classification module. The spectral-spatial diffusion module adopts forward and reverse spectral-spatial diffusion processes to achieve adaptive construction of sample relationships without requiring prior knowledge of graphical structure or neighborhood information. It captures spectral-spatial distribution and contextual information of objects in HSI and mines unsupervised spectral-spatial diffusion features within the reverse diffusion process. Finally, these features are fed into the attention-based classification module for per-pixel classification. The diffusion features can facilitate cross-sample perception via reconstruction distribution, leading to improved classification performance. Experiments on three public HSI datasets demonstrate that the proposed method can achieve better performance than state-of-the-art methods. For the sake of reproducibility, the source code of SpectralDiff will be publicly available at https://github.com/chenning0115/SpectralDiff.

I. INTRODUCTION

HSI classification uses pixel-level spectral and spatial information, but existing methods struggle to model spectral-spatial relationships across samples. SpectralDiff uses diffusion to construct those relationships and extract contextual features without predefined graph or neighborhood information.

I. INTRODUCTION

  • HSI classification assigns pixels to land-cover classes, while high dimensionality and spatial heterogeneity and homogeneity complicate effective feature extraction.Spectral-only features do not fully use HSI characteristics, motivating joint spectral-spatial methods.Source: §I · p. 1
  • Existing DNN methods have limited ability to model spectral-spatial relationships across samples, while GNN approaches require graph or neighborhood design and capture data distributions and context ineffectively.The paper identifies graph design as adding complexity and subjectivity.Source: §I · p. 2
  • SpectralDiff combines forward and reverse spectral-spatial diffusion with attention-based per-pixel classification, using iterative denoising to derive contextual features and sample relationships.The framework models Gaussian noise addition and removal, then feeds extracted diffusion features to the classification module.Source: §I · p. 2
  • The paper presents a generative perspective for acquiring spectral-spatial features and reports that experiments show superiority over existing state-of-the-art techniques.It also identifies this as, to the authors’ knowledge, the first application of diffusion models to HSI classification.Source: §I · p. 2

II. RELATED WORK

The related work reviews HSI classification methods and diffusion models, placing SpectralDiff among approaches to spectral-spatial feature extraction and cross-sample relationship modeling.

II. RELATED WORK

  • The section reviews traditional spectral-spatial feature extraction and deep-learning methods alongside diffusion-model background to contextualize SpectralDiff.The authors also outline diffusion-model developments and challenges.Source: §II · p. 2

A. Hyperspectral Image Classification

  • HSI classification assigns pixels to land-cover classes, and spectral-only feature extraction omits spatial relationships; joint spectral-spatial methods address limitations of using spectral features alone.The review covers statistical, nonlinear, and spatial-structure feature extraction approaches.Source: A. Hyperspectral Image Classification · pp. 2–3
  • Deep-learning methods learn spectral-spatial features, but global sample relationships remain challenging; GNN approaches incur high computation and memory costs and slow gradient decline.The paper proposes diffusion models for global relationship modeling and reports comparative experiments to test the approach.Source: A. Hyperspectral Image Classification · p. 3

B. Diffusion Models

  • Diffusion models add Gaussian noise over forward time steps and learn a reverse process that denoises the data; their learned representations can capture patterns and relationships between samples.The review situates SpectralDiff as applying hierarchical denoising to multichannel HSI data.Source: B. Diffusion Models · p. 3

III. METHODOLOGY

SpectralDiff learns sample relationships by training a denoising network across forward and reverse spectral-spatial diffusion. The forward process progressively adds Gaussian noise to local hyperspectral instances according to a variance schedule.

III. METHODOLOGY

  • Training the spectral-spatial denoising network to estimate forward-added noise establishes relationships among HSI samples.Source: §III · p. 3

A. Spectral-Spatial Diffusion Module

1) Forward Spectral-Spatial Diffusion Process:

  • Each K × K hyperspectral neighborhood follows a Markov forward process, with Gaussian noise added progressively and each step conditioned on the previous noisy instance.The variance schedule αt controls the noise magnitude; the noisy instance at timestep t can be generated from the timestep, schedule, and sampled Gaussian noise.Source: 1) Forward Spectral-Spatial Diffusion… · pp. 4–5

2) Reverse Spectral-Spatial Diffusion Process:

The reverse process uses a denoising network to estimate the added noise and progressively recover the preceding hyperspectral instance.

2) Reverse Spectral-Spatial Diffusion Process:

B. Loss Function of Spectral-Spatial Diffusion Process

Training samples noisy hyperspectral instances at random timesteps and trains the diffusion network to predict the noise added to them.

B. Loss Function of Spectral-Spatial Diffusion Process

C. Structure of the Spectral-Spatial Denoising Network

The spectral-spatial denoising network uses a U-Net-like encoder-decoder to predict noise from a timestep and noisy HSI patch.

C. Structure of the Spectral-Spatial Denoising Network

  • The network receives timestep t and noisy HSI patch HIt, then predicts the added Gaussian noise.Source: C. Structure of the Spectral-Spatial… · p. 6
  • Three-dimensional convolutions encode spectral and spatial features through three down-sampling blocks and decode them through three up-sampling blocks.Spatial dimensions are preserved while spectral dimensions are compressed and restored; down-sampling uses stride-2 3D convolution and up-sampling uses stride-2 3D deconvolution.Source: C. Structure of the Spectral-Spatial… · p. 6

D. Attention-based Classification Module

The attention-based classifier uses patches of diffusion features around each pixel to produce per-pixel classifications. Experiments use Indian Pines, Pavia University, and Salinas, with training and test samples drawn from each dataset.

D. Attention-based Classification Module

  • Diffusion features are extracted from down- and up-sampling activations of a noise-corrupted hyperspectral image, and their information can vary by timestamp and layer.The extracted features retain the raw features’ spatial dimensions and include original spectral information alongside deep features.Source: D. Attention-based Classification Module · p. 6
  • For each pixel, the classifier processes a centered patch of diffusion features, maps it into spatial tokens, and uses attention to produce a classification.The classifier combines Conv2D feature mapping with Transformer encoding and an MLP output layer.Source: D. Attention-based Classification Module · p. 6

1) Datasets:

  • The experiments evaluate SpectralDiff on Indian Pines, Pavia University, and Salinas, with training and test samples distributed across land-cover categories.Indian Pines uses 200 selected bands from a 145×145 image; Pavia University uses 103 bands from a 610×340 image; Salinas has 224 bands and 512×217 pixels.Source: 1) Datasets · pp. 6–7

2) Evaluation Metrics:

The evaluation section frames the experiments as comparisons of algorithm effectiveness. Training uses separate optimization settings for diffusion and classification.

2) Evaluation Metrics:

  • The evaluation section states that the algorithm’s effectiveness will be compared.The supplied passage does not specify the comparison metrics.Source: 2) Evaluation Metrics · p. 7

3) Training Details:

  • The diffusion model uses Adam with a 1e-4 learning rate, batch size 256, patch size 64, and more than 30,000 training epochs.The study recommends maximizing patch size within available computational resources to model a larger range of pixel relationships.Source: 3) Training Details · p. 7
  • The model was trained with batch size 64 and converged rapidly, requiring fewer than 50 epochs to achieve the desired classification accuracy on all three datasets.Source: 3) Training Details · p. 8

B. Performance Analysis

SpectralDiff achieves the best overall performance on the Indian Pines, Pavia University, and Salinas datasets. Its overall accuracy exceeds the second-best algorithm on each dataset, with performance also examined across class balance and sample sizes.

B. Performance Analysis

  • The comparison includes CNN, Transformer, graph-based, and unsupervised-feature baselines, alongside SpectralDiff’s two-stage diffusion and classification pipeline.The pipeline uses 64×64 patches for diffusion training and 13-sized patches for classification.Source: B. Performance Analysis · p. 8

1) Quantitative Results:

  • 0.81%, 0.93%, and 1.97% OA gains over the second-best algorithm on Indian Pines, Pavia University, and Salinas, respectively.The authors report the best overall performance on all three datasets; the supplied result visuals include the Indian Pines and Pavia University tables.Source: 1) Quantitative Results · p. 8, Table II · p. 9, Table III · p. 9
  • SpectralDiff shows relatively balanced subcategory results on Indian Pines and Salinas, while its subcategory balance is weaker than SSRN on Pavia University despite high overall AA.The reported balance assessment also notes higher AA and κ on Indian Pines and Salinas.Source: 1) Quantitative Results · p. 8
  • Across varying sample sizes, SpectralDiff consistently outperforms other models; at 10 samples per class, it leads on Pavia University but has no significant advantage on Indian Pines or Salinas.The authors suggest that the larger number of diffusion feature channels may contribute to classifier overfitting.Source: 1) Quantitative Results · p. 9

2) Qualitative results:

Diffusion features improve classification metrics over raw features on the three datasets, while qualitative maps are reported to show less noise and closer alignment with ground truth.

2) Qualitative results:

  • Classification maps for the proposed method are reported to have less noise, closer ground-truth alignment, and better overall land-cover segmentation than the compared methods.CNN1D, CNN2D, SpectralFormer, and miniGCN maps are described as noisy, with discontinuous land-cover blocks and rough classification results.Source: 2) Qualitative results · p. 9

1) Ablation study:

  • 0.78%, 5.87%, and 3.17% OA improvements from diffusion features over raw features on Indian Pines, Pavia University, and Salinas, respectively.For 20 to 80 training samples per class, diffusion features consistently outperform raw features; at 10 samples, they do not improve results on Indian Pines or Pavia University, potentially due to overfitting from high feature dimensionality.Source: 1) Ablation study · pp. 9–10, Table V · p. 10

2) Diffusion Model Analysis:

The analysis examines reverse-diffusion reconstruction, feature-extraction settings, and inference time. Classification maps report higher overall accuracy for SpectralDiff than the listed comparison methods on all three datasets.

2) Diffusion Model Analysis:

  • Reverse diffusion progressively restores the Indian Pines image and woods spectral curve from states resembling noise toward their original forms.The image and spectral-curve reconstructions are shown across timestamps, with the ground-truth image also shown.Source: 2) Diffusion Model Analysis · p. 10
  • For Pavia University and Salinas, Timestamp = 5 generally performs best, and Layerindex 0 outperforms 1 and 2; Indian Pines shows fluctuations without significant changes.Larger timestamps are associated with fewer denoising iterations and more noise in the input, which the authors link to classification-performance deviation.Source: 2) Diffusion Model Analysis · pp. 10, 11, 13, Table VI · p. 11
  • Ours leads on all three datasets: Indian Pines OA 93.146% vs SSGRN 92.336%, Pavia University OA 94.775% vs SSGRN 93.850%, and Salinas OA 98.971% vs DMVL 97.005%.These values are reported in the classification-map captions.Source: Figure 6 · p. 11, Figure 7 · p. 12, Figure 8 · p. 12
  • Inference takes longer than with simple CNN and GCN methods, but is at the same level as complex CNN and transformer methods.The timing analysis counts only the classification stage of the two-stage method.Source: 2) Diffusion Model Analysis · p. 13, Table VII · p. 13

V. CONCLUSION

SpectralDiff constructs HSI spectral-spatial distributions from a generative perspective, adaptively forming sample relationships without prior graph or neighborhood information. The authors report that it outperforms state-of-the-art techniques and identify out-of-distribution generalization and detection as future research directions.

V. CONCLUSION

  • SpectralDiff adaptively constructs sample relationships without prior graph structures or neighborhood information, captures HSI distribution and contextual information, and achieves cross-sample perception.The authors report that experiments show it outperforms state-of-the-art techniques.Source: §V · p. 13
  • The authors identify out-of-distribution generalization and detection in hyperspectral imaging as future directions for diffusion models.They discuss potential generalization to unseen examples outside the training distribution and diffusion models' detection performance.Source: §V · pp. 13–14

Related papers

Loading 2304.05961v2…