PaperlayerView exact revision on arXiv

Source-linked AI summary

Hyperspectral Image Classification with Markov Random Fields and a Convolutional Neural Network

Xiangyong Cao, Feng Zhou, Lin Xu, Deyu Meng, Zongben Xu, John Paisley

arXiv:1705.00727v2cs.CV

TL;DR

HSI classification requires using spectral and spatial information, while many prior approaches rely on hand-crafted features or do not jointly formulate deep learning and MRFs in a Bayesian framework. This paper combines a CNN with a smooth MRF prior and iteratively updates parameters and labels. Across one synthetic and two real benchmark datasets, the method outperforms state-of-the-art methods, including deep and non-deep models.

  • Problem

    Many HSI methods use hand-crafted features, and prior deep-learning methods do not jointly formulate deep learning and MRFs within a Bayesian framework.

  • Method

    The method combines a CNN for spectral-spatial feature extraction and pseudo-annotations with a smooth MRF prior, iteratively updating CNN parameters and pixel labels.

  • Results

    The method outperforms state-of-the-art deep and non-deep methods on one synthetic HSI dataset and two real HSI datasets.

  • Takeaways & Limitations

    Joint CNN-MRF modeling provides a supervised HSI classification approach that uses spectral and spatial information in a unified Bayesian framework.

Abstract

from arXiv · show

This paper presents a new supervised classification algorithm for remotely sensed hyperspectral image (HSI) which integrates spectral and spatial information in a unified Bayesian framework. First, we formulate the HSI classification problem from a Bayesian perspective. Then, we adopt a convolutional neural network (CNN) to learn the posterior class distributions using a patch-wise training strategy to better use the spatial information. Next, spatial information is further considered by placing a spatial smoothness prior on the labels. Finally, we iteratively update the CNN parameters using stochastic gradient decent (SGD) and update the class labels of all pixel vectors using an alpha-expansion min-cut-based algorithm. Compared with other state-of-the-art methods, the proposed classification method achieves better performance on one synthetic dataset and two benchmark HSI datasets in a number of experimental settings.

I. INTRODUCTION

HSI classification benefits from combining spectral and spatial information, but many existing methods rely on hand-crafted features. The paper proposes a supervised Bayesian method integrating CNN representations with an MRF prior and reports improved performance across synthetic and benchmark datasets.

  • HSI sensors capture hundreds of continuous narrow spectral bands, enabling material-related spectral and spatial information for applications requiring pixel-level class labels.
  • A. Related work: spectral vs spectral-spatial based methods: Spectral-only methods use pixel spectra or manually extracted features, while spectral-spatial methods incorporate correlations among neighboring pixels.
  • A. Related work: spectral vs spectral-spatial based methods: MRFs incorporate spatial information by encouraging neighboring pixels to receive the same class label, improving HSI classification accuracy in prior work.
  • A. Related work: spectral vs spectral-spatial based methods: Hand-crafted features depend on manually specified strategies and domain knowledge, whereas deep learning learns data-dependent hierarchical representations from raw data.
  • B. Contributions of our approach: The proposed method formulates HSI classification in a Bayesian framework and combines a CNN with a smooth MRF prior.
  • B. Contributions of our approach: The CNN extracts spectral-spatial features from 3D patches, while the MRF prior acts on labels and the model alternates updates to CNN parameters and pixel labels.
  • B. Contributions of our approach: Experiments on one synthetic and two real HSI datasets report that the proposed method outperforms state-of-the-art deep and non-deep methods.

II. PROBLEM FORMULATION

The formulation assigns a class label to each patch’s central pixel by combining classifier outputs with spatial label modeling. An intermediate pseudo-annotation representation enables alternating classifier retraining and label optimization over the full image.

  • 1) Problem notation:: The HSI is represented as H ∈ R^h×w×d, with h and w spatial dimensions and d spectral bands.
  • 1) Problem notation:: Each k × k × d patch produces a CNN feature vector whose label corresponds to the patch’s central pixel, not to every pixel within the patch.
  • 1) Problem notation:: The label set contains K classes, and sliding-window extraction with padding yields n = hw patches.
  • 2) Problem setup:: HSI classification estimates all labels y by maximizing the conditional distribution P(y|X, Θ), combining MRF spatial modeling with deep-learning discrimination.
  • 2) Problem setup:: An intermediate variable ey assigns each patch an initial probabilistic pseudo-annotation generated from the classifier.
  • 2) Problem setup:: The computational formulation replaces summation over ey with a point estimate and separates pseudo-annotation computation from label optimization.
  • 2) Problem setup:: The classifier is first trained on labeled data, then retrained on all patches using current labels, with both steps repeated until stopping criteria are met.
  • 2) Problem setup:: The CNN is used as classifier f, and a smoothness prior is placed on labels to enforce spatial consistency.

III. PROPOSED APPROACH

The proposed approach combines a CNN classifier with an MRF labeling model to use spectral and spatial information jointly. It alternates CNN learning, spatially regularized label inference, and algorithm-level updating.

  • The approach introduces a CNN classifier, places a smoothness prior on labels, formulates classification as an MRF labeling problem, and summarizes the resulting iterative algorithm.

A. Discriminative CNN-based classifier

The discriminative classifier processes 3D HSI patches directly through convolutional, pooling, and fully connected layers. Softmax outputs provide pseudo-label distributions, and SGD updates the CNN parameters.

  • CNNs exploit local spatial correlations through convolutional connectivity, unlike standard fully connected feed-forward networks.
  • The adopted network contains an input layer, two convolution-max-pooling pairs, two fully connected layers, and an output layer.
  • Each input sample is a k × k × d 3D patch that remains unflattened at the CNN input.
  • The first convolution uses 100 filters of size 5 × 5 × d, producing 100 feature maps before subsequent pooling and feature processing.
  • After the second pooling stage, the resulting feature maps are flattened and passed to fully connected layers with learned weights, biases, and ReLU activations.
  • The final CNN vector is passed through softmax to produce a label distribution, from which the pseudo-classification label is selected by arg max.
  • The CNN uses cross-entropy loss with one-hot labels, and its parameters are optimized using stochastic gradient descent.
  • Weights are randomly initialized, biases are zero-initialized, and dropout with rate 0.5 is applied in the fifth and sixth fully connected layers.

B. Label smoothness prior and MRF optimization

The model combines CNN-derived label pseudo-annotations with a spatially smooth MRF prior and optimizes the resulting labeling objective approximately with α-expansion min-cut.

  • Label smoothness prior: CNN pseudo-annotations provide the likelihood term, while the label prior supplies spatial regularization in the classification objective.The indicator function defines the likelihood contribution for assigning labels to pixels.
  • Label smoothness prior: The smoothness prior increases probability when neighboring labels agree, encouraging piecewise-smooth segmentations.The parameter µ controls smoothness, and δ assigns higher values when neighboring labels are equal.
  • MRF formulation: The final objective is an MRF energy over an image graph whose nodes are pixels and whose edges represent neighboring relationships.Its unary term reflects class-assignment cost from pixel probabilities, while pairwise terms encourage neighboring labels to match.
  • MRF optimization: Because the optimization is NP-hard, the method uses α-expansion min-cut for approximate inference because of its reported performance and computation speed.The optimization follows calculation of label pseudo-annotations from the current classifier.

C. CNN-MRF classification algorithm

The CNN-MRF algorithm alternates between CNN training and MRF-based label updates, repeating these steps until a stopping criterion is satisfied.

  • Initialization: The algorithm first trains the CNN classifier using the labeled training data.Its inputs include HSI patches, training data, learning rate, smoothness parameter, and batch size.
  • Alternating updates: It computes CNN label pseudo-annotations and updates all pixel labels with α-expansion using the smoothness parameter µ.This label-update step is performed after the initial CNN training and again during later iterations.
  • Alternating updates: The CNN is retrained using the HSI patches and current labels, followed by another pseudo-annotation computation and α-expansion label update.The retraining and label-update pair forms the algorithm’s iterative core.
  • Termination: Steps that retrain the CNN and update labels repeat until the stopping criterion is satisfied.The procedure therefore alternates parameter learning with label optimization.

IV. EXPERIMENTS

Experiments evaluate CNN-MRF on one synthetic and two real-world benchmark HSI datasets against classical, MRF-regularized, and deep-learning methods using standard accuracy criteria.

  • Datasets: The study tests the method on one synthetic dataset and two real-world benchmark datasets.The synthetic data are generated from five randomly extracted endmembers with 162 spectral bands and 40,000 vectors.
  • Compared methods: Comparisons include SVM-GC, MLRsubMLL, SVM-3DDWT-GC, SS-DCNN, SPP-DCNN, and DC-CNN.SVM-GC and MLRsubMLL integrate MRFs into SVM and MLR methods, respectively.
  • Compared methods: The experiments distinguish classification methods without MRF from regularized classification methods that incorporate MRF modeling.The distinction is applied when organizing comparisons between methods with and without MRF components.
  • Evaluation: Performance is compared using overall accuracy, average accuracy, and the kappa coefficient, with larger values indicating better classification performance.OA is total correct samples divided by total test samples, AA averages class accuracies, and κ incorporates omission and commission errors.

2) Impact of parameter settings:

Parameter studies examine CNN kernel size, width, depth, patch size, and MRF smoothness, selecting settings based on accuracy, robustness, or runtime trade-offs.

  • Kernel size: Larger first-layer kernels improve overall accuracy, so the experiments use a kernel size of 5.The paper attributes the improvement to capturing more structure and texture.
  • Network width: Classification results are not very sensitive to second-layer network width, so the default width is set to 200.The first convolutional layer width is fixed at 100 during this study.
  • Network depth: Increasing depth does not always improve results, and depth 7 achieves the best performance among the tested depths.The tested networks have depths 5, 7, 9, 11, and 13; the paper notes possible gradient vanishing in deeper networks.
  • Patch size: Larger patches improve results but increase training time, leading to a default patch size of 9 as a performance–runtime trade-off.The tested patch sizes are k = {1, 3, 5, 9, 13}.
  • Smoothness parameter: Setting the smoothness parameter to µ = 20 produces the best classification results and is used in subsequent experiments.The smoothness parameter study reports this choice from Table V.
  • Other parameters: Other settings show consistent performance under variation, including pooling kernel size 2, fully connected widths 200 and 100, learning rate α = 0.001, and batch size 100.The stated batch size applies to the synthetic and Indian Pines datasets.

3) Experimental result of the synthetic HSI data:

On the synthetic HSI dataset, CNN-MRF outperformed competing methods across OA, AA, and κ, with classification maps visually closest to ground truth.

  • Synthetic HSI results: CNN-MRF achieved better OA, AA, and κ than competing methods on the synthetic dataset.The comparison included non-deep-learning MRF-based methods and other deep-learning methods.
  • Experimental setup: The experiments used a tuned CNN structure before comparing CNN-MRF with competing methods.
  • Synthetic HSI results: The synthetic-data comparison reported overall accuracy, average accuracy, κ statistics, and running time.

2) ROSIS Pavia University Data:

On the Pavia University dataset, CNN-MRF again achieved the best performance across OA, AA, and κ under a balanced, small-sample evaluation.

  • Experimental setup: The Pavia University experiment trained with 40 samples per class and tested on the remaining samples, repeating each method 20 times.This created a balanced training sample setting with limited labeled data.
  • Pavia University results: CNN-MRF achieved the best performance on Pavia University for OA, AA, and κ.The method also produced a much smoother classification map than the other methods.
  • Pavia University results: CNN-MRF obtained about 2% OA improvement over CNN on Pavia University.
  • Pavia University results: MLRsubMLL showed about 30% OA improvement over MLRsub when using the MRF prior.

C. Limited training data scenarios

Under highly limited training-data conditions, CNN and MRF each contributed to improved classification accuracy, while CNN-MRF performance evolved non-monotonically across training.

  • Limited training data scenarios: Training proportions of 0.1% to 0.5% per class were evaluated on Pavia University, with data augmentation used for CNN training.
  • Limited training data scenarios: CNN outperformed the other methods for every evaluated training-set size without the spatial prior.
  • Limited training data scenarios: Adding the MRF spatial prior significantly improved the corresponding OA results across the limited-training experiments.
  • CNN-MRF interaction: CNN-MRF OA increased quickly during the first 30 epochs, rose more slowly for about 40 epochs, then decreased slightly or fluctuated.The experiments therefore used a maximum of 60 epochs and reported the resulting output.
  • Deep-learning comparisons: The study also compared CNN-MRF with SS-DCNN, SPP-DCNN, and DC-CNN on Indian Pines and Pavia University.The authors reported improved performance for CNN-MRF in OA, AA, and κ on both datasets, while identifying running time as a potential drawback.

F. Comparison with other label regularization methods

The method is compared with alternative label-regularization approaches under matched experimental settings across synthetic, Indian Pines, and Pavia University datasets. The reported experiments show that the proposed method outperforms state-of-the-art methods, while future work targets computational efficiency on large datasets.

  • Comparison setup: CNN-MF and CNN-MV replace the MRF optimization with median filtering and majority voting within the same algorithm framework.The methods are evaluated alongside the proposed approach on synthetic, Indian Pines, and Pavia University data using the same experimental settings.
  • Comparison setup: The comparison includes overall-accuracy results for label-regularization methods across the synthetic, Indian Pines, and Pavia University datasets.Table XIV reports overall accuracy for all label-regularization methods on these three datasets.
  • Results: The proposed CNN–MRF method outperforms state-of-the-art deep and non-deep methods on one synthetic and two real benchmark HSI datasets.The conclusion summarizes the cross-dataset experimental outcome rather than isolating a single metric or dataset.
  • Limitations and future work: Future work includes dimensionality reduction to make the method more efficient, especially for large-scale datasets.The authors also plan to extend the MRF-based regularization regime to different application scenarios.
Loading 1705.00727v2…