PaperlayerView exact revision on arXiv

Source-linked AI summary

FPGA: Fast Patch-Free Global Learning Framework for Fully End-to-End Hyperspectral Image Classification

Zhuo Zheng, Yanfei Zhong, Ailong Ma, Liangpei Zhang

arXiv:2011.05670v1cs.CVeess.IV

TL;DR

Patch-based HSI classifiers are computationally costly and limited in spatial context. FPGA uses GS2 sampling with a fully end-to-end FreeNet FCN to process whole images globally, and experiments on three benchmark datasets report advantages in speed and accuracy over patch-based learning.

  • Problem

    Patch-based deep HSI classifiers repeatedly process overlapping patches, increasing computational cost and limiting the spatial context available for long-range dependency modeling.

  • Method

    FPGA combines GS2 stochastic stratified sampling with FreeNet, an encoder-decoder FCN using spectral attention, a lightweight decoder, and lateral feature fusion.

  • Results

    Experiments on three public benchmark datasets suggest that FPGA is superior to patch-based learning in both inference speed and HSI classification accuracy.

  • Takeaways & Limitations

    FreeNet directly processes the entire HSI and outputs a classification map without dimension reduction or post-processing.

Abstract

from arXiv · show

Deep learning techniques have provided significant improvements in hyperspectral image (HSI) classification. The current deep learning based HSI classifiers follow a patch-based learning framework by dividing the image into overlapping patches. As such, these methods are local learning methods, which have a high computational cost. In this paper, a fast patch-free global learning (FPGA) framework is proposed for HSI classification. In FPGA, an encoder-decoder based FCN is utilized to consider the global spatial information by processing the whole image, which results in fast inference. However, it is difficult to directly utilize the encoder-decoder based FCN for HSI classification as it always fails to converge due to the insufficiently diverse gradients caused by the limited training samples. To solve the divergence problem and maintain the abilities of FCN of fast inference and global spatial information mining, a global stochastic stratified sampling strategy is first proposed by transforming all the training samples into a stochastic sequence of stratified samples. This strategy can obtain diverse gradients to guarantee the convergence of the FCN in the FPGA framework. For a better design of FCN architecture, FreeNet, which is a fully end-to-end network for HSI classification, is proposed to maximize the exploitation of the global spatial information and boost the performance via a spectral attention based encoder and a lightweight decoder. A lateral connection module is also designed to connect the encoder and decoder, fusing the spatial details in the encoder and the semantic features in the decoder. The experimental results obtained using three public benchmark datasets suggest that the FPGA framework is superior to the patch-based framework in both speed and accuracy for HSI classification. Code has been made available at: https://github.com/Z-Zheng/FreeNet.

I. INTRODUCTION

HSI classification benefits from deep spectral-spatial learning, but patch-based methods incur redundant computation and limited spatial context. FPGA addresses these constraints with global, patch-free FCN learning, GS2 sampling, FreeNet, and encoder-decoder feature fusion.

  • Deep CNN-based classifiers significantly improve HSI classification accuracy by learning hierarchical spectral-spatial representations end to end.
  • Overlapping patches create redundant computation, slowing inference, while limited patch sizes constrain long-range spatial dependency modeling.
  • FPGA introduces an encoder-decoder FCN that shares spatial computation to process whole HSIs and exploit global spatial information.
  • GS2 transforms all training samples into stochastic stratified sequences, producing diverse gradients that support FCN convergence and effective parameter updates.
  • FreeNet combines a spectral-attention encoder, lightweight decoder, and lateral connections that fuse spatial details with semantic features for clearer classification maps.

III. FPGA: FAST PATCH-FREE GLOBAL LEARNING FRAMEWORK FOR HSI CLASSIFICATION

The FPGA framework replaces overlapping patch processing with a patch-free FCN that shares spatial computation across the whole image. Its encoder-decoder design uses lateral connections to combine semantic and spatial features.

  • FPGA addresses the patch-based speed bottleneck by sharing computation in the spatial dimension through a patch-free global learning framework.
  • The patch-free framework maps the whole image rather than repeatedly processing overlapping local patches.
  • Lateral connections fuse finer spatial details from encoder features with stronger semantic features from the decoder.

A. Patch-Free Global Learning

Patch-free global learning replaces explicit image patches with an encoder-decoder FCN’s implicit receptive field, enabling one-shot computation over the whole HSI while using sampled positions for supervision.

  • Patch-free global learning replaces explicit patching with the model’s implicit receptive field to avoid redundant overlap computation and capture wider spatial context.
  • The framework processes the entire HSI through an encoder-decoder FCN, producing a predicted probability cube for all spatial positions.
  • FPGA combines a GS2 sampler, an encoder-decoder FCN, and lateral connections between the encoder and decoder.
  • During training, all pixels participate in forward computation, but only sampled positions receive supervised signals in each iteration.
  • Because training and testing both use one-shot forward computation, patch-free global learning significantly increases inference speed and exploits spatial context.

B. Global Stochastic Stratified Sampling Strategy

GS2 transforms all training samples into a stochastic sequence of class-stratified samples, targeting stable, diverse gradients and FCN convergence with limited HSI training data.

  • GS2 transforms all training samples into a stochastic sequence of stratified samples to ensure FCN convergence and class-balanced training.
  • Algorithm 1 repeatedly fetches class-specific elements and combines them into successive stratified-label sets until the available sample groups are exhausted.
  • The resulting sequence is randomly shuffled during training to preserve gradient stochasticity and prevent overfitting.
  • The algorithm groups samples by class, shuffles each class’s order, and stratifies its samples into sets before recombination.
  • The sampling procedure uses α as the mini-batch size per class, with smaller α values producing more gradient orientations for optimization.

C. FreeNet in FPGA

FreeNet is a lightweight encoder-decoder FCN that processes entire HSIs, uses spectral attention to reweight feature maps, and progressively restores spatial detail through lateral feature fusion.

  • Network architecture: FreeNet uses an encoder to compute hierarchical convolutional features over the entire HSI and a decoder to output a same-size classification probability map.A compression factor β controls feature-map counts to trade off speed and accuracy.
  • Encoder network: The encoder contains a stem block, four hybrid blocks, and 3×3 convolutional, group-normalization, and ReLU-based basic modules.The stem transforms variable input band counts to 64 channels, while blocks #1–#3 downsample and block #4 does not.
  • Spectral attention: Spectral attention uses global spatial context and channel scaling to model feature-map interdependencies without changing tensor dimensions.Two fully connected layers and sigmoid gating compute the scaling vector T, while reduction ratio r balances capacity and computational cost.
  • Decoder network: The decoder progressively upsamples semantic features and aggregates finer encoder features through SSF before pixel classification.Its head applies 3×3 and 1×1 convolutions, with the latter using N filters for N categories.
  • Lateral feature fusion: Lateral SSF connects encoder and decoder features using a 1×1 convolution and pointwise addition, refining semantics with spatial details.The interpolated decoder feature is the baseline term, while the transformed encoder feature supplies the residual term; this supports lossless gradient flow into shallow layers.

D. Fully End-to-End HSI Classification Using FreeNet in FPGA

After FreeNet converges, FPGA performs one-shot forward computation on the whole HSI. This shares spatial computation for faster inference, subject to an input-size constraint.

  • Inference: FreeNet performs patch-free inference over the whole HSI by sharing computation in the spatial dimension.This enables faster inference than patchwise HSI classification after training convergence.
  • Implementation constraint: The HSI spatial dimensions must be multiples of 8 because FreeNet contains three 2× upsampling blocks.The implementation preserves the raw HSI while satisfying this requirement.

IV. EXPERIMENTAL RESULTS AND ANALYSIS

Experiments evaluate FreeNet against patch-based classifiers on three benchmark datasets using standardized architecture, optimization, and classification metrics. On Pavia University, FreeNet produces stronger visual and quantitative results than the compared patch-based methods.

  • Experimental settings: Experiments compare FreeNet and its variants with SVM and four patch-based deep-learning classifiers on three benchmark HSI datasets.The datasets are ROSIS-03 Pavia University, Salinas, and CASI University of Houston.
  • Experimental settings: The same standard FreeNet architecture is used across all three datasets without dataset-specific tuning.The configuration is FreeNet with β = 1.0.
  • Experimental settings: Evaluation uses per-class accuracy, overall accuracy, average accuracy, and the Kappa coefficient.These four metrics are applied to assess the proposed methods.
  • Pavia University results: FreeNet’s Pavia University classification map has smoother edges and finer spatial detail than the compared methods.The paper attributes this to global spatial context, lateral SSF, and decoder design.
  • Pavia University results: 99.81% OA is reported for FreeNet, exceeding DFFN’s 98.57% OA under 200 training samples per class.DFFN also reports 99.16% AA and 0.9808 Kappa among patch-based methods, while FreeNet is reported as more accurate overall.

C. Experiment 2: Salinas Dataset

On Salinas, FreeNet achieved the highest reported classification accuracy and produced a cleaner map than comparable methods, while the dataset’s performance was near saturation.

  • Dataset: Salinas contains 512 × 217 pixels, 204 spectral bands, and 16 ground-truth classes.The dataset has a 3.7 m spatial resolution.
  • Results: FreeNet produced less noise than DFFN despite their similar accuracy, yielding a better visual classification map.The comparison suggests that global spatial information contributes to visual map quality.
  • Results: FreeNet achieved 99.92% OA, exceeding the patch-based DFFN by 0.2%.FreeNet slightly surpassed DFFN across OA, AA, and Kappa.
  • Analysis: Cross-layer feature fusion appears in both FreeNet’s lateral SSF and DFFN’s multi-layer fusion, indicating its importance for HSI classification.FreeNet fuses shallow encoder spatial details with deep decoder semantic information.

D. Experiment 3: CASI University of Houston Dataset

On the CASI University of Houston benchmark, FreeNet achieved a state-of-the-art result and approximately 2% improvement over the best patch-based method, with clearer maps for several connected classes.

  • Dataset: CASI University of Houston contains 349 × 1905 pixels, 144 spectral bands, and 15 ground-truth classes.Its officially predefined training and test samples support more reliable benchmark evaluation.
  • Visual analysis: FreeNet’s classification maps showed better connectivity for Road, Highway, and Railway than the compared methods.These classes also had higher accuracy than in the other methods according to Table VII C9-C11.
  • Results: FreeNet achieved a state-of-the-art result, improving approximately 2% over DFFN, the best patch-based method.The result supports stronger performance from patch-free global learning on this more difficult benchmark.
  • Interpretation: FreeNet exploits global spatial context through its spectral attention module and lateral connection-based SSF.The larger receptive field supplies global context, while SSF incorporates spatial details.
  • Error analysis: CASI accuracy was lower because GrassHealthy, Commercial, and Railway pixels in shadow areas were often misclassified as Water.The analysis indicates that spectral information is sensitive to observation conditions.

V. FPGA SENSITIVITY ANALYSIS

The sensitivity analysis shows that GS2 resolves FCN convergence problems by producing more diverse gradients, while FreeNet remains stable across a broad range of α values.

  • GS2 sampling: GS2 increased baseline FCN OA from 15.23% to 65.12% at β = 0.75 and from 12.49% to 65.16% at β = 1.0.The paper treats approximately 10% OA as evidence that the model fails to converge.
  • GS2 sampling: GS2 splits training samples into equal-spatial-size mini-batches to obtain diverse gradients and partially supervised signals.These signals make it easier for optimization to skip local minimum points.
  • Hyperparameter sensitivity: FreeNet showed stable performance when α was within 30% of the total number of training samples.The result indicates robustness of GS2 with respect to α.

B. Lateral Connection Based SSF

Lateral connection-based SSF improves FreeNet by transferring encoder spatial details to decoder semantic features, while the patch-free design substantially accelerates inference by avoiding overlapping-patch computation.

  • Module effectiveness: Adding lateral connection and spectral attention improved OA from 65.12% to 84.23% at β = 0.75 and from 65.16% to 84.91% at β = 1.0.These results reached performance similar to state-of-the-art patch-based HSI classifiers.
  • SSF mechanism: SSF passes shallow-layer spatial details to the decoder, progressively refining deep semantic features into spatially finer fused representations.Pointwise addition also alleviates gradient vanishing and simplifies optimization.
  • Spectral attention: Spectral attention further improved OA from 84.23% to 85.49% at β = 0.75 and from 84.91% to 86.61% at β = 1.0.The module models encoder feature-map interdependencies using global spatial context.
  • Inference speed: Patch-free methods achieved approximately 560× practical and 480× theoretical speedups over patch-based methods.The practical ratio is larger because theoretical calculations ignore parallel computation.
  • Deployment: FreeNet’s inference time was much smaller than its training time, supporting offline training followed by online UAV or satellite inference.This deployment pattern is explicitly identified as a suitable application scenario.

E. The Impact of the Number of Training Samples

Reducing the training samples generally lowers FreeNet’s performance across the evaluated benchmark datasets. The effect varies by dataset, with Salinas easier than Pavia University and CASI University of Houston remaining steady until samples reach 5%.

  • Reducing the training samples generally results in a performance drop for FreeNet.This trend was observed across the Pavia University, Salinas, and CASI University of Houston datasets.
  • Salinas classification is easier than Pavia University classification for FreeNet.The authors speculate that Salinas benefits from more spectral information because it has more bands, while spatial-information contributions may have saturated.
  • For CASI University of Houston, performance remains steady until the training samples are reduced to 5% of the labeled samples.
Loading 2011.05670v1…