PaperlayerView exact revision on arXiv

Source-linked AI summary

Classification of Hyperspectral and LiDAR Data Using Coupled CNNs

Renlong Hang, Zhu Li, Pedram Ghamisi, Danfeng Hong, Guiyu Xia, Qingshan Liu

arXiv:2002.01144v1cs.CVeess.IV

TL;DR

Hyperspectral and LiDAR fusion must combine complementary information while addressing limited training samples and costly fusion layers. The paper proposes coupled CNNs with shared parameters and joint feature- and decision-level fusion, reporting best performance on Houston data and very high performance on Trento data.

  • Problem

    Existing fusion models face limited training samples, while fully connected fusion layers can contain many parameters and supervised information may be lost in unsupervised representation learning.

  • Method

    The framework uses coupled CNNs for hyperspectral and LiDAR feature learning, then combines feature-level fusion with weighted decision-level fusion.

  • Results

    The proposed models achieve the best performance on the Houston data and very high performance on the Trento data.

  • Takeaways & Limitations

    Simultaneous feature-level and decision-level fusion is more effective than feature-level fusion alone on the Trento data, with maximization producing the best results.

Abstract

from arXiv · show

In this paper, we propose an efficient and effective framework to fuse hyperspectral and Light Detection And Ranging (LiDAR) data using two coupled convolutional neural networks (CNNs). One CNN is designed to learn spectral-spatial features from hyperspectral data, and the other one is used to capture the elevation information from LiDAR data. Both of them consist of three convolutional layers, and the last two convolutional layers are coupled together via a parameter sharing strategy. In the fusion phase, feature-level and decision-level fusion methods are simultaneously used to integrate these heterogeneous features sufficiently. For the feature-level fusion, three different fusion strategies are evaluated, including the concatenation strategy, the maximization strategy, and the summation strategy. For the decision-level fusion, a weighted summation strategy is adopted, where the weights are determined by the classification accuracy of each output. The proposed model is evaluated on an urban data set acquired over Houston, USA, and a rural one captured over Trento, Italy. On the Houston data, our model can achieve a new record overall accuracy of 96.03%. On the Trento data, it achieves an overall accuracy of 99.12%. These results sufficiently certify the effectiveness of our proposed model.

I. INTRODUCTION

Hyperspectral data provide rich spectral-spatial information, but spectrally similar objects remain difficult to distinguish, motivating fusion with LiDAR elevation data. The paper addresses this challenge with coupled CNNs and simultaneous feature- and decision-level fusion.

  • Hyperspectral data offer richer spectral information than multispectral data, combined with spatial information for land-use and land-cover classification.
  • Spectrally similar objects in complex urban and rural scenes remain difficult to discriminate using hyperspectral information alone.
  • LiDAR records object elevation and therefore provides information complementary to hyperspectral data.
  • Existing fusion methods include feature-level approaches that combine heterogeneous features and decision-level approaches that integrate outputs from multiple classifiers.
  • Deep learning offers end-to-end learning of high-level semantic features, and CNNs have been applied to hyperspectral and remote-sensing classification tasks.
  • The proposed framework uses two coupled CNNs and simultaneously applies feature-level and decision-level fusion, achieving 96.03% overall accuracy on Houston data.

II. METHODOLOGY

The proposed framework uses coupled CNNs to learn complementary spectral-spatial and elevation features from hyperspectral and LiDAR data, then combines them through feature-level and decision-level fusion. Parameter sharing reduces model complexity and allows the two networks to learn from each other.

  • Coupled CNN architecture: The framework contains an HS network for spectral-spatial feature learning and a LiDAR network for elevation feature learning.The HS branch applies PCA and extracts a local cube, while the LiDAR branch extracts a spatially corresponding patch.
  • Coupled CNN architecture: PCA reduces redundant hyperspectral information before corresponding hyperspectral cubes and LiDAR patches are processed by three convolutional layers.For each pixel, the inputs are a p×p×k hyperspectral cube and a p×p LiDAR patch.
  • Coupled CNN architecture: The second and third convolutional layers share parameters across the HS and LiDAR networks.This coupling reduces the number of parameters by twice and lets gradients from both networks influence shared-layer learning.
  • Fusion module: The fusion module first combines the learned representations and then sends the individual and fused features to separate output layers.The three outputs are integrated afterward through decision-level fusion.
  • Fusion module: Feature-level fusion evaluates concatenation, element-wise maximization, and summation strategies for combining the hyperspectral and LiDAR representations.The summation method adds the two representations, whereas maximization selects the larger value element by element.
  • Fusion module: Decision-level fusion uses a weighted summation of three output vectors, with class-specific weights determined by each output layer’s training accuracy.The weights correspond to the three outputs connected to the hyperspectral, LiDAR, and fused representations.

D. Network Training and Testing

The network is trained end-to-end with cross-entropy losses supervising hyperspectral, LiDAR, and fused outputs, then uses class-accuracy-based weights for test prediction.

  • The whole network is trained end-to-end on labeled training samples using cross-entropy losses for three outputs.The outputs correspond to hyperspectral, LiDAR, and fused representations.
  • L3 supervises the fused feature, while L1 and L2 supervise the hyperspectral and LiDAR features, respectively.
  • The final loss combines L1, L2, and L3 with weight parameters λ1 and λ2.The weights are empirically set to 0.01 in the experiments.
  • L1 and L2 also act as regularization terms for L3, reducing overfitting risk during network training.
  • Test-time decision weights are computed from each output's per-class accuracy before combining the three predictions.The class-accuracy calculation uses an indicator function over the training set.

A. Data Description

The study evaluates hyperspectral–LiDAR fusion on two standard datasets: an urban Houston scene and a rural Trento scene, each with aligned imagery and labeled samples.

  • The experiments use two hyperspectral and LiDAR fusion datasets.
  • Houston Data: The Houston dataset covers 349 × 1905 pixels at 2.5 m resolution and contains 144 hyperspectral bands.It includes 15 classes and standard training and test sets.
  • Houston Data: Figure 4 presents Houston hyperspectral and LiDAR images alongside training and test data maps.The hyperspectral visualization uses bands 64, 43, and 22 as R, G, and B.
  • Trento Data: Figure 5 shows Trento hyperspectral and LiDAR images with corresponding training and test maps.The hyperspectral visualization uses bands 40, 20, and 10 as R, G, and B.
  • Trento Data: The Trento dataset is a rural scene with 166 × 600 pixels, 1 m resolution, and 63 hyperspectral bands.It contains six classes and uses standard training and test sets.

B. Experimental Setup

The evaluation compares single-source baselines and fusion variants on both datasets, then benchmarks the proposed model against state-of-the-art methods on Houston.

  • CNN-HS and CNN-LiDAR are used as single-source baselines for comparing fusion methods on Houston and Trento.
  • The proposed model is additionally compared with numerous state-of-the-art models on the Houston data.
  • Deep learning models are implemented in PyTorch and optimized with Adam using batch size 64, learning rate 0.001, and 200 epochs.
  • Performance is evaluated using overall accuracy, average accuracy, per-class accuracy, and the Kappa coefficient.OA is the fraction of correctly classified test pixels, while AA averages class accuracies.

1) Comparison with different fusion models:

Feature-level fusion generally improves on hyperspectral-only classification, while combining feature- and decision-level fusion is especially effective, with maximization best on Trento.

  • Different fusion models: CNN-HS outperforms CNN-LiDAR for each Houston class, indicating stronger discrimination from hyperspectral spectral-spatial information.
  • Different fusion models: All three Houston feature-level fusion variants exceed CNN-HS in most classes, consistent with complementary LiDAR information.The variants are concatenation, maximization, and summation.
  • Different fusion models: On Trento, all models exceed 90% OA, feature-level fusion improves CNN-HS, and joint feature- and decision-level fusion performs better than feature-level fusion alone.
  • Different fusion models: The best Trento results are obtained with maximization fusion.
  • State-of-the-art models: The proposed CNN-DF-M and CNN-DF-S models outperform comparable two-branch CNNs and E-UGF in the Houston state-of-the-art comparison.
  • State-of-the-art models: Table V compares models using OA, AA, and Kappa, with prior results taken directly from original papers because reproduction was not possible.

D. Analysis on the proposed model

The reduced hyperspectral dimensionality k initially improves OA before reaching a stable state, with k=20 selected to balance computation complexity and classification performance on both datasets.

  • OA initially increases with k and then tends to stabilize on both the Houston and Trento datasets.
  • k=20 is selected for both datasets based on computation complexity and classification performance.

2) Analysis on the neighboring size:

The neighboring size affects single-source models unevenly, while coupling reduces parameters substantially and slightly improves fused-model OA.

  • 2) Analysis on the neighboring size:: Increasing the neighboring size from 9 to 11 improves CNN-HS and CNN-LiDAR OA by more than 1 percent on Houston.
  • 2) Analysis on the neighboring size:: 11 is selected as a reasonable neighboring size for CNN-HS and CNN-LiDAR on both datasets.This choice follows the observed OA changes across candidate sizes and agrees with prior works and.
  • 2) Analysis on the neighboring size:: CNN-LiDAR OA increases by more than 1 percent from 9 to 11 and decreases from 11 to 13 on Trento.
  • 3) Analysis on the coupling strategy:: The coupling strategy reduces trainable parameters by about 47% on both datasets.For CNN-DF-M and CNN-DF-S, Houston parameters decrease from 196128 to 103968, while Trento parameters decrease from 192672 to 100512.
  • 3) Analysis on the coupling strategy:: Coupling slightly improves the OA of CNN-DF-C, CNN-DF-M, and CNN-DF-S.

4) Analysis on the computation cost:

The proposed fusion models require more training time than single-source and feature-level models but retain high test efficiency, while the study also evaluates key hyper-parameters.

  • 4) Analysis on the computation cost:: Testing the proposed fusion models takes no more than 2 seconds after training.This test efficiency is close to the time costs of the other models.
  • 4) Analysis on the computation cost:: The proposed decision-level and feature-level fusion models cost more training time than single-source and feature-level fusion models.Single-source models train faster because they process only one data source without interactions between sources.
  • 5) Analysis on the weight parameters:: The highest OA occurs at λ2=0.01, and the optimal values for λ1 and λ2 are both 0.01.
  • IV. CONCLUSIONS: The conclusion reports the proposed model's best performance on Houston and very high performance on Trento.The paper also evaluates reduced dimensionality and neighboring size as classification hyper-parameters.
Loading 2002.01144v1…