PaperlayerView exact revision on arXiv

Source-linked AI summary

More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification

Danfeng Hong, Lianru Gao, Naoto Yokoya, Jing Yao, Jocelyn Chanussot, Qian Du, Bing Zhang

arXiv:2008.05457v1cs.CVeess.IV

TL;DR

Single-modality remote sensing classification can miss complementary spectral, spatial, and other scene information, limiting detailed characterization. This paper proposes a unified multimodal deep-learning framework with five fusion strategies for pixel-level and spatial-spectral classification, reporting average gains of around 10% over single modalities across three indices.

  • Problem

    Single-modality remote sensing data provide limited information diversity, although combining modalities can characterize scenes more precisely.

  • Method

    The MDL-RS framework unifies pixel-level and CNN-based spatial-spectral classification with five fusion modules, including newly proposed cross fusion for multimodal and cross-modal learning.

  • Results

    Average OA, AA, and κ increased by 10% over single modalities, while compactness-based fusion outperformed concatenation-based fusion in multimodal and cross-modal settings.

  • Takeaways & Limitations

    The framework provides a unified baseline for blending multimodal remote sensing features across pixel-based and spatial-spectral classification tasks.

Abstract

from arXiv · show

Classification and identification of the materials lying over or beneath the Earth's surface have long been a fundamental but challenging research topic in geoscience and remote sensing (RS) and have garnered a growing concern owing to the recent advancements of deep learning techniques. Although deep networks have been successfully applied in single-modality-dominated classification tasks, yet their performance inevitably meets the bottleneck in complex scenes that need to be finely classified, due to the limitation of information diversity. In this work, we provide a baseline solution to the aforementioned difficulty by developing a general multimodal deep learning (MDL) framework. In particular, we also investigate a special case of multi-modality learning (MML) -- cross-modality learning (CML) that exists widely in RS image classification applications. By focusing on "what", "where", and "how" to fuse, we show different fusion strategies as well as how to train deep networks and build the network architecture. Specifically, five fusion architectures are introduced and developed, further being unified in our MDL framework. More significantly, our framework is not only limited to pixel-wise classification tasks but also applicable to spatial information modeling with convolutional neural networks (CNNs). To validate the effectiveness and superiority of the MDL framework, extensive experiments related to the settings of MML and CML are conducted on two different multimodal RS datasets. Furthermore, the codes and datasets will be available at https://github.com/danfenghong/IEEE_TGRS_MDL-RS, contributing to the RS community.

I. INTRODUCTION · II. RELATED WORK · A. Shallow Models for MML

Remote-sensing classification increasingly requires multimodal learning because single modalities provide limited information, while complementary observations enable more detailed scene characterization. The paper proposes MDL-RS, a unified framework addressing modality selection, fusion mechanisms, and fusion locations for both multimodality and cross-modality learning, alongside reviewing shallow multimodal models.

  • I. INTRODUCTION: Remote-sensing image classification supports applications including urban planning, forest monitoring, soil analysis, and disaster response.
  • I. INTRODUCTION: Most existing remote-sensing classification methods target single modalities such as hyperspectral, multispectral, LiDAR, SAR, or OpenStreetMap data.
  • I. INTRODUCTION: Combining modalities can exploit complementary properties, such as hyperspectral spectral discrimination, multispectral spatial detail, and SAR’s ability to address cloud-related missing information.
  • I. INTRODUCTION: MDL-RS is proposed as a unified multimodal deep-learning framework that addresses what to fuse, how to fuse, and where to fuse in remote-sensing classification.
  • I. INTRODUCTION: The framework supports both multimodality learning and cross-modality learning, including compact modality blending when one modality is absent during inference.
  • I. INTRODUCTION: Five plug-and-play fusion modules are investigated: early, middle, late, encoder-decoder, and cross fusion, with cross fusion introduced to transfer information across modalities more effectively.
  • II. RELATED WORK: Related work introduces multimodality learning and cross-modality learning as the two topics organizing the subsequent discussion.
  • A. Shallow Models for MML: Earlier shallow multimodal models used morphological operators, attribute profiles, graph-based subspace learning, sparse and low-rank modeling, feature stacking, and manifold alignment for remote-sensing fusion.

B. Deep Models for MML … B. Extraction Network (Ex-Net)

The paper reviews multimodal deep-learning approaches for remote-sensing classification, including pixel-level networks and cross-modality learning, then introduces MDL-RS as a generic end-to-end framework with modality-specific extraction and fusion modules. Its Ex-Net extracts hierarchical features from heterogeneous modalities before fusion, using separate modality streams and a fusion stream.

  • B. Deep Models for MML: Deep multimodal RS models are broadly categorized into two groups, including common pixel-level multimodal classification networks.Examples use CNN-based feature extraction and fusion for hyperspectral, LiDAR, and other multisource RS data.
  • B. Deep Models for MML: Pixel-level approaches include deep feature fusion, end-to-end two-CNN/one-DNN fusion, and two-branch CNNs with cascade blocks.These designs target automatic feature extraction and fusion from multimodal RS inputs.
  • B. Deep Models for MML: A fused FCN was reported for MS-LiDAR and HS data, while segmentation networks require abundant labeled images and high-resolution sources.The passage notes that these requirements increase time and cost challenges.
  • C. CML: A Special Case of MML: Cross-modality learning trains a model to achieve the same or closer performance when inference uses either one modality or multiple modalities.The passage describes CML as a special family of multimodal learning and notes growing research attention.
  • A. Method Overview: The proposed MDL-RS is a generic end-to-end multimodal network for RS imagery classification with pixel-wise FC-Net and spatial-spectral CNN forms.Both forms contain an Extraction Network and a Fusion Network for multimodal feature representation learning.
  • B. Extraction Network (Ex-Net): Ex-Net extracts hierarchical representations from different modalities so heterogeneous data can be blended more effectively in feature space than in the original space.The method is motivated by the difficulty of directly fusing data from different sensors.
  • B. Extraction Network (Ex-Net): Ex-Net models two aligned modality inputs sharing one-hot label information across C categories and N pixels.The inputs have dimensions d1×N and d2×N, respectively, with aligned pixel pairs x1,i and x2,i.
  • B. Extraction Network (Ex-Net): The Ex-Net uses separate streams for the two modalities and a fusion stream, with learned linear-regression operations such as encoders or convolutions.The stream index is s = 1, 2 for modalities and s = 0 for fusion.

zBN

The zBN procedure computes a z-score result before passing it through a nonlinear activation. Its learnable parameters are stream-specific, and the activation is implemented with ReLU.

  • zBN: zBN produces s,i as the z-score result of z(l).
  • zBN: The learnable parameters γs and βs are associated with the s-th network or modality stream.
  • zBN: The zBN output is imported into the next block before producing the activated output a(l) s,i.
  • zBN: The nonlinear activation function u(·) is implemented by ReLU.

C. Fusion Network (Fu-Net)

The Fu-Net receives Ex-Net encoded features end-to-end and learns multimodal representations through nonlinear mappings. It organizes fusion by how and where modalities are combined, including early, middle, late, and compact fusion strategies.

  • C. Fusion Network (Fu-Net): Ex-Net encoded features are fed into the Fu-Net as new inputs in an end-to-end network.The encoded features are denoted as {A_s} and serve as Fu-Net inputs.
  • C. Fusion Network (Fu-Net): The Fu-Net output is generalized through a nonlinear mapping function composed of several network blocks.A block consists of encoder or convolution operations, batch normalization, and nonlinear activation.
  • C. Fusion Network (Fu-Net): The framework examines fusion strategies by how modalities are combined, including concatenation-based, encoder-decoder, and cross fusion.Encoder-decoder and cross fusion are identified as compactness-based fusion methods.
  • C. Fusion Network (Fu-Net): Fusion is categorized by where it occurs as early fusion, middle fusion, or late fusion.These strategies determine how vector representations are formed from different modality streams.
  • C. Fusion Network (Fu-Net): Concatenation-based fusion stacks outputs from different streams but has limited ability to blend heterogeneous modality properties.The Fu-Net therefore also considers more compact feature fusion.

2) Compactness-based fusion: · 1) Concatenation-based fusion:

The paper presents cross fusion as a compactness-oriented module that interactively updates subnetworks across modalities. It also describes concatenation-based fusion through shared parameters, cross-modal weights, and feature interactions that produce compact representations for subsequent layers.

  • 2) Compactness-based fusion:: En-De fusion is a representative compactness-based approach formulated by minimizing a reconstruction loss.The formulation uses an encoder and a reconstruction-based decoder with respect to estimated variable sets.
  • 2) Compactness-based fusion:: Cross fusion interactively updates parameters across modality-specific subnetworks to learn more compact cross-modal feature representations.Each stream learns its own properties while incorporating supplementary information from another modality.
  • 2) Compactness-based fusion:: The cross-fusion design enables each modality stream to combine modality-specific properties with diversified information from another stream.This setting targets more sufficient information blending across modalities.
  • 1) Concatenation-based fusion:: Concatenation-based fusion combines modality-specific transformed features through additive cross-modal interactions.The supplied formulation shows each output using transformations of both modality inputs.
  • 1) Concatenation-based fusion:: The three components of v_i share the same learnable parameters and can be treated as three new samples for the next layer.This shared-parameter interpretation enforces a more compact fusion.
  • 1) Concatenation-based fusion:: Interactive crossing of weights and features enables compact fusion representations across network streams.The process is illustrated in Fig. 4(e), with learned weights used across modalities.

D. Significance of Compact Blending in CML · E. Network Architecture for MDL-RS

CML addresses limited large-scale availability of spatially rich imagery by learning feature representations from multiple source data. MDL-RS provides a baseline architecture with single-stream or two-stream networks, configurable fusion locations, and specialized CNN pooling for spatial information.

  • D. Significance of Compact Blending in CML: CML can mitigate classification bottlenecks by learning better feature representations from multiple source data.Large-scale EO data such as MS and SAR are available, whereas spatially richer HS imagery is difficult to acquire extensively.
  • D. Significance of Compact Blending in CML: Concatenation-based fusion creates massive same-modality connections while activating few neurons across modalities.The observed activations can favor one modality while inhibiting another during learning.
  • E. Network Architecture for MDL-RS: MDL-RS is a baseline multimodal remote-sensing classification network designed to accommodate plug-and-play modules.Its basic architecture includes pixel-wise FC-Nets and spatial-spectral CNNs, with layer-wise configurations listed in Table I.
  • E. Network Architecture for MDL-RS: Single-modality and early-fusion configurations use a single-stream network based on Ex-Net or Fu-Net.The remaining fusion configurations use two-stream Ex-Net structures.
  • E. Network Architecture for MDL-RS: Fusion occurs at the input for early fusion, Block 5 of Fu-Net for middle, en-de, and cross fusion, and Block 7 for late fusion.These locations define where modality representations are combined in the basic architecture.
  • E. Network Architecture for MDL-RS: En-de fusion uniquely learns additional parameters to reconstruct fused features from Fu-Net Block 4.Its reconstruction module resembles Ex-Net but removes batch normalization and replaces ReLU with Sigmoid.
  • E. Network Architecture for MDL-RS: CNN pooling layers are spaced to improve spatial feature extraction, with average pooling in Block 6 reducing spatial-information loss.Convolution and pooling strides are both set to 1 for patch-based inputs.

IV. EXPERIMENTS … 1) Implementation details:

Experiments assess the multimodal deep learning framework on HS-LiDAR and MS-SAR datasets, using specified data configurations and training procedures. The setup also evaluates multimodal inference and cross-modality inference by removing one modality.

  • A. Data Description: Two multimodal datasets, HS-LiDAR and MS-SAR, are used for quantitative and qualitative performance assessment.
  • 1) HS-LiDAR Houston2013 data:: The Houston2013 HS-LiDAR scene contains 144 hyperspectral bands, one LiDAR band, 349×1905 pixels, and 15 LULC-related categories.The hyperspectral wavelengths span 364nm–1046nm at a 10nm spectral interval.
  • 2) MS-SAR LCZ data:: LCZ data combine Sentinel-2 multispectral imagery with 10 spectral bands and Sentinel-1 dual-polarimetric SAR represented by a four-component PolSAR covariance matrix.Training uses Berlin while inference uses a separate area to avoid information leakage.
  • B. Experimental Setup: The networks are implemented in TensorFlow, with hyperparameters selected by grid search on validation sets created through ten random 8:2 training–validation splits.The models are trained on the training set, and the validation splits support final hyperparameter selection.
  • 1) Implementation details:: Training uses Adam with an exponential learning-rate policy, batch sizes of 64 and 256 for the first and second datasets, respectively, and ℓ2-norm weight regularization.Training stops when validation loss fails to decrease.
  • 1) Implementation details:: For HS-LiDAR, attribute profiles extracted from the single-band LiDAR image produce 21-band profiles to exploit spatial information and support FC-Net learning.
  • 1) Implementation details:: Models are trained with multiple modalities and evaluated both with multimodal inputs and in the cross-modality setting by zeroing one modality.The passage gives bi-modality as an example.

2) Evaluation metric:

The MDL-RS framework is evaluated on pixel-level remote-sensing image classification using Overall Accuracy, Average Accuracy, and the Kappa Coefficient. These indices quantify classification performance, with Kappa accounting for hypothetical chance agreement.

  • Evaluation metric: Pixel-level classification performance is quantified using Overall Accuracy (OA), Average Accuracy (AA), and Kappa Coefficient (κ).These three commonly used indices are calculated for evaluating the proposed MDL-RS framework.
  • Evaluation metric: Overall Accuracy uses Nc, the number of correctly classified samples, and Na, the total number of samples.The metric formulations define these quantities as the correct and total sample counts, respectively.
  • Evaluation metric: Kappa Coefficient (κ) incorporates Pe, the hypothetical probability of chance agreement.Pe is computed from the numbers of real and predicted samples for each class.

3) Comparison with state-of-the-art baselines: · C. Result and Analysis on Houston Data · 1) Quantitative comparison:

The Houston-data comparison evaluates single modalities and five fusion strategies with FC-Nets and CNNs across multimodal and cross-modality settings. Compactness-based cross fusion achieves the strongest overall results, while CNNs improve compared methods by 2%∼3% on the three main indices.

  • 3) Comparison with state-of-the-art baselines:: The study compares early, middle, and late concatenation-based fusion, en-de and cross compactness-based fusion, and single modalities using FC-Nets and CNNs.CNN inputs generally use a patch centered by a pixel.
  • 1) Quantitative comparison:: FC-Net comparisons report overall accuracy, average accuracy, κ, and per-category accuracy across MML, CML-HSI, and CML-LiDAR settings.These results are listed in Table IV.
  • 1) Quantitative comparison:: Single HSI exceeds single LiDAR by over 15% OA, while multimodal inputs outperform single modalities because they provide greater feature diversity.The LiDAR input uses pre-extracted APs before network processing.
  • 1) Quantitative comparison:: Compactness-based fusion generally surpasses concatenation-based fusion by at least 1% OA, AA, and κ, with cross fusion achieving the best classification results.Middle and late fusion are more effective than early fusion.
  • 1) Quantitative comparison:: In CML, early fusion falls to 28.13% OA in CML-HSI and 12.76% OA in CML-LiDAR when one modality is missing during inference.Other concatenation-based strategies are lower than single-modality results, indicating limited feasibility for CML.
  • 1) Quantitative comparison:: Cross fusion overcomes competing methods in both MML and CML, transfers information between modalities effectively, and exceeds single modalities in classification accuracy.Compactness-based networks also outperform concatenation-based models in per-class performance.
  • 1) Quantitative comparison:: CNN-based architectures outperform FC-Net architectures overall, increasing all compared algorithms by 2%∼3% across the three main indices.The CNN design extracts semantically meaningful information from locally neighboring pixels.

2) Visual comparison:

Visual comparisons show that multimodal deep learning reduces semantic-labeling errors and that CNNs produce smoother classification maps than FC-Nets. Fusion strategies further improve recognition under spectral variability and complex materials, with category-specific advantages for cross or compactness-based fusion.

  • Visual comparison:: MDL reduces semantic-labeling errors relative to single modalities, while compactness-based fusion produces more realistic classification maps.
  • Visual comparison:: CNNs generate smoother classification maps than FC-Nets by removing noisy pixels.
  • Visual comparison:: Multimodal fusion improves robustness to spectral variability, including cloud cover in optical imagery, by leveraging sources such as LiDAR.
  • Visual comparison:: Cross fusion identifies important visual, spectral, and other cues in complex materials, producing results closer to ground truth.
  • Visual comparison:: En-de or cross fusion better recognizes Residential and Commercial, whereas CML-HSI with compactness-based fusion better identifies cloud-covered Road, Highway, and Grass.The latter advantage is attributed to more effective information transfer from LiDAR data.

D. Result and Analysis on LCZ Data … V. CONCLUSION

On LCZ MS-SAR data, the MDL-RS framework improves multimodal classification over single modalities, with cross fusion producing smoother and more detailed maps. The paper concludes that fusion location and strategy matter, while sample dependence motivates future weakly or self-supervised learning.

  • D. Result and Analysis on LCZ Data: LCZ classification is more difficult than Houston classification because 100m pixels contain complex categories, MS and SAR use different imaging mechanisms, and cross-city transferability is required.These factors help explain inferior LCZ performance, particularly when comparing SAR with LiDAR in cross-modality learning.
  • 1) Quantitative comparison:: 10% average increases in OA, AA, and κ distinguish MDL-RS from single-modality results, while CNN-based methods exceed FC-based methods by around 10% across all three indices.The passage describes these trends as consistent with results on the HS-LiDAR Houston2013 datasets.
  • 2) Visual comparison:: Cross fusion produces smoother and more detailed classification maps than other MML fusion approaches by using cross learning to reduce the modality gap.En-de fusion shows a similar visual conclusion but achieves slightly lower accuracy than cross fusion.
  • V. CONCLUSION: The proposed MDL-RS framework uses Ex-Net and Fu-Net subnetworks as a baseline for pixel-level remote-sensing image classification with multimodal data.The framework investigates multiple fusion strategies within this architecture.
  • V. CONCLUSION: MDL-RS addresses “what,” “where,” and “how” to fuse, using FC-Nets for pixel-based classification and CNNs for spatial-spectral classification while extending analysis from MML to CML.The framework generalizes four well-known fusion modules, although the supplied passage truncates before listing them all.
  • V. CONCLUSION: Middle fusion and late fusion tend to produce better classification results than early fusion, with middle fusion performing particularly well in quantitative and qualitative assessments.En-de fusion and cross fusion follow the middle-fusion architecture.
  • V. CONCLUSION: Concatenation-based fusion often performs poorly in CML, whereas compactness-based en-de and cross fusion blend multimodal features effectively in both MML and CML.The conclusion presents cross fusion as newly proposed and en-de fusion as part of the compactness-based strategies.
  • V. CONCLUSION: Dependence on sample quality and quantity is especially strong for deep-learning models, motivating future weakly supervised or self-supervised techniques and better-designed fusion modules.The authors identify this dependence as a performance bottleneck for MDL.
Loading 2008.05457v1…