PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Interpretable Multimodal Fusion: Heat Conduction Modeling for Hyperspectral and LiDAR Joint Classification

Kan Wei, Jiahui Cui, Jing Yao, Xinyu Zhao, Lei Wang, Pedram Ghamisi

arXiv:2609.11040v1cs.CVcs.AI

TL;DR

HSI–LiDAR fusion must combine complementary spectral and structural information while addressing heterogeneous modalities and efficient long-range modeling. M2Heat uses heat-conduction-guided vHeat, enhanced FVEs, and Cross-Frequency Fusion to perform interpretable sub-quadratic multimodal modeling. It achieves competitive performance across Trento, Houston2013, and Augsburg while producing smoother, more spatially consistent classification maps in challenging urban regions.

  • Problem

    HSI–LiDAR fusion seeks to combine complementary spectral and structural cues, but existing methods struggle with heterogeneous representations, long-range dependencies, and computational efficiency.

  • Method

    M2Heat uses a heat conduction operator with enhanced Frequency Value Embeddings and Cross-Frequency Fusion to propagate and align HSI–LiDAR features in spatial and frequency domains.

  • Results

    M2Heat achieves competitive overall performance on the Trento, Houston2013, and Augsburg benchmarks and produces smoother, more spatially consistent classification maps.

  • Takeaways & Limitations

    Heat-conduction-guided neural operators provide an interpretable perspective for efficient multimodal remote-sensing fusion.

Abstract

from arXiv · show

The fusion of hyperspectral (HS) and Light Detection and Ranging (LiDAR) data plays a crucial role in enhancing land-cover classification by jointly exploiting spectral, spatial, and structural cues. However, existing multimodal fusion methods still struggle to model long-range dependencies and complex anisotropic interactions while maintaining computational efficiency. This paper introduces M2Heat, a physics-inspired framework that investigates multimodal fusion through the lens of heat conduction. At its core, a physics-driven visual heat conduction module (vHeat) and enhanced Frequency Value Embeddings (FVEs) simulate anisotropic information flow, enabling the capture of global dependencies with sub-quadratic complexity and physical interpretability. This mechanism, combined with a hybrid spatial-frequency fusion strategy named Cross-Frequency Fusion (CFF) module, produces highly discriminative and robust feature representations. M2Heat achieves competitive overall performance on three benchmarks, i.e., Trento, Houston2013, and Augsburg, while providing an interpretable heat-conduction-guided perspective for multimodal feature fusion. These results indicate the potential of heat-conduction-guided neural operators for efficient and interpretable RS multimodal fusion. The source code is publicly available at https: /github.com/Weikan0425/M2Heat_HSI_LiDAR.

I. INTRODUCTION

HSI provides detailed spectral information while LiDAR contributes structural cues, but multimodal fusion still faces heterogeneous representations, scale disparities, and difficulty modeling long-range dependencies efficiently. M2Heat addresses this gap with heat-conduction-based global modeling and frequency-domain cross-modal fusion.

  • I. INTRODUCTION: HSI and LiDAR provide complementary information, respectively offering fine spectral signatures and rich structural cues for challenging land-cover scenes.Multimodal integration is motivated by the limitations of single-modality methods in environments with diverse land covers, shadows, and structural occlusions.
  • I. INTRODUCTION: Convolution captures local patterns, self-attention models global context with quadratic complexity, and Mamba offers linear-complexity global modeling but limited interpretability for structured remote sensing data.These trade-offs motivate an alternative that combines efficient global modeling with a physically interpretable mechanism.
  • I. INTRODUCTION: M2Heat combines a heat conduction operator with Cross-Frequency Fusion to model global spatial–spectral dependencies and align heterogeneous HSI–LiDAR representations.The framework uses learnable thermal diffusivity through Frequency Value Embeddings and performs fusion across spatial and frequency domains.
  • I. INTRODUCTION: Physics-inspired priors are used to improve interpretability and structural consistency by constraining learned representations according to diffusion and thermodynamic principles.Prior work motivates applying these constraints to remote sensing problems involving energy transport, material interactions, and multimodal alignment.
  • I. INTRODUCTION: M2Heat extends heat-based visual modeling to HSI–LiDAR classification by treating feature propagation as deterministic heat conduction rather than iterative generative diffusion.The formulation targets coupled spectral–spatial–structural propagation and cross-modal alignment.

B. Multimodal Alignment and Feature Fusion: From Shallow Interaction to Hybrid Integration

M2Heat addresses heterogeneous HSI–LiDAR fusion by combining modality-specific and shared heat-conduction processing with frequency-aware integration. The framework uses this pipeline to align complementary spectral and structural representations for pixel-wise classification.

  • B. Multimodal Alignment and Feature Fusion: From Shallow Interaction to Hybrid Integration: HSI and LiDAR provide complementary spectral and structural cues, but disparities in resolution, distributions, and semantics make robust alignment difficult.These challenges motivate unified, dynamically adaptive fusion rather than shallow interaction alone.
  • A. Overall Framework of M2Heat: M2Heat processes co-registered HSI and LiDAR patches through Stem Layers, modality-specific HCO Layers, a shared HCO Layer, CFF, and a classifier.The shared HCO aligns heterogeneous representations in a unified latent space before CFF-based multi-frequency interaction.
  • 1) Heat Conduction Operator Formulation:: The framework is grounded in heat conduction, modeling feature evolution as diffusion across spatial–spectral dimensions.The formulation treats feature maps as temperature-like fields and uses diffusion to represent information propagation and structural smoothing.
  • 1) Heat Conduction Operator Formulation:: DCT-domain diffusion preserves low-frequency global structures while attenuating high-frequency components, providing efficient multi-scale feature evolution.The use of DCT supports reflective boundary conditions for finite patches and yields approximately O(N^1.5) complexity.

2) Heat Conduction Operator Formulation:

The HCO formulation separates modality-specific feature extraction from shared cross-modal alignment. Learnable FVEs generate non-negative, potentially spatially and channel-varying diffusion coefficients for adaptive heat conduction.

  • 2) Heat Conduction Operator Formulation:: M2Heat projects HSI and LiDAR inputs into a shared latent space, then splits features for modality-specific processing and shared HCO-based alignment.The split supports separate extraction of discriminative modality cues alongside joint representation alignment.
  • 2) Heat Conduction Operator Formulation:: FVEs θ_q are mapped through a learnable linear transformation and ReLU to produce the effective diffusion coefficient k_q.The ReLU mapping preserves the non-negative diffusivity constraint required by the forward heat-conduction formulation.
  • 2) Heat Conduction Operator Formulation:: FVE-based coefficients can vary across spatial positions and channels, adapting diffusion behavior to heterogeneous modalities and scenes.This makes the propagation pattern responsive to the differing structures of HSI and LiDAR features.

3) HCO Architecture and Theoretical Properties:

The HCO architecture combines local convolution, gated conduction, normalization, residual updates, and feed-forward refinement. Its physical core supports interpretable anisotropic diffusion while auxiliary neural components adapt the flow.

  • 3) HCO Architecture and Theoretical Properties:: Each HCO Layer follows an LN–Operator–LN–FFN structure, with depthwise convolution supplying local spatial dependencies before heat-conduction processing.The architecture combines transformer-style normalization and feed-forward structure with physically interpretable dynamics.
  • 3) HCO Architecture and Theoretical Properties:: Parallel linear projections form gating and conduction paths, whose element-wise modulation implements adaptive diffusion control.The gating branch modulates the conduction response rather than replacing the physical flow.
  • 3) HCO Architecture and Theoretical Properties:: CFF combines frequency-domain fusion with dynamic fusion to support effective multimodal feature interaction.The module uses amplitude- and phase-based representations for cross-modal alignment and complementary feature enhancement.
  • 3) HCO Architecture and Theoretical Properties:: HCO explicitly learns anisotropic diffusion patterns that vary directionally with input geometry and texture structure.Its DCT-domain attenuation solves the discretized heat equation through Laplacian eigenmode modulation, while neural components provide data-adaptive calibration.
  • 3) HCO Architecture and Theoretical Properties:: The heat-conduction formulation links physics-based diffusion theory with deep spectral–spatial modeling and interpretable multimodal propagation.The paper describes this integration as providing enhanced robustness, generalization, and physical plausibility relative to purely data-driven fusion mechanisms.

1) Frequency-domain decomposition:

CFF decomposes HSI and LiDAR features into frequency-domain amplitude and phase components before jointly learning aligned representations. Inverse transformation reconstructs spatial features containing complementary semantic and geometric information.

  • 1) Frequency-domain decomposition:: Each modality’s feature map is transformed into the frequency domain with a 2D FFT and decomposed into amplitude and phase.Amplitude represents spatial-frequency energy distribution, while phase preserves high-frequency structural details.
  • 1) Frequency-domain decomposition:: The frequency representation separates low-frequency semantic structures from high-frequency geometric details for complementary modeling.This decomposition supports distinct treatment of global semantic content and fine structural information.
  • 1) Frequency-domain decomposition:: The amplitude and phase components from both modalities are concatenated and processed by a shared convolution-GELU-convolution MLP.This shared processing performs joint frequency-aware learning for cross-modal alignment.
  • 3) Spatial reconstruction:: Inverse FFT transforms the fused frequency representations back into spatial features integrating complementary information from both modalities.Element-wise complex reconstruction combines the learned amplitude and phase components before spatial-domain output.

4) Weighted integration with HCO features:

The method combines HCO-aligned and frequency-fused representations through learnable weighting, then classifies the resulting modality-aware features with a lightweight prediction head.

  • 4) Weighted integration with HCO features:: Learnable nonnegative weights α and β combine the frequency-fused and HCO-aligned features, balancing spatial- and frequency-domain contributions.A 1 × 1 convolution aligns channels and adds nonlinear refinement before the weighted fusion.
  • 4) Weighted integration with HCO features:: The CFF module combines frequency-domain amplitude–phase interactions with spatial-domain correlations, while DCT supports boundary-consistent heat diffusion in HCO.CFF uses FFT for complex-valued amplitude–phase separation, whereas HCO uses DCT for real-valued diffusion over finite patches.
  • 4) Weighted integration with HCO features:: The classification head reduces channels, applies batch normalization and LeakyReLU, and uses global average pooling to produce a compact feature vector.The fused feature ˆU_t ∈ R^H×W×C is transformed into z ∈ R^C for each input sample.
  • 4) Weighted integration with HCO features:: A second 1 × 1 convolution and softmax project the compact vector into normalized probabilities over K classes.The model is trained with standard cross-entropy over labeled pixels, using ground-truth indicators and predicted probabilities.

IV. EXPERIMENT AND DISCUSSION

The experiments evaluate M2Heat on three HSI–LiDAR benchmarks using quantitative metrics, classification maps, and comparisons with existing methods.

  • IV. EXPERIMENT AND DISCUSSION: M2Heat is evaluated on the Trento, Houston2013, and Augsburg HSI–LiDAR benchmarks against nine state-of-the-art methods, with ablation and supplementary analyses.The evaluation includes dataset descriptions, experimental protocols, metrics, and robustness analyses.
  • IV. EXPERIMENT AND DISCUSSION: Trento contains co-registered HSI and LiDAR data from rural Italy, with 63 spectral bands, 1 m resolution, and six labeled land-use classes.The scene has spatial size 166 × 600 pixels and 30,214 labeled samples.
  • IV. EXPERIMENT AND DISCUSSION: The evaluation reports Overall Accuracy, Average Accuracy, and the Kappa coefficient, alongside classification maps for the benchmark datasets.The supplied figures and tables identify the Trento result table and classification-map visualizations.

3) Augsburg [61]:

Augsburg provides a benchmark setting with hyperspectral and LiDAR-derived DSM data, and M2Heat uses dataset-specific training choices selected through validation experiments.

  • 3) Augsburg [61]:: Augsburg combines spaceborne HSI with a LiDAR-derived DSM, resampled to 30 m ground sampling distance for computational consistency.The HSI contains 180 bands spanning 0.4–2.5 µm, while the DSM supplies elevation information.
  • 3) Augsburg [61]:: The experiment uses Adam for 500 epochs with StepLR scheduling, spectral normalization, and dataset-specific learning rates, batch sizes, and patch sizes.The implementation runs in PyTorch 2.0 on an RTX 4090 GPU.
  • 3) Augsburg [61]:: Validation identifies a learning rate of 5e-4 and batch size of 64 as the best Augsburg configuration, reaching 87.48% OA.Moderate batch sizes and mid-range learning rates performed best in this dataset.
  • 3) Augsburg [61]:: Patch-size testing finds the highest Augsburg OA at 90.30% with a 7 × 7 patch, while performance remains stable across the tested neighborhood range.The learning rate and batch size were fixed to the dataset-specific settings during this analysis.
  • 3) Augsburg [61]:: The final Augsburg hyperparameters are learning rate 5e-4, batch size 64, and patch size 7 × 7.The settings are selected separately from those used for Trento and Houston2013.

D. Comparison Result and Analysis

Across Trento and Houston2013, M2Heat combines strong quantitative performance with spatially coherent classification maps, while ablations examine fusion representations, efficiency, and robustness.

  • 1) Trento Dataset:: On Trento, single-source HSI methods reach 94.51% and 96.06% OA, whereas multisource methods generally perform better by using complementary spatial and elevation cues.The comparison includes SF and MambaHSI as HSI-only baselines.
  • 1) Trento Dataset:: 99.64% OA on Trento makes M2Heat the top-performing method, exceeding S2CMamba at 99.21% and preserving near-best classwise accuracy with κ=0.9952.M2Heat also performs strongly on the challenging sixth class, reaching 97.91% accuracy.
  • 1) Trento Dataset:: Trento classification maps show spatially coherent, sharply delineated regions with better preservation of fine structural details and boundaries.Competing maps contain more noise and misclassification patches, especially for classes with subtle spectral differences.
  • 2) Houston2013 Dataset:: 91.20% OA, 92.32% AA, and κ=0.9045 on Houston2013 give M2Heat favorable overall performance and the highest Kappa coefficient among compared methods.The model performs best on several challenging urban classes, including Residential, Commercial, Highway, and Railway.
  • 2) Houston2013 Dataset:: Houston2013 maps indicate improved handling of shadows and mixed pixels, including shadowed regions caused by vegetation and tall buildings.The supplied passage also notes competing-model confusion between Water and Railway classes.

3) Augsburg Dataset:

On Augsburg, M2Heat achieves the strongest reported classification performance and produces smoother, more spatially consistent maps. Ablations further support the contributions of its dual fusion components and frequency-specific designs.

  • 90.30% OA and 0.8607 Kappa are the highest reported Augsburg results, confirming M2Heat’s robustness in integrating spectral and elevation information.
  • M2Heat improves challenging Residential Area, Low Plants, Allotment, and Water classes, although Industrial Area and Commercial Area remain difficult because of limited samples and class imbalance.
  • Smoother, more spatially consistent maps indicate reduced noise and misclassification in complex regions.
  • S1 improves OA by +1.40%, +0.25%, and +1.33%, while S2 adds +1.34%, +1.61%, and +0.67% across Trento, Houston, and Augsburg.
  • The full model combining S1 and S2 reaches 99.64%, 91.20%, and 87.48% OA on Trento, Houston2013, and Augsburg, respectively, exceeding partial configurations.
  • FFT-AP is the strongest CFF representation, outperforming the best alternative by 0.13%, 0.52%, and 0.42% across the three datasets.

2) Comparison Between Different Operators:

The operator comparison shows that HCO combines strong classification, global feature propagation, and moderate computational cost. Its learned diffusivity also remains effective under reduced supervision and yields more separable feature clusters.

  • Replacing HCO with convolution, MSA, or Mamba causes noticeable degradation; MSA incurs prohibitive overhead, while Mamba lacks explicit frequency-aware fusion.
  • HCO achieves 99.64%, 91.20%, and 87.48% OA on the three datasets while using 555.62k parameters, 5.334G FLOPs, and 0.53 s inference time.
  • HCO performs two-dimensional global propagation in the frequency domain while preserving image-patch spatial organization.
  • HCO retains the highest OA at every reduced training ratio and the lowest average ROD: 0.252%, 0.742%, and 0.490% on Trento, Houston2013, and Augsburg.
  • M2Heat produces clearer class separation than convolutional, MSA, and Mamba alternatives, including compact distributions on challenging Houston2013.

3) Analysis of Learnable Diffusion Patterns in HCO:

Learnable diffusivity patterns make HCO adaptive to spatial location, spectral channel, and modality-specific or shared representations. Perturbation experiments associate higher learned diffusivity with greater prediction sensitivity.

  • P × P × C diffusivity adapts jointly across spatial positions and spectral channels, producing richer variation than spatial-only FVEs.
  • The shared fusion branch learns larger diffusion coefficients than HSI and LiDAR branches, suggesting stronger diffusion for cross-modal interaction and aggregation.
  • Channel-averaged maps and channel-wise curves show distinct HSI, LiDAR, and shared response patterns, indicating modality-specific diffusion regulation.
  • Higher diffusivity regions are more sensitive to perturbation, with larger OA drops at higher ratios on Houston2013 and Augsburg.
  • M2Heat combines anisotropic spectral–spatial propagation, sub-quadratic global modeling, and cross-modal fusion through physics-inspired HCO and enhanced FVEs.
  • HCO remains more computationally expensive than linear-complexity operators such as Mamba, motivating further efficiency improvements.
Loading 2609.11040v1…