Source-linked AI summary
A3CLNN: Spatial, Spectral and Multiscale Attention ConvLSTM Neural Network for Multisource Remote Sensing Data Classification
Heng-Chao Li, Wen-Shuai Hu, Wei Li, Jun Li, Qian Du, Antonio Plaza
TL;DR
HSI and LiDAR provide complementary information, yet existing multisource classification methods may not fully exploit class-dependent scales or effectively integrate spectral and spatial information. The paper proposes dual-channel A3CLNN with attention-based feature extraction, three-level fusion, and stepwise training. Experiments report better performance than CNN-, capsule-network-, and ConvLSTM-based alternatives.
Problem
Existing HSI–LiDAR fusion methods may miss class-dependent multiscale information and fail to effectively integrate spectral and spatial information.
Source: §I · p. 3Method
Dual-channel A3CLNN uses spatial, spectral, and multiscale attention with three-level fusion and stepwise training for HSI–LiDAR classification.
Source: §I · pp. 3–4Results
The proposed model performs better than CNN-based, capsule-network-based, and baseline ConvLSTM-based approaches in the reported experiments.
Source: §V · p. 15Takeaways & Limitations
The reported findings support using dual-channel A3CLNN to integrate HSI and LiDAR information for multisource remote-sensing classification.
Source: §V · p. 15
Abstract
from arXiv · showhide
The problem of effectively exploiting the information multiple data sources has become a relevant but challenging research topic in remote sensing. In this paper, we propose a new approach to exploit the complementarity of two data sources: hyperspectral images (HSIs) and light detection and ranging (LiDAR) data. Specifically, we develop a new dual-channel spatial, spectral and multiscale attention convolutional long short-term memory neural network (called dual-channel A3CLNN) for feature extraction and classification of multisource remote sensing data. Spatial, spectral and multiscale attention mechanisms are first designed for HSI and LiDAR data in order to learn spectral- and spatial-enhanced feature representations, and to represent multiscale information for different classes. In the designed fusion network, a novel composite attention learning mechanism (combined with a three-level fusion strategy) is used to fully integrate the features in these two data sources. Finally, inspired by the idea of transfer learning, a novel stepwise training strategy is designed to yield a final classification result. Our experimental results, conducted on several multisource remote sensing data sets, demonstrate that the newly proposed dual-channel A3CLNN exhibits better feature representation ability (leading to more competitive classification performance) than other state-of-the-art methods.
I. INTRODUCTION
HSI and LiDAR offer complementary spectral and elevation information, but existing fusion methods may miss class-dependent scales and fail to integrate spectral and spatial information effectively. The paper introduces dual-channel A3CLNN, combining attention-based feature extraction, multilevel fusion, and stepwise training; it also identifies underused multiscale information and long-term dependencies in ConvLSTM3D-based HSI classification.
I. INTRODUCTION
- The motivation for fusing HSI and LiDAR is their complementary information: HSI provides spectral detail, while LiDAR provides elevation information.The introduction notes that LiDAR data are less affected by atmospheric interferers and can improve characterization of HSI scenes.Source: §I · p. 2
- Earlier HSI–LiDAR fusion approaches include decision fusion, morphological feature extraction, and deep models, but may not fully integrate spectral and spatial information.The introduction names two-branch CNN, patch-to-patch CNN, and dual-channel capsule network examples.Source: §I · p. 3
- Dual-channel A3CLNN combines HSI and LiDAR pipelines with three-level fusion and stepwise training to integrate their features for multisource classification.The fusion network reuses LiDAR features at its top because HSI features have a much larger order of magnitude.Source: §I · p. 3
- Learnable spectral and spatial attention modules extract enhanced features from HSI and LiDAR data.The introduction also proposes composite attention learning that combines spectral and spatial attention.Source: §I · p. 4
- A learnable multiscale residual attention module addresses the possibility that fixed-scale extraction misses scale information that differs across classes.The paper notes that fixed convolution-kernel sizes across classes may leave multiscale information unrepresented.Source: §I · pp. 3–4
A. ConvLSTM2D and ConvLSTM3D
- ConvLSTM3D was developed to better preserve HSI structure, but SSCL3DNN does not consider multiscale information or fully use long-term dependencies.ConvLSTM2D requires decomposing 3-D HSI data into a 2-D sequence, which may lose intrinsic structure.Source: A. ConvLSTM2D and ConvLSTM3D · p. 4
B. Attention Mechanism
The section motivates attention for feature enhancement and describes a dual-channel framework for combining complementary HSI and LiDAR information through composite attention and three-level fusion.
B. Attention Mechanism
- The section frames attention through Q, K, and V inputs and softmax normalization, identifying additive and dot-product attention as two common attention functions.It also identifies WQ and WK as parameter weights and T as the matrix transpose.Source: B. Attention Mechanism · p. 4
- Prior attention approaches include channel-wise, spatial-temporal, spatial-channel, and spatial-spectral feature enhancement, but effective attention implementations for multisource remote-sensing fusion and classification were lacking.Source: B. Attention Mechanism · p. 4
A. Architecture Overview
- The architecture uses two branches to exploit complementary HSI spectral and LiDAR height information, with three-level fusion and a multi-task-loss stepwise training strategy.Source: A. Architecture Overview · pp. 4–5
B. Composite Attention Learning
SeAB learns spectral-enhanced features by modeling dependencies across spectral components and using a normalized attention vector to weight its input.
- SeAB learns a spectral-enhanced representation by producing a normalized spectral attention vector that weights its input.Source: 1) Spectral Attention Block (SeAB) · p. 5
- The input is split into 2-D spectral components and fed sequentially to a 3×3 ConvLSTM2D layer to model long-range spectral dependencies, followed by a 1×1 layer that generates an attention map.Source: 1) Spectral Attention Block (SeAB) · p. 5
- Spatial pooling converts the attention map to a vector before softmax normalization; SeAB can be added at any network layer and is placed after ConvLSTM3D in the experiments.Source: 1) Spectral Attention Block (SeAB) · pp. 5–6
2) Spatial Attention Block (SaAB):
SaAB extracts spatial-enhanced features from LiDAR data, while MSRAB learns multiscale representations and adaptively attends to important areas at each scale.
2) Spatial Attention Block (SaAB):
- SaAB uses 3×3 and 1×1 ConvLSTM2D layers followed by softmax to produce a spatial attention map for enhancing LiDAR features.In the experiments, SaAB follows the ConvLSTM2D layer in the LiDAR branch.Source: 2) Spatial Attention Block (SaAB) · p. 6
- Composite attention learning jointly exploits spectral and spatial information to learn spatial-spectral features.The method applies spatial attention in SaAB to spectral-enhanced features in the fusion network.Source: 2) Spatial Attention Block (SaAB) · p. 6
C. Multiscale Residual Attention Block (MSRAB)
- Because different classes may contain different scale information, MSRAB is designed to enhance multiscale representation and provide the model with a larger receptive field.A ConvLSTM2D fusion layer models long-term dependencies across scales, while residual learning supports feature reuse.Source: C. Multiscale Residual Attention Block… · p. 6
- MSRAB extracts features at 1×1, 3×3, and 5×5 scales, then uses attention to adaptively focus on important areas at each scale.Its ConvLSTM2D layers cascade multiscale features in the time-step dimension before nonlinear attention learning.Source: C. Multiscale Residual Attention Block… · p. 6
D. Multiscale Spectral Attention Neural Network (MSSeA) for the HSI Branch
The MSSeA branch processes HSI neighborhoods with ConvLSTM3D layers, spectral attention, and multiscale feature learning. It maps the resulting features to class probabilities for classification.
D. Multiscale Spectral Attention Neural Network (MSSeA) for the HSI Branch
- For each HSI pixel, MSSeA takes an s×s neighborhood across the first K principal components, decomposes it into a sequence, and feeds it to cascaded ConvLSTM3D layers.The sequence length τ is fixed to 1 in the described experiments.Source: D. Multiscale Spectral Attention… · p. 7
- The HSI branch applies SeAB after ConvLSTM3D layers for spectral enhancement, then uses pooling and MSRAB to learn multiscale information.PCA is used for spectral dimensionality reduction because the original HSI data contain redundant information.Source: D. Multiscale Spectral Attention… · p. 7
- A GAP layer maps MSSeA features to class space, replacing a fully connected layer to reduce parameter count and limit overfitting to some degree.The paper also states that this choice relaxes input-resolution limitations.Source: D. Multiscale Spectral Attention… · p. 7
- A softmax classification layer predicts class probabilities, and cross entropy optimizes the HSI branch.Source: D. Multiscale Spectral Attention… · p. 7
E. Multiscale Spatial Attention Neural Network (MSSaA) for the LiDAR Branch
The MSSaA branch processes LiDAR neighborhoods with ConvLSTM2D layers, spatial attention, and multiscale feature learning, then predicts class probabilities.
E. Multiscale Spatial Attention Neural Network (MSSaA) for the LiDAR Branch
- MSSaA takes an s×s local spatial-context window around each LiDAR pixel and decomposes it into a sequence for cascaded ConvLSTM2D layers.The sequence length τ is set to 1 in the experiments.Source: E. Multiscale Spatial Attention Neural… · p. 7
- The LiDAR branch applies SaAB after ConvLSTM2D layers to enhance spatial information, then uses pooling and MSRAB to learn multiscale features.Source: E. Multiscale Spatial Attention Neural… · p. 7
- A GAP layer maps the LiDAR multiscale features to category space, and softmax probabilities are optimized with cross entropy.Source: E. Multiscale Spatial Attention Neural… · p. 7
F. Three-Level Fusion Strategy
The fusion network combines HSI and LiDAR features at three levels, using composite attention, feature fusion, and reuse of LiDAR features.
F. Three-Level Fusion Strategy
- At the first fusion level, LiDAR spatial attention is applied to HSI spectral-enhanced features for composite attention learning, with residual learning used to avoid degradation.Source: F. Three-Level Fusion Strategy · p. 8
- At the second level, MSRAB outputs from both branches are concatenated along the spectral dimension and fused using a 1×1 ConvLSTM3D layer and GAP.Source: F. Three-Level Fusion Strategy · p. 8
- At the third level, LiDAR GAP features are reused by concatenating them with the features from the preceding fusion stage.The paper motivates this reuse by noting that HSI feature magnitudes are much larger than LiDAR feature magnitudes.Source: F. Three-Level Fusion Strategy · p. 8
GAP , XL
The fusion network produces a probability distribution and uses cross-entropy loss for optimization.
- The fusion output is fed into a softmax function to yield probability distribution P F.Source: GAP , XL · p. 8
- Cross-entropy is used as LossF to optimize the fusion network.Source: GAP , XL · p. 8
G. Loss Function and Network Training Strategy
The model is trained in stages: the HSI and LiDAR branches are pretrained, then used to initialize and train the fusion network with a multi-task loss. Experiments evaluate the approach using the Houston and Trento HSI–LiDAR datasets.
G. Loss Function and Network Training Strategy
- The multi-task loss combines LossH, LossL, and LossF with scalar weights α, β, and γ, each fixed to 1 in the experiments.Source: G. Loss Function and Network Training… · p. 8
- The HSI and LiDAR branches are pretrained separately, then initialize the fusion network for joint training with the multi-task loss.The authors report that this stepwise strategy accelerates convergence of the whole model.Source: G. Loss Function and Network Training… · p. 8
IV. EXPERIMENTAL RESULTS
- The classification comparison includes ELM, SVM, SSCL3DNN, SaCL2DNN, two-branch CNN, and dual-channel CapsNet, with OA, AA, and κ averaged over 10 runs.Source: §IV · p. 8
1) Houston Data:
- The experiments use Houston HSI–LiDAR data captured over the University of Houston campus and surrounding area, with 144 spectral bands and 15 class labels.The Houston data have 349 × 1905 pixels and 2.5 m spatial resolution.Source: A. Experimental Data · p. 9, 1) Houston Data · p. 9
2) Trento Data:
The Trento data comprise 600 × 166 pixels, 63 spectral bands, and six ground-truth classes. Parameter experiments select settings for the proposed model on the Trento dataset.
2) Trento Data:
- The Trento data comprise 600 × 166 pixels, 1 m spatial resolution, 63 spectral bands, and six ground-truth classes.Source: 2) Trento Data · pp. 9–10
B. Parameter Settings
- The selected local spatial window is 11 × 11 for Trento, and the optimal number of PCA components is K = 10 for both datasets.Source: B. Parameter Settings · p. 10
- For Trento, the selected feature-map counts are {16, 32, 64}, while dropout is set to 0.5 for both datasets.Source: B. Parameter Settings · p. 10
- With lr fixed at 0.0001, the selected [lrH, lrL] for Trento is [0.001, 0.0005], and Nstep1 and Nstep2 are both 500.The reported parameter experiments also identify lr = 0.0001 for both datasets.Source: B. Parameter Settings · pp. 10–11
C. Classification Performance
Across the Houston and Trento datasets, dual-channel A3CLNN outperforms the tested comparison methods in the reported classification results. The authors also report gains over single-source branches and describe classification maps with fewer mislabeled areas and better-delineated class boundaries.
C. Classification Performance
- Using HSI and LiDAR yields OA gains of 3.55% and 1.08% over the HSI branch for Houston and Trento, respectively.The reported OA values for the LiDAR branch are 59.83% and 89.31%, and for the HSI branch 87.00% and 97.65%, respectively.Source: Table VII · p. 11, C. Classification Performance · p. 13
- The authors attribute the model’s performance to ConvLSTM gate mechanisms, attention mechanisms, three-level fusion, and stepwise training.They state that the fusion and training strategies integrate complementary HSI and LiDAR information and accelerate convergence.Source: C. Classification Performance · pp. 11, 13
- 90.55% classification accuracy on Houston, with a gain over 7.40% relative to the two-branch CNN; on Trento, OA gains are 6.04%, 7.88%, and 3.37% versus SVM, ELM, and two-branch CNN, respectively.Source: Table VIII · p. 12, Table IX · p. 12, Figure 8 · p. 12
- The classification maps have fewer mislabeled areas and better-delineated class boundaries than the ground-truth comparisons, especially for classes 6, 8, and 11 in Houston and 2, 3, and 5 in Trento.Source: Figure 8 · p. 12, Figure 9 · p. 12, C. Classification Performance · pp. 13–14
- The model gains 3.94% and 1.98% over dual-channel CapsNet on the two datasets, and 4.54% and 2.27% in OA over SSCL3DNN, respectively.Source: C. Classification Performance · p. 13
D. Ablation Study
The ablations assess attention modules, multiscale design, and stepwise training, while the conclusion describes a dual-channel A3CLNN with three-level fusion. The authors report stronger performance than several CNN-, capsule network-, and ConvLSTM-based approaches.
D. Ablation Study
- SeAB improves the proposed model(H+L) classification accuracy by 0.50% and 0.62% on the two data sets.The ablation compares the full model with the version without SeAB.Source: (1) The effectiveness of SeAB · p. 14, Table XI · p. 14
- Adding SaAB yields 1.22% and 0.56% gains over the proposed model(H+L, without SaAB) on the two data sets.The reported comparison examines the model with versus without SaAB.Source: (2) The structure analysis of SaAB · p. 14, Table XII · p. 14
- The model with composite attention gains 1.55% and 1.10% over the version without it on the two data sets.The study also reports that the version without composite attention gains 2.99% and 1.17% in OA over SSCL3DNN(H+L).Source: (3) The analysis of two-level… · p. 14, Table XIII · p. 14
- MSRAB brings 3.32% and 1.15% improvements over variants replacing it with ConvLSTM2D or ConvLSTM3D on the two data sets.The authors identify composite attention and MSRAB as main reasons for performance improvements.Source: (4) The architecture study of MSRAB · pp. 14–15, Table XIV · p. 14
- With stepwise training, the Houston loss curve converges faster and more stably than without it, while both data-set curves tend to converge stably.Fig. 10 presents learning curves for Houston and Trento; the reported comparison of convergence speed and stability is for Houston.Source: (5) The analysis of stepwise training… · p. 15, Figure 10 · p. 15
V. CONCLUSION
- The model uses separate HSI and LiDAR pipelines with spatial, spectral, and multiscale residual attention, plus three-level fusion and stepwise training.The fusion strategy is intended to integrate spatial and spectral information from the two sources.Source: §V · p. 15
- The dual-channel A3CLNN provides better performance than state-of-the-art CNN-based, capsule network-based, and baseline ConvLSTM-based approaches.The comparisons include two-branch CNN, dual-channel CapsNet, and SSCL3DNN.Source: §V · p. 15