Source-linked AI summary
Semantic-Guided Fusion Network for Multi-Source Remote Sensing Image Classification
Yuwei Zhao, Chuanzheng Gong, Baogui Huan, Feng Gao, Junyu Dong, Qian Du
TL;DR
Multi-source remote sensing classification still faces limited semantic-context modeling and unreliable fusion under slight spatial misalignment. SGFNet combines semantic-aware dynamic convolution with frequency-domain cross-modal fusion, and is reported to outperform state-of-the-art methods on the Augsburg and Houston 2018 datasets.
Problem
Existing methods have limited adaptive semantic-context modeling, and slight cross-modal spatial misalignment can weaken reliable feature fusion.
Source: §I · p. 1Method
SGFNet uses SMCB to generate semantic-aware kernels from contextual relationships and FMFB for adaptive cross-modal interaction in the frequency domain.
Source: §I · p. 1Results
SGFNet consistently outperforms several state-of-the-art methods on two benchmark datasets; on Augsburg, its Overall Accuracy is 92.38%, outperforming all other methods.
Source: §I · p. 2, C. Experimental Results and Analysis · p. 4Takeaways & Limitations
The proposed frequency-aware fusion enhances complementary information integration and improves the discriminative capability of multi-source feature representations.
Source: §I · p. 1
Abstract
from arXiv · showhide
Multi-source remote sensing image classification has attracted increasing attention due to the complementary spectral, structural, and geometric information. However, existing methods still suffer from two limitations: insufficient semantic contextual modeling and unreliable feature fusion caused by slight spatial misalignment. To address these issues, we propose a Semantic-Guided Fusion Network (SGFNet) for multi-source remote sensing image classification. Specifically, the Semantic Mixing Convolution Block (SMCB) is designed to dynamically generate semantic-aware convolution kernels according to contextual relationships among feature representations. In addition, the Frequency Modulated Fusion Block (FMFB) is introduced to perform cross-modal interaction in the frequency domain, which effectively alleviates the influence of slight spatial misalignment and improves complementary information fusion. Extensive experiments conducted on the Augsburg and Houston 2018 datasets demonstrate that the proposed SGFNet consistently outperforms several state-of-the-art methods. The codes are publicly available at https://github.com/oucailab/SGFNet .
I. INTRODUCTION
Multi-source remote sensing classification combines complementary spectral, structural, and geometric information, but current methods face gaps in semantic-context modeling and reliable fusion under slight spatial misalignment. SGFNet addresses these gaps with semantic-aware dynamic convolution and frequency-domain fusion, and is reported to outperform several state-of-the-art methods on two benchmarks.
I. INTRODUCTION
- Existing methods lack adaptive semantic and contextual representations, while slight cross-modal misalignment can weaken reliable feature fusion.The misalignment is associated with differing imaging geometries and sensor characteristics.Source: §I · p. 1
- SMCB dynamically generates semantic-aware convolution kernels from contextual relationships among feature representations.The proposed block targets semantic and contextual feature representations, in contrast to conventional fixed, spatially shared kernels.Source: §I · p. 1
- FMFB performs adaptive cross-modal interaction in the frequency domain to alleviate slight misalignment and enhance complementary information integration.The paper focuses on joint classification using HSI with SAR or LiDAR data.Source: §I · pp. 1–2
- SGFNet is reported to consistently outperform several state-of-the-art methods on two benchmark datasets.The supplied passages do not provide comparative scores for this overall result.Source: §I · p. 2
II. METHODOLOGY
SGFNet uses a dual-branch architecture to progressively refine semantic representations and fuse multi-modal features through stacked Semantic-Guided Fusion Modules.
II. METHODOLOGY
- The dual-branch framework processes HSI and SAR/LiDAR data separately at first, then fuses them through stacked Semantic-Guided Fusion Modules.Each module uses SMCB for semantic context and FMFB for complementary frequency-domain characteristics; two convolutional layers refine the fused features for classification.Source: A. Overall Framework of the Proposed… · p. 2
B. Semantic Mixing Convolution Block (SMCB)
SMCB addresses the limits of fixed convolutions by generating semantic-aware dynamic filters from contextual affinities, then applying them to local value patches alongside a local enhancement branch.
B. Semantic Mixing Convolution Block (SMCB)
- SMCB derives an affinity map from query and key features to generate a dynamic convolution kernel for each spatial position.The affinity map relates each position to K^2 sampling locations in a local window, and a linear layer and Softmax produce the kernel.Source: B. Semantic Mixing Convolution Block… · p. 2
- The dynamic kernel aggregates each channel’s unfolded value features by weighted summation over a local K × K neighborhood.The input feature is projected by a 1 × 1 convolution to obtain value and query features, while adaptive pooling extracts compact contextual information.Source: B. Semantic Mixing Convolution Block… · p. 2
- A local enhancement branch applies a 3 × 3 depth-wise convolution to the input, and its feature is added residually to the dynamically aggregated feature.The combined feature forms the block output.Source: B. Semantic Mixing Convolution Block… · p. 3
C. Frequency Modulated Fusion Block
FMFB addresses slight cross-modal misalignment by fusing HSI and SAR/LiDAR features in the frequency domain. It transforms the modulated features back to spatial space and refines them with an FFN and residual connections.
C. Frequency Modulated Fusion Block
- FMFB applies adaptive cross-modal fusion in the frequency domain to enhance feature consistency under slight spatial misalignment.The motivation is that small translations mainly affect phase while largely preserving the frequency-energy distribution, making FMFB less sensitive to slight misalignment than pixel-wise spatial fusion.Source: C. Frequency Modulated Fusion Block · p. 3
- The block takes HSI and SAR/LiDAR features, applies 2D DCT, and computes their element-wise frequency-domain difference.The joint representation generates query and key features, while each DCT-transformed input is separately projected into value features.Source: C. Frequency Modulated Fusion Block · p. 3
- Attention uses the query and key features with channel dimension d_k as a scaling factor to stabilize training gradients.The resulting modulated frequency-domain features are transformed back to spatial space through 2D inverse DCT.Source: C. Frequency Modulated Fusion Block · p. 3
- An FFN and residual connections further refine the reconstructed feature.This follows the inverse-DCT transformation of the modulated features.Source: C. Frequency Modulated Fusion Block · p. 3
A. Datasets and Experimental Settings
SGFNet is evaluated for land-cover classification on two multi-source remote-sensing benchmarks: Augsburg and Houston 2018. The experiments compare it with five state-of-the-art methods using OA, AA, and Kappa.
A. Datasets and Experimental Settings
- Evaluation uses the Augsburg HSI–SAR dataset and the Houston 2018 HSI–LiDAR dataset for land-cover classification.Augsburg contains ground-truth annotations for seven land-cover classes; Houston 2018 is an urban land-cover and land-use benchmark over the University of Houston campus and surrounding areas.Source: A. Datasets and Experimental Settings · p. 3
- SGFNet is compared with S2ENet, DFINet, ExViT, DFFNet, and MSFM using OA, AA, and Kappa.Training used an NVIDIA GeForce RTX 5080 GPU for 100 epochs, with AdamW at learning rate 0.001 and batch size 32.Source: A. Datasets and Experimental Settings · p. 3
B. Parameter Analysis
Parameter analysis selects two SGFMs, 30 HSI principal components, and dynamic kernel size K = 7. The reported choices correspond to the best classification performance or OA peak in the tested settings.
B. Parameter Analysis
- Two SGFMs achieve the best performance on both datasets among the tested counts from one to four.The accompanying parameter plot relates OA to the number of SGFMs.Source: B. Parameter Analysis · p. 3, Figure 5 · p. 4
- Thirty HSI principal components give the best classification performance among the tested PCA settings.The authors note that retaining too few components discards crucial discriminative spectral features.Source: B. Parameter Analysis · p. 4, Figure 5 · p. 4
- OA on both datasets peaks at K = 7 among the tested dynamic kernel sizes.The authors attribute the trade-off to insufficient semantic sampling with smaller kernels and potentially redundant or noisy context with larger kernels.Source: B. Parameter Analysis · p. 4, Table II · p. 4
C. Experimental Results and Analysis
SGFNet outperforms the compared methods on Augsburg and Houston2018, including on several challenging land-cover categories. Its Houston2018 results partially alleviate class-imbalance effects but do not fundamentally solve the problem.
C. Experimental Results and Analysis
- 92.38% OA on Augsburg, with SGFNet outperforming all other methods.It achieves the best class-wise performance in Allotment and remains highly competitive in Industrial.Source: C. Experimental Results and Analysis · p. 4
- On Houston2018, SGFNet obtains the highest OA and Kappa and performs best in several challenging categories.These include Evergreen trees, Deciduous trees, Buildings, Roads, Sidewalks, Crosswalks, and Paved parking lots.Source: C. Experimental Results and Analysis · pp. 4–5
- SGFNet partially mitigates class imbalance, improving minority-class results and achieving the highest AA of 96.71%, but does not fundamentally solve imbalance.The paper notes that it lacks balanced sampling, loss reweighting, and data augmentation.Source: C. Experimental Results and Analysis · p. 5
D. Ablation Study
The ablation results indicate that SMCB and FMFB each improve classification performance and contribute complementary benefits to SGFNet.
D. Ablation Study
- Both SMCB and FMFB improve classification performance, and their complementary contributions jointly support more robust classification.The paper associates SMCB with enhanced contextual representation and FMFB with alleviating slight cross-modal spatial misalignment.Source: D. Ablation Study · p. 5
E. Robustness to Spatial Misalignment
Spatial-shift tests on Houston2018 show a smaller OA drop for SGFNet than for DFFNet and MSFMamba. The conclusion describes SGFNet’s semantic-kernel and frequency-domain fusion approach and reports consistent outperformance on two benchmark datasets.
E. Robustness to Spatial Misalignment
- Under 1, 2, 3, and 5-pixel shifts to the LiDAR input, SGFNet's OA drops 1.40%, versus 2.60% for DFFNet and 2.03% for MSFMamba.The HSI branch remains fixed; the authors attribute SGFNet's lower drop to FMFB's frequency-domain modulation.Source: E. Robustness to Spatial Misalignment · p. 5, Table V · p. 5
IV. CONCLUSIONS
- SGFNet combines SMCB's contextual-affinity-based semantic kernels with FMFB's adaptive cross-modal interaction in the frequency domain.Experiments on Augsburg and Houston2018 show consistent outperformance over state-of-the-art methods.Source: §IV · p. 5