Source-linked AI summary
Adaptive Frequency Enhancement Network for Remote Sensing Image Semantic Segmentation
Feng Gao, Miao Fu, Jingchao Cao, Junyu Dong, Qian Du
TL;DR
Remote sensing segmentation methods must adapt to varied land-cover distributions while strengthening spatial–frequency interaction. AFENet addresses this with adaptive frequency separation and selective fusion of contextual and detailed features, and it outperforms compared methods across three datasets, including a 54.83% mIoU on LoveDA.
Problem
Existing methods have limited adaptation to different land-cover distributions and insufficient interaction between spatial and frequency features.
Method
AFENet combines AFSIM, which adaptively separates and modulates high- and low-frequency features, with SFM, which selectively fuses global context and local detail.
Results
AFENet outperforms compared methods across three datasets, achieving the highest LoveDA mIoU of 54.83%.
Takeaways & Limitations
Adaptive frequency modulation and selective fusion support segmentation across diverse rural, urban, and complex land-cover scenarios.
Abstract
from arXiv · showhide
Semantic segmentation of high-resolution remote sensing images plays a crucial role in land-use monitoring and urban planning. Recent remarkable progress in deep learning-based methods makes it possible to generate satisfactory segmentation results. However, existing methods still face challenges in adapting network parameters to various land cover distributions and enhancing the interaction between spatial and frequency domain features. To address these challenges, we propose the Adaptive Frequency Enhancement Network (AFENet), which integrates two key components: the Adaptive Frequency and Spatial feature Interaction Module (AFSIM) and the Selective feature Fusion Module (SFM). AFSIM dynamically separates and modulates high- and low-frequency features according to the content of the input image. It adaptively generates two masks to separate high- and low-frequency components, therefore providing optimal details and contextual supplementary information for ground object feature representation. SFM selectively fuses global context and local detailed features to enhance the network's representation capability. Hence, the interactions between frequency and spatial features are further enhanced. Extensive experiments on three publicly available datasets demonstrate that the proposed AFENet outperforms state-of-the-art methods. In addition, we also validate the effectiveness of AFSIM and SFM in managing diverse land cover types and complex scenarios. Our codes are available at https://github.com/oucailab/AFENet.
I. INTRODUCTION
High-resolution remote sensing segmentation supports land-cover understanding, but existing methods struggle with varying land-cover distributions and weak spatial–frequency interaction. AFENet addresses these challenges through adaptive frequency modulation and selective feature fusion.
- High-resolution remote sensing segmentation is important for interpreting and understanding land cover across applications such as land-use monitoring.
- Existing CNN-based methods capture spatial hierarchies but have limited receptive fields for global context modeling.
- Different land covers vary substantially in object types, distributions, and textures, but many methods use the same parameters for every input image.
- Spatial and frequency features require stronger interaction because existing CNN-based methods do not sufficiently model frequency variations across land-cover types.
- AFENet combines adaptive frequency and spatial feature interaction with selective feature fusion to model ground objects using both domains.
- AFENet is evaluated on three publicly available datasets, with reported results outperforming state-of-the-art methods and showing scalability across land-cover types.
- AFSIM generates input-dependent masks for high- and low-frequency components, while SFM selectively fuses contextual and detailed features.
C. Frequency Domain Feature Learning
The method learns frequency-domain information alongside spatial features by adaptively separating high- and low-frequency components. These components are then aligned, interacted with spatial features, and selectively fused to provide complementary detail and context.
- Frequency-domain features support nonlocal representation learning, and Fourier analysis decomposes images into components at different frequencies.
- AFENet uses adaptive frequency separation to extract high- and low-frequency information at scales determined by each image’s geospatial distribution.
- The decoder inserts Adaptive Frequency Enhancement Blocks to refine frequency and spatial features, while Transformer Blocks fuse multiscale information.
- AFSIM contains adaptive frequency separation, spatial feature fusion, and cross-domain feature interaction.
- The Adaptive Window-mask Module computes frequency boundaries from each image’s land-cover distribution to separate high- and low-frequency information.
- The module computes adaptive ratios from globally pooled features, uses them to define a mask window, and applies Fourier-domain masking for frequency separation.
- SFM processes high- and low-frequency features with pooling and concatenation to generate weights for selective feature fusion.
- The resulting adaptive separation adjusts mask size according to input content, supplying detail and contextual information to spatial features.
C. Selective feature Fusion Module (SFM)
SFM selectively fuses high- and low-frequency features using spatial attention, combining contextual information with detailed edge features. Its residual structure dynamically adjusts feature weights and adds nonlinear transformation.
- High-frequency features capture texture and edges, while low-frequency features represent contextual and structural information.
- SFM concatenates the two feature types and uses average and max pooling to obtain global and local spatial information.
- A 7×7 convolution and sigmoid activation generate a spatial attention mask for feature selection.
- Element-wise multiplication combines the attention maps with high- and low-frequency features to adjust their contributions.
- The resulting feature map contains semantic information about spatial distributions and detailed edge features.
D. Loss Function
The loss function combines Cross-Entropy and Dice losses to balance pixel-wise classification accuracy with regional segmentation quality, including under class imbalance.
- The composite loss Ltotal combines Cross-Entropy loss Lce and Dice loss Ldice.
- Cross-Entropy loss measures discrepancy between predicted class probabilities and true labels across samples and classes.
- Dice loss is incorporated to address class imbalance by emphasizing high-confidence predictions.
- The combined objective improves pixel classification accuracy and segmentation quality by optimizing overlap between predicted and true regions.
IV. EXPERIMENTAL RESULTS AND ANALYSIS
The experiments include ablation studies, comparative evaluations, and model-complexity analyses to examine AFENet’s components and performance.
- The experimental setup covers datasets, evaluation metrics, and implementation details.
- Ablation studies, comparative evaluations, and model-complexity analyses assess the contributions of AFENet’s key components.
- The analyses are presented to demonstrate the model’s reported performance.
1) Datasets:
AFENet is evaluated on three public remote-sensing benchmarks spanning diverse land-cover types, environmental conditions, and scene complexities.
- Experiments use the ISPRS Vaihingen, ISPRS Potsdam, and LoveDA datasets.
- These datasets provide benchmarks for semantic segmentation in remote-sensing imagery.
- Vaihingen contains 33 orthophotos with six annotated semantic classes and multispectral, DSM, and normalized DSM data.
- LoveDA contains 5,987 high-resolution images from urban and rural areas, annotated across seven semantic classes.
2) Evaluation Metrics:
The evaluation uses OA, mean F1 score, and mIoU, supplemented by per-class F1 and IoU values for granular category-level assessment.
- OA, mean F1 score, and mIoU are the primary metrics for evaluating segmentation performance.
- TP_i, FP_i, TN_i, and FN_i denote class-specific confusion counts, while N is the number of classes excluding clutter/background.
- Per-class F1 and IoU scores provide a more granular evaluation across semantic categories.
3) Implementation Details:
Experiments combine specified optimization, augmentation, testing, and complexity-analysis procedures with ablations of AFENet components on the Vaihingen dataset.
- Training uses Adam with learning rate 0.0006 and weight decay 0.01 in PyTorch 2.0.0 on an NVIDIA GTX 4090.
- Vaihingen and Potsdam images are cropped into 512 × 512 patches and augmented with scaling and vertical and horizontal flips.
- Test-time horizontal and vertical flips are used, with batch size 8 and 100 batches per epoch during training.
- Complexity is assessed using FLOPs and parameter counts for a 3×512×512 input image.
- Ablation studies evaluate mIoU and mean F1 after removing or adding components, with results averaged across multiple runs.
- Removing any module reduces performance; AFSIM branch removal affects frequency-feature interactions, while AWM+Fixed and SFM+Add replace adaptive separation and selective fusion.
- Visual ablations link the high-frequency branch to small-object segmentation, the low-frequency branch to large-object semantics, AWM to scale handling, and SFM to feature fusion.
- Progressive ablations begin with a ResNet-18 encoder and Transformer Block-based decoder using only spatial-domain features.
2) Impacts of AFSIM:
AFSIM complements low-frequency context with high-frequency detail, while SFM improves context-detail fusion; adaptive windows adjust frequency emphasis across land-use scenarios.
- Impacts of AFSIM: The complete AFSIM improves mIoU by 0.62% over AFSIM-Low and by 0.51% over AFSIM-High alone.
- Impacts of AFSIM: The low-frequency branch improves contextual recognition of large structures, whereas the high-frequency branch improves small-object and vegetation-edge precision.
- Impacts of AFSIM: Combining both AFSIM branches provides complementary contextual and edge-detail benefits for superior segmentation.
- Impacts of SFM: Adding SFM increases mIoU by 0.58% on the baseline and by 0.63% when AFSIM is already present.
- Impacts of SFM: SFM improves building-edge segmentation, detail accuracy, and segmentation consistency when combined with the baseline or AFSIM.
- Adaptive window: Adaptive windows prioritize low-frequency information in homogeneous rural areas while retaining high-frequency details for farmland and building edges.
- Adaptive window: In complex urban regions, adaptive windows retain more high-frequency detail by reducing redundant low-frequency components.
- Adaptive window: The AWM’s adaptive frequency balance improves segmentation across diverse and complex land-use scenarios.
C. Comparative Experiments
Across Vaihingen, Potsdam, and LoveDA, AFENet is reported to outperform the compared methods, with visual analyses emphasizing balanced spatial-frequency integration for boundaries and semantics.
- AFENet consistently outperforms the compared methods across the ISPRS Vaihingen, ISPRS Potsdam, and LoveDA datasets.
- Vaihingen: On Vaihingen, AFENet outperforms all compared approaches on mF1, mIoU, and OA.
- Vaihingen: Its dual separation and selective fusion of high- and low-frequency information supports precise delineation of vehicles, building structures, vegetation, and tree boundaries.
- Potsdam: On Potsdam, AFENet achieves the highest scores across all reported metrics.
- Potsdam: The Potsdam analysis reports smoother building and low-vegetation edges than spatially focused or frequency-centric alternatives.
- Potsdam: Spatially focused models can compromise boundary accuracy, while frequency-centric models can lose semantics or struggle with large-scale objects.
3) Results on the LoveDA Dataset:
On LoveDA, AFENet achieves the highest reported mIoU and improves segmentation across rural and urban scenes by combining adaptive frequency modulation with spatial-feature interaction. Its advantage is especially evident for edges, varied object scales, and intricate boundaries.
- Quantitative comparison: 54.83% mIoU is achieved by AFENet on LoveDA, surpassing all other evaluated models.Table VII reports IoU for seven semantic classes and the overall mIoU on the official test set.
- Scene coverage: LoveDA’s rural and urban scenes test segmentation of agricultural land, roads, and complex building clusters.The visualization includes two rural scenes and one urban scene, with key regions highlighted for inspection.
- Comparison with existing methods: Conventional, attention-based, multi-scale, and frequency-based methods each perform well on particular scene characteristics but do not satisfy multiple-scene requirements simultaneously.The cited comparisons describe strengths on uniform regions, contextual regions, building areas, agricultural fields, and building edges, while identifying limited coverage across scene types.
- AFENet analysis: AFENet’s adaptive frequency-separating mechanism supplements different scenes with high- and low-frequency information tailored to ground-object distributions.The mechanism is described as improving multi-scale representation and fine-detail recognition across buildings, roads, and agricultural regions.
- AFENet analysis: AFENet improves semantic coherence and edge-segmentation accuracy, particularly for objects with varied scales and intricate boundaries.The reported advantage follows from more comprehensive multi-scale information and enhanced recognition of fine details.
- Efficiency context: AFENet is evaluated alongside computational complexity, parameter count, and mIoU comparisons under consistent settings on Vaihingen.The comparison uses specified VGG-16 and ResNet-18 backbones for classic and recent methods.