PaperlayerView exact revision on arXiv

Source-linked AI summary

SFFNet: A Wavelet-Based Spatial and Frequency Domain Fusion Network for Remote Sensing Segmentation

Yunsong Yang, Genji Yuan, Jinjiang Li

arXiv:2405.01992v1cs.CV

TL;DR

Remote sensing segmentation must retain spatial information while handling complex objects and areas with large grayscale variations. SFFNet uses two-stage spatial and frequency-domain mapping with wavelet decomposition and feature alignment, and its experiments report improved segmentation accuracy and robustness.

  • Problem

    Remote sensing segmentation must handle complex objects, large grayscale variations, and the information loss caused when frequency-domain features replace spatial features alone.

  • Method

    SFFNet uses two stages for spatial feature extraction and mapping, combining global and local spatial branches with a Haar-wavelet branch and MDAF for cross-domain alignment and feature selection.

  • Results

    Experiments report that the Global branch performs best in FLOPs and segmentation accuracy among the compared global-mapping modules, while the Global and Local branches improve large-object and small-object segmentation, respectively.

  • Takeaways & Limitations

    Combining spatial and frequency-domain features enables SFFNet to preserve spatial and semantic information while more comprehensively considering grayscale-varying regions in remote sensing images.

Abstract

from arXiv · show

In order to fully utilize spatial information for segmentation and address the challenge of handling areas with significant grayscale variations in remote sensing segmentation, we propose the SFFNet (Spatial and Frequency Domain Fusion Network) framework. This framework employs a two-stage network design: the first stage extracts features using spatial methods to obtain features with sufficient spatial details and semantic information; the second stage maps these features in both spatial and frequency domains. In the frequency domain mapping, we introduce the Wavelet Transform Feature Decomposer (WTFD) structure, which decomposes features into low-frequency and high-frequency components using the Haar wavelet transform and integrates them with spatial features. To bridge the semantic gap between frequency and spatial features, and facilitate significant feature selection to promote the combination of features from different representation domains, we design the Multiscale Dual-Representation Alignment Filter (MDAF). This structure utilizes multiscale convolutions and dual-cross attentions. Comprehensive experimental results demonstrate that, compared to existing methods, SFFNet achieves superior performance in terms of mIoU, reaching 84.80% and 87.73% respectively.The code is located at https://github.com/yysdck/SFFNet.

I. INTRODUCTION

Remote sensing segmentation supports applications across Earth-observation tasks, but complex grayscale variation and spatial–frequency trade-offs challenge existing methods. SFFNet addresses these issues by combining spatial and frequency-domain feature mappings in a two-stage architecture.

  • Remote sensing image segmentation supports land-cover mapping, change detection, environmental protection, and object extraction.
  • CNN-based methods struggle with complex remote sensing objects, while attention-based approaches may not directly encode clear global context.
  • Spatial-only segmentation is challenged by grayscale variations in edges and shadows, whereas frequency features are more sensitive to these regions.
  • Using only frequency-domain features can lose spatial information, while spatial features better capture category semantics and accurate spatial structure.
  • SFFNet uses two-stage spatial extraction and mapping, adding Haar-wavelet high- and low-frequency features through WTFD alongside global and local spatial features.
  • MDAF aligns frequency- and spatial-domain features across scales and selects representative features through multiscale vertical convolution and dual-cross attention.

A. CNN-based Remote Sensing Image Semantic Segmentation

Remote sensing segmentation methods use CNNs, multiscale features, attention, Transformers, and wavelets to address complex scenes. SFFNet combines these directions while retaining spatial information alongside frequency-domain representations.

  • CNN-based models such as UNet and its variants improve semantic segmentation, but remote sensing complexity includes small targets, occlusion, and limited receptive fields.
  • Multiscale feature extraction, including spatial pyramid pooling and multiscale CNN designs, addresses varied object scales in remote sensing images.
  • Attention mechanisms improve performance but often remain dependent on convolutional operations without directly encoding global context.
  • Transformer-based methods directly encode global context, but applying SwinTransformer or ViT for global mapping has high computational complexity.
  • Wavelet-based segmentation methods can improve performance and generalization, but using only frequency-domain features may lose crucial spatial information.
  • SFFNet retains spatial diversity through global and local mapping, adds WTFD frequency features, and uses MDAF to align and fuse both domains.

A. SFFNet Structure

SFFNet uses a two-stage architecture that extracts spatial features, maps them through global, local, and wavelet branches, and fuses aligned spatial-frequency representations for segmentation.

  • A. SFFNet Structure: SFFNet first extracts spatial features, then maps them through global, local, and wavelet branches before final segmentation.The latter three extracted features are merged after convolution and interpolation to combine semantic and spatial information.
  • A. SFFNet Structure: The Global, Local, and WTFD branches produce global, multiscale local, low-frequency, and high-frequency features for subsequent fusion.These mapped features share dimensions of C × (H/4) × (W/4).
  • A. SFFNet Structure: MDAF aligns spatial and frequency-domain feature pairs, selects representative information, and combines them with extracted features for final prediction.The two aligned feature groups are combined with the first-layer extraction output through the segmentation header.
  • A. SFFNet Structure: The architecture retains spatial information while expanding the representation space without using a complex decoder.Its components include the Global branch, Local branch, WTFD, and MDAF.
  • A. SFFNet Structure: Vertical stripwise convolutions establish interactions between pixels in different windows, enabling remote dependencies across pre-segmented windows.Convolution lengths equal to the window size connect features across windows.

B. Feature Mapping

Feature mapping combines efficient global modeling with multiscale local mapping. The Global branch uses window processing and vertical stripe convolutions, while the Local branch uses multiscale max-pooling.

  • 1) Global Branch: The Global branch downsamples features, applies window partitioning and self-attention, then uses vertical stripe convolutions for inter-window interaction.This replaces the more complex SW-Trans block for inter-window information exchange.
  • 1) Global Branch: The Global branch outputs features with dimensions (C/3) × (H/2) × (W/2).The branch uses window size ws and batch normalization in its computation.
  • 1) Global Branch: Vertical stripe convolutions support efficient global mapping and capture directional features such as roads, rivers, and edges.The convolutions learn features in both horizontal and vertical directions.
  • 2) Local Branch: The Local branch applies multi-scale max-pooling with 5×5, 9×9, and 13×13 kernels to downsampled features.These pooling scales capture feature information at different spatial scales.

3) Wavelet Transform Feature Decomposer :

WTFD uses Haar wavelets to decompose spatial features into low-frequency and directional high-frequency components, then maps these components into features for segmentation.

  • 3) Wavelet Transform Feature Decomposer: The Haar wavelet process applies low-pass and high-pass filtering, with each decomposition reducing feature size by half.Two iterations yield A, H, V, and D components.
  • 3) Wavelet Transform Feature Decomposer: WTFD transforms spatial features into one low-frequency component and horizontal, vertical, and diagonal high-frequency components.The high-frequency components represent local details, while the low-frequency component captures global characteristics.
  • 3) Wavelet Transform Feature Decomposer: The three high-frequency components are concatenated and passed through nonlinear point-convolution mapping to produce high-frequency features.Channel reduction filters redundant information during feature mapping.
  • 3) Wavelet Transform Feature Decomposer: Low-frequency information is processed through convolution to obtain low-frequency features, while the resulting low- and high-frequency outputs support segmentation.The outputs are represented as Fl and Fh with dimensions (C, H/2, W/2).
  • 3) Wavelet Transform Feature Decomposer: WTFD provides segmentation with both local-detail and global-characteristic feature information instead of replacing spatial features entirely.The design addresses spatial information loss associated with using frequency features as substitutes for spatial features.

C. Multiscale Dual-Representation Alignment Filter

MDAF aligns and selects spatial- and frequency-domain features by mapping them to a unified scale and applying domain-cross attention. Multiscale convolutions produce Q, K, and V representations, which DAF weights and concatenates into the output feature.

  • MDAF addresses semantic differences between spatial and frequency features through semantic alignment and feature selection.
  • Multiscale convolutions map spatial and frequency features to a unified scale before producing Q, K, and V matrices.Different-scale vertical bar-shaped convolutions are concatenated and followed by 1×1 convolutions.
  • DAF uses each domain to query the counterpart and its own key-value pairs, then weights and concatenates the resulting features.
  • Attention weights emphasize features with similar semantics and suppress or ignore features with different semantics.This dynamic weighting supports the intended semantic alignment during feature selection.
  • Multi-scale Ortho-Convolution applies bar-shaped convolutions and layer normalization to generate same-sized Q, K, and V matrices.The matrices have size (C, H, W), while the two input features each have dimensions C×H×W.
  • The MDAF operation combines the two DAF outputs into an output tensor with C channels and spatial dimensions H×W.

D. Loss function

The loss function combines Dice loss with cross-entropy loss. Cross-entropy is defined over samples and classes using one-hot labels and the network's softmax outputs.

  • The training loss combines Dice loss Ldice and cross-entropy loss Lce.
  • Cross-entropy uses N samples and K classes, comparing one-hot semantic labels with corresponding softmax outputs.The softmax output ˆy(n)_k represents sample n's confidence for class k.

IV. EXPERIMENT

The experiments evaluate remote-sensing segmentation on Vaihingen and Potsdam using standard datasets, augmentation and training procedures, and multiple accuracy and efficiency metrics. The evaluation emphasizes class-wise segmentation quality alongside overall accuracy and computational cost.

  • Datasets: Vaihingen and Potsdam provide standard remote-sensing segmentation benchmarks with varied land-cover categories and environmental conditions.The datasets include classes such as buildings, roads, and trees and support comparability across studies.
  • Datasets: Vaihingen contains 33 high-resolution image blocks and six land-cover classes, including impervious surfaces, buildings, vegetation, trees, cars, and clutter.
  • Datasets: Potsdam contains 38 aerial image blocks at 5-centimeter ground sampling distance, with TOP, DSM, and four multispectral bands.
  • Implementation: Images were randomly cropped into 512×512 patches and trained with AdamW, a 6e-4 base learning rate, cosine scheduling, and 105 epochs.
  • Implementation: Training used random scaling, vertical and horizontal flipping, and rotation, while testing used multiscale evaluation and random flipping augmentation.
  • Evaluation metrics: Evaluation included OA, F1 score, mIoU, Parameters, and FLOPs, with precision and recall defined using true-positive and error counts.
  • Evaluation metrics: OA is computed as (tp + tn) divided by (tp + fp + fn + tn).
  • Evaluation metrics: mIoU averages the IoU values across categories, while overall F1 averages class-specific F1 scores.

B. Ablation Experiment

Ablation experiments examine SFFNet's branches and fusion components on Vaihingen and Potsdam, using mIoU and meanF1 as primary metrics. Removing components weakens segmentation, while local, global, low-frequency, and high-frequency features improve distinct visual regions.

  • Ablation setup: Ablations on Vaihingen and Potsdam focus primarily on mIoU and meanF1, with results averaged across multiple trials.
  • Ablation setup: The component study distinguishes Global, Local, WTFD-L, WTFD-H, and MDAF, and compares MDAF against direct concatenation or addition.
  • Reported comparisons: The ablation results are organized in tables covering component removal, module additions, global mapping comparisons, and backbone parameter counts.
  • Incremental additions: Adding the Global branch improves segmentation of large continuous regions relative to the baseline.
  • Incremental additions: Adding the Local branch improves segmentation of local details, including small objects.
  • Incremental additions: Low-frequency WTFD features noticeably affect objects that are difficult to segment under shadows.
  • Incremental additions: Adding high-frequency WTFD features improves edge regions in the segmentation results.
  • Component removal: Removing any SFFNet branch or component results in weaker segmentation performance.

2) Impact of Global branch:

The Global branch improves SFFNet’s segmentation of large continuous regions and contributes to accuracy on both Vaihingen and Potsdam. It also provides efficient global mapping compared with mainstream alternatives.

  • Impact of Global branch: The Global branch improves segmentation of large continuous objects and increases feature discriminability in large continuous regions.
  • Impact of Global branch: SFFNet’s Global branch performed best in FLOPs and segmentation accuracy against ViT, Swin-T, Mobile-ViT, and Fast-ViT global-mapping modules.

3) Impact of Local branch:

SFFNet’s Local branch improves small-object segmentation and local-detail discrimination, while frequency components and MDAF further support performance across Vaihingen and Potsdam comparisons.

  • Impact of Local branch: Removing the Local branch reduced mIoU by 1.05% on both Vaihingen and Potsdam, while F1 decreased by 0.62% and 0.6%, respectively.
  • Impact of Local branch: The Local branch improves segmentation of small local objects and helps distinguish local details.
  • Frequency and fusion components: Removing low-frequency features reduced mIoU by 0.81% on Vaihingen and 1.18% on Potsdam, while adding them increased baseline mIoU by 1.18%.
  • Frequency and fusion components: Removing high-frequency features reduced mIoU by 0.69% on Vaihingen and 0.8% on Potsdam.
  • Frequency and fusion components: Replacing MDAF with Concat reduced Vaihingen and Potsdam mIoU by 0.59% and 0.56%, while Addition reduced them by 0.78% and 0.72%.
  • Comparative experiments: SFFNet achieved 84.80% mIoU on Vaihingen and 87.73% on Potsdam, surpassing the compared segmentation methods.
  • Comparative experiments: Frequency-aware models perform better in shadows, edges, and high-texture regions, while SFFNet preserves spatial awareness that XNet lacks.

V. CONCLUSION

SFFNet combines spatial and frequency-domain feature processing in a two-stage segmentation network to address grayscale variation while preserving spatial information.

  • Conclusion: SFFNet first extracts spatial features, then maps them through global and local spatial branches to preserve extensive spatial information.
  • Conclusion: WTFD uses the Haar wavelet transform to map features into low-frequency and high-frequency components for frequency-domain processing.
  • Conclusion: The paper aims to encourage practical solutions for grayscale-variation segmentation and further exploration of Haar wavelet decomposers in remote sensing.
Loading 2405.01992v1…