Source-linked AI summary
A Novel Transformer Based Semantic Segmentation Scheme for Fine-Resolution Remote Sensing Images
Libo Wang, Rui Li, Chenxi Duan, Ce Zhang, Xiaoliang Meng, Shenghui Fang
TL;DR
Fine-resolution remote-sensing segmentation needs stronger contextual modeling than conventional FCN-based approaches provide. The paper combines a Swin Transformer encoder with a DCFAM decoder, achieving effective results on the Vaihingen and Potsdam datasets.
Problem
FCN-based segmentation methods have coarse decoder outputs and limited long-range dependency, while existing extensions remain based on convolutional architectures.
Method
The proposed DC-Swin uses Swin Transformer as the encoder and a densely connected feature aggregation module as the decoder.
Results
DC-Swin achieves 90.71% mean F1-score, 91.63% OA, and 83.22% mIoU on Vaihingen, plus 93.25%, 92.00%, and 87.56%, respectively, on Potsdam.
Takeaways & Limitations
Experiments on the Vaihingen and Potsdam datasets demonstrate the effectiveness of the proposed scheme for semantic-segmentation accuracy.
Abstract
from arXiv · showhide
The fully convolutional network (FCN) with an encoder-decoder architecture has been the standard paradigm for semantic segmentation. The encoder-decoder architecture utilizes an encoder to capture multilevel feature maps, which are incorporated into the final prediction by a decoder. As the context is crucial for precise segmentation, tremendous effort has been made to extract such information in an intelligent fashion, including employing dilated/atrous convolutions or inserting attention modules. However, these endeavors are all based on the FCN architecture with ResNet or other backbones, which cannot fully exploit the context from the theoretical concept. By contrast, we introduce the Swin Transformer as the backbone to extract the context information and design a novel decoder of densely connected feature aggregation module (DCFAM) to restore the resolution and produce the segmentation map. The experimental results on two remotely sensed semantic segmentation datasets demonstrate the effectiveness of the proposed scheme.Code is available at https://github.com/WangLibo1995/GeoSeg
I. INTRODUCTION
FCN encoder-decoder networks provide a common semantic-segmentation framework, but coarse decoding and limited long-range dependency constrain precise segmentation. The paper introduces Swin Transformer and DCFAM to address these limitations.
- Motivation: FCN uses a contracting encoder to extract features and an expanding decoder to recover positional information.The encoder progressively downsamples feature maps, while the decoder progressively increases the receptive field.
- Motivation: FCN predictions can remain coarse because of the decoder’s over-simplified design.
- Motivation: FCN-based methods have limited long-range dependency because convolutional locality restricts contextual modeling.Prior responses include large kernels, dilated convolutions, feature pyramids, and attention mechanisms.
- Transformer Motivation: Transformer architectures use multihead attention to model long-range dependencies in sequence modeling and transduction.
- Contribution: The paper introduces Swin Transformer for fine-resolution remote-sensing segmentation and proposes DCFAM for multi-scale relation-enhanced semantic features.Together they form the Densely Connected Swin Transformer scheme.
II. METHODOLOGY
DC-Swin uses an encoder-decoder architecture with Swin Transformer as the encoder and DCFAM as the decoder.
- Architecture: DC-Swin adopts Swin Transformer for encoding and the proposed DCFAM for decoding.
A. Swin Transformer
The Swin Transformer backbone partitions RGB images into patch tokens, transforms them through hierarchical stages, and offers configurable complexity. The study selects ImageNet-pretrained Swin-S as its encoder backbone.
- Input Representation: Swin Transformer partitions the input RGB image into non-overlapping patches and represents each patch using concatenated RGB values.
- Feature Transformation: A linear embedding projects patch features to dimension C before paired Swin Transformer blocks extract semantic features.The blocks maintain the token count within the stage.
- Backbone Variants: Four backbones—Swin-T, Swin-S, Swin-B, and Swin-L—vary by embedding dimension C and the number of Transformer blocks.
- Backbone Selection: The study chooses ImageNet-pretrained Swin-S, with 50M parameters compared with 45M for ResNet-101.
B. Densely Connected Feature Aggregation Module
DCFAM addresses confusing, multi-scale geospatial objects by enhancing spatial and channel relationships and aggregating transformer features across scales. Its connections combine low- and high-level features while using upsampling and dilated convolutions to restore resolution and capture context.
- DCFAM targets multi-scale and confusing geospatial objects that affect fine-resolution remote-sensing segmentation.
- The downsample connection fuses low-level and high-level transformer features.
- The Large field Upsample Connection embeds dilated convolution to capture multi-scale context effectively.The described composite functions use dilation rates of 12 and 6, alongside convolution and transpose convolution operations.
- Shared Spatial Attention models long-range dependencies across the spatial dimension.
- Shared Channel Attention extracts long-range dependencies among feature channels.
- Four aggregation features, AF1 through AF4, are computed to combine the module outputs.The aggregation equations include bilinear upsampling with a scale factor of 2.
A. Dataset
The evaluation uses the ISPRS Vaihingen and Potsdam semantic labelling datasets, with predefined training/testing splits and 1024×1024-pixel input patches. Vaihingen contains 33 tiles, while Potsdam contains 38.
- The experiments use the ISPRS Vaihingen and Potsdam semantic labelling datasets.
- Vaihingen contains 33 tiles, with 16 images for training and 17 for testing.The tiles have an average size of 2494×2064 pixels.
- Potsdam contains 38 tiles, with 24 tiles for training and 14 for testing.Each tile is sized 6000×6000 pixels.
- Image tiles are cropped into 1024×1024-pixel patches as input, without using DSMs in the experiments.
B. Experimental Setting
Experiments use PyTorch with AdamW optimization, a 0.0003 learning rate, and soft cross-entropy loss. Performance is assessed using OA, mIoU, and F1-score, with class-wise confusion counts defining the metrics.
- Experiments are implemented in PyTorch using AdamW with a 0.0003 learning rate and soft cross-entropy loss.
- Overall accuracy, mean Intersection over Union, and F1-score are the evaluation indices.
- The metrics use true-positive, false-positive, true-negative, and false-negative counts for each class.
- Overall accuracy is computed across all categories, including the background.
C. Semantic Segmentation Results and Analysis
DC-Swin achieves strong semantic-segmentation performance on both Vaihingen and Potsdam, surpassing most ResNet-based methods and competing favorably with contextual, multi-scale, and transformer alternatives. The authors attribute this performance to Swin-S global context modeling and DCFAM.
- 90.71% mean F1-score, 91.63% OA, and 83.22% mIoU are achieved on Vaihingen.
- 93.25% mean F1-score, 92.00% OA, and 87.56% mIoU are achieved on Potsdam.
- DC-Swin outperforms most ResNet-based methods with highly competitive accuracy.
- The scheme surpasses DeepLabV3+, PSPNet, EaNet, DDCM-Net, BoTNet, and ResT in the reported comparisons.
1) Performance Comparison:
The ablation study separates the contributions of the Swin-S backbone, dense connections, and shared attention modules across the Vaihingen and Potsdam datasets. Swin-S+DCFAM achieves the highest accuracy on both datasets.
- Replacing ResNet-101 with Swin-S increases mIoU by 3% on Vaihingen and 4.05% on Potsdam.
- Dense connections substantially improve the performance of both ResNet101+DC and Swin-S+DC over their corresponding baseline methods.
- Shared SCA and SSA attention modules further increase accuracy, while the nonshared DCFAM-NS variant scores lower than DCFAM.
- Swin-S+DCFAM obtains the highest accuracy on both datasets, benefiting from long-range dependencies and a shared multi-scale structure.
IV. CONCLUSION
The paper introduces Transformer-based semantic segmentation for fine-resolution remote sensing images and combines Swin Transformer with a densely connected feature aggregation module. Experiments on Vaihingen and Potsdam demonstrate segmentation-accuracy effectiveness and motivate broader exploration of Transformers in remote sensing.
- The paper introduces Transformer into semantic segmentation of fine-resolution remote sensing images for the first time.
- The densely connected feature aggregation module captures multi-scale relation-enhanced semantic features to increase segmentation accuracy.
- Numerical experiments on the ISPRS Vaihingen and Potsdam datasets demonstrate the scheme's effectiveness in segmentation accuracy.
- The authors envisage broader exploration of Transformer potential and feasibility in remote sensing and Earth observation.