Source-linked AI summary
SCAttNet: Semantic Segmentation Network with Spatial and Channel Attention Mechanism for High-Resolution Remote Sensing Images
Haifeng Li, Kaijian Qiu, Li Chen, Xiaoming Mei, Liang Hong, Chao Tao
TL;DR
High-resolution remote sensing images pose semantic-segmentation challenges because their backgrounds and ground-object differences are complex and diverse. SCAttNet uses spatial and channel attention to adaptively refine features, and experiments on ISPRS Vaihingen and Potsdam show improved segmentation results.
Problem
Complex and diverse backgrounds and pronounced ground-object differences make high-resolution remote sensing image semantic segmentation challenging.
Method
SCAttNet is a semantic segmentation network that uses spatial and channel attention modules to improve high-resolution remote sensing image segmentation accuracy.
Results
SCAttNet achieves improved semantic segmentation results on the ISPRS Vaihingen and Potsdam datasets, including higher reported metrics than baseline models.
Takeaways & Limitations
The attention modules improve segmentation performance, including results for small objects such as cars.
Takeaways & Limitations
The study uses only two common attention modules, leaving more effective module design and discriminative-feature capture for future work.
Abstract
from arXiv · showhide
High-resolution remote sensing images (HRRSIs) contain substantial ground object information, such as texture, shape, and spatial location. Semantic segmentation, which is an important task for element extraction, has been widely used in processing mass HRRSIs. However, HRRSIs often exhibit large intraclass variance and small interclass variance due to the diversity and complexity of ground objects, thereby bringing great challenges to a semantic segmentation task. In this paper, we propose a new end-to-end semantic segmentation network, which integrates lightweight spatial and channel attention modules that can refine features adaptively. We compare our method with several classic methods on the ISPRS Vaihingen and Potsdam datasets. Experimental results show that our method can achieve better semantic segmentation results. The source codes are available at https://github.com/lehaifeng/SCAttNet.
I. INTRODUCTION
HRRSI semantic segmentation is challenging because complex, diverse scenes produce substantial ground-object differences and make effective feature representation difficult. SCAttNet addresses this challenge with spatial and channel attention, achieving competitive results and improved performance on small objects.
- Semantic segmentation classifies every image pixel and supports applications including change detection, element extraction, and military target recognition.
- HRRSIs present wide imaging ranges, complex backgrounds, and diverse ground objects that complicate semantic segmentation.
- Existing attention methods select meaningful channel features but may not enhance feature representation along the spatial dimension.
- SCAttNet is proposed as a semantic segmentation network using spatial and channel attention to improve HRRSI segmentation accuracy.
- Experiments on Vaihingen and Potsdam demonstrate competitive results and remarkable performance improvements on small objects.
A. Overview of SCAttNet
SCAttNet combines a feature-extraction backbone with cascaded channel and spatial attention. Two variants use SegNet or ResNet50 as their backbone.
- The network consists of a backbone for feature extraction and an attention module composed of cascaded channel and spatial attention.
- Extracted features are refined first across channels and then across spatial positions before semantic segmentation output is obtained.
- SCAttNet V1 uses SegNet as its backbone, whereas SCAttNet V2 uses ResNet50.
C. Attention Module
The attention module adaptively refines feature representations in channel and spatial dimensions. Channel attention emphasizes informative semantic channels, while spatial attention aggregates location information, especially for small objects.
- Channel Attention: Channel attention learns a 1D weight for each feature channel using relationships among channels, then multiplies those weights with the corresponding channels.
- Spatial Attention: Spatial attention learns a 2D spatial weight map from relationships among positions and applies it to corresponding spatial locations.
- Spatial Attention: Spatial attention is useful for aggregating spatial information in HRRSIs with varied object sizes and complicated distributions, especially for small ground objects.
- Complexity: The channel attention module has C × C/4 parameters and the spatial attention module has 98 parameters, while placement only at the backbone’s end layer seldom adds computation complexity.
- Attention Integration: The channel and spatial modules are integrated sequentially, selecting significant feature maps before choosing considerable neuron activity within each map.
III. EXPERIMENT
The experiments evaluate SCAttNet on the ISPRS Vaihingen and Potsdam datasets.
- The evaluation measures network performance on the ISPRS Vaihingen and ISPRS Potsdam datasets.
- The study compares semantic segmentation performance across two ISPRS remote-sensing benchmarks.
- The experimental section assesses the proposed network using dataset-level semantic segmentation results.
A. Datasets and evaluation metrics
The study evaluates semantic segmentation using the ISPRS Vaihingen and Potsdam datasets and the MIoU, AF, and OA metrics.
- Datasets: The ISPRS Vaihingen dataset contains 33 orthophoto maps, 16 labeled maps, three spectral bands, and six categories.Its average image size is 2494 × 2064 at 9 cm resolution.
- Datasets: The ISPRS Potsdam dataset contains 38 orthophoto maps and related normalized DSMs, with 24 labeled maps.
- Evaluation metrics: Semantic segmentation performance is evaluated with mean intersection over union, average F1-score, and overall accuracy.
B. Implementation details
The experiments use selected labeled Vaihingen images for evaluation and training, generate cropped patches, and train several models from scratch under specified backbone settings.
- Data split: The Vaihingen labeled dataset is split into four evaluation images and 12 training images.
- Data preparation: Training images are randomly cropped into 256 × 256 patches, yielding 27,000 training patches.
- Data selection: The experiment excludes datasets without DSMs to retain wide application value.
- Model training: All models are trained from scratch without bells and whistles, using VGG-16 or ResNet50 backbones according to the model configuration.
C. Results of Vaihingen dataset
On the Vaihingen dataset, attention modules improve segmentation performance over the corresponding SegNet and ResNet50 baselines, with particularly notable gains for cars.
- Overall results: 1.4%/1.25%/0.56% MIoU/AF/OA gains result from adding channel attention to original SegNet.
- Attention-module analysis: 3.08%/3.21% IoU/F1-score gains for cars result from adding the spatial attention module.
- Overall results: 2.9%/2.59% MIoU/AF gains over original SegNet are achieved after channel-refined features are input to the spatial attention module.
- Overall results: 1.21%/0.83%/0.90% MIoU/AF/OA gains over ResNet50 are achieved by SCAttNet V2.
- Visualization and local evaluation: 64.92%/77.74%/85.49% MIoU/AF/OA are obtained by SCAttNet V1 on Vaihingen ID32, versus 60.57%/74.01%/82.78% for SegNet.
D. Results of Potsdam dataset
On Potsdam, SCAttNet variants outperform their comparative models across reported metrics, while visualizations indicate improved building segmentation and reduced interference from other categories.
- Overall results: 1.12%/1.08%/1.03% MIoU/AF/OA gains over ResNet50 are achieved by SCAttNet V2.
- Overall results: 2.08%/1.74%/1.35% MIoU/AF/OA gains over SegNet are achieved by SCAttNet V1.
- Visualization and local evaluation: 59.50%/71.16%/85.04% MIoU/AF/OA are obtained by SCAttNet V1 on Potsdam ID 5 12, versus 57.19%/68.87%/83.81% for SegNet.
- Visualization: The Potsdam visualization compares original images, SegNet outputs, SCAttNet V1 outputs, and ground truth in separate Vaihingen and Potsdam rows.
- Attention visualization: Attention modules suppress impervious-surface responses in car and building visualizations, reducing interference from other categories.
IV. CONCLUSION
The study proposes an attention-based semantic segmentation network that adaptively refines features and demonstrates effectiveness on two ISPRS datasets. It remains limited to two common attention modules, leaving more discriminative feature capture as future work.
- SCAttNet adaptively refines features based on attention modules for semantic segmentation.
- Experiments on the ISPRS Vaihingen and Potsdam datasets demonstrate the method’s effectiveness.
- Using only two common attention modules limits the study, motivating more effective attention designs for capturing discriminative semantic features.