Source-linked AI summary
An Attention-Fused Network for Semantic Segmentation of Very-High-Resolution Remote Sensing Imagery
Xuan Yang, Shanshan Li, Zhengchao Chen, Jocelyn Chanussot, Xiuping Jia, Bing Zhang, Baipeng Li, Pan Chen
TL;DR
Very-high-resolution remote sensing segmentation must fuse multisource inputs while retaining both abstract semantics and precise spatial information. The paper proposes AFNet with multipath encoding and attention-fused modules for these two fusion problems, reporting state-of-the-art performance on the ISPRS Vaihingen and Potsdam 2D datasets. Overfitting remains a scope-bound concern because the authors note that its use may only be suitable for these two datasets.
Problem
Deep convolutional networks face difficulty fusing multisource remote sensing data and combining high-level abstract features with low-level spatial features.
Method
AFNet integrates a multipath encoder with MAFB for multisource fusion and RAFB for high-level and low-level feature fusion.
Results
AFNet achieves state-of-the-art performance on the ISPRS Vaihingen 2D and Potsdam 2D datasets.
Takeaways & Limitations
The proposed architecture simultaneously addresses multipath feature fusion and multilevel feature fusion in very-high-resolution remote sensing semantic segmentation.
Takeaways & Limitations
The authors note that overfitting may only be used on the ISPRS Vaihingen and Potsdam datasets and is not recommended for most datasets.
Abstract
from arXiv · showhide
Semantic segmentation is an essential part of deep learning. In recent years, with the development of remote sensing big data, semantic segmentation has been increasingly used in remote sensing. Deep convolutional neural networks (DCNNs) face the challenge of feature fusion: very-high-resolution remote sensing image multisource data fusion can increase the network's learnable information, which is conducive to correctly classifying target objects by DCNNs; simultaneously, the fusion of high-level abstract features and low-level spatial features can improve the classification accuracy at the border between target objects. In this paper, we propose a multipath encoder structure to extract features of multipath inputs, a multipath attention-fused block module to fuse multipath features, and a refinement attention-fused block module to fuse high-level abstract features and low-level spatial features. Furthermore, we propose a novel convolutional neural network architecture, named attention-fused network (AFNet). Based on our AFNet, we achieve state-of-the-art performance with an overall accuracy of 91.7% and a mean F1 score of 90.96% on the ISPRS Vaihingen 2D dataset and an overall accuracy of 92.1% and a mean F1 score of 93.44% on the ISPRS Potsdam 2D dataset.
1. Introduction
Remote sensing semantic segmentation must handle multisource data and preserve both abstract and precise spatial information. AFNet addresses these feature-fusion challenges with multipath extraction and attention-based fusion modules.
- Very-high-resolution imagery preserves detailed textures that help distinguish target objects in remote sensing scenes.
- Remote sensing inputs include multiband imagery and auxiliary data such as NDVI and DSM, which differ from ordinary RGB images.
- DCNN pooling preserves abstract representations but loses spatial information, creating a need to fuse high-level and low-level features for pixel classification.
- Mainstream DCNNs struggle with both multipath fusion from multisource data and multilevel fusion between abstract and spatial features.
- The MAFB uses attention to weight and fuse features from different inputs, while the RAFB fuses high-level abstract and low-level spatial features.
- AFNet combines a multipath encoder, MAFB, and RAFB to address multipath and multilevel feature-fusion problems.
2. Related Work
Related segmentation networks use encoder-decoder structures, dilated convolutions, pooling, skip connections, and attention to balance abstraction, context, and spatial precision. The paper positions AFNet as addressing limitations in existing feature-fusion strategies.
- FCN introduced skip connections and upsampling for semantic segmentation but provides insufficient high-level abstraction and inaccurate boundaries after eight-times upsampling.
- Backbone-style networks such as PSPNet and DeepLabV3 use dilated convolutions and pyramid or atrous spatial pyramid pooling to reduce downsampling and fuse features.
- RefineNet uses multipath refinement but ignores global context, while related methods use global pooling or concatenate features from different receptive fields.
- DANet combines channel and spatial attention, but its branches independently extract single-path features and fuse them through addition.
- DFN fuses abstract and spatial features with channel attention but does not consider multipath inputs or spatial weights for refining high-level features.
- AFNet is presented as an architecture overview integrating multipath and multilevel feature fusion through its MPE, MAFB, and RAFB components.
3. Methodology
AFNet addresses multisource and multilevel feature-fusion challenges through multipath encoding and attention-guided fusion. Its MAFB and RAFB modules weight complementary feature information before producing segmentation outputs.
- Multipath encoding: The multipath encoder uses separate main and auxiliary branches instead of directly stacking multisource inputs.The branches independently extract features from image and auxiliary data before fusion.
- Multipath attention fusion: The variant spatial-attention module extracts key spatial features alongside channel attention.It extends channel-based weighting with spatial-dimension weighting.
- Multipath attention fusion: MAFB concatenates branch features, computes spatial and channel attention weights, and fuses the resulting weighted features.The module is intended to learn useful information from image and auxiliary data while suppressing interference.
- Refinement attention fusion: RAFB uses high-level features to weight low-level features and low-level spatial information to refine high-level feature localization.This addresses the trade-off between abstraction and spatial accuracy in encoder-decoder segmentation networks.
- AFNet architecture: AFNet combines a DFN baseline with a multipath encoder, MAFB for multipath fusion, and RAFB for multilevel feature fusion.The architecture is designed for multipath inputs and for combining abstract and spatial information.
4.1. Datasets
The experiments use the ISPRS Vaihingen and Potsdam 2D aerial-image benchmarks, each containing six land-cover categories and separate online-evaluation test tiles. Potsdam additionally provides near-infrared/RGB imagery and DSM data for training and inference.
- Vaihingen 2D: The Vaihingen 2D dataset contains six land-cover categories and 33 aerial-image tiles from Vaihingen, Germany.The categories are impervious surfaces, buildings, low vegetation, trees, cars, and clutter/background.
- Vaihingen 2D: Vaihingen tiles average 2494 × 2064 pixels at 9 cm resolution, with 17 tiles reserved for online evaluation.The remaining 16 samples are divided into training and validation sets.
- Potsdam 2D: The Potsdam 2D dataset contains six land-cover categories and 38 aerial-image tiles from Potsdam, Germany.Its imagery includes near-infrared/RGB data and DSM information.
- Potsdam 2D: Potsdam tiles are 6000 × 6000 pixels at 5 cm resolution, with 14 tiles used for online evaluation.The other 24 samples are divided into training and validation sets.
- Preprocessing: Image slicing uses 50% overlap, with the valid inference range distinguished from the slice range.The figure also marks the mirror axis used in the slicing procedure.
4.2. Implementation Details
The implementation preprocesses multisource IRRG, DSM, and NDVI data, slices large images with overlap, augments training and inference inputs, and uses different training and inference tile sizes. TTA combines predictions from augmented inputs before final classification.
- Data preprocessing: IRRG and DSM data are normalized, while NDVI is computed from the IR and R channels and retained without normalization because it lies between −1 and 1.The processed inputs are divided into IRRG image data and NDVI/DSM auxiliary data.
- Image slicing: Large IRRG and NDVI/DSM tiles are sliced with overlap to reduce boundary-related edge effects after stitching.The overlap addresses incomplete semantic information near slice boundaries, although increasing overlap increases the number of slices.
- Data augmentation: Training augmentation uses random horizontal and vertical flips, random rotation, and random cropping to improve generalization and avoid overfitting.
- Data augmentation: Inference augmentation applies horizontal and vertical flips and rotation, then integrates multiple predictions through test-time augmentation.The combined inference results are used to obtain the final classification result.
- Implementation settings: Training uses 640 × 640-pixel network inputs with 800 × 800-pixel slices and 400-pixel overlap, while inference uses 1920 × 1920-pixel inputs and 960-pixel overlap.The implementation uses one NVIDIA TITAN Xp GPU with 12 GB memory.
- Training schedule: The total number of iterations is defined as the total number of epochs multiplied by the iterations per epoch.The learning-rate passages define variables for WarmUp and Step strategies but do not state their specific schedules here.
4.3. Evaluation Metrics
Evaluation uses overall accuracy and category-level F1 scores, with mean F1 added because overall accuracy is insufficiently sensitive to small categories. Overall accuracy measures the fraction of correctly classified pixels.
- Overall accuracy: Overall accuracy is the ratio of correctly classified pixels to the total number of pixels.The numerator counts correctly classified pixels, and the denominator counts all pixels.
- F1 score: Category accuracy is evaluated with F1 scores calculated from precision and recall.True positives occupy the confusion matrix diagonal, while false positives and false negatives aggregate off-diagonal column and row elements, respectively.
- Overall evaluation: Mean F1 averages category-level F1 scores because overall accuracy is not adequately sensitive to small categories.
4.4. Experiments on the Vaihingen Dataset
Experiments on ISPRS Vaihingen evaluate AFNet through module ablations, test-time augmentation, repeated runs, and comparisons with state-of-the-art methods. The proposed architecture progressively improves segmentation and achieves the best reported performance in the supplied comparisons.
- Ablation Study: The ablation study evaluates MPE, RAFB, MAFB, and TTA by progressively modifying a DFN baseline on the Vaihingen test set.The baseline uses IRRG input with ResNet-50; module variants isolate multipath encoding, attention-fused fusion, refinement fusion, and test-time augmentation.
- Ablation Study: 91.1% OA and 90.28% mean F1 score: AFNet combines MAFB and RAFB within the multipath architecture before TTA.The AFNet configuration is identified as MPVN-RM in the ablation study.
- Ablation Study: 91.7% OA: TTA raises AFNet accuracy by 0.6% on Vaihingen.Across networks, TTA increases OA by approximately 0.4% to 0.6%.
- Ablation Study: The qualitative ablation results show progressively fewer building and road errors as MPE, MAFB, RAFB, and TTA are added.The reported improvements include recovering missing building corners, reducing road misclassification, and repairing fine details.
- Comparing Methods: AFNet achieves the best performance among the compared methods on the ISPRS Vaihingen 2D dataset.The comparison includes methods using ensembles, post-processing, or different training settings; AFNet is reported without multimodel ensemble learning or post-processing.
4.5. Experiments on the Potsdam Dataset
Experiments on ISPRS Potsdam evaluate AFNet with the same training and inference settings and report ablations, TTA effects, and comparisons with other methods. AFNet achieves state-of-the-art performance on this dataset.
- Experiments on the Potsdam Dataset: The Potsdam experiments evaluate MPE, RAFB, MAFB, and TTA using the same training and inference settings as the Vaihingen experiments.Ablation results are reported in Tables 5 and 6, with qualitative results in Figures 16 and 17.
- Experiments on the Potsdam Dataset: 92.1% OA and 93.44% mean F1 score: AFNet achieves these results on the ISPRS Potsdam 2D dataset.The reported result follows application of the stated inference settings and TTA evaluation.
- Experiments on the Potsdam Dataset: AFNet outperforms the compared state-of-the-art methods on the ISPRS Potsdam 2D dataset.The comparison is presented numerically in Table 7 and qualitatively in Figure 17.
5. Discussion
The discussion evaluates AFNet’s architectural and training choices, including encoder branches, input slice size, optimization, and overfitting behavior. These experiments motivate the selected settings and identify a dataset-specific boundary for using overfitting during training.
- Encoder selection: ResNet-50 was retained for the main encoder branch because ResNet-18/34 produced validation accuracy approximately 1% lower.The weaker performance was attributed to insufficient feature abstraction for the task’s complexity.
- Encoder selection: ResNet-18 was selected for the auxiliary branch because NDVI/DSM features require lower complexity, while larger auxiliary encoders showed almost identical performance.The auxiliary data provided only limited improvement, and the smaller branch reduced parameter demands.
- Slice size: 640 produced the highest validation accuracy among the tested slice sizes of 256, 512, and 640.Larger slices helped address complex scenes and widely varying object scales, but GPU memory and batch normalization created a trade-off.
- Optimization: The Step learning-rate strategy improved convergence stability and slightly improved network performance compared with the WarmUp strategy.The learning rate was multiplied by 0.1 every 200 epochs.
- Optimization: Adam was selected because AFNet converged slowly with SGD, whose loss remained significantly greater in the experiments.The authors considered SGD difficult to tune for AFNet’s multi-branch encoder and complex decoder.
- Overfitting: Validation loss began increasing at epoch 194, whereas validation accuracy reached its maximum at epoch 409, indicating slight overfitting without reduced validation accuracy.The final probability feature map was thresholded to obtain predictions, and the remaining reliability stayed above the correct-classification threshold.
- Overfitting: The retraining procedure was applied to both datasets, but using overfitting as a training strategy was limited to the ISPRS Vaihingen and Potsdam datasets.The authors explicitly state that overfitting remains inadvisable for most datasets.
6. Conclusions
The paper concludes that AFNet addresses feature fusion in very-high-resolution remote sensing imagery through asymmetric multipath extraction and attention-based fusion. Its MAFB and RAFB modules respectively fuse different data types and combine abstract with spatial features.
- Conclusions: AFNet combines the MPE, MAFB, and RAFB to address multipath data fusion and multilevel feature fusion in very-high-resolution remote sensing imagery.The MPE extracts IRRG and NDVI/DSM features, while the attention modules learn effective channel and spatial information.
- Conclusions: The asymmetric MPE extracts different feature types from different inputs while saving hardware resources and maintaining accuracy.Its branches are designed according to the characteristics of the input data.
- Conclusions: MAFB fuses different data types, and RAFB fuses high-level abstract features with low-level spatial features.Both modules use attention structures to learn effective information for their respective fusion tasks.