Source-linked AI summary
Feature Pyramid Network for Multi-Class Land Segmentation
Selim S. Seferbekov, Vladimir I. Iglovikov, Alexander V. Buslaev, Alexey A. Shvets
TL;DR
Multi-class land segmentation from satellite imagery remains challenging, while high-resolution optical imagery offers broad coverage for monitoring Earth’s surface. The paper develops an FPN-based model with a pretrained ResNet50 encoder and reports a public leaderboard score of 0.493 for the land-cover classification challenge.
Problem
Multi-class land segmentation is needed to categorize satellite imagery for monitoring Earth’s surface and supporting applications such as urban planning and disaster prevention.
Method
The model uses a Feature Pyramid Network with a pretrained ImageNet ResNet50 bottom-up encoder, top-down pathways with lateral connections, and a seven-channel segmentation output.
Results
0.493 public leaderboard score was achieved by the model on the land-cover classification challenge.
Takeaways & Limitations
The FPN-based approach provides a practical multi-class land-segmentation model evaluated on the public leaderboard.
Takeaways & Limitations
The training labels are imperfect because annotating multi-class segmentation masks is costly, and small objects are not consistently annotated.
Abstract
from arXiv · showhide
Semantic segmentation is in-demand in satellite imagery processing. Because of the complex environment, automatic categorization and segmentation of land cover is a challenging problem. Solving it can help to overcome many obstacles in urban planning, environmental engineering or natural landscape monitoring. In this paper, we propose an approach for automatic multi-class land segmentation based on a fully convolutional neural network of feature pyramid network (FPN) family. This network is consisted of pre-trained on ImageNet Resnet50 encoder and neatly developed decoder. Based on validation results, leaderboard score and our own experience this network shows reliable results for the DEEPGLOBE - CVPR 2018 land cover classification sub-challenge. Moreover, this network moderately uses memory that allows using GTX 1080 or 1080 TI video cards to perform whole training and makes pretty fast predictions.
1. Introduction
High-resolution optical satellite imagery supports broad land-surface monitoring, while fully convolutional methods enable dense pixel-wise segmentation. The paper applies an FPN-based approach to multi-class land segmentation in high-resolution satellite imagery.
- Motivation: High-resolution optical satellite imagery is widely available and can support land-surface monitoring and topographic databases.Its 0.5–1.0 m geometric resolution is described as sufficient for land-cover categorization.
- Prior approaches: Fully convolutional networks replace fully connected layers with convolutions to produce spatial feature maps and dense pixel-wise outputs.The maps are upsampled, allowing end-to-end segmentation with input images of arbitrary sizes.
- Proposed approach: FPN uses bottom-up and top-down pathways with lateral connections to construct feature pyramids at multiple scales.The paper selects a ResNet50 encoder pre-trained on ImageNet for the bottom-up pathway.
- Proposed approach: The proposed network targets multi-class segmentation of high-resolution satellite imagery.The work generalizes ideas from prior convolutional approaches to this remote-sensing setting.
2. Dataset
The challenge dataset contains high-resolution RGB satellite images paired with seven-class color-coded land-cover masks. Its labels have annotation limitations, and validation images require submitted predictions for evaluation.
- Dataset composition: The training data contains 803 RGB satellite images, each sized 2448x2448 pixels at 50 cm resolution.The imagery was collected by DigitalGlobe satellites, and each training image has a paired land-cover mask.
- Annotations: Each mask encodes seven land-cover classes using RGB color coding.The supplied passage begins listing classes including urban and agriculture land.
- Annotations: Mask values may not be pure 0 and 255, so threshold 128 is recommended for binarization.This affects preparation of the color-coded annotations.
- Evaluation: The labels are imperfect because multi-class mask annotation is costly, and small objects are intentionally left unannotated.The validation set contains 171 images without masks whose predictions are uploaded to the DeepGlobe website.
3. Model
The model addresses multi-scale land-object segmentation with an FPN built from a pre-trained ResNet50 encoder and a custom decoder. It merges feature maps across scales before producing seven-channel full-resolution outputs.
- Motivation: FPN is used because segmenting objects at different scales, especially small objects, is challenging.The general FPN architecture is presented as the model’s solution to this problem.
- Encoder: The bottom-up pathway uses pre-trained ResNet50 modules for feature extraction, reducing spatial resolution while increasing hierarchical representation.Outputs from convolution modules C_i are passed to the top-down pathway.
- Decoder: The top-down pathway upsamples features, combines them element-wise with lateral bottom-up features, and applies convolutions to form multi-scale feature maps.Nearest-neighbor upsampling by 2 is used between successive top-down stages.
- Output: The decoder concatenates four same-resolution feature maps, applies convolution, normalization, activation, and dropout, then reduces channels to seven classes.The final seven-channel map is upsampled to the original image size using bilinear interpolation.
4. Training
Training combines one-hot masks, IoU-based evaluation, categorical cross entropy, and a generalized loss, with augmentation and dropout used to address limited and imperfect data. Predictions use whole-image inference and test-time augmentation.
- Objective: Masks are converted to seven-channel one-hot images, and performance is evaluated with the Jaccard index, also called IoU.The metric measures similarity between sets represented by masks and predictions.
- Objective: The training objective combines categorical cross entropy with an intersection-based term weighted by α and β.The selected values α = 1 and β = 0.5 provided the best performance in validation.
- Augmentation: Training uses spatial and color augmentation, including scale transformations from 0.6−1.4, 30-degree rotations, random 448x448 crops, and color changes.A quarter of the dataset is held out for validation.
- Regularization: Spatial dropout with p = 0.5 is used to reduce over-fitting, while training and validation IoU become close to 0.55.The reported standard deviation is around 0.13 between classes and 0.03 between iterations.
- Inference: Whole 2448x2448 images are padded by 8 pixels, and test-time augmentation averages four predictions from 90-degree rotations.Padding makes the image side divisible by 32 and helps prevent bottom-up pathway artifacts.
5. Conclusions
The authors developed a multi-class land segmentation system using FPN with a pretrained ResNet50 encoder and a tailored loss function. The model achieved a best public leaderboard score of 0.493, while addressing over-fitting with spatial dropout and test-time augmentation.
- The system uses an FPN with an ImageNet-pretrained ResNet50 in the bottom-up pathway and a designed loss function.
- Inaccurate class labeling is identified as the main difficulty in the multi-class segmentation problem.
- Strong spatial dropout on the final layer and test-time augmentation were used to prevent over-fitting.
- 0.493 was the model's best public leaderboard score.