Source-linked AI summary
Unsupervised Domain Adaptation using Generative Adversarial Networks for Semantic Segmentation of Aerial Images
Bilel Benjdira, Yakoub Bazi, Anis Koubaa, Kais Ouni
TL;DR
Aerial semantic-segmentation models lose accuracy when transferred to new cities or sensor domains because of domain shift, making new pixel-labeled target datasets costly. The paper uses GAN-based image translation and labeled-source fine-tuning to adapt the model, improving Potsdam-to-Vaihingen performance and sensor-affected classes.
Problem
Domain shift reduces the accuracy of segmentation models when they are applied to new aerial-image domains, while collecting target-domain pixel labels is costly.
Method
The method translates source images toward the target domain with a cyclic-loss GAN, then fine-tunes the pretrained segmentation model using the translated images and original labels.
Results
The method increases average target-dataset accuracy by 17% and also improves precision, recall, F1, and IoU.
Takeaways & Limitations
The method targets sensor-related domain shift without affecting classes associated with other or absent domain shifts, while requiring no labeling data or manual work.
Takeaways & Limitations
The GAN can make translated source images resemble the target domain without guaranteeing that their structural content is preserved.
Abstract
from arXiv · showhide
Segmenting aerial images is being of great potential in surveillance and scene understanding of urban areas. It provides a mean for automatic reporting of the different events that happen in inhabited areas. This remarkably promotes public safety and traffic management applications. After the wide adoption of convolutional neural networks methods, the accuracy of semantic segmentation algorithms could easily surpass 80% if a robust dataset is provided. Despite this success, the deployment of a pre-trained segmentation model to survey a new city that is not included in the training set significantly decreases the accuracy. This is due to the domain shift between the source dataset on which the model is trained and the new target domain of the new city images. In this paper, we address this issue and consider the challenge of domain adaptation in semantic segmentation of aerial images. We design an algorithm that reduces the domain shift impact using Generative Adversarial Networks (GANs). In the experiments, we test the proposed methodology on the International Society for Photogrammetry and Remote Sensing (ISPRS) semantic segmentation dataset and found that our method improves the overall accuracy from 35% to 52% when passing from Potsdam domain (considered as source domain) to Vaihingen domain (considered as target domain). In addition, the method allows recovering efficiently the inverted classes due to sensor variation. In particular, it improves the average segmentation accuracy of the inverted classes due to sensor variation from 14% to 61%.
1. INTRODUCTION
Semantic segmentation enables pixel-level understanding of aerial scenes, but models trained in one city or sensor domain can lose accuracy after deployment elsewhere. The paper proposes GAN-based adaptation that translates source images toward the target domain before fine-tuning the segmentation model.
- Semantic segmentation assigns every input pixel a label describing its enclosing region, enabling complete scene understanding.
- CNN-based segmentation can exceed 80% accuracy when trained on a robust, semantically labeled dataset.
- Performance falls dramatically when images differ in lighting, sensor type, resolution, or object representation across source and target domains.
- Target-domain pixel labeling is costly and impractical because collecting large annotated aerial datasets is time-consuming and expensive.
- The proposed method translates source images to target-domain characteristics with a cyclic-loss GAN, then fine-tunes the segmentation model using original labels.
2. Related works
Prior domain-adaptation work largely addresses classification, regression, synthetic-to-real transfer, or vehicle-mounted urban imagery. The paper targets the comparatively unaddressed problem of adapting semantic segmentation across aerial-image domains.
- Existing domain-adaptation efforts focus mainly on classification, regression, and improving deep-learning adaptability.
- Several semantic-segmentation methods use simulated data to improve performance on real images.
- FCNs in the wild use pixel-level adversarial loss to encourage domain-invariant features, while CyCADA translates synthetic images toward real-image style.
- The paper identifies aerial semantic-segmentation adaptation as insufficiently treated because most prior methods use urban scenes captured from car-mounted cameras.
- Experiments study adaptation from the Potsdam dataset to the Vaihingen dataset in the ISPRS semantic-segmentation benchmark.
3. Generative Adversarial Networks (GANs)
GANs train a generator and discriminator in competition, and image-to-image translation applies this framework to changing images between domains. The paper distinguishes paired translation, which needs corresponding examples, from unpaired translation, which can use randomly sampled images from each domain.
- 3.1. Generator and Discriminator: A GAN contains a generator that produces data resembling real samples and a discriminator that distinguishes real from generated data.
- 3.1. Generator and Discriminator: The generator and discriminator compete through an adversarial zero-sum game during simultaneous training.
- 3.1. Generator and Discriminator: Equation 1 defines the GAN objective as a two-player minimax game involving real samples, noise, generated samples, and discriminator outputs.
- 3.2. Image-to-Image Translation: Image-to-image translation converts an image from one domain into an image that mimics another domain's appearance.
- 3.2. Image-to-Image Translation: Paired translation learns from corresponding source-target image pairs, whereas unpaired translation learns bidirectionally from two non-corresponding image sets.
4. Proposed method
The proposed method uses GAN-based source-to-target image translation, followed by segmentation-model training on translated images with reused source labels. Adversarial and reconstruction objectives seek target-domain appearance while preserving source-image structure.
- GAN architecture: The method translates source-domain images into target-domain appearance using an unpaired image-to-image translation GAN.The translation targets sensor type, image quality, and resolution characteristics of the target domain.
- GAN architecture: A reverse target-to-source mapping and reconstruction loss are added to preserve image content and structure during translation.The method enforces approximate recovery of source images after forward and inverse mappings.
- GAN architecture: The generator uses an encoder-decoder architecture with four downsampling and four upsampling convolutional layers.Leaky ReLU is used for downsampling and standard ReLU for upsampling.
- GAN architecture: The discriminator classifies generated images as real or fake after encoding them into a 256-dimensional feature vector.It uses five convolutional layers, softmax output, Leaky ReLU activations, and Instance Normalization except in the first layer.
- Domain adaptation algorithm: The four-stage algorithm trains a source segmentation model, trains the GAN, translates source images, and trains a target model on translated images.The target model is initialized from the source model with fixed weights as a starting point and trained using reused source pixel labels.
- Domain adaptation algorithm: The GAN maps source samples toward target-domain appearance, while the target segmentation model learns from the translated images and their original labels.The adversarial discriminator is trained to distinguish generated images from target-domain images.
- Domain adaptation algorithm: The adversarial loss can ensure general resemblance to the target domain but cannot guarantee preservation of the source image’s structural content.This limitation motivates the inverse mapping and reconstruction constraint.
5. Experimental results
This section states that the experiments evaluate the efficiency of the proposed algorithm through implemented experiments and discussion of the obtained results.
- Experimental setup: The experiments are designed to evaluate the efficiency of the proposed domain-adaptation algorithm.The section presents the implemented experiments and discusses their obtained results.
5.1. Datasets and evaluation metrics
The evaluation uses the ISPRS aerial-image semantic-segmentation benchmark, with Potsdam and Vaihingen datasets containing six labeled ground-object classes. Metrics include accuracy, precision, recall, F1 score, and mean IoU, while domain shift is analyzed through sensor, resolution, and class-representation differences.
- Datasets: The experiments use the ISPRS 2D Semantic Labeling benchmark with publicly available Potsdam and Vaihingen aerial-image datasets.Only image data are used, although Digital Surface Model data are provided.
- Datasets: The datasets provide labels for six classes: building, tree, car, impervious surfaces, low vegetation, and clutter/background.Very High Resolution imagery provides detailed object information for aerial-image segmentation.
- Datasets: The six classes are imbalanced, with impervious surfaces and buildings more represented than cars and clutter.Table 1 reports each class percentage relative to the total number of dataset pixels.
- Domain shift analysis: The Potsdam-to-Vaihingen domain shift combines imaging-sensor, resolution, and class-representation factors.Potsdam uses RGBIR imagery at 5 centimeters per pixel, whereas Vaihingen uses IRRG imagery at 9 centimeters per pixel.
- Evaluation metrics: Segmentation efficiency is measured using accuracy, precision, recall, and F1 score based on true-positive, true-negative, false-positive, and false-negative pixels.These measures are defined per class using the corresponding confusion-count quantities.
- Evaluation metrics: Intersection over Union is calculated separately for each of the six classes, then averaged as mean IoU.Mean IoU summarizes the class-wise IoU values.
5.2. Experimental settings
The experiments train BiSeNet on Potsdam, translate source images toward Vaihingen using a GAN, and fine-tune on the translated dataset. Target-domain accuracy improves during fine-tuning.
- 5.2.1. Step1: Training the segmentation model: BiSeNet was trained on Potsdam using ResNet101 features for 80 epochs without image augmentation.Training used ADAM with a 0.0001 learning rate and batch size 1.
- 5.2.1. Step1: Training the segmentation model: Potsdam validation accuracy rapidly exceeded 86% within a few epochs.
- 5.2.2. Training the proposed GAN: The GAN was trained on randomly sampled Potsdam and Vaihingen image patches to translate source-domain imagery toward the target domain.Each dataset contained 400 512 ∗512 images, split into 300 training and 100 test images.
- 5.2.4. Step4: Fine-tuning the segmentation model with the translated dataset: Fine-tuning on the translated dataset increased Vaihingen average accuracy from 34% to values between 39% and 52%.The largest reported increase was 17% after eight epochs.
- 5.2.4. Step4: Fine-tuning the segmentation model with the translated dataset: Trees and Buildings improved substantially over fine-tuning epochs, while other classes remained practically unchanged.
6. Discussion
The proposed adaptation improves target-domain segmentation, particularly for classes whose domain shift is strongly associated with sensor variation. Its effect is limited for classes dominated by other shift factors.
- Overall results: The algorithm increased target-dataset average accuracy by up to 17%, with similar improvements in precision, recall, F1, and IoU.The improvement also produced visibly better predicted segmentation masks.
- Class-specific effects: Classes strongly affected by sensor variation benefited most, whereas Impervious surfaces, Car, Clutter background, and Low vegetation showed no practical accuracy change.The latter classes were either unaffected by domain shift or mainly affected by factors other than the sensor.
- Class-specific effects: Building accuracy increased from 0.23 to 0.71 when the domain shift was associated with the sensor factor.
- Scope of effectiveness: The method preserves accuracy when there is no domain shift or when other factors dominate the shift.This scope allows combination with techniques targeting other domain-shift factors.
7. Conclusion
The paper proposes a GAN-based method for adapting aerial-image semantic segmentation across domains. It targets sensor-related domain shift without labeled target data, while future work extends toward semi-supervised settings.
- The paper proposes a GAN-based method for domain adaptation in aerial-image semantic segmentation.
- The method targets sensor variation and does not require labeled target data or other manual work.It uses randomly collected source and target images and applies the adaptation steps automatically.
- Future work will extend the method to semi-supervised domain scenarios where a few target-domain labels are available.