Source-linked AI summary
Self-Ensembling with GAN-based Data Augmentation for Domain Adaptation in Semantic Segmentation
Jaehoon Choi, Taekyung Kim, Changick Kim
TL;DR
Semantic segmentation adaptation is hindered by costly pixel annotations and domain shift, while manual augmentations can misalign self-ensembling predictions. The paper combines target-guided, cycle-free GAN augmentation with self-ensembling and reports state-of-the-art benchmark performance.
Problem
Pixel-level annotation is costly, and manual geometric augmentation used for self-ensembling is not suited to reducing semantic-segmentation domain shift.
Method
The framework uses target-guided, cycle-free GAN augmentation that preserves semantic content and collaborates with self-ensembling.
Results
The approach achieves state-of-the-art performance on challenging unsupervised domain adaptation benchmarks.
Takeaways & Limitations
GAN-based augmentation provides a target-style alternative to geometric transformations for strengthening self-ensembling in semantic segmentation.
Takeaways & Limitations
Adversarial domain alignment can trigger negative transfer by aligning target features with incorrect source semantic categories.
Abstract
from arXiv · showhide
Deep learning-based semantic segmentation methods have an intrinsic limitation that training a model requires a large amount of data with pixel-level annotations. To address this challenging issue, many researchers give attention to unsupervised domain adaptation for semantic segmentation. Unsupervised domain adaptation seeks to adapt the model trained on the source domain to the target domain. In this paper, we introduce a self-ensembling technique, one of the successful methods for domain adaptation in classification. However, applying self-ensembling to semantic segmentation is very difficult because heavily-tuned manual data augmentation used in self-ensembling is not useful to reduce the large domain gap in the semantic segmentation. To overcome this limitation, we propose a novel framework consisting of two components, which are complementary to each other. First, we present a data augmentation method based on Generative Adversarial Networks (GANs), which is computationally efficient and effective to facilitate domain alignment. Given those augmented images, we apply self-ensembling to enhance the performance of the segmentation network on the target domain. The proposed method outperforms state-of-the-art semantic segmentation methods on unsupervised domain adaptation benchmarks.
1. Introduction
The paper addresses synthetic-to-real semantic segmentation by combining target-guided GAN augmentation with self-ensembling. This design preserves semantic content while transferring target-domain style and improves unsupervised adaptation performance.
- Semantic segmentation requires costly, time-consuming pixel-level annotation, motivating adaptation from labeled synthetic data to unlabeled real data.
- Adversarial domain alignment can cause negative transfer by matching target features to incorrect source semantic categories.
- Self-ensembling uses teacher and student networks, but geometric augmentations can create spatial misalignment between their segmentation predictions.
- TGCF-DA uses a target-guided GAN generator and semantic constraint to synthesize target-styled images while preserving semantic content.
- The proposed framework unifies TGCF-DA with self-ensembling, training segmentation networks on source, target, and augmented data.
- The approach achieves state-of-the-art performance on challenging unsupervised domain adaptation benchmarks.
2. Related work
Related work spans adversarial, cycle-consistency, style-transfer, and self-training approaches for semantic segmentation adaptation. The proposed framework builds on self-ensembling and GAN-based image translation while avoiding cycle-consistency.
- Semantic segmentation adaptation commonly uses adversarial training to produce domain-invariant features through domain confusion.
- Other methods jointly adapt pixel- and feature-level representations using cycle-consistency loss or style transfer.
- Self-training methods such as CBST generate pseudo labels and retrain the segmentation network with them.
- Self-ensembling applies consistency regularization using teacher predictions or exponential moving averages of model weights.
- GAN-based unpaired image translation commonly uses cycle-consistency to map images to the target domain and reconstruct the originals.
3. Proposed Method
TGCF-DA combines target-guided GAN augmentation with self-ensembling to align source representations with target styles while preserving semantic content. The student uses supervised losses on source and augmented images, while target samples contribute a consistency loss with an EMA teacher.
- Target-guided generator: TGCF-DA translates source images into target-domain styles using a target-guided generator while retaining their spatial and semantic structure.The generator extracts target style information through AdaIN and preserves source-image structure during translation.
- Semantic constraint: A fixed pretrained segmentation model supplies a pixel-level semantic constraint that preserves objects and stabilizes adversarial training without cycle-consistency.The constraint is implemented as a cross-entropy loss between predictions for images before and after translation.
- Target-guided and cycle-free data augmentation: The GAN uses a target-guided generator and discriminator, with adversarial training designed to produce target-like images without losing source semantic content.The framework uses the LSGAN objective and spectral normalization to stabilize GAN training.
- Self-ensembling: The student learns from labeled source and GAN-augmented images with supervised cross-entropy, while unlabeled target images contribute a student–teacher consistency loss.The teacher network is updated from the student using an exponential moving average, and target-sample augmentation is used for consistency training.
- Self-ensembling: The total training objective combines supervised and consistency losses, with the consistency weight ramped up during training.The paper reports that weight ramp-up is necessary to improve the effectiveness of the consistency loss.
4. Experiments
The experiments evaluate synthetic-to-real adaptation from GTA5 or SYNTHIA to Cityscapes, using segmentation models, augmentation, self-ensembling, and ablation studies. The proposed TGCF-DA + SE framework achieves strong adaptation performance and improves over its components and baselines.
- Experimental setup: Experiments evaluate adaptation from synthetic GTA5 or SYNTHIA to real-world Cityscapes, using Cityscapes validation as the evaluation set.GTA5 includes 24,966 urban scene images with pixel-level annotations; the evaluation reports class IoU and mean IoU.
- Experimental setup: The augmentation network, TGCF-DA, combines a generator, discriminator, and segmentation model for synthetic data generation.Its generator uses source and target encoders plus a decoder and is based on an autoencoder architecture.
- Experimental results: 14.2% and 13.1% are the proposed framework’s improvements over the baseline on GTA5→Cityscapes and SYNTHIA→Cityscapes, respectively.The paper reports state-of-the-art mIoU scores on both adaptation experiments.
- Ablation studies: SE alone improves mIoU by 4.3% and 4.9% in the two experiments, whereas TGCF-DA improves mIoU by 7.1% and 11.0%.The reported comparisons are against the baseline in Table 1-(a) and Table 1-(b), respectively.
- Ablation studies: The teacher and student networks in the proposed method enhance their performance simultaneously during training.The authors attribute this result to TGCF-DA enhancing self-ensembling for semantic segmentation.
- Ablation studies: On GTA5→Cityscapes, Figure 4 compares testing mIoUs for SE and the proposed method across training epochs.SE is shown with blue and orange lines, while the proposed method is shown with grey and yellow lines.
5. Analysis
The analysis examines how self-ensembling, GAN-based augmentation, and training settings affect adaptation quality, image semantics, and class-wise performance.
- Self-ensembling visualization: Teacher-student consistency predictions improve boundary refinement later in training, while teacher predictions provide useful early training targets.The consistency loss concentrates on object boundaries at later training stages.
- Image synthesis comparison: TGCF-DA preserves semantic structures more effectively than CycleGAN, UNIT, and MUNIT while avoiding cycle-consistency modules and their computational burden.CycleGAN and UNIT can distort objects, whereas MUNIT may cause style spillover across semantic classes.
- Per-class IoU analysis: Self-ensembling generally improves per-class IoUs more for majority classes such as road than for minority classes such as bus.Teacher pseudo-labels can reinforce incorrect minority-class predictions under class imbalance.
- Hyperparameter sensitivity: The consistency-loss weight uses δ = 1+δ0e−5(1−x)2 with δ0 = 30, while EMA decay changes from 0.99 early to 0.999 later.The added one ensures that consistency loss contributes at the beginning of training.
- Hyperparameter sensitivity: A semantic-constraint weight of λsem = 10 preserves local and global image structures, whereas λsem = 1 encourages object and scene mixing.These examples support semantic consistency as a key constraint for the augmentation generator.
6. Conclusion
The paper proposes a complementary framework combining target-guided GAN augmentation with self-ensembling for unsupervised semantic-segmentation domain adaptation. Without cycle consistency, the framework produces semantically consistent augmented images and achieves superior performance to existing state-of-the-art approaches.
- 6. Conclusion: The framework combines target-guided GAN-based data augmentation with self-ensembling for unsupervised domain adaptation in semantic segmentation.Augmented images support pseudo-label transfer from the teacher network to the student network.
- 6. Conclusion: The augmentation network avoids cycle consistency while producing augmented images for domain alignment.The conclusion identifies cycle-free generation as part of the proposed framework.
- 6. Conclusion: Experimental results verify that the proposed model is superior to existing state-of-the-art approaches.