Source-linked AI summary
Unsupervised Intra-domain Adaptation for Semantic Segmentation through Self-Supervision
Fei Pan, Inkyu Shin, Francois Rameau, Seokju Lee, In So Kweon
TL;DR
Synthetic data reduce annotation demands but remain difficult to transfer to real images, and prior adaptation methods largely overlook distribution differences within the target domain. The paper combines inter-domain adaptation with entropy-based easy–hard splitting and self-supervised easy-to-hard adaptation. Across reported benchmarks, the method outperforms existing adaptation approaches, while its effectiveness is limited by large source–target divergence and source error.
Problem
Synthetic-to-real semantic segmentation remains difficult, while prior UDA methods mainly address the source–target gap and overlook substantial distribution differences within target data.
Method
The method first performs inter-domain adaptation, ranks target images by entropy into easy and hard splits, then adapts from easy to hard using pseudo labels.
Results
The method outperforms existing adaptation algorithms across synthetic-to-real traffic experiments and reaches 46.3% mean IoU, a 2.5% improvement over AdvEnt.
Takeaways & Limitations
Combining inter-domain and intra-domain alignment provides a self-supervised strategy that can be combined with existing domain-adaptation approaches.
Takeaways & Limitations
Performance is affected by source error and source–target divergence, making the model less efficient when the domain gap is large.
Abstract
from arXiv · showhide
Convolutional neural network-based approaches have achieved remarkable progress in semantic segmentation. However, these approaches heavily rely on annotated data which are labor intensive. To cope with this limitation, automatically annotated data generated from graphic engines are used to train segmentation models. However, the models trained from synthetic data are difficult to transfer to real images. To tackle this issue, previous works have considered directly adapting models from the source data to the unlabeled target data (to reduce the inter-domain gap). Nonetheless, these techniques do not consider the large distribution gap among the target data itself (intra-domain gap). In this work, we propose a two-step self-supervised domain adaptation approach to minimize the inter-domain and intra-domain gap together. First, we conduct the inter-domain adaptation of the model; from this adaptation, we separate the target domain into an easy and hard split using an entropy-based ranking function. Finally, to decrease the intra-domain gap, we propose to employ a self-supervised adaptation technique from the easy to the hard split. Experimental results on numerous benchmark datasets highlight the effectiveness of our method against existing state-of-the-art approaches. The source code is available at https://github.com/feipan664/IntraDA.git.
1. Introduction
Semantic segmentation needs extensive pixel-level annotation, motivating synthetic-to-real adaptation. This work addresses both the source–target gap and the distribution gap within real target data through two-step self-supervision.
- Pixel-level annotation is expensive and labor-intensive, making large-scale semantic-segmentation datasets difficult to collect.
- Synthetic simulator and game-engine images provide precise annotations, but models trained on them transfer poorly to real images because of cross-domain differences.
- Previous domain-adaptation methods primarily adapt models from labeled source data to unlabeled target data, addressing the inter-domain gap.
- Real target data also exhibit an intra-domain gap caused by diverse scenes, moving objects, and weather conditions.
- The proposed approach performs inter-domain adaptation, entropy-based easy–hard target splitting, and intra-domain adaptation using pseudo labels from the easy split.
2. Related Works
Related work has addressed domain alignment through adversarial learning at feature, image, and output levels, while entropy and curriculum strategies support confidence-based or staged adaptation. The proposed approach builds on these directions by ranking target images with entropy and adapting from easier to harder samples.
- Unsupervised Domain Adaptation: Adversarial UDA aligns source and target distributions by training a generator and discriminator to learn domain-invariant features.
- Unsupervised Domain Adaptation: Domain alignment has also been performed at the image level with CycleGAN and at the output level with structural output alignment.
- Uncertainty via Entropy: Entropy has been used to minimize target uncertainty and measure confidence when transferring samples across domains.
- Curriculum Domain Adaptation: Curriculum domain adaptation processes easier samples before harder ones, sometimes using intermediate domains to decompose domain discrepancies.
3. Approach
The approach combines inter-domain adaptation, entropy-based target ranking, and intra-domain adaptation in a two-step self-supervised pipeline. It ranks target images into easy and hard splits, then aligns the hard split to the easy split using pseudo labels and adversarial learning.
- 3.1. Inter-domain Adaptation: The method first adapts a segmentation model between labeled source data and unlabeled target data using inter-domain adversarial learning.The inter-domain discriminator predicts domain labels while the generator is trained to fool it, alongside segmentation loss optimization.
- 3.2. Entropy-based Ranking: Target images are ranked by mean prediction entropy and divided into easy and hard splits using a ratio hyperparameter λ.The easy split contains the selected proportion of target images, while the remaining images form the hard split; no dataset-specific threshold is introduced.
- 3.3. Intra-domain Adaptation: The intra-domain stage aligns the easy and hard target splits by using easy-split predictions from Ginter as pseudo labels.Soft segmentation maps from the inter-domain generator are converted to one-hot pseudo labels for supervised optimization of Gintra.
- 3.3. Intra-domain Adaptation: An intra-domain discriminator distinguishes easy- and hard-split entropy maps, while Gintra is trained to confuse it.This adversarial alignment is applied to entropy maps from both splits to bridge their distribution gap.
- 3. Approach: Training proceeds in three stages: inter-domain optimization, target pseudo-label generation and ranking, then intra-domain optimization.The complete objective is not minimized in a single training stage because the proposed model is a two-step self-supervised approach.
4. Experiments
Experiments evaluate the two-step adaptation approach across synthetic-to-real semantic segmentation and digit-domain benchmarks. Results show gains from intra-domain alignment, self-training, entropy normalization, and combined adaptation.
- Experimental setup: The experiments use GTA5, SYNTHIA, and Synscapes as synthetic sources and Cityscapes as the unlabeled real target, evaluated with mean IoU.Cityscapes validation performance uses the PASCAL VOC intersection-over-union metric.
- GTA5→Cityscapes: 46.3% mean IoU on GTA5→Cityscapes improves AdvEnt by 2.5% through the proposed intra-domain adaptation.The baseline AdvEnt reaches 43.8% mIoU, while the proposed method reaches 46.3%.
- Discussion: Entropy normalization raises GTA5→Cityscapes performance to 47.0% mIoU by dividing mean entropy by the number of predicted rare classes.The normalization moves images containing many objects toward the easy split.
- SYNTHIA→Cityscapes: 41.7% and 48.9% mean IoU are achieved on SYNTHIA→Cityscapes using 16-class and 13-class evaluation settings, respectively.The method is reported as more accurate on car and motor bike classes than existing techniques.
- Synscapes→Cityscapes: 54.2% mIoU is achieved on Synscapes→Cityscapes, higher than the AdaptSegNet baseline.The experiment uses vanilla-GAN for the comparison and combines inter-domain and intra-domain adaptation.
- Digit adaptation: Digit adaptation also outperforms CyCADA, reaching 95.8±0.1% on MNIST→USPS, 97.8±0.1% on USPS→MNIST, and 95.1±0.3% on SVHN→MNIST.The digit experiments use CyCADA-based inter-domain and intra-domain networks.
5. Conclusion
The paper presents a self-supervised adaptation method that jointly addresses inter-domain and intra-domain gaps. It combines inter-domain training, entropy-based target splitting, and intra-domain adaptation, outperforming existing adaptation algorithms.
- The method jointly minimizes inter-domain and intra-domain gaps through self-supervised domain adaptation.
- It first applies inter-domain adaptation, then uses target-image entropy maps to split the target data into easy and hard subsets.
- It finally performs intra-domain adaptation to further narrow the gap between the target subsets.
- The model can be combined with existing domain adaptation approaches and outperforms existing adaptation algorithms in experiments on synthetic-to-real traffic images.