PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Domain Adaptation in Semantic Segmentation: a Review

Marco Toldo, Andrea Maracani, Umberto Michieli, Pietro Zanuttigh

arXiv:2005.10876v1cs.CVcs.LGeess.IV

TL;DR

Deep semantic segmentation requires extensive labeled data, while real-world target domains often differ from available source data. This paper surveys deep UDA methods, organizing them by adaptation level and methodological category, and compares their performance in autonomous-driving scenarios. The review finds substantial variation across methods and backbones, while noting that UDA remains below supervised target-domain training and includes open problems such as open-set and boundless-set adaptation.

  • Problem

    Semantic segmentation requires huge amounts of labeled training data, while practical settings often lack suitable annotations and exhibit source–target domain shifts.

  • Method

    The paper comprehensively reviews deep UDA for semantic segmentation, categorizing methods by adaptation level and seven non-mutually-exclusive methodological families.

  • Results

    The survey compares methods on synthetic-to-real road-scene adaptation and reports that results vary greatly by method, evaluation protocol and backbone; GTA5-to-Mapillary reaches 41.9 mIoU for the highest-performing cited approach.

  • Takeaways & Limitations

    UDA for semantic segmentation remains an open research field with substantial room for improvement and possible relevance to continual learning under changing data distributions.

  • Takeaways & Limitations

    Standard UDA assumes shared source and target classes, although real target images may contain unseen categories; UDA performance also remains below supervised target-domain training.

Abstract

from arXiv · show

The aim of this paper is to give an overview of the recent advancements in the Unsupervised Domain Adaptation (UDA) of deep networks for semantic segmentation. This task is attracting a wide interest, since semantic segmentation models require a huge amount of labeled data and the lack of data fitting specific requirements is the main limitation in the deployment of these techniques. This problem has been recently explored and has rapidly grown with a large number of ad-hoc approaches. This motivates us to build a comprehensive overview of the proposed methodologies and to provide a clear categorization. In this paper, we start by introducing the problem, its formulation and the various scenarios that can be considered. Then, we introduce the different levels at which adaptation strategies may be applied: namely, at the input (image) level, at the internal features representation and at the output level. Furthermore, we present a detailed overview of the literature in the field, dividing previous methods based on the following (non mutually exclusive) categories: adversarial learning, generative-based, analysis of the classifier discrepancies, self-teaching, entropy minimization, curriculum learning and multi-task learning. Novel research directions are also briefly introduced to give a hint of interesting open problems in the field. Finally, a comparison of the performance of the various methods in the widely used autonomous driving scenario is presented.

1. Introduction

Semantic segmentation assigns a semantic label to every pixel, making it a challenging dense-labeling task that depends on extensive annotation. Domain adaptation addresses distribution shifts by transferring knowledge from related labeled source domains to different target domains, and this survey organizes deep UDA methods and reviews their applications.

  • Semantic segmentation: Semantic segmentation assigns a label to each pixel, producing dense semantic masks rather than image-level or region-level labels.This provides a more detailed understanding of image content but makes the task more demanding than classification or object detection.
  • Motivation: Deep segmentation models can achieve impressive performance, but training requires massive labeled datasets whose pixel-wise annotation is expensive and time-consuming.Datasets such as Cityscapes, Mapillary, Pascal VOC, MS-COCO, ADE20K, NYUD-v2 and SUN-RGBD address different scene and object settings.
  • Domain adaptation: Domain adaptation transfers labeled knowledge from one or more related source domains to a different target domain to address distribution shifts that degrade model performance.The closer the source and target domains are, the easier adaptation becomes, making source-data selection important.
  • Unsupervised domain adaptation: Unsupervised domain adaptation uses labeled source samples and unlabeled target samples, typically adapting a source-trained model without direct target-domain supervision.The standard setting assumes that source and target domains differ but remain related, such as synthetic source data and real-world target data.
  • Survey scope: The survey analyzes deep UDA methods for semantic segmentation, organizes the literature into categories, and reviews adaptation stages, techniques, applications and future directions.Its structure covers problem formulation, adaptation at different network stages, state-of-the-art methods, synthetic-to-real road-scene adaptation, and comparative results grouped by architecture and evaluation scenario.

2. Unsupervised Domain Adaptation for Semantic Segmentation

UDA addresses distribution shifts between labeled source domains and unlabeled, related target domains in semantic segmentation. Adaptation can align inputs, features, outputs, or additional network-level spaces, while preserving semantic consistency.

  • Problem Formulation: Semantic segmentation maps each image pixel to a label, making its required pixel-level annotations particularly costly to acquire.This motivates using domain adaptation to reduce dependence on target-domain annotations.
  • Problem Formulation: Domain adaptation scenarios include closed-set, partial, open-set, open-partial, and boundless settings, distinguished by source-target category overlap.In open-set adaptation, target-only categories are generally treated as an unknown class, whereas boundless adaptation learns them individually.
  • Adaptation Spaces: Input-level adaptation matches visual appearance, feature-level adaptation aligns latent representations, and output-level adaptation aligns segmentation prediction spaces.Output spaces retain semantic cues while offering lower-dimensional representations for adaptation.
  • Adaptation Spaces: Marginal distribution alignment alone may preserve no semantic consistency because class-conditional distributions can still differ across domains.The review describes reconstruction, prediction-uniformity, and engineered low-level-statistics constraints as ways to address this issue.
  • Adaptation Spaces: Some methods align ad-hoc activation spaces to capture high-level patterns through gradients flowing across multiple network levels.The review presents these approaches alongside input-, feature-, and output-level adaptation techniques.

3. Review of Unsupervised Domain Adaptation strategies

The review organizes UDA methods for semantic segmentation into seven main categories and distinguishes adversarial and generative approaches by their adaptation mechanisms.

  • Taxonomy: The review groups UDA approaches into seven main categories for semantic segmentation.The categories are presented through a visual overview in Figure 5.
  • Adversarial Learning: Adversarial discriminative approaches use adversarial learning to address domain adaptation.The supplied passage introduces them as one of the review’s principal categories.
  • Generative-Based Approaches: Generative-based approaches translate data between domains or generate target-like training data from source data.They may also transform source data into representations closer to target-domain characteristics before network processing.

3.1. Weakly- and Semi- Supervised Learning

Weakly and semi-supervised segmentation methods preceded UDA and reduce reliance on pixel-level labels through image tags, bounding boxes, or limited strong annotations.

  • Motivation and Background: Early domain-adaptation work for segmentation adapted classification methods before approaches explicitly modeled segmentation’s spatial structure.Weakly and partially supervised techniques historically preceded UDA.
  • Motivation and Background: Weakly supervised segmentation uses image-level labels, while semi-supervised segmentation combines many weakly labeled samples with few pixel-level annotations.Pixel-level semantic maps are more expensive and time-consuming to obtain than image tags or bounding boxes.
  • Weakly Supervised Learning: Multiple Instance Learning methods estimate unobserved pixel-label probabilities from image-label probabilities and impose output-space constraints.A new loss function is introduced to optimize the constraint set.
  • Weakly Supervised Learning: A simple-to-complex framework distinguishes single-object clean images from cluttered images containing multiple objects and categories.The framework first uses salient-object detection to compute semantic information.
  • Semi-Supervised Learning: A semi-supervised architecture combines classification and segmentation networks through bridging layers trained with weak and strong annotations.Class-specific activation maps transfer information from the classification network to simplify segmentation training.
  • Semi-Supervised Learning: Another iterative approach trains segmentation using bounding-box annotations rather than pixel-level semantic maps.The supplied passage introduces region proposal methods as the first step of this procedure.

3.2. Domain Adversarial Discriminative

Domain adversarial approaches align source and target representations by training domain discriminators against segmentation networks, operating at feature, reconstruction, or output levels.

  • Adversarial Learning: Generative adversarial networks introduced adversarial learning for generating samples with distributions similar to real data.This generative objective motivated later extensions to domain adaptation.
  • Feature Adversarial Adaptation: Feature adversarial adaptation repurposes the real-fake discriminator as a source-target domain classifier.The segmentation network receives source supervision while the discriminator distinguishes source from target representations.
  • Feature Adversarial Adaptation: Some feature-level methods add target-guided distillation by matching target activations with those from an ImageNet-pretrained network.The authors argue that this decreases overfitting to source data.
  • Feature Adversarial Adaptation: Other methods align latent embeddings with reconstruction constraints and apply adversarial learning in reconstructed image space.The reconstruction constraint encourages latent embeddings to retain enough information to recover their input images.
  • Output Adversarial Adaptation: Output adversarial adaptation aligns prediction maps from source and target inputs in the segmentation network’s lower-dimensional output space.A domain discriminator is trained to identify the origin domain of the prediction maps.
  • Output Adversarial Adaptation: Tsai et al. improve adversarial signal flow by applying discriminators to multiple dense classification outputs at different network depths.Later methods combine output-space adversarial adaptation with additional modules.
  • Output Adversarial Adaptation: Recent methods extract meaningful patterns from segmentation outputs to guide discriminators toward more functional source-target alignment.Entropy minimization is one strategy used for this purpose.

3.3. Generative-Based Approaches

Generative-based approaches adapt segmentation by translating images between source and target domains, aiming to reduce pixel-level statistical differences while preserving source semantics. Methods span CycleGAN-based bidirectional translation, lighter one-way alternatives, target-to-source translation with pseudo-labeling, and style-transfer decompositions.

  • Image-to-Image Translation: Image-to-image translation learns cross-domain image mappings from unpaired data, but its ill-posedness requires constraints to obtain acceptable joint distributions.The task seeks a joint distribution across domains, and marginal distributions alone do not determine a unique solution.
  • Image-to-Image Translation: Input-level adaptation transfers target visual attributes onto source images while preserving source semantic information to reduce covariate shift.The resulting images can serve as target-distributed training data while retaining original source annotations.
  • CycleGAN-Based Methods: CycleGAN is widely used for bidirectional source-to-target and target-to-source translation through two coupled generative adversarial models.The framework concurrently performs conditional translation in both directions and is used by many domain-adaptation methods.
  • Semantic Preservation: Several methods enforce semantic consistency by requiring translated images to preserve content recognized by the segmentation predictor or a semantic-aware discriminator.Other work adds phase consistency, based on the observation that image semantics are mostly encoded in Fourier phase rather than amplitude.
  • Alternative Generative Designs: Lightweight alternatives discard CycleGAN’s backward projection branch, while target-to-source translation can generate source-like target images for pseudo-label-based supervised training.The one-way designs reduce the computational burden associated with adding at least four networks to the semantic predictor.
  • Style Transfer: Style-transfer methods disentangle images into domain-specific style and domain-invariant content representations to achieve image-level appearance invariance.Style captures low-level texture information, whereas content expresses domain-invariant structure.

3.4. Classifier Discrepancy

Classifier-discrepancy methods address a limitation of feature-level adversarial adaptation: global alignment can ignore category-level structure and place features near decision boundaries. Adversarial Dropout Regularization uses prediction disagreement to align target features away from those boundaries.

  • Adversarial Feature Alignment: Original feature-level adversarial adaptation aligns source and target representations using a domain critic alongside source-supervised segmentation training.The feature extractor competes with a domain discriminator while the task objective maintains segmentation discriminativeness.
  • Limitations: Semantic-unaware critics can miss category-level joint distributions, placing features near class boundaries and transferring target representations to incorrect categories.These effects arise even when the critic achieves global marginal-distribution alignment.
  • Adversarial Dropout Regularization: Adversarial Dropout Regularization perturbs the dense classifier to produce two predictions, then minimizes their discrepancy to push target features away from decision boundaries.The classifier maximizes output variation so it can detect less-adapted target features, while the extractor reduces that variation.

3.5. Self-Training

Self-training adapts segmentation models by converting confident target predictions into pseudo-labels for additional supervision. Because iterative self-teaching can propagate errors, methods filter, weight, or ensemble predictions to improve pseudo-label reliability.

  • Core Strategy: Self-training uses highly confident predictions on unlabeled target data as pseudo-labels that reinforce predictor training.The approach transfers self-generated supervision from target predictions into subsequent optimization.
  • Core Strategy: Iterative self-teaching can progressively propagate prediction mistakes, so most methods restrict learning to top-confidence target predictions.Confidence filtering implicitly assumes that the most confident predictions are more likely to be correct.
  • Pseudo-Label Filtering: Offline methods compute dataset-wide confidence thresholds, filter target segmentation maps, and train with both pseudo-labeled target data and annotated source data.The threshold is recomputed by examining the entire training set at each update step.
  • Pseudo-Label Filtering: Weighted pseudo-labeling learns a confidence threshold that jointly refines and weights labels, balancing uncertain pixels while computing labels over a single batch.The learned threshold retains a global view despite batch-level pseudo-label computation.
  • Reliability Estimation: Adversarial output discriminators can estimate prediction reliability, allowing reliable target predictions to guide pseudo-label filtering.This strategy applies the discriminative adaptation module directly over segmentation outputs.
  • Ensembling and Teachers: Prediction ensembling combines multiple segmenters or low- and high-level predictions to generate more reliable target pseudo-labels.Mean predictions from different segmenter instances support more robust adaptation across multiple self-training rounds.
  • Ensembling and Teachers: Self-ensembling uses a teacher network formed from averaged student weights, with the student trained to follow the teacher’s target predictions.The teacher supplies auxiliary pseudo-label-like guidance for unlabeled samples.

3.6. Entropy Minimization

Entropy-minimization methods use prediction uncertainty to adapt on unlabeled target data, extending pixel-level objectives with structural, distributional, or class-balance mechanisms. The survey highlights gradient imbalance and source-to-target class-statistics mismatch as key concerns.

  • Motivation: Entropy minimization is closely related to semi-supervised learning and can be applied to unlabeled target data once source and target distributions are matched.The survey presents UDA and semi-supervised learning as closely related after distribution alignment.
  • Pixel-Level Objectives: Pixel-level entropy minimization treats each spatial prediction unit as an independent contribution, limiting its access to semantic-map structure.Later methods add global adversarial alignment and class-wise priors to obtain more meaningful adaptation.
  • Structural and Distributional Extensions: Global adversarial optimization aligns source and target entropy-map distributions while source-derived class-wise priors reduce imbalance toward easy classes.The discriminator captures global patterns that distinguish entropy behavior across domains.
  • Limitations and Extensions: Entropy minimization can be dominated by high-confidence predictions because gradients increase rapidly toward low-uncertainty regions and may tend to infinity.This gradient predominance motivates additional weighting or balancing arrangements.
  • Limitations and Extensions: Category-wise weighting can use target prediction distributions instead of source annotations when source class statistics significantly differ from target statistics.This addresses the possibility that source-derived class priors are not representative of the target domain.

3.7. Curriculum Learning

Curriculum-learning methods solve easier target-domain tasks first, then use inferred properties to guide semantic segmentation. Unlike self-training, their pseudo-labels encode target-domain statistics rather than estimated task annotations.

  • Curriculum learning solves easier tasks first to infer useful target-domain properties for training a more challenging segmentation task.
  • Its pseudo-labels represent inferred target-domain statistics, whereas self-training uses estimated target annotations directly during training.
  • The first methods estimate global image and local landmark-superpixel label distributions, which are considered less sensitive to domain discrepancy.
  • A source-trained logistic regression model estimates the target global label distribution because target labels are unavailable in unsupervised adaptation.
  • Samples whose semantic maps do not follow the estimated target label distribution remain penalized, motivating a second local clue.

3.8. Multi-Tasking

Multi-tasking methods use additional source-domain information, such as depth maps, alongside segmentation to learn more invariant image representations. GIO-Ada extends this idea by using semantic and geometric information for both input- and output-level adaptation.

  • Multi-tasking jointly trains segmentation with auxiliary tasks such as depth regression to build an invariant and generic image embedding.
  • SPIGAN combines source-to-target image translation with prediction of privileged information, using source depth maps as regularization for domain adaptation.
  • GIO-Ada uses source semantic maps and depth maps to generate target-like images through an adversarially trained transform network.
  • At the output level, GIO-Ada feeds predicted semantic and depth maps to a discriminator distinguishing outputs from real versus adapted images.

3.9. New Research Directions

New research directions relax standard UDA assumptions by addressing unseen target classes and changing domain distributions. BUDA separates adaptation of shared classes from zero-shot transfer to unseen classes, while continual learning handles incremental domain changes.

  • Standard UDA assumes source and target domains share a predefined set of semantic categories, although novel-domain images may contain unseen classes.
  • BUDA explicitly handles unseen target classes using minimal class-name semantics, decoupling domain adaptation from zero-shot learning.
  • BUDA adapts categories shared across domains with entropy minimization designed to avoid incorrect alignment over unseen target classes.
  • A generative model then synthesizes visual features conditioned on class descriptors to transfer knowledge from seen to unseen classes.
  • Continual learning addresses settings where domain distributions change incrementally and models must perform well across all encountered distributions.

4. A Case Study: Synthetic to Real Adaptation for Semantic Understanding of Road Scenes

The case study examines synthetic-to-real adaptation for urban-scene semantic segmentation, using annotated synthetic sources and largely unlabeled real-world targets. It compares methods across common datasets and backbones, with performance varying by method, protocol, and architecture.

  • Scenario and datasets: UDA transfers knowledge from labeled synthetic domains to real-world target domains without requiring expensive target-domain training labels.A limited number of labeled target samples may still be used for testing or validation.
  • Source datasets: GTA5 provides 24,966 synthetic 1914 × 1052 px street images with pixel-level annotations, while SYNTHIA-RAND-CITYSCAPES provides 9,400 synthetic 1280 × 760 px annotated images.The datasets use different simulators and visual settings for synthetic urban scenes.
  • Scenario and datasets: The case study focuses on GTA5 and SYNTHIA as source datasets, with Cityscapes and Mapillary among the principal real-world targets.Other datasets are possible but less commonly used in the summarized comparisons.
  • Methods comparison: For GTA5-to-Cityscapes adaptation, results vary substantially with the method and evaluation protocol, while ResNet-based approaches generally outperform competitors.ResNet-101 and VGG-16 are the most widespread backbones, and ResNet-101 is presented as the best option for comparison with existing literature.
  • Methods comparison: For SYNTHIA-to-Cityscapes adaptation, results are reported over either 13 or 16 classes; VGG-16 is most popular, while ResNet-101 generally achieves higher results.The corresponding mIoU16 values are grouped by backbone to summarize commonly used techniques and outcomes.
  • Methods comparison: Mapillary is less extensively compared but is described as more variable than Cityscapes; the highest reported GTA5-to-Mapillary result reaches a mIoU of 41.9.The survey identifies this variability as a consideration for future work.

5. Conclusion and Future Directions

The paper surveys a broad range of UDA approaches for semantic segmentation and organizes them by adaptation level and methodological category. It concludes that the field remains open, with current methods still trailing supervised target-domain training.

  • The survey reviews UDA approaches for semantic segmentation, motivated by the limited availability of large labeled training datasets.
  • Existing methods are organized by adaptation location—input, feature, output, or ad-hoc network levels—and by seven non-mutually-exclusive methodological categories.
  • The survey evaluates synthetic-to-real adaptation for road-scene understanding and compares reported accuracies across works grouped by backbone architecture.
  • UDA for semantic segmentation remains an open research field with substantial room for improvement because leading approaches remain far below supervised training on the target dataset.
  • Future directions include open-set and boundless-set UDA, while UDA knowledge may also benefit continual learning under repeated distribution changes.
Loading 2005.10876v1…