Source-linked AI summary
DACS: Domain Adaptation via Cross-domain Mixed Sampling
Wilhelm Tranheden, Viktor Olsson, Juliano Pinto, Lennart Svensson
TL;DR
Semantic segmentation UDA must adapt models from labelled source domains to unlabelled target domains despite domain shifts that create faulty pseudo-labels. DACS mixes source images and labels with target images and pseudo-labels, improving results on synthetic-to-real benchmarks, including state-of-the-art GTA5 →Cityscapes performance.
Problem
Semantic segmentation models often generalize poorly across domains, while target annotations may be unavailable and domain shifts can produce low-quality pseudo-labels.
Method
DACS creates training samples by mixing source images and ground-truth labels with target images and pseudo-labels before training.
Results
DACS outperforms existing methods on GTA5 →Cityscapes with 52.14% mIoU and achieves 48.34% mIoU on the 16-class SYNTHIA →Cityscapes formulation.
Takeaways & Limitations
Cross-domain mixing solves the class-conflation problem and improves the state of the art for GTA5 →Cityscapes UDA.
Takeaways & Limitations
Distribution Alignment is not legitimate in realistic unlabelled settings because the ground-truth target class distribution is unknown.
Abstract
from arXiv · showhide
Semantic segmentation models based on convolutional neural networks have recently displayed remarkable performance for a multitude of applications. However, these models typically do not generalize well when applied on new domains, especially when going from synthetic to real data. In this paper we address the problem of unsupervised domain adaptation (UDA), which attempts to train on labelled data from one domain (source domain), and simultaneously learn from unlabelled data in the domain of interest (target domain). Existing methods have seen success by training on pseudo-labels for these unlabelled images. Multiple techniques have been proposed to mitigate low-quality pseudo-labels arising from the domain shift, with varying degrees of success. We propose DACS: Domain Adaptation via Cross-domain mixed Sampling, which mixes images from the two domains along with the corresponding labels and pseudo-labels. These mixed samples are then trained on, in addition to the labelled data itself. We demonstrate the effectiveness of our solution by achieving state-of-the-art results for GTA5 to Cityscapes, a common synthetic-to-real semantic segmentation benchmark for UDA.
1. Introduction
Unsupervised domain adaptation addresses semantic segmentation when labelled source data and unlabelled target data differ substantially. DACS mixes cross-domain images with ground-truth labels and pseudo-labels to reduce class conflation and improve adaptation.
- Motivation: Domain shift occurs when training data differ significantly from the data encountered during inference, often requiring adaptation to the target domain.Target annotation and fine-tuning may be infeasible when no target annotations are accessible.
- Problem setting: UDA trains with labelled source-domain data while learning from unlabelled target-domain data, making synthetic data useful for applications with costly annotations.Semantic segmentation UDA is especially relevant to applications such as autonomous driving.
- Challenge: Naive pseudo-labelling can bias predictions toward easy-to-predict classes, causing difficult classes to stop being predicted during training.The problem arises because artificial training targets are created from the network’s class predictions.
- DACS: DACS forms augmented samples by extracting source classes and pasting them onto target images, then combining source labels with target pseudo-labels.The resulting mixed samples are trained alongside labelled data, and source ground-truth regions replace parts of target pseudo-labels.
- Contribution: DACS addresses class-conflation problems while retaining training data and learning from the entire unlabelled target dataset during training.The method is evaluated on the GTA5 →Cityscapes and SYNTHIA →Cityscapes synthetic-to-real benchmarks.
2. Related Work
Related UDA methods use adversarial alignment, pseudo-labelling, or consistency regularization. DACS instead adapts mixed-sample augmentation across domains and enforces consistency on cross-domain mixed images.
- Domain alignment: UDA methods align domains adversarially at pixel, feature-map, or semantic levels, often exploiting similarities in semantic-map structure.Semantic-level alignment treats the segmentation network as a generator encouraged to predict realistic semantic maps.
- Domain alignment: Other alignment strategies transfer target texture or low-frequency spectral information to source images, or align feature maps channel-wise.
- Pseudo-labelling: Pseudo-labelling trains directly on target images using artificial targets from predictions, but domain gaps can produce faulty labels and merge classes.Existing corrections include specialized sampling and uncertainty handling.
- DACS: DACS injects source-label information into target pseudo-labels through cross-domain mixing, while using mixed images for consistency training.Unmixed target pseudo-labels are not used for training until images and labels have been mixed across domains.
- DACS: Unlike consistency-based UDA methods that combine consistency regularization with image-distribution alignment, DACS enforces consistency between target and cross-domain mixed predictions.The approach adapts mixed-sample augmentation from semantic-segmentation SSL to cross-domain UDA.
3. Method
DACS addresses weaknesses of naive target-only mixing by combining source images with ground-truth labels and target images with pseudo-labels. These cross-domain augmentations are trained alongside source data to reduce class conflation and improve UDA performance.
- 3.1. Naive Mixing to UDA: Naive Mixing combines target images and pseudo-labels, then trains on the augmented samples together with source-domain images.This adaptation performs poorly because target predictions can conflate semantically similar classes.
- 3.1. Naive Mixing to UDA: Class conflation causes rare or difficult target classes such as sidewalk and rider to be confused with frequent or similar classes such as road and person.The reported pattern affects target-domain images but not source-domain images.
- 3.2. DACS: DACS creates augmented samples by mixing pixels from source and target images, using source ground-truth labels and target predictions to form mixed pseudo-labels.The default strategy selects half of the source classes, cuts out their corresponding pixels, and pastes them onto a target image.
- 3.2. DACS: Cross-domain mixing helps the network handle the domain gap and effectively solves the class conflation problem, yielding considerably better performance than Naive Mixing.The overall procedure trains on source-domain images and cross-domain augmentations and is called DACS.
- 3.3. Loss Function: DACS minimizes a loss combining cross-entropy on source images with cross-entropy on mixed images, weighted by λ.The algorithm samples source and target batches, generates mixed images and pseudo-labels, computes predictions and loss, and updates the network by stochastic gradient descent.
4. Experiments
DACS is evaluated on two synthetic-to-real UDA benchmarks and compared with prior methods, source-only baselines, and alternative mixing or pseudo-labelling strategies. It achieves strong benchmark performance while reducing class conflation, although evaluation and domain-similarity conditions constrain interpretation.
- Experimental setup: Experiments evaluate DACS on GTA5 →Cityscapes and SYNTHIA →Cityscapes using semantic-segmentation benchmarks and comparisons with existing methods.The target Cityscapes dataset contains 2,975 labelled training images and 19 classes; GTA5 and SYNTHIA provide synthetic source images.
- GTA5 →Cityscapes results: 52.14% overall mIoU makes DACS the strongest reported method for GTA5 →Cityscapes, with the best results for seven individual classes.Comparisons include source-only training and previous methods, with results averaged over three runs.
- SYNTHIA →Cityscapes results: 48.34% mIoU on 16 classes and 54.81% on 13 classes are reported for DACS on SYNTHIA →Cityscapes.DACS also achieves the strongest per-class results for 4 of the 16 classes.
- Evaluation conditions: Using the evaluation validation set for early stopping or hyper-parameter selection can substantially increase reported results and exploit validation-set variance.The authors report that early stopping would raise DACS to 53.84% for GTA5 and to 55.98% for SYNTHIA’s 13-class setting.
- Additional experiments: Naive Mixing conflates classes, including classifying sidewalks as road, whereas DACS distinguishes these classes in the qualitative examples.Seven classes have per-class IoUs below 1% for Naive Mixing, and pseudo-labelling alone performs worse than the source baseline.
- Additional experiments: Distribution Alignment represents all classes and supports the hypothesis that artificial entropy injection can help avoid class conflation, but it is not legitimate without target class-distribution information.Replacing ClassMix with CutMix or CowMix slightly lowers performance while preserving the solution to class conflation.
5. Conclusion
The paper proposes DACS for unsupervised domain adaptation in semantic segmentation and evaluates it on two domain-adaptation benchmarks. DACS outperforms existing methods on GTA5 →Cityscapes and SYNTHIA →Cityscapes.
- DACS is proposed as a novel algorithm for unsupervised domain adaptation in semantic segmentation.
- Naive adaptation of mixing-based consistency regularization from SSL to UDA produces systematic prediction problems, which DACS modifies to correct.
- DACS is evaluated on the GTA5 →Cityscapes and SYNTHIA →Cityscapes domain-adaptation benchmarks.
- DACS outperforms existing methods and pushes the state of the art on GTA5 →Cityscapes and SYNTHIA →Cityscapes.