Source-linked AI summary
Multi-source Domain Adaptation for Semantic Segmentation
Sicheng Zhao, Bo Li, Xiangyu Yue, Yang Gu, Pengfei Xu, Runbo Hu, Hua Chai, Kurt Keutzer
TL;DR
The paper addresses semantic segmentation adaptation from multiple labeled source domains to an unlabeled target domain, extending beyond single-source settings. MADAN combines source-specific pixel-level adaptation, domain aggregation, and feature-level alignment, and experiments report consistent improvements over several baselines. Its scope remains limited by a large gap to the target-supervised oracle and by potential gains from better discriminator-loss weighting.
Problem
Existing simulation-to-real semantic segmentation adaptation methods mainly address a single source, leaving multiple sources with different distributions insufficiently handled.
Method
MADAN generates a semantically consistent adapted domain for each source, aggregates adapted domains with two discriminators, and aligns aggregated and target domains at the feature level.
Results
MADAN improves mIoU by 15.6%, 1.6%, 4.1%, and 12.0% over best source-only, best single-source DA, source-combined DA, and other multi-source DA, respectively, on Cityscapes from GTA and SYNTHIA.
Takeaways & Limitations
Experiments across synthetic GTA and SYNTHIA and real Cityscapes and BDDS demonstrate the effectiveness of MADAN for multi-source semantic segmentation adaptation.
Takeaways & Limitations
A large performance gap remains between adaptation algorithms and the target-supervised oracle, and better aggregation may require dynamic discriminator-loss weighting and source-domain knowledge.
Abstract
from arXiv · showhide
Simulation-to-real domain adaptation for semantic segmentation has been actively studied for various applications such as autonomous driving. Existing methods mainly focus on a single-source setting, which cannot easily handle a more practical scenario of multiple sources with different distributions. In this paper, we propose to investigate multi-source domain adaptation for semantic segmentation. Specifically, we design a novel framework, termed Multi-source Adversarial Domain Aggregation Network (MADAN), which can be trained in an end-to-end manner. First, we generate an adapted domain for each source with dynamic semantic consistency while aligning at the pixel-level cycle-consistently towards the target. Second, we propose sub-domain aggregation discriminator and cross-domain cycle discriminator to make different adapted domains more closely aggregated. Finally, feature-level alignment is performed between the aggregated domain and target domain while training the segmentation network. Extensive experiments from synthetic GTA and SYNTHIA to real Cityscapes and BDDS datasets demonstrate that the proposed MADAN model outperforms state-of-the-art approaches. Our source code is released at: https://github.com/Luodian/MADAN.
1 Introduction
Simulation-to-real semantic segmentation usually adapts from one source, whereas multiple sources with different distributions create additional alignment challenges. MADAN addresses this setting with pixel-level adaptation, domain aggregation, and feature-level alignment.
- Single-source simulation-to-real adaptation methods do not address the practical setting of multiple labeled sources with different distributions.
- Directly combining sources and applying single-source adaptation may perform poorly because source images can have different distributions.
- Classification-oriented multi-source methods are insufficient for segmentation because segmentation requires structured pixel-wise prediction.
- MADAN generates an adapted domain for each source using cycle-consistent pixel-level alignment and dynamic semantic consistency.
- Sub-domain aggregation and cross-domain cycle discriminators make adapted domains more closely aggregated, followed by feature-level alignment for semantic segmentation.
- Experiments adapt synthetic GTA and SYNTHIA data to real Cityscapes and BDDS datasets to evaluate MADAN.
2 Problem Setup
The problem setup contains multiple labeled source domains and one unlabeled target domain. The domains share an image space and label set while differing in distributions, and the goal is target-label prediction from source supervision.
- The unsupervised multi-source setting contains multiple labeled source domains and one unlabeled target domain.
- Each source domain provides observed data and corresponding labels drawn from its source distribution.
- The homogeneity assumption places data from different domains in the same image space despite different distributions.
- The closed-set assumption requires all domains to share the same label space.
- The adaptation model is learned from labeled source datasets and unlabeled target data to predict target-domain labels.
3 Multi-source Adversarial Domain Aggregation Network
MADAN adapts multiple source domains toward a target while preserving semantic information and aggregating the adapted domains. It then trains one segmentation model with task supervision and feature-level alignment to the target.
- Framework overview: MADAN comprises Dynamic Adversarial Image Generation, Adversarial Domain Aggregation, and Feature-aligned Semantic Segmentation.The framework is trained through generators, discriminators, and a target segmentation model.
- Dynamic Adversarial Image Generation: For each source, a generator maps source images toward the target while a pixel-level discriminator distinguishes adapted images from real target images.The reverse mapping and cycle-consistency loss constrain the forward translation.
- Dynamic Adversarial Image Generation: Dynamic semantic consistency uses the task segmentation model on adapted images and KL divergence to preserve source semantic information during translation.The adapted-domain model dynamically replaces direct use of the source-domain model for translated images.
- Adversarial Domain Aggregation: Adversarial domain aggregation uses a sub-domain aggregation discriminator and a cross-domain cycle discriminator to reduce discrepancies among adapted source domains.The sub-domain discriminator makes adapted domains indistinguishable, while the cross-domain cycle discriminator transfers adapted images back toward individual source domains.
- Feature-aligned Semantic Segmentation: The unified adapted domain trains one segmentation model with task loss, while feature-level alignment matches its encoder features for adapted and target images.The feature map is taken from the encoder’s last convolutional layer, and the aligned features are intended to improve target inference.
- Overall optimization: The overall MADAN objective combines adversarial image generation, cycle and semantic consistency, domain aggregation, task supervision, and feature-level alignment.The training process optimizes the generators and discriminators jointly with the segmentation model.
4 Experiments
Experiments evaluate MADAN for simulation-to-real semantic segmentation using GTA and SYNTHIA as sources and Cityscapes and BDDS as targets. Results compare source-only, single-source, multi-source, and oracle settings, with qualitative analyses and ablations of MADAN’s components.
- Experimental settings: Experiments use synthetic GTA and SYNTHIA as source domains and real Cityscapes and BDDS as target domains, evaluating segmentation with mIoU.The comparisons include source-only, single-source domain adaptation, multi-source domain adaptation, and an oracle trained on target labels.
- Quantitative comparison: Source-only transfer performs worst in most adaptation settings, while simply combining source domains performs better than using each single source.The result reflects domain shift between source and target distributions and differences among source domains.
- Quantitative comparison: Single-source domain adaptation improves over source-only transfer, but combining sources and applying single-source adaptation can degrade performance.This pattern is reported for comparisons involving CyCADA on GTA and SYNTHIA.
- Quantitative comparison: MADAN achieves the highest mIoU among adaptation methods by jointly using multiple sources with pixel-level and feature-level alignment, consistency losses, and domain aggregation.The method also significantly outperforms source-combined domain adaptation.
- Qualitative analysis: Qualitative results show improved segmentation after adaptation, while the final pixel-level alignment makes translated images resemble Cityscapes and preserves semantic information.The visualizations compare source-only, CycleGAN-based, component combinations, and the full MADAN model.
- Ablation study: Ablations show that SAD and CCD improve performance, SAD outperforms CCD, DSC further improves mIoU, and feature-level alignment contributes without introducing performance degradation.The ablation findings indicate that the modules are orthogonal to some extent.
5 Conclusion
MADAN addresses multi-source domain adaptation for synthetic-to-real semantic segmentation by adapting and aggregating source domains before segmentation. It improves mIoU over source-only, single-source, source-combined, and other multi-source approaches on both Cityscapes and BDDS.
- MADAN generates adapted images for each source using dynamic semantic consistency, then aggregates the adapted domains with dedicated discriminators.The framework combines pixel- and feature-level alignment with cycle-consistency.
- 15.6%, 1.6%, 4.1%, and 12.0% mIoU improvements are reported on Cityscapes over best source-only, best single-source DA, source-combined DA, and other multi-source DA, respectively.
- 11.7%, 0.6%, 2.6%, and 11.3% mIoU improvements are reported on BDDS against the same four comparison categories, respectively.
- The authors identify multi-modal domain adaptation and improved computational efficiency as directions for further study.Examples include combining image and LiDAR data and using neural architecture search.