Source-linked AI summary
All about Structure: Adapting Structural Information across Domains for Boosting Semantic Segmentation
Wei-Lun Chang, Hui-Po Wang, Wen-Hsiao Peng, Wei-Chen Chiu
TL;DR
The paper addresses unsupervised domain adaptation for semantic segmentation from labeled synthetic images to unlabeled real images. DISE disentangles domain-invariant structure from domain-specific texture, uses structure for adaptation, and supports image translation and label transfer; experiments report superiority over several state-of-the-art approaches.
Problem
Unsupervised adaptation must transfer segmentation knowledge from labeled synthetic data to unlabeled real images despite domain shift and the difficulty of aligning structured segmentation outputs.
Method
DISE disentangles images into domain-invariant structure and domain-specific texture representations, aligns structure across domains, and enables cross-domain image translation for label transfer.
Results
DISE achieves state-of-the-art performance of 45.4 mIoU on GTA5-to-Cityscapes and is reported superior to several state-of-the-art baselines.
Takeaways & Limitations
The experiments support using shared high-level structure while filtering domain-specific texture for semantic-segmentation adaptation.
Takeaways & Limitations
The approach is motivated by the assumption that the entire feature or output space need not be aligned and that high-level structure is the most decisive segmentation information.
Abstract
from arXiv · showhide
In this paper we tackle the problem of unsupervised domain adaptation for the task of semantic segmentation, where we attempt to transfer the knowledge learned upon synthetic datasets with ground-truth labels to real-world images without any annotation. With the hypothesis that the structural content of images is the most informative and decisive factor to semantic segmentation and can be readily shared across domains, we propose a Domain Invariant Structure Extraction (DISE) framework to disentangle images into domain-invariant structure and domain-specific texture representations, which can further realize image-translation across domains and enable label transfer to improve segmentation performance. Extensive experiments verify the effectiveness of our proposed DISE model and demonstrate its superiority over several state-of-the-art approaches.
1. Introduction
Unsupervised adaptation transfers segmentation knowledge from labeled synthetic images to unlabeled real images despite domain shift. DISE separates domain-invariant structure from domain-specific texture and aligns only structure to support adaptation and label transfer.
- Motivation: Synthetic datasets provide pixel-level labels, but models trained on them often perform poorly on real-world scenes because low-level textures differ across domains.This difference is the domain-shift problem motivating adaptation.
- Motivation: Domain adaptation commonly matches feature distributions across synthetic and real domains to learn domain-invariant representations.Prior criteria include second-order statistics and domain-adversarial training.
- Limitation: Existing approaches assume the entire feature or output spaces can be aligned while remaining discriminative for semantic segmentation.The paper identifies this as a strong assumption underlying conventional adaptation.
- Proposed approach: DISE disentangles images into a domain-invariant structure component and a domain-specific texture component, then aligns the structure distributions across domains.The framework is based on the hypothesis that high-level structure is most effective for segmentation prediction.
- Contributions: Within one framework, DISE combines explicit structure-texture representation, structure-only invariance, cross-domain image translation, and label transfer.Experiments on standardized datasets are reported to confirm superiority over several state-of-the-art baselines.
2. Related Work
Prior semantic-segmentation adaptation methods use distribution alignment, image translation, and label transfer in different ways. DISE combines these strategies while explicitly separating domain-invariant structure from domain-specific texture.
- Overview: Semantic segmentation is harder to adapt than image classification because its output is a highly structured and contextual segmentation map.The related-work review organizes methods around distribution alignment, image translation, and label transfer.
- Distribution alignment: Prior methods apply distribution matching in feature or output space, but whole-space alignment assumes both domains can be aligned effectively.Output-space approaches use spatial contextual similarities between source and target segmentation maps.
- Strategy taxonomy: Table 1 categorizes prior methods by image translation, distribution alignment, label transfer, and the order in which these strategies are applied.IT, DA, and LT denote Image Translation, Distribution Alignment, and Label Transfer.
- Image translation and label transfer: Image translation can augment training data, facilitate a common feature space, and transfer source labels to target-domain images.Direct translation may also carry source-specific information into the target domain, potentially harming learning.
- DISE: DISE uses common and private encoders to disentangle high-level domain-invariant structure from low-level domain-specific texture while employing all three strategies.Its design differs from prior work through explicit structure-texture separation and integrated translation for label transfer.
3. Method
DISE disentangles each source- or target-domain image into shared structure and domain-specific texture, then uses these components for reconstruction, translation, and segmentation. Its losses preserve structure and texture properties while aligning structure-based predictions across domains and transferring source labels to translated target-like images.
- Framework overview: DISE uses a shared encoder, domain-specific private encoders, a shared decoder, and a pixel-wise classifier to represent images with structure and texture components.The common encoder captures domain-invariant high-level structure, while private encoders capture domain-specific low-level texture.
- Framework overview: Source images use supervised cross-entropy on ground-truth labels, while target images are unannotated and receive predictions from the shared structure representation.The classifier operates on structure components in both domains; source labels train the common encoder and classifier.
- Image translation: Swapping private texture components between source and target representations lets the decoder produce cross-domain translated images while retaining the originating image’s structure.Translated images are trained with domain-adversarial and perceptual losses to match the counterpart domain and preserve perceptual content.
- Image translation: Source ground-truth labels become pseudo-labels for translated source images because translation preserves their structure while changing texture appearance.This transfers annotated source information to target-domain-like images within the same framework.
- Learning objectives: Output-space adversarial training aligns source and target segmentation predictions by training a patch-level discriminator and updating the encoder and classifier to fool it.The discriminator distinguishes source and target predictions, while the segmentation model reverses the target discrimination signal.
- Learning objectives: DISE regularizes structure and texture separation through reconstruction, translation structure and texture losses, perceptual losses, and output-space adversarial training.Higher VGG layers receive greater perceptual-loss weight because they represent high-level structure more strongly than lower layers.
4. Experimental Results
Experiments evaluate DISE for unsupervised adaptation from synthetic GTA5 or SYNTHIA to unlabeled Cityscapes. Ablations, quantitative comparisons, and qualitative translations support gains from structure–texture disentanglement and label transfer.
- Datasets: Experiments use GTA5 or SYNTHIA as annotated synthetic source domains and unlabeled Cityscapes as the real-world target domain.Evaluation uses Cityscapes validation data under the common adaptation protocol.
- Performance Comparison: 45.4 mIoU gives DISE state-of-the-art performance for GTA5-to-Cityscapes adaptation.Class-level gains are especially reported for Road, Sidewalk, Wall, Fence, Building, and Sky.
- Performance Comparison: On SYNTHIA-to-Cityscapes adaptation, DISE remains superior on Road, Sidewalk, Building, and Sky, although a prior method performs closely in mIoU.Results are evaluated over 16 semantic classes.
- Ablation Study: The ablation compares Source Only, Seg-map Adaptation, DISE without label transfer, and full DISE under distinct training objectives.The settings progressively introduce output-space adaptation, structure–texture disentanglement, and label transfer.
- Ablation Study: 39.8 mIoU is achieved by Source Only, 42.6 by Seg-map Adaptation, and DISE without label transfer gains 4.3 over Source Only.Full DISE achieves the best performance after adding augmented data through label transfer.
- Image-to-Image Translation: DISE translates between domains while preserving structure and producing the desired texture appearance, supporting source-label transfer to translated images.The qualitative S2T and T2S results validate using source ground-truth labels as pseudo labels for target-like translated images.
5. Conclusion
The paper proposes DISE for unsupervised semantic-segmentation domain adaptation by separating domain-invariant structure from domain-specific texture. Experiments report superiority over several state-of-the-art methods and support the hypothesis that high-level structure is decisive for segmentation.
- Conclusion: DISE disentangles images into domain-invariant structure and domain-specific texture components for semantic-segmentation adaptation.The structure component advances adaptation, while translated images enable source-label transfer for additional target-domain supervision.
- Conclusion: Extensive simulation results on typical datasets confirm DISE’s superiority over several state-of-the-art methods.The conclusion presents this result as support for the paper’s initial structure-invariance hypothesis.