Source-linked AI summary
Learning Texture Invariant Representation for Domain Adaptation of Semantic Segmentation
Myeongjin Kim, Hyeran Byun
TL;DR
Synthetic-data training reduces the burden of pixel-level annotation but generalizes poorly across the synthetic–real domain gap, particularly in texture. The paper diversifies synthetic textures with style transfer, then uses self-training for direct target-texture supervision, achieving state-of-the-art GTA5-to-Cityscapes performance. It also analyzes stylized-model properties and compares the approach with CycleGAN-based methods.
Problem
Pixel-level annotation is laborious, while synthetic-data-trained segmentation models struggle to generalize to real images because of the synthetic–real domain gap.
Method
The method diversifies synthetic textures with style transfer to learn texture-invariant representations, then fine-tunes with self-training for direct target-texture supervision.
Results
The method achieves state-of-the-art performance on the GTA5-to-Cityscapes benchmark.
Takeaways & Limitations
Texture-diversified stylized datasets regularize segmentation models toward texture invariance, while self-training adapts the learned representation to the target texture.
Takeaways & Limitations
The method does not handle the layout gap, and performance is lower for small classes such as t-light, t-sign, and person when shape differences are more decisive.
Abstract
from arXiv · showhide
Since annotating pixel-level labels for semantic segmentation is laborious, leveraging synthetic data is an attractive solution. However, due to the domain gap between synthetic domain and real domain, it is challenging for a model trained with synthetic data to generalize to real data. In this paper, considering the fundamental difference between the two domains as the texture, we propose a method to adapt to the texture of the target domain. First, we diversity the texture of synthetic images using a style transfer algorithm. The various textures of generated images prevent a segmentation model from overfitting to one specific (synthetic) texture. Then, we fine-tune the model with self-training to get direct supervision of the target texture. Our results achieve state-of-the-art performance and we analyze the properties of the model trained on the stylized dataset with extensive experiments.
1. Introduction
Synthetic data reduces annotation costs but leaves a synthetic–real domain gap, especially in texture. The paper diversifies source textures and uses self-training to adapt segmentation models, achieving state-of-the-art GTA5-to-Cityscapes performance.
- Pixel-level annotation is laborious, taking 90 min per Cityscapes image in one example.
- Synthetic datasets provide automatic labels, but models trained on them struggle to generalize to real images because of the domain gap.
- CycleGAN reduces visual differences but does not fully translate synthetic texture, leaving models vulnerable to synthetic-texture overfitting.
- Style transfer generates texture-diversified source images, increasing texture variation and guiding the model toward texture-invariant representations.
- Self-training then provides direct supervision from the target texture during fine-tuning.
- The method achieves state-of-the-art performance on the GTA5-to-Cityscapes benchmark and analyzes stylized-data properties and prior CycleGAN-based methods.
2. Related Work
Prior domain-adaptation work transfers knowledge from labeled source data to unlabeled target data through pixel-level adaptation or self-training. Motivated by texture dependence, this paper uses style transfer to promote texture-invariant segmentation representations for target-domain adaptation.
- Unsupervised domain adaptation uses labeled source inputs and labels together with unlabeled target inputs to improve target-domain performance.
- Pixel-level adaptation translates synthetic images toward the target style to reduce visual differences such as texture and lighting.
- Self-training generates pseudo-labels from confident target predictions and fine-tunes the model with them.
- Prior comparisons suggest combining pixel-level adaptation with self-training can outperform pixel-level adaptation alone and directly supervise target texture.
- 2.3. Texture and shape: The paper adopts style-transfer-based texture diversification as a regularizer against overfitting to one texture, aiming for texture-invariant representations.
3. Method
The method diversifies synthetic textures to learn texture-invariant representations, then adapts those representations to target-domain texture through self-training and auxiliary alignment.
- Method overview: The pipeline uses Style-swap to diversify source textures, CycleGAN to translate source images toward target style, and two training stages.The stylized and translated datasets retain a one-to-one mapping with the original source datasets.
- Stage 1: Texture-invariant representation: Stage 1 trains on stylized images to learn texture-invariant representations and on translated images to guide the model toward target style.Stylized and translated inputs are alternated because of memory limitations.
- Texture diversification: Style-swap is selected to remove synthetic texture while preserving structure, remaining time-efficient, and transferring diverse styles.AdaIN was rejected because it can distort content structure with wave patterns, which is problematic for pixel-level segmentation.
- Texture diversification: Stylized images lose synthetic texture and acquire varied random textures, encouraging shape-dependent, texture-invariant representations.Each texture comes from a different style image, increasing texture variation during training.
- Stage 2: Target-texture adaptation: Stage 2 fine-tunes the model with high-confidence target pseudo-labels and translated source images, iteratively providing direct supervision from target texture.Predictions above the 0.9 confidence threshold are used as pseudo-labels.
- Training objective: The training objective combines source segmentation loss with adversarial alignment, while the discriminator classifies source versus target features.Source images provide ground-truth labels, and target images are used for adversarial training and pseudo-label supervision.
4. Experiments
Experiments evaluate the method on GTA5-to-Cityscapes and SYNTHIA-to-Cityscapes using standard datasets and segmentation architectures. Results show strong performance overall, with advantages concentrated in large-area classes while layout differences constrain small-class transfer.
- Datasets: GTA5 contains 24,966 synthetic images with 19-class labels compatible with Cityscapes, while SYNTHIA uses 9,400 images and evaluates 13 common classes.
- Datasets: Cityscapes provides 5,000 densely annotated images, with experiments using 2,975 training and 500 validation images.
- Benchmark results: Our method outperforms previous state-of-the-art models on GTA5-to-Cityscapes, surpassing BDL with two segmentation-training iterations versus BDL’s six.
- Benchmark results: On SYNTHIA-to-Cityscapes, the method performs especially well on large-area classes such as road and sidewalk, which are more affected by texture.
- Class-wise analysis: Performance is lower than other methods on small classes including t-light, t-sign, and person because the unhandled layout gap changes shape distributions across domains.
- Class-wise analysis: Stylized training outperforms Translated and Original training for large-area classes such as road and sidewalk, where layout distributions are similar and texture is especially important.
5. Discussion
The discussion shows that Style-swap better diversifies synthetic texture than CycleGAN-based alternatives, while ablations and robustness tests support the model’s texture-invariant behavior.
- 5.1. Comparison with CycleGAN-based methods: CycleGAN can generate inappropriate artifacts, whereas Style-swap avoids these patterns through local-patch style transfer.CycleGAN produces vegetation-like sky artifacts and blurred people; Style-swap does not show such patterns.
- 5.1. Comparison with CycleGAN-based methods: Style-swap diversifies texture while CycleGAN mainly matches auxiliary-domain color distributions and preserves synthetic texture.The paper identifies texture diversification as the key difference from DRPC.
- 5.1. Comparison with CycleGAN-based methods: 11,026 styles are used because Style-swap requires no additional training for each style, unlike costly CycleGAN-based randomization.DRPC uses 15 auxiliary domains, while the proposed approach handles many styles with fixed computation.
- 5.2. Ablation study: The stylized-source model outperforms other pixel-level methods, and Stage 2 self-training converges after three iterations.The study selects three self-training iterations for the reported results.
- 5.2. Ablation study: Using stylized and translated source images together with output-level adversarial loss outperforms using stylized images alone.The authors attribute this to combining texture-invariant learning with guidance toward the target style.
- 5.3. Texture-invariance analysis: The model is more robust to Gaussian, Impulse, Shot, and Speckle noise than AdaptSegNet and remains robust across varied textures.The noise tests preserve object shape while perturbing image texture.
6. Conclusion
The paper adapts semantic segmentation models to target-domain texture by generating stylized synthetic datasets and applying self-training. Experiments report texture-invariant behavior and new state-of-the-art performance on GTA5-to-Cityscapes.
- 6. Conclusion: Style transfer generates Stylized GTA5 and Stylized SYNTHIA, whose varied textures regularize segmentation models toward texture-invariant representations.The paper evaluates this behavior qualitatively on varied textures and quantitatively on noisy validation sets.
- 6. Conclusion: Self-training provides direct supervision of the target texture after texture diversification of the source dataset.The method combines style transfer, texture-invariant representation learning, and target-texture adaptation.
- 6. Conclusion: The approach achieves new state-of-the-art performance on the GTA5-to-Cityscapes benchmark.The paper also analyzes class-wise texture influence and compares Style-swap with CycleGAN-based methods.