PaperlayerView exact revision on arXiv

Source-linked AI summary

Intra-Source Style Augmentation for Improved Domain Generalization

Yumeng Li, Dan Zhang, Margret Keuper, Anna Khoreva

arXiv:2210.10175v2cs.CVcs.AIcs.LG

TL;DR

Domain shifts challenge deep-learning models, and segmentation training data from multiple source domains can be costly to collect. ISSA uses StyleGAN2 inversion with a masked noise encoder to mix styles while preserving scene content; it reports up to 12.4% mIoU improvement on driving-scene domain generalization. The approach increases style diversity within the source domain, while its global mixing does not selectively change local-object styles.

  • Problem

    Domain shifts challenge deep-learning models, while diverse semantic-segmentation data are costly to collect and target domains may be inaccessible during training.

    Source: §1 · p. 1

  • Method

    ISSA uses a masked noise encoder for StyleGAN2 inversion to mix styles from a single source domain while preserving semantic content.

    Source: §1 · p. 2

  • Results

    Up to 12.4% mIoU improvement is reported for driving-scene domain generalization across network architectures, even with limited source diversity and no target-domain access.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ISSA increases training-data diversity by mixing style and content combinations within the source domain, aiming to alleviate spurious correlation.

    Source: §1 · p. 2

    ISSA performs global style transformations and cannot selectively alter the style of local objects such as a vehicle’s color.

    Source: §S.4 · p. 15

Abstract

from arXiv · show

The generalization with respect to domain shifts, as they frequently appear in applications such as autonomous driving, is one of the remaining big challenges for deep learning models. Therefore, we propose an intra-source style augmentation (ISSA) method to improve domain generalization in semantic segmentation. Our method is based on a novel masked noise encoder for StyleGAN2 inversion. The model learns to faithfully reconstruct the image preserving its semantic layout through noise prediction. Random masking of the estimated noise enables the style mixing capability of our model, i.e. it allows to alter the global appearance without affecting the semantic layout of an image. Using the proposed masked noise encoder to randomize style and content combinations in the training set, ISSA effectively increases the diversity of training data and reduces spurious correlation. As a result, we achieve up to $12.4\%$ mIoU improvements on driving-scene semantic segmentation under different types of data shifts, i.e., changing geographic locations, adverse weather conditions, and day to night. ISSA is model-agnostic and straightforwardly applicable with CNNs and Transformers. It is also complementary to other domain generalization techniques, e.g., it improves the recent state-of-the-art solution RobustNet by $3\%$ mIoU in Cityscapes to Dark Z\"urich.

1. Introduction

ISSA uses StyleGAN2 inversion and a masked noise encoder to diversify style within a single source domain while preserving scene content. The introduction reports improved driving-scene segmentation under domain shifts, including a gain of up to 12.4% mIoU.

1. Introduction

  • Open-world variation in illumination and adverse weather complicates deployment, while target domains may be inaccessible or unpredictable during training.The paper frames generalization as important for safety-critical applications such as autonomous driving.Source: §1 · p. 1
  • In a snow-domain example, Cityscapes-trained HRNet segments a truck with ISSA but fails to do so with the baseline.The example evaluates on ACDC.Source: Figure 1 · p. 1
  • Improving generalization from one source domain is especially compelling because diverse semantic-segmentation data are costly to annotate.Many existing approaches assume access to multiple source domains.Source: §1 · p. 1
  • ISSA mixes styles extracted within the source domain to diversify style-content combinations without changing semantic content.The approach aims to reduce spurious correlations and requires neither extra labeling effort nor additional data sources.Source: §1 · p. 2
  • Randomly masking predicted StyleGAN2 noise encourages latent prediction, helping separate content from style for realistic mixing in complex driving scenes.The encoder is proposed to address the difficulty of faithfully reconstructing complex scenes while enabling style mixing.Source: §1 · p. 2
  • Up to 12.4% mIoU improvement is reported for driving-scene domain generalization across network architectures, even with limited source diversity and no target-domain access.The snow-domain example shows ISSA-trained HRNet segmenting a truck that the baseline fails to segment.Source: Figure 1 · p. 1, §1 · p. 2

2. Related Work

Related work covers domain-generalization strategies, data augmentation, and GAN inversion, noting limits for dense prediction and complex scenes. ISSA performs image-level style mixing and is presented as a model-agnostic alternative that preserves content for segmentation.

2. Related Work

  • Domain-generalization approaches include augmentation, alignment, meta-learning, and ensembles, while comparatively few address pixel-level tasks such as semantic segmentation.Some segmentation methods use extra data in the wild, while RobustNet uses instance-selective whitening.Source: §2 · p. 2
  • Feature-level methods MixStyle and DSU perturb normalization statistics, can distort semantic content, and require adaptation to network architecture; ISSA mixes styles at image level and can complement them.The paper notes a slight source-domain performance drop with DSU and MixStyle, while combining them with ISSA boosts their performance.Source: §2 · p. 2
  • Prior augmentation changes style, content, or both, but mixup is poorly suited to dense prediction and synthetic corruptions can look artificial.CutMix and geometric transformations increase content diversity, whereas Hendrycks corruptions alter appearance without changing content.Source: §2 · p. 2
  • On Cityscapes, the masked noise encoder preserves finer details than pSp† and the Feature-Style encoder in the qualitative comparison.The caption says pSp† struggles with details and Feature-Style fails to faithfully reconstruct small objects while blurring some objects.Source: Figure 2 · p. 3
  • StyleMix also combines content and style, but its AdaIN-based mixing does not meet pixel-wise label-preservation requirements; ISSA preserves content so ground-truth maps can be reused.The cited comparison describes ISSA samples as natural-looking and style-altered.Source: §2 · pp. 2–3
  • Earlier GAN-inversion work includes optimization, encoder, and hybrid approaches, but most prior applications focus on object-centric datasets rather than complex scenes.The paper positions its masked noise encoder as suitable for segmentation augmentation on complex scenes.Source: §2 · p. 3

3. Method

ISSA uses GAN inversion to recombine styles and semantic content within the source domain, while the masked noise encoder supports faithful reconstruction and style mixing. Its training objective combines reconstruction and regularization losses.

3.1. Intra-Source Style Augmentation (ISSA)

  • ISSA recombines styles and content from source-domain images while preserving content, allowing the original ground-truth label maps to be reused.This diversifies the training set and reduces spurious style-content correlations without additional annotation.Source: §3.1 · pp. 3–4

3.2. Masked Noise Encoder

  • Randomly masking noise-map patches limits their encoding capacity, encouraging latent codes to control style while the noise map captures scene semantics.The encoder combines style codes from one image with the noise map from another to synthesize a style-mixed image.Source: §3.2 · p. 5

3.3. Encoder Training Loss

  • Encoder training combines pixel MSE, LPIPS, adversarial, and regularization losses; regularization sparsifies noise prediction and aligns predicted latent codes with generator codes.Reconstruction losses are relaxed at positions masked during training.Source: §3.3 · pp. 5–6

4. Experiments

The experiments evaluate reconstruction and single-source generalization under unseen geographic, weather, and day-to-night shifts. ISSA improves domain-generalization results across tested models and scenarios, while its masked noise encoder preserves scene details better than the compared inversion baselines.

4. Experiments

  • Experiments train on Cityscapes and test on unseen BDD100K, ACDC, and Dark Zürich domains representing geographic, weather, and day-to-night shifts.The segmentation experiments use HRNet, SegFormer, and DeepLab variants.Source: §4 · p. 6

4.1. Masked Noise Encoder

  • The masked noise encoder outperforms pSp, pSp†, and the Feature-Style encoder on all three reconstruction metrics.Table 1 reports MSE, LPIPS, and FID for Cityscapes at 128 × 256 resolution.Source: §4.1 · p. 6, Table 1 · p. 7
  • The masked noise encoder preserves semantic layout and fine details, whereas pSp† and the Feature-Style encoder fail to reconstruct some small objects.The paper notes that using such imperfect reconstructions for augmentation can confuse segmentation models and degrade performance.Source: §4.1 · p. 7
  • With noise masking, ISSA improves generalization on BDD100K, ACDC, and Dark Zürich despite slightly lowering Cityscapes performance.The masked encoder produces more pronounced gains, including more than 10% mIoU improvement from Cityscapes to Dark Zürich.Source: §4.1 · p. 7, Table 2 · p. 7

4.2. Domain Generalization

  • ISSA achieves the best mIoU and consistently improves all four ACDC scenarios with both HRNet and SegFormer.The comparison uses Cityscapes for training and ACDC as an unseen target domain.Source: §4.2 · p. 7, Table 3 · p. 8
  • 23.09% mIoU with ISSA versus 20.11% for RobustNet alone on Cityscapes-to-Dark Zürich.ISSA also improves RobustNet's generalization in the more challenging day-to-night setting.Source: §4.2 · p. 8, Table 5 · p. 8

5. Conclusion

The paper proposes ISSA, a GAN-inversion-based augmentation method for domain-generalized semantic segmentation from restricted single-source data. Its masked noise encoder preserves fine-grained content while enabling style mixing, and experiments show effectiveness across datasets and network architectures.

5. Conclusion

  • ISSA uses GAN inversion for semantic-segmentation augmentation with restricted training data from a single source domain.The masked noise encoder is the key enabler.Source: §5 · p. 8
  • The masked noise encoder preserves fine-grained content details while allowing style mixing without changing semantic content.Source: §5 · pp. 8–9
  • Extensive experiments demonstrate ISSA's effectiveness across different datasets and network architectures.Source: §5 · p. 9

Supplementary Material

The supplementary material adds encoder ablations and complexity analysis, expands the domain-generalization comparisons, and assesses ISSA against unsupervised adaptation methods. It also includes a limitations and future-work discussion.

Supplementary Material

  • Appendix S.1 adds noise-masking and noise-map-resolution ablations, encoder-comparison visuals, and computational-complexity information.Source: Supplementary Material · p. 12
  • Appendix S.2 provides detailed augmentation comparisons, segmentation visuals, and evidence of ISSA's plug-and-play ability.Source: Supplementary Material · p. 12
  • Appendix S.3 compares ISSA with unsupervised domain-adaptation methods and reports competitiveness without target-domain data; S.4 discusses limitations and future directions.Source: Supplementary Material · p. 12

S.1. Masked Noise Encoder

The supplementary encoder study selects masking and noise-map settings that balance reconstruction quality with style mixing. Qualitative comparisons and runtime figures further characterize the encoder.

S.1. Masked Noise Encoder

  • Patch size P = 4 with masking ratio ρ = 25% gives the best reconstruction performance and is used for ISSA.The patch-size effect is minor, while higher masking ratios negatively affect reconstruction.Source: §S.1 · p. 12, Table S.1 · p. 12
  • A 32×64 noise map, one fourth of the input resolution, reconstructs better than lower-resolution maps and is used in experiments.Full-resolution noise can encode nearly all details and weaken style mixing; very low resolution cannot faithfully reconstruct the scene.Source: §S.1 · pp. 12–13, Figure S.1 · p. 13
  • The masked noise encoder preserves fine details and reconstructs small objects such as pedestrians and traffic signs better than the compared encoders.The comparison includes pSp, pSp†, and the Feature-Style encoder.Source: §S.1 · p. 12, Figure S.2 · p. 13
  • Encoder training takes around 7 days on two GPUs at 256 × 512 resolution, while inference takes 0.1 seconds per image versus 55.7 seconds for PTI.StyleGAN2 training takes a similar amount of time to encoder training.Source: §S.1 · p. 12

S.2. Domain Generalization

Across driving-scene domain shifts, ISSA consistently outperforms other data-augmentation techniques across datasets and network architectures. Additional experiments report gains for daytime-to-night evaluation and show that ISSA can use a generator and encoder trained on a different dataset without retraining.

S.2. Domain Generalization

  • Among Hendrycks corruption types, noise and blur have larger negative performance impacts, while weather and digital corruptions offer little generalization benefit.Source: §S.2 · p. 13, Table S.3 · p. 14
  • ISSA consistently outperforms other data-augmentation techniques across unseen datasets and network architectures, and leads on ACDC with both HRNet and SegFormer.On ACDC, the comparison covers Rain, Fog, Snow, and Night; the tables also compare ISSA combined with CutMix.Source: Table S.4 · p. 14, Table S.3 · p. 14
  • For BDD100K-Daytime training, ISSA still outperforms other augmentation methods on unseen domains when evaluation focuses on ACDC-Night.Training uses 2,526 daytime BDD100K images, excluding dawn and nighttime images.Source: §S.2 · pp. 13–14, Table S.5 · p. 15
  • The qualitative examples show same-content samples receiving random styles from Cityscapes or BDD100K-Daytime, alongside Cityscapes-to-ACDC segmentation outputs.Source: §S.2 · p. 14
  • ISSA remains effective with a generator and encoder trained on BDD100K, even when the segmenter is trained on Cityscapes and the encoder-generator never saw Cityscapes samples.The pipeline does not require retraining for this cross-dataset use.Source: §S.2 · p. 14, Table S.6 · p. 15

S.3. Comparison with Unsupervised Domain Adaptation Methods

The authors compare ISSA with unsupervised domain adaptation (UDA), which uses unlabeled target-domain samples in addition to source data. Their Cityscapes-to-ACDC comparison reports that ISSA is on par with or better than UDA without access to target-domain images or labels.

S.3. Comparison with Unsupervised Domain Adaptation Methods

  • The comparison evaluates Cityscapes-to-ACDC adaptation/generalization against UDA methods that use source data and extra unlabeled target-domain samples.Source: §S.3 · p. 15
  • A supplementary figure shows style mixing on web-crawled images using a generator and encoder trained only on Cityscapes.All but the content images in the first two rows are web-crawled.Source: Figure S.3 · p. 15
  • ISSA is on par with or better than UDA methods on Cityscapes-to-ACDC, despite using no target-domain images or labels during training.The UDA methods use extra unlabeled target-domain samples; their results are from [60].Source: §S.3 · p. 15

S.4. Limitations and Future Work

The authors identify global style mixing as a limitation because it cannot selectively alter the style of local objects. They propose class-aware mixing and text-conditioned styles as future directions.

S.4. Limitations and Future Work

  • Global style mixing cannot specifically change a local object's style, such as changing a vehicle from red to black, though global changes inevitably modify local areas.Source: §S.4 · p. 15
  • Future directions include class-aware style mixing and using a pretrained vision-language model such as CLIP [56] to condition styles on text.The example goal is adding snow to a road while leaving other semantic classes unchanged.Source: §S.4 · p. 15
  • Supplementary examples show same-content images with random styles drawn from Cityscapes or BDD100K-Daytime.Source: Figure S.4 · p. 16
Loading 2210.10175v2…