PaperlayerView exact revision on arXiv

Source-linked AI summary

Style-Hallucinated Dual Consistency Learning for Domain Generalized Semantic Segmentation

Yuyang Zhao, Zhun Zhong, Na Zhao, Nicu Sebe, Gim Hee Lee

arXiv:2204.02548v2cs.CV

TL;DR

Synthetic-to-real domain generalized semantic segmentation seeks robust performance on unseen real-world scenes despite limited source variation and a large synthetic-to-real gap. SHADE combines style hallucination with style and retrospection consistency to learn domain-invariant representations. It achieves state-of-the-art performance and outperforms prior methods on three real-world datasets in both single- and multi-source settings.

  • Problem

    Synthetic-to-real domain generalized semantic segmentation must generalize from synthetic source data to unseen real-world scenes despite limited source variation and a large distribution gap.

  • Method

    SHADE combines Style Consistency, Retrospection Consistency, and a style hallucination module that generates diverse source styles from representative basis styles.

  • Results

    SHADE achieves state-of-the-art performance on three real-world datasets under both single-source and multi-source domain generalization settings.

  • Takeaways & Limitations

    Generating diverse and realistic source samples catalyzes dual consistency learning for domain-invariant representation in synthetic-to-real domain generalization.

Abstract

from arXiv · show

In this paper, we study the task of synthetic-to-real domain generalized semantic segmentation, which aims to learn a model that is robust to unseen real-world scenes using only synthetic data. The large domain shift between synthetic and real-world data, including the limited source environmental variations and the large distribution gap between synthetic and real-world data, significantly hinders the model performance on unseen real-world scenes. In this work, we propose the Style-HAllucinated Dual consistEncy learning (SHADE) framework to handle such domain shift. Specifically, SHADE is constructed based on two consistency constraints, Style Consistency (SC) and Retrospection Consistency (RC). SC enriches the source situations and encourages the model to learn consistent representation across style-diversified samples. RC leverages real-world knowledge to prevent the model from overfitting to synthetic data and thus largely keeps the representation consistent between the synthetic and real-world models. Furthermore, we present a novel style hallucination module (SHM) to generate style-diversified samples that are essential to consistency learning. SHM selects basis styles from the source distribution, enabling the model to dynamically generate diverse and realistic samples during training. Experiments show that our SHADE yields significant improvement and outperforms state-of-the-art methods by 5.05% and 8.35% on the average mIoU of three real-world datasets on single- and multi-source settings, respectively.

1 Introduction

Synthetic-to-real domain generalized semantic segmentation must handle limited source variation and a large synthetic-to-real distribution gap without extra real-world data. SHADE addresses these challenges through style hallucination and dual consistency constraints, achieving state-of-the-art performance.

  • Motivation: Domain generalized semantic segmentation trains on annotated source data to handle unseen conditions, where synthetic-to-real shifts remain substantial.The shift reflects both limited source environmental variation and differences in image styles and object characteristics.
  • Proposed framework: SHADE learns style-invariant representations across diversified source samples and narrows the synthetic-to-real gap through retrospective real-world knowledge.Its two constraints are style consistency and retrospection consistency.
  • Style hallucination module: Style hallucination selects representative basis styles and combines them to generate diverse, realistic source samples for consistency learning.Farthest point sampling helps balance style diversity and realism by including rare styles.
  • Results: SHADE achieves new state-of-the-art performance on single-source and multi-source domain-generalization benchmarks.The experiments evaluate the method on real-world target datasets under both settings.

2 Related Work

Prior domain-generalization methods diversify training data with real-world templates or learn domain-invariant features within source domains. SHADE instead generates new styles using only synthetic source data and learns invariant representations across them.

  • Domain Generalization: Domain generalization aims to improve performance on unseen domains using one or multiple source domains, commonly using synthetic data for semantic segmentation.This setting addresses expensive pixel-level annotation while targeting unseen real-world domains.
  • Domain Generalization: Existing approaches either diversify training data with real-world templates or reduce domain-specific information within source data.Examples include instance normalization and whitening transformations.
  • Proposed direction: SHADE differs by generating new styles only from the synthetic source domain and learning invariant representations across those styles.Its framework combines style hallucination with dual consistency constraints.

3 Methodology

SHADE combines Style Consistency and Retrospection Consistency with a Style Hallucination Module to learn domain-invariant representations from synthetic data while generating diverse styles. The framework uses style-aware feature transformations and representative basis styles to support consistency learning.

  • Framework overview: SHADE combines Style Consistency, Retrospection Consistency, and the Style Hallucination Module to reduce the gap between synthetic and real-world data.The dual constraints learn domain-invariant representations, while SHM dynamically generates diverse styles.
  • Dual consistency constraints: Style Consistency minimizes Jensen-Shannon Divergence between predictions for original and semantically aligned style-diversified samples.The two samples share semantic content but differ in style, encouraging stable pixel-level predictions across styles.
  • Dual consistency constraints: Retrospection Consistency aligns segmentation-model bottleneck features with those from an ImageNet-pretrained model for original and style-diversified samples.It uses real-world representation knowledge to guide learning without requiring extra real-world data.
  • Style Hallucination Module: Style hallucination represents the style space with C basis styles and combines them through AdaIN-style transformations to generate new samples.The transformation uses channel-wise means and standard deviations from content and style feature maps.
  • Style Hallucination Module: FPS basis styles cover rare source styles more broadly than Kmeans centers and generate more diverse styles, including styles closer to real-world ones.The paper reports that Kmeans-generated styles remain within source distributions and can miss rare styles.
  • Training objective: The overall objective averages cross-entropy losses on original and style-hallucinated samples and adds weighted Style and Retrospection Consistency losses.λ_SC and λ_RC control the two consistency terms.

4 Experiments

Experiments evaluate SHADE across single-source, multi-source, ablation, style-variation, parameter, and cross-domain settings using mIoU on real-world datasets. Results consistently support the effectiveness of style hallucination and dual consistency, with sensitivity to SHM placement and basis-style refresh frequency.

  • Single-source DG: 42.42% average mIoU on three real-world targets, improving 15.00% over baseline and 5.05% over ISW in single-source DG.This result uses GTAV training data with a ResNet-50 backbone.
  • Multi-source DG: 14.28% average-mIoU improvement over baseline, with SHADE surpassing ISW and IBN-Net by 8.35% and 9.84% in multi-source DG.The multi-source protocol trains on GTAV and SYNTHIA with ResNet-50.
  • Ablation studies: SHM improves performance alone, while SC adds 3.99% average mIoU and RC adds 4.61% over SHM; removing RC reduces SHADE by 2.75% mIoU.The ablation attributes gains to diverse samples, style-invariant learning, and real-world knowledge for narrowing the synthetic-to-real gap.
  • Style variation methods: MixStyle and CrossNorm achieve 41.10% and 40.72% average mIoU, while random styles and Kmeans bases suffer from unrealistic or dominant-style generation.SHM selects basis styles with FPS to better use the source style space.
  • Parameter analysis: Inserting SHM after the first convolutional layer performs best, whereas insertion after ResNet layer3 causes drastic degradation because deep activations contain more semantic information and biased style patterns.The shallow-layer placement better represents style information through channel-wise statistics.
  • Parameter analysis: Frequent basis-style reselection at intervals k <= 3 maintains consistently good performance as the source-style distribution changes during training.The interval analysis compares periodic reselection with selecting basis styles only once at initialization.
  • Further evaluation: SHADE consistently outperforms ISW and IBN-Net across real and synthetic evaluation datasets when trained on CityScapes.This cross-domain evaluation supports the method's versatility beyond the primary synthetic-to-real setting.

5 Conclusion

SHADE learns domain-invariant representations for synthetic-to-real domain generalization through dual consistency constraints and style hallucination. Experiments report state-of-the-art performance under single- and multi-source settings.

  • 5 Conclusion: SHADE uses Style Consistency and Retrospection Consistency to learn representations consistent across styles and guided by retrospective knowledge.The framework targets synthetic-to-real domain generalized semantic segmentation.
  • 5 Conclusion: The style hallucination module generates diverse and realistic source samples to catalyze dual consistency learning.SHM is integrated into SHADE as the source-sample generation component.
  • 5 Conclusion: SHADE achieves state-of-the-art performance on three real-world datasets under both single- and multi-source domain generalization settings with different backbones.

A Visualization

The qualitative comparison evaluates SHADE against several baselines across real-world datasets and target conditions. SHADE performs consistently across class types and environmental conditions.

  • A Visualization: SHADE is compared with baseline, IBN-Net, and ISW on CityScapes, BDD100K, and Mapillary.The comparison is presented as qualitative segmentation results.
  • A Visualization: SHADE consistently outperforms the other methods under sunny, cloudy, and overcast target conditions.
  • A Visualization: SHADE handles both “stuff” classes such as road and “things” classes such as bus and bicycle.

B Per-class IoU

Per-class IoU analysis on CityScapes examines the effect of retrospection consistency for models trained on GTAV with a ResNet-50 backbone. RC improves both “things” and “stuff” classes, with a larger gain for “things.”

  • B Per-class IoU: RC improves per-class IoU for “things” classes by 2.47% and for “stuff” classes by 1.54%.The analysis uses CityScapes evaluation with GTAV training and a ResNet-50 backbone.
  • B Per-class IoU: The improvement from RC is more significant for “things” classes than for “stuff” classes.
Loading 2204.02548v2…