PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Style Augmentation for Domain Generalized Urban-Scene Segmentation

Zhun Zhong, Yuyang Zhao, Gim Hee Lee, Nicu Sebe

arXiv:2207.04892v2cs.CV

TL;DR

Synthetic-to-real domain generalization seeks robust segmentation models trained without target data, but domain shifts can degrade performance on unseen domains. AdvStyle learns adversarial image styles to generate hard training examples. It improves performance on two synthetic-to-real benchmarks and is also reported to reach state-of-the-art accuracy on two image-classification datasets.

  • Problem

    Domain generalization seeks segmentation models trained on labeled source data that work on unseen domains, where shifts in weather, season, and light can degrade performance.

    Source: §1 · p. 1

  • Method

    AdvStyle adversarially updates image-level style features and uses the learned styles to generate hard examples for robust model training.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    AdvStyle achieves state-of-the-art DG performance on two synthetic-to-real benchmarks and state-of-the-art accuracy on two image-classification datasets.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The paper reports that dynamically generated hard styles help the segmentation model generalize to style variations in unseen domains.

    Source: §5 · p. 10

    AdvStyle nearly doubles baseline training cost and cannot represent severe environmental changes such as rain or snow through style features alone.

    Source: I Limitations · p. 15

Abstract

from arXiv · show

In this paper, we consider the problem of domain generalization in semantic segmentation, which aims to learn a robust model using only labeled synthetic (source) data. The model is expected to perform well on unseen real (target) domains. Our study finds that the image style variation can largely influence the model's performance and the style features can be well represented by the channel-wise mean and standard deviation of images. Inspired by this, we propose a novel adversarial style augmentation (AdvStyle) approach, which can dynamically generate hard stylized images during training and thus can effectively prevent the model from overfitting on the source domain. Specifically, AdvStyle regards the style feature as a learnable parameter and updates it by adversarial training. The learned adversarial style feature is used to construct an adversarial image for robust model training. AdvStyle is easy to implement and can be readily applied to different models. Experiments on two synthetic-to-real semantic segmentation benchmarks demonstrate that AdvStyle can significantly improve the model performance on unseen real domains and show that we can achieve the state of the art. Moreover, AdvStyle can be employed to domain generalized image classification and produces a clear improvement on the considered datasets.

1 Introduction

The introduction frames synthetic-to-real domain generalization as a response to target-data constraints and domain shifts, then motivates adversarial style augmentation as a way to improve generalization across styles.

1 Introduction

  • Domain adaptation uses unlabeled target data, whereas domain generalization trains only on labeled source data and can use synthetic data such as GTAV.The motivation includes the cost and time of pixel-wise annotation and the degradation that occurs when models encounter unseen domains.Source: §1 · p. 1
  • Image-level style differences across datasets are substantial, and changing a test image’s style feature can sharply reduce GTAV mIoU.The paper defines style feature as the channel-wise mean and standard deviation; CityScapes-style replacement causes lower performance than other style changes.Source: §1 · p. 2
  • AdvStyle learns adversarial image styles and uses them to generate hard training examples, while training the model on both adversarial and original images.The style feature is a learnable parameter updated with adversarial segmentation loss; the resulting adversarial images are generated dynamically from the current model.Source: §1 · p. 2
  • Experiments on two synthetic-to-real DG benchmarks show that AdvStyle achieves new state-of-the-art DG performance.The introduction also reports state-of-the-art accuracy on two image-classification datasets.Source: §1 · p. 3
  • AdvStyle can be applied to domain-generalized image classification and achieves state-of-the-art accuracy on two datasets.Source: §1 · p. 3

2 Related Work

Related work covers source-data augmentation and domain-invariant feature learning, adversarial training for domain generalization, and style-based data generation.

2 Related Work

  • Prior semantic-segmentation DG methods augment with extra real-world data or learn domain-invariant features; AdvStyle instead learns adversarial styles using synthetic source data alone.Source: §2 · p. 3
  • Earlier adversarial DG methods target image classification, while this work generates adversarial samples through image style for semantic segmentation.Source: §2 · p. 3
  • Style augmentation has used style exchange, mixing, or learnable modules; AdvStyle instead learns styles adversarially to train on hard stylized examples.The cited colorization-based approach [2] requires a pretrained colorization model and aims to impair model performance, unlike AdvStyle’s generalization objective.Source: §2 · p. 3

3 Method

The method targets synthetic-to-real domain generalization by dynamically generating a source domain with adversarially learned image styles. It trains on original and stylized images, while using style variation to preserve semantic content for segmentation.

3 Method

  • Synthetic-to-real domain generalization trains on one labeled synthetic domain for performance on unseen real domains; the method generates a dynamic source domain to reduce domain shifts.The task is formulated as a worst-case problem, with semantic-space distribution distance constrained for semantic consistency.Source: §3 · p. 4

3.1 Overview

  • AdvStyle decomposes images into normalized content and channel-wise mean and standard deviation style features, then updates styles adversarially each iteration.The model trains on original and adversarial images; both steps generate hard stylized samples for the current model.Source: §3.1 · p. 4

3.2 Adversarial Style Learning

  • Adversarial style learning initializes learnable style features from image statistics and updates them using the segmentation loss gradient.One update achieves similar results to multi-step learning while being more efficient, so the experiments use one step.Source: §3.2 · pp. 4–5

3.3 Robust Model Training

  • Robust model training composes an adversarial image from the normalized image and learned style features, then optimizes on original and adversarial samples.At test time, the model receives original samples without AdvStyle.Source: §3.3 · p. 5

3.4 Discussion

  • Unlike pixel-wise adversarial perturbations, AdvStyle varies image style to maintain most pixels’ semantic content, supporting pixel-wise consistency in segmentation.The paper reports that Table 3 demonstrates AdvStyle’s superiority over pixel-wise adversarial learning.Source: §3.4 · p. 5

4 Experiments

Experiments evaluate AdvStyle on synthetic-to-real segmentation with GTAV or SYNTHIA sources, real-world target datasets, and multiple models, augmentations, and DG settings. The reported comparisons show gains across segmentation and image-classification benchmarks, including state-of-the-art results under specified settings.

4.1 Experimental Setup

  • Evaluation uses GTAV or SYNTHIA as the synthetic source and CityScapes, BDD-100K, and Mapillary as real-world targets, with DeepLabV3+ and three backbones.Models use SGD and are trained for 40K iterations; the reported evaluation uses the last training checkpoint and averages three runs.Source: §4.1 · pp. 5–6

4.2 Evaluation

  • For GTAV, AdvStyle raises average mIoU over the baseline for MobileNetV2, ResNet-50, and ResNet-101, and further improves IBN-Net and ISW across settings.The reported baseline-to-AdvStyle average mIoU pairs are 26.03% to 32.11%, 27.42% to 37.39%, and 31.47% to 37.34%, respectively.Source: §4.2 · p. 6, Table 1 · p. 6, Table 2 · p. 6
  • 35.32% average mIoU with AdvStyle versus 22.91% without it; AdvStyle also outperforms the compared augmentation methods and complements color jittering and Gaussian blur.The comparison uses GTAV and ResNet-50; the paper reports further mIoU gains across target domains when the three augmentations are combined.Source: §4.2 · p. 7, Table 3 · p. 7
  • AdvStyle ranks best among the compared style-aware methods, while adversarial learning rates from 1 to 10 yield the best results; a rate of 40 may produce unrealistic styles and hamper training.The comparisons include MixStyle, CrossStyle, and RandStyle; even small learning rates improve performance on all target domains.Source: §4.2 · p. 7, Table 4 · p. 7, Figure 4 · p. 7

4.3 Comparison with State-of-The-Art Methods

  • AdvStyle reaches state-of-the-art mIoU with ResNet-50; IBN-Net+AdvStyle achieves new state-of-the-art results with ResNet-50 and ResNet-101, while matching FSDR comparably with ResNet-101.The compared methods differ in training data, strategies, auxiliary data, and evaluation checkpoints, so the paper cautions that comparisons are not absolutely fair.Source: Table 5 · p. 8, §4.3 · p. 8

4.4 Performance under Multi-Source Setting

  • With GTAV and SYNTHIA as joint sources, AdvStyle consistently improves ISW results.The multi-source evaluation uses ResNet-50.Source: §4.4 · p. 9, Table 6 · p. 9

4.5 Visualization

  • Qualitative results show improved baseline, IBN, and ISW segmentations, especially for easily confused road–sidewalk and building–sky classes.The t-SNE visualization shows generated styles differing from the original distribution, sometimes near other datasets' distributions or outside the four visualized dataset distributions.Source: §4.5 · p. 9, Figure 5 · p. 9, Figure 6 · p. 10

4.6 Evaluation on Image Classification Task

  • On Digits, AdvStyle improves ERM across target domains and raises ME-ADA average accuracy by 8.0%; on PACS, ERM+AdvStyle exceeds L2D by 2.4% in average accuracy.AdvStyle also improves RSC and L2D on PACS, and the reported Digits and PACS results support its use in single-domain-generalized image classification.Source: §4.6 · p. 10, Table 7 · p. 10, Table 8 · p. 10

5 Conclusion

AdvStyle generates hard stylized training images through adversarial image-level style learning for domain-generalized segmentation. The paper reports improved unseen-domain performance, state-of-the-art results in two synthetic-to-real settings, and gains in image classification.

5 Conclusion

  • AdvStyle dynamically generates hard stylized images by learning adversarial image-level style features to encourage training with more diverse samples.It can be integrated with different models without modifying network structures or learning strategies.Source: §5 · p. 10
  • AdvStyle improves generalization on unseen real domains and achieves state-of-the-art performance in two synthetic-to-real settings.Source: §5 · p. 10
  • AdvStyle also produces significant improvement when applied to domain-generalized image classification.Source: §5 · p. 10

A Details of Datasets in Domain Generalized Semantic Segmentation

The synthetic-to-real evaluation uses GTAV or SYNTHIA as the source and three real-world datasets as targets. Dataset sizes, splits, and validation-set evaluation are specified for these benchmarks.

A Details of Datasets in Domain Generalized Semantic Segmentation

  • The benchmark uses GTAV or SYNTHIA as the synthetic source and CityScapes, BDD-100K, and Mapillary as real-world evaluation domains.Source: A Details of Datasets in Domain… · p. 13
  • GTAV has 24,966 images at 1914×1052, split into 12,403 training, 6,382 validation, and 6,181 test images; SYNTHIA has 9,400 images at 960×720, with 6,580 used for training.Source: A Details of Datasets in Domain… · p. 13
  • Evaluation uses validation sets of 500 CityScapes images, 1,000 BDD-100K images, and 2,000 Mapillary images; CityScapes images are collected primarily in Germany.The respective image sizes are 2048×1024, 1280×720, and 1920×1080.Source: A Details of Datasets in Domain… · p. 13

B Details of Single Domain Generalization in Image Classification

The image-classification experiments evaluate domain generalization across Digits and PACS, using one domain as source and the remaining domains as targets. The appendix also defines the ERM baseline and reports training configurations.

B Details of Single Domain Generalization in Image Classification

  • Digits uses MNIST as the source and evaluates on SVHN, MNIST-M, SYN, and USPS across five domains and 10 classes.The setup uses ConvNet, Adam with learning rate 10−4, 10,000 training iterations, and batch size 32.Source: B Details of Single Domain… · p. 13
  • PACS evaluates one of Artpaint, Cartoon, Sketch, or Photo as source and the other three as target domains across seven classes.The setup uses ImageNet-pretrained ResNet18 with a fully connected classification head and SGD for 30 epochs, batch size 128, and an initial learning rate of 0.004 reduced tenfold after 24 epochs.Source: B Details of Single Domain… · p. 13
  • The baseline is vanilla empirical risk minimization, which trains directly on the source domain using classification loss.Source: B Details of Single Domain… · p. 13

C Position of AdvStyle

The position study finds that image-level AdvStyle performs best among the tested injection positions. Performance gains occur at positions 0–2, while deeper-layer injection does not improve or can hurt results.

C Position of AdvStyle

  • AdvStyle injection at positions 0–2 improves performance, with the best result at position 0; injection at positions 3–4 fails to improve or can hurt performance.Table 9 reports the impact of injecting AdvStyle at different positions.Source: C Position of AdvStyle · p. 13, Table 9 · p. 13
  • Deeper layers capture more semantic content rather than style, which the paper gives as the reason positions 3–4 fail to improve or can hurt performance.Source: C Position of AdvStyle · p. 13
  • The authors use a large AdvStyle learning rate because the style-feature gradient is very small without a batch-normalization layer.Source: C Position of AdvStyle · p. 13

D Comparison of Adversarial Augmentations

AdvStyle outperforms pixel-wise AdvPixel in mean mIoU, and combining both augmentations improves performance further. The paper attributes AdvStyle’s suitability for segmentation to changing image style while retaining the semantic content of most pixels.

D Comparison of Adversarial Augmentations

  • 3.42% in mean mIoU is the margin by which AdvStyle outperforms AdvPixel; randomly mixing their augmentations improves mean mIoU by 0.81%.Both augmentations improve performance, according to the comparison.Source: D Comparison of Adversarial… · p. 14, Table 10 · p. 14
  • AdvPixel learns pixel-wise adversarial examples, whereas AdvStyle varies image style and retains the semantic content of most pixels.The authors argue this better preserves pixel-wise semantic consistency for segmentation.Source: D Comparison of Adversarial… · p. 14

E Variants of AdvStyle

The section explores patch-wise and LAB-space variants, analyzes dataset-distribution distances, describes the training algorithm, and provides additional visualizations.

E Variants of AdvStyle

  • AdvStyle-Patches uses four learnable 6-dim features, one for each image quadrant, and improves BDD and Mapillary results, though mean improvement across domains is marginal.The patches are formed by splitting each image evenly into four.Source: E Variants of AdvStyle · p. 14, Table 11 · p. 15
  • LAB-based AdvStyle significantly improves unseen-domain performance but scores below RGB-based AdvStyle on two of three benchmarks, while RGB↔LAB conversion adds training time.The LAB procedure obtains learnable mean and standard deviation in LAB space, then converts back to RGB for adversarial learning and optimization.Source: E Variants of AdvStyle · p. 14, Table 11 · p. 15

F Quantitative Understanding of AdvStyle

G Algorithm and Pytorch-Like Pseudo-Code

  • Training alternates adversarial style learning, which updates initialized channel-wise mean and standard deviation, with robust model training on original and adversarial images.The model is optimized using the sum of segmentation losses on both image sets.Source: G Algorithm and Pytorch-Like Pseudo-Code · pp. 14–15

H More Visualizations

  • Further segmentation results are provided in Figures 8, 9, and 10, comparing the baseline with “baseline+AdvStyle” across the reported visual results.Source: H More Visualizations · p. 15
  • The section also shows further examples generated by AdvStyle.These examples use GTAV and a ResNet-50 backbone.Source: H More Visualizations · p. 15

I Limitations

AdvStyle increases training cost and does not address severe environmental changes through style features alone; manually adding such conditions is proposed as mitigation.

I Limitations

  • AdvStyle’s computational cost is almost double the baseline because generating style-adversarial examples requires an additional forward-backward process.The paper identifies increased training time as its main limitation.Source: I Limitations · p. 15
  • Style features alone cannot represent severe environmental changes such as rainy or snowy weather.The limitation also names fog as an example of such a condition.Source: I Limitations · p. 15
  • Rain, snow, and fog can be manually added to source and adversarially augmented samples to alleviate this limitation.The paper presents this as a mitigation, not a guarantee of resolving the limitation.Source: I Limitations · p. 15
Loading 2207.04892v2…