Source-linked AI summary
Domain Randomization and Pyramid Consistency: Simulation-to-Real Generalization without Accessing Target Domain Data
Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto Sangiovanni-Vincentelli, Kurt Keutzer, Boqing Gong
TL;DR
Synthetic-to-real semantic segmentation must generalize to unseen real domains without target-domain data, unlike domain adaptation settings that use target images during training. The paper combines stylization-based domain randomization with pyramid consistency, and reports superior domain-generalization performance, on par with or better than state-of-the-art domain adaptation.
Problem
Visual differences between synthetic training data and real scenes reduce real-image segmentation performance, while domain adaptation requires target-domain data that may be unavailable in advance.
Method
The method stylizes synthetic images into multiple auxiliary real-image domains and enforces pyramid consistency across domains and within images.
Results
The method achieves superior performance over state-of-the-art domain-generalization and domain-adaptation methods, with adaptation results using target data during training.
Takeaways & Limitations
Simulation-based semantic segmentation can generalize across multiple real-world target domains without accessing their training data.
Takeaways & Limitations
Direct regularization on raw feature maps does not work well because it requires nearly identical representations across training domains, including harder-to-fit domains.
Abstract
from arXiv · showhide
We propose to harness the potential of simulation for the semantic segmentation of real-world self-driving scenes in a domain generalization fashion. The segmentation network is trained without any data of target domains and tested on the unseen target domains. To this end, we propose a new approach of domain randomization and pyramid consistency to learn a model with high generalizability. First, we propose to randomize the synthetic images with the styles of real images in terms of visual appearances using auxiliary datasets, in order to effectively learn domain-invariant representations. Second, we further enforce pyramid consistency across different "stylized" images and within an image, in order to learn domain-invariant and scale-invariant features, respectively. Extensive experiments are conducted on the generalization from GTA and SYNTHIA to Cityscapes, BDDS and Mapillary; and our method achieves superior results over the state-of-the-art techniques. Remarkably, our generalization results are on par with or even better than those obtained by state-of-the-art simulation-to-real domain adaptation methods, which access the target domain data at training time.
1. Introduction
Simulation provides abundant labeled imagery but differs visually from real scenes, motivating domain generalization without target-domain training data. The approach stylizes synthetic images and enforces pyramid consistency to improve domain-invariant representations and generalization.
- Motivation: Simulation offers large amounts of low-cost, diverse labeled imagery for training semantic segmentation networks.Its scale supports training deep convolutional neural networks with millions of parameters.
- Motivation: Visual differences between synthetic and real scenes can reduce semantic segmentation performance on real images.Domain adaptation addresses this mismatch but requires access to unlabeled target-domain data during training.
- Problem Setting: The proposed domain generalization setting trains without target-domain data while aiming to generalize across multiple unseen real-world domains.This avoids needing to know in advance the deployment city, weather, or time of day.
- Approach: Domain randomization transfers synthetic source images into multiple real-image styles using image-to-image translation and auxiliary domains.The method does not require controlling the simulator and can select auxiliary domains from the real world.
- Approach: Pyramid consistency training regularizes internal activations across stylized domains and within images to encourage domain-invariant representations.The network is trained to fit both the synthetic source domain and stylized auxiliary domains while maintaining consistent predictions.
- Results: Experiments report superior performance to state-of-the-art domain-generalization and domain-adaptation methods, with adaptation methods using target data during training.The results are described as on par with or better than state-of-the-art domain-adaptation results.
2. Related Work
Related work spans semantic-segmentation domain adaptation, domain generalization, domain randomization, and data augmentation. This paper distinguishes its stylization-based randomization from simulator manipulation and applies it to semantic segmentation.
- Domain Adaptation: Semantic-segmentation domain adaptation commonly learns domain-invariant representations through adversarial feature alignment and label-distribution alignment.Prior methods include global and superpixel-level label alignment and class-balanced self-training.
- Domain Generalization: Domain adaptation accesses unlabeled images from a known target domain during training, whereas domain generalization evaluates on unseen domains.The distinction concerns target-domain availability and whether the test domain is known during training.
- Domain Randomization: Traditional domain randomization varies simulated textures, colors, lighting, poses, and camera settings to reduce the reality gap.These approaches generally modify objects or conditions inside the simulation environment.
- Domain Randomization: The paper instead uses image-to-image stylization to create auxiliary domains from synthetic images without modifying simulator objects.Its auxiliary image sets use ImageNet classes to provide different styles while preserving the synthetic image content.
- Data Augmentation: Data augmentation supplements datasets with transformed versions of existing samples, including rotations, translations, and blurring.The related work distinguishes these modifications from the paper’s cross-domain stylization strategy.
3. Approach
The approach stylizes synthetic images into auxiliary real domains, then uses pyramid consistency to learn representations invariant to appearance and robust to scale changes.
- Domain Randomization with Stylization: Synthetic images are mapped to multiple auxiliary real domains, expanding the training set to K + 1 times its original size.Image-to-image translation creates the auxiliary domains while retaining most semantic content.
- Domain Randomization with Stylization: Training on the augmented domains generalizes better to unseen targets than training on the single synthetic source domain.The improvement may reflect both increased training-set size and exposure to styles closer to real images.
- Pyramid Consistency across Domains: Pyramid consistency regularizes activations for stylized versions of the same content so their high-level features and predictions remain similar.The loss uses operators such as average pooling and an ℓ1 distance to guide domain-invariant representations.
- Pyramid Consistency across Domains: Spatial pyramid pooling accommodates representation differences while enforcing consistency at multiple scales through 1 × 1, 2 × 2, 4 × 4 and 8 × 8 pooling levels.The pooled outputs are squeezed and concatenated into a lower-dimensional tensor than the original feature map.
- Pyramid Consistency across Domains: The cross-domain consistency scheme compares activation maps for images with the same annotation across training domains at several deep layers.Figure 3 depicts the activation maps at the last few layers and the corresponding pyramid consistency loss.
- Pyramid Consistency within an Image: Consistency between randomly cropped and full images artificially randomizes scale, improving robustness to domain gaps from scene layout and object scale.Each crop is resized to the full image size before being fed to the network, and corresponding feature regions are compared.
4. Experiments and Results
The experiments evaluate target-domain-free semantic segmentation across synthetic-to-real settings, using cross-domain validation, a proposed generalizability metric, ablations, and comparisons with domain generalization and adaptation methods. Domain randomization and pyramid consistency improve performance across datasets, while the full approach compares favorably with existing methods.
- Experimental Settings: Models are trained without target-domain data and tested on unseen domains, distinguishing this evaluation from domain adaptation.Cross-validation selects a model using 500 images from another real-world dataset rather than the target dataset.
- Experimental Settings: Experiments use GTA and SYNTHIA as source domains, Cityscapes, BDDS, and Mapillary as test domains, and auxiliary datasets for stylizing source images.The study also evaluates multiple FCN backbones and two auxiliary-domain sets.
- Evaluation: Mean IoU averages per-class IoU values, while Gperf measures performance over unseen domains drawn from a distribution of possible real-world domains.The experiment uses three unseen test domains for Gperf.
- Domain Randomization: Accuracy increases as the number of auxiliary domains grows, then saturates, suggesting that additional domains eventually provide limited benefit under the tested settings.The trend is observed for both auxiliary-domain sets when training on GTA and testing on Cityscapes, BDDS, and Mapillary.
- Ablation Study: DR raises mIoU from 29.81 to 34.64 on Cityscapes, from 24.59 to 30.14 on BDDS, and from 26.63 to 31.64 on Mapillary; PCD and PCI further improve results.The full approach reaches 36.11, 31.56, and 32.25 mIoU on Cityscapes, BDDS, and Mapillary, respectively.
- Generalization and Comparison: Across source, backbone, and target combinations, the proposed techniques improve generalizability by 5%∼12% and outperform IBN-Net in the GTA-to-Cityscapes comparison.The method is also reported as on par or better than state-of-the-art adaptation methods in GTA→Cityscapes and SYNTHIA→Cityscapes settings.
5. Conclusion
The paper presents a simulation-to-real domain generalization approach for semantic segmentation that requires no target-domain training data. It combines synthetic-image style randomization with pyramid consistency and reports superior performance across domain generalization and adaptation settings.
- The approach generalizes semantic segmentation networks from simulation to real-world scenes without accessing target-domain data during training.
- It randomizes synthetic images using auxiliary datasets and enforces pyramid consistency across domains and within images.
- Experiments show superior performance over state-of-the-art methods in both domain generalization and domain adaptation.
Appendix
The appendix finds that the number of auxiliary style domains affects results more than the particular domains selected. Different style sets, including real weather styles, produce similar outcomes.
- Different style sets, including real weather styles, lead to similar results.
- The number of domains influences results more than the specific domains used.
A. Detailed Comparison with Other Works.
Detailed class-wise comparisons complement mean-IoU evaluations across domain generalization and domain adaptation settings. The method performs better in many classes and achieves higher mIoU than state-of-the-art methods.
- The detailed tables compare class-wise accuracies alongside mean IoU for domain generalization and domain adaptation settings.Comparisons cover GTA to Cityscapes for both settings and SYNTHIA to Cityscapes for domain adaptation.
- The method performs better in many classes and outperforms state-of-the-art methods in mIoU under both evaluation settings.
B. Additional Experiments on auxiliary domains and color augmentation.
Additional experiments vary the auxiliary style sets and replace style transfer with color augmentations. Results remain close across style choices, color augmentation is slightly worse, and pyramid consistency remains effective.
- B. Additional Experiments on auxiliary domains and color augmentation.: The approach is rerun with 15 BDD real-world styles spanning weather conditions and time of day.
- B. Additional Experiments on auxiliary domains and color augmentation.: The style-transfer step is replaced with 15 color augmentations that vary hue, saturation, grayscale, and contrast while preserving object semantics.
- B. Additional Experiments on auxiliary domains and color augmentation.: Results for random, semantic, BDD, and color-augmentation styles are close, although color augmentation is slightly worse than the others.
- B. Additional Experiments on auxiliary domains and color augmentation.: Pyramid consistency is effective across all tested style-set and augmentation cases.
C. More Discussion.
The discussion reports that color augmentation performs slightly worse than style transfer and attributes this to limited appearance, color, and texture variation. It also identifies non-uniform color-shift policies as future work.
- Color augmentation performs a little worse than style transfer, according to Table 6.
- The proposed explanation is that color augmentation adds no real-image appearances and randomizes only color, not texture.
- Learning an optimal non-uniform color-shift policy is identified as a future research direction.