Source-linked AI summary
Semantic-Aware Domain Generalized Segmentation
Duo Peng, Yinjie Lei, Munawar Hayat, Yulan Guo, Wen Li
TL;DR
Domain generalized semantic segmentation must handle unseen target domains without target-domain data, while standard global normalization may weaken category discrimination. The paper proposes SAN for category-level center alignment and SAW for distributed alignment, and reports state-of-the-art performance across domain generalization and adaptation settings, with remaining source-target performance gaps.
Problem
Semantic segmentation performance degrades on unseen domains, while domain adaptation requires target-domain data that may be unavailable.
Method
SAN and SAW sequentially perform category-level center alignment and distributed alignment to produce domain-invariant and discriminative features.
Results
The approach achieves state-of-the-art performance in domain generalization and domain adaptation, improving upon previous DGSS methods across benchmarks and backbones.
Takeaways & Limitations
SAN and SAW improve generalization without target-domain training data and can be integrated across different backbone architectures.
Takeaways & Limitations
Style-invariant features may still contain source-domain-specific cues, causing a significant performance gap between source and target domains.
Abstract
from arXiv · showhide
Deep models trained on source domain lack generalization when evaluated on unseen target domains with different data distributions. The problem becomes even more pronounced when we have no access to target domain samples for adaptation. In this paper, we address domain generalized semantic segmentation, where a segmentation model is trained to be domain-invariant without using any target domain data. Existing approaches to tackle this problem standardize data into a unified distribution. We argue that while such a standardization promotes global normalization, the resulting features are not discriminative enough to get clear segmentation boundaries. To enhance separation between categories while simultaneously promoting domain invariance, we propose a framework including two novel modules: Semantic-Aware Normalization (SAN) and Semantic-Aware Whitening (SAW). Specifically, SAN focuses on category-level center alignment between features from different image styles, while SAW enforces distributed alignment for the already center-aligned features. With the help of SAN and SAW, we encourage both intra-category compactness and inter-category separability. We validate our approach through extensive experiments on widely-used datasets (i.e. GTAV, SYNTHIA, Cityscapes, Mapillary and BDDS). Our approach shows significant improvements over existing state-of-the-art on various backbone networks. Code is available at https://github.com/leolyj/SAN-SAW
1. Introduction
The paper targets semantic segmentation that generalizes to unseen domains without target-domain training data. It introduces SAN and SAW to align category-level feature distributions while preserving discriminative segmentation features.
- Domain adaptation methods require target-domain samples, which are impractical to obtain for rarely occurring concepts.
- The proposed SAN and SAW modules collaboratively align category-level distributions to enhance feature discriminability.
- Semantic-aware center alignment and distributed alignment enable discriminative and compact matching of features from different styles.
- Extensive benchmark evaluations improve upon previous DGSS methods and set new state-of-the-art performance.
- SAN and SAW are plug-and-play across backbone architectures and consistently improve out-of-domain generalization without target-domain training data.
2. Background
Domain adaptation aligns source and target distributions but requires target images during training. Domain generalization instead evaluates on totally unseen domains, while DGSS remains relatively underexplored.
- Domain adaptation aligns source and target feature distributions to improve model generalization.
- Existing domain adaptation for semantic segmentation broadly uses adversarial training and self-training based methods.
- Domain adaptation methods require target-domain samples, limiting their use on totally unseen target domains.
- Domain generalization is evaluated on totally unseen domains, unlike domain adaptation with accessible unlabeled target images.
- Domain generalization for semantic segmentation is still in its infancy.
3. Preliminaries
The preliminaries describe instance normalization and whitening as distribution-standardization tools. They motivate semantic-aware alignment because global matching can damage semantic category relationships and segmentation boundaries.
- An intermediate feature map F has dimensions N, K, H, and W for batch sample, channel, height, and width.
- Instance Normalization standardizes each channel of each sample using its mean and standard deviation.
- Instance Normalization produces zero-mean, unit-standard-deviation channels, but channel joint distributions may remain mismatched.
- Instance Whitening decorrelates channels by making each sample’s covariance matrix close to the identity matrix.
- SAN transforms feature map F into category-level normalized features eF through a multi-branch normalization strategy.
- Whitening-based global alignment can map features to incorrect semantic categories and produce poor segmentation boundaries.
4. Proposed Method
The proposed method sequentially uses SAN for category-level center alignment and SAW for distributed alignment, aiming to preserve semantic content while improving feature discrimination across domains.
- Semantic-Aware Normalization (SAN): SAN transforms intermediate features into category-level-centered representations using segmentation masks and category-specific normalization parameters.The masks highlight category regions, while shared affine parameters map features of the same category into a common space.
- Semantic-Aware Normalization (SAN): CFR refines rough predicted category masks before regional normalization is applied within category regions.It adaptively adjusts highlighted features using pooled feature representations and a convolution-sigmoid transformation.
- Semantic-Aware Whitening (SAW): SAW performs distributed alignment on SAN’s semantic-centred features while avoiding the overly strong whitening that can damage semantic information.Unlike GIW’s fixed neighboring-channel groups, SAW rearranges channels so each group contains channels associated with different categories.
- Semantic-Aware Whitening (SAW): SAW assigns classifier-weighted channels to groups containing different-category-related channels and optimizes a whitening loss over those groups.Channel relevance is obtained from classifier weights, whose largest absolute values determine selected channel indexes for each category.
- Semantic-Aware Whitening (SAW): SAW is applied only during training because its operations do not alter the main network’s forward-pass features.SAN and SAW are sequentially embedded in the proposed network, with SAN producing the semantic-centred features used by SAW.
5. Experiments
Experiments evaluate SAN and SAW across synthetic-to-real and arbitrary-domain generalization settings, multiple datasets, and backbones. Results show consistent gains on unseen domains, retained source-domain performance, and benefits from both modules and their design choices.
- Experimental setup: Experiments cover GTA5, SYNTHIA, Cityscapes, BDDS, and Mapillary, using VGG-16, ResNet-50, and ResNet-101 across five source-to-unseen-domain settings.Each experiment is repeated three times, with average results reported; IoU is the evaluation metric.
- Domain generalization results: SAN and SAW consistently achieve the best performance across evaluated generalization settings and backbones, also comparing favorably with methods trained using target-domain data.Qualitative comparisons are provided in Figure 4, while quantitative comparisons appear in Table 1 and Appendix B.
- Source-domain performance: The method largely retains source-domain performance and performs comparably with a model trained without domain generalization.This addresses the common source-domain performance degradation associated with domain-generalization methods.
- Module ablation: SAN and SAW provide average mIoU improvements of 8.71% and 7.93%, respectively, when progressively integrated on ResNet-50 from GTA5 to four target datasets.The best performance is achieved when both modules are combined.
- SAN ablation: Removing CFR reduces SAN performance, while CFR contributes an average gain of 2.30%.This ablation supports the usefulness of Category-level Feature Refinement within SAN.
- SAW ablation: Category-related grouping in SAW reaches 37.54%, 34.97%, 39.85% and 28.46% on G → C, B, M and S, respectively.The comparison evaluates baseline IW, general grouping, and the proposed category-related grouping.
- Hyper-parameter analysis: Hyper-parameter analysis identifies C = 4 as the optimal category count for semantic-aware feature alignment.C selects the leading categories according to their proportions in the training data, subject to channel divisibility.
6. Conclusion and limitations
The paper concludes that SAN and SAW provide category-level center and distributed alignment for domain-generalized segmentation, with state-of-the-art performance. It also acknowledges remaining feature differences and dataset diversity limitations.
- Conclusion: SAN and SAW sequentially perform category-level center alignment and distributed alignment to produce domain-invariant and discriminative features.The conclusion presents these modules as the paper’s central approach to out-of-domain semantic segmentation.
- Conclusion: Comprehensive experiments report state-of-the-art performance in both domain generalization and domain adaptation.The stated conclusion summarizes the empirical effectiveness of SAN and SAW.
- Limitations: Style-invariant features may still contain source-domain-specific cues, leaving a significant performance difference between source and target domains.The paper identifies meta-learning, ensemble learning, and disentangled representation learning as possible directions for addressing this gap.
- Limitations: The evaluated datasets lack diversity and contain biases because they are mostly captured in the developed world.This limits the representativeness of the empirical evaluation.
Appendix A. Evaluation on other DG settings
The evaluation expands domain generalization beyond three commonly tested source-target settings and reports state-of-the-art performance across the additional settings and backbones.
- Evaluation scope: Existing DG methods commonly evaluate only three source-target settings, leaving two settings underexplored.The omitted settings are B → G, S, C, M and M → G, S, C, M.
- Evaluation scope: The evaluation includes generalization from B and M to the remaining domains.
- Results: The model consistently achieves state-of-the-art results across all evaluated settings and backbones.
Appendix B. Comparison with DA methods
The method is evaluated against domain adaptation approaches despite using no target-domain data, and it outperforms most compared adaptation techniques.
- Setting: Unlike DA methods, the proposed DG method requires no access to target-domain data during training.
- Results: The method outperforms the adaptation performance of most compared state-of-the-art DA techniques.The comparisons cover GTA5→Cityscapes and SYNTHIA→Cityscapes.
- Implication: Target-domain independence gives the method broader applicability than approaches requiring target-domain access.
Appendix C. Further Implementation Details
SAN and SAW are applied at early convolutional stages, where shallow features encode more style information, using the backbone architecture shown for VGG and ResNet.
- Placement: SAN and SAW are imposed after stages 1 and 2 of each backbone network.
- Rationale: The modules are placed early because shallow convolutional layers encode more style information.
- Architecture: The architecture is illustrated with VGG and ResNet backbones.
Appendix D. Computational complexity
The proposed modules add little computational cost because they operate only in the first two layers and on four main categories.
- Runtime: Training and inference time increase negligibly compared with the baseline.
- Design scope: The modules are implemented only in the first two network layers and for four main categories.The passage attributes the low overhead to this restricted placement and category count.
- Memory: Additional memory overhead remains below 2G.
Appendix E. More qualitative results
Qualitative results show that SAN and SAW remain effective under unseen weather and illumination changes. Compared with the baseline, the method produces clearer segmentation of roads and sidewalks in these conditions.
- Under GTA5 → Mapillary weather changes, the proposed approach produces fairer segmentation results than the baseline.
- Under GTA5 → BDDS illumination changes, the method identifies roads and sidewalks more clearly than the baseline.
- The qualitative comparisons demonstrate semantic-aware feature matching across multiple unseen-domain conditions.