Source-linked AI summary
Learning to Optimize Domain Specific Normalization for Domain Generalization
Seonguk Seo, Yumin Suh, Dongwan Kim, Geeho Kim, Jongwoo Han, Bohyung Han
TL;DR
Domain generalization must learn transferable representations without target-domain examples, while normalization choices can struggle with domain shifts or reduce semantic discrimination. The paper introduces DSON, which learns domain-specific mixtures of batch and instance normalization with separate affine parameters and statistics. It reports state-of-the-art accuracy across standard benchmarks, including under substantial label noise, and shows viability for unsupervised domain adaptation.
Problem
Domain generalization seeks transferable representations for unseen domains without accessing target-domain examples during training, while batch normalization can be limited by domain shift and instance normalization can reduce semantic discrimination.
Method
DSON combines multiple normalization methods by learning separate affine parameters, statistics, and mixture weights for each source domain while sharing other network parameters.
Results
The algorithm achieves state-of-the-art accuracy consistently on multiple standard benchmarks, including with substantial label noise, and is well-suited for unsupervised domain adaptation.
Takeaways & Limitations
Domain-specific optimized normalization provides a domain generalization technique that constructs domain-invariant representations by combining normalization methods and removing domain-specific style information.
Takeaways & Limitations
The approach assumes that reasonably good target-domain representations can be obtained from information in source domains alone because target-domain information is unavailable during training.
Abstract
from arXiv · showhide
We propose a simple but effective multi-source domain generalization technique based on deep neural networks by incorporating optimized normalization layers that are specific to individual domains. Our approach employs multiple normalization methods while learning separate affine parameters per domain. For each domain, the activations are normalized by a weighted average of multiple normalization statistics. The normalization statistics are kept track of separately for each normalization type if necessary. Specifically, we employ batch and instance normalizations in our implementation to identify the best combination of these two normalization methods in each domain. The optimized normalization layers are effective to enhance the generalizability of the learned model. We demonstrate the state-of-the-art accuracy of our algorithm in the standard domain generalization benchmarks, as well as viability to further tasks such as multi-source domain adaptation and domain generalization in the presence of label noise.
1 Introduction
The paper targets domain generalization by designing domain-specific normalization layers that balance domain-invariant representations with semantic discrimination. DSON combines heterogeneous normalization methods and reports state-of-the-art accuracy on standard benchmarks.
- Motivation: Domain generalization trains models on multiple source domains without target examples, aiming for transferable representations that work in unseen domains.This setting is more challenging than unsupervised domain adaptation because target-domain data are unavailable during training.
- Motivation: Batch normalization can regularize heterogeneous-domain representations, but its benefit is limited under significant domain shift and domain-specific styles.The paper motivates normalization-layer design as an architectural route to domain generalization.
- Approach: DSON optimizes the normalization combination separately for each domain while learning separate normalization parameters, statistics, and mixture weights.Other network parameters, including convolutional layers, remain shared across domains.
- Approach: The approach combines heterogeneous normalization methods, using instance normalization to remove domain-specific information while retaining semantically discriminative features.The contribution specifically frames instance normalization as balancing cross-category variance and domain invariance.
- Results: DSON achieves state-of-the-art accuracy on multiple standard benchmark datasets and outperforms established normalization methods.This is presented as a principal contribution of the proposed domain-specific normalization technique.
2 Related Work
The related work places DSON among domain generalization methods based on losses, architectures, and meta-learning, alongside multi-source adaptation and neural-network normalization techniques. It distinguishes DSON by optimizing domain-specific combinations of normalization types for generalization.
- Domain Generalization: Domain generalization methods include loss-function approaches that encourage representations to generalize across domains.The related-work discussion identifies domain-invariant representation learning as one major category.
- Multi-Source Domain Adaptation: Multi-source domain adaptation uses multiple source domains plus unlabeled target examples, placing it between domain adaptation and domain generalization.The related work reviews adversarial, mixture-of-experts, and domain-specific batch-normalization approaches for this setting.
- Normalization: Batch normalization operates over mini-batch channel activations, whereas instance normalization performs the corresponding operation per instance.These normalization techniques were originally developed to regularize models and improve generalization performance.
- Normalization: BIN, SN, and SSN combine multiple normalization types, while DSBN uses separate batch-normalization layers for each domain.BIN combines batch and instance normalization; SN additionally includes layer normalization.
3 Domain-Specific Optimized Normalization for Domain Generalization
DSON learns domain-specific normalization choices and parameters while sharing the remaining feature-extractor parameters across source domains. It combines batch and instance normalization to balance semantic discriminability with domain invariance, then ensembles domain-specific predictions for unseen targets.
- Overview: DSON trains domain-specific classifiers and ensembles their predictions in a joint embedding space intended to remain valid for unseen target domains.Feature extractors share parameters except in normalization layers, while each source domain has its own classifier pathway.
- Normalization Motivation: Fine-tuning batch-normalization parameters in cross-domain training can degrade target-domain generalization by overfitting to source domains.The PACS ResNet-18 experiment compares fixed BN statistics and parameters with fine-tuned BN settings.
- Instance Normalization for Domain Generalization: Instance normalization reduces domain-specific style information and overfitting, but also reduces inter-class variance and feature discriminability relative to batch normalization.The paper motivates combining IN and BN to address this trade-off.
- Domain-Specific Optimized Normalization: DSON transforms domain-specific channel-wise whitened activations using domain-specific affine parameters γ_d and β_d.The whitening uses domain-specific means and variances before the affine transformation.
- Domain-Specific Optimized Normalization: The method optimizes each domain’s mixture of normalization techniques and separate normalization parameters while sharing convolutional parameters across domains.The domain-specific mixture weights are shared across layers within each domain and are optimized with the source-domain classification losses.
- Inference: For inference on an unknown target domain, the example is passed through all source-domain feature extractors, and their logits are averaged before softmax.The method assigns separate instance-normalization mixture weights to source domains and aggregates their predictions.
4 Experiments
The experiments evaluate DSON across three domain generalization benchmarks using standard protocols and ResNet backbones. Mixture weights are shared across layers to improve optimization and accuracy.
- Benchmarks: DSON is evaluated on PACS, Office-Home, and five digit-recognition datasets under domain generalization protocols.PACS and Office-Home use three source domains for training and the remaining domain for testing; the digit experiments use MNIST, MNIST-M, USPS, SVHN, and Synthetic Digits.
- Implementation: Experiments use ResNet backbones with ImageNet-pretrained convolutional and batch-normalization layers.The setup is chosen for fair comparison with prior methods.
- Evaluation: Tables 2 and 3 compare domain generalization accuracy on PACS and Office-Home, respectively, with target domains represented by columns.The PACS comparison includes ResNet-18 and ResNet-50, while Office-Home uses ResNet-18.
- Implementation: The learning rate follows ηp = η0 (1+αp)^β with η0 = 0.02, α = 10, and β = 0.75 over 10K SGD-M iterations.Training uses momentum 0.9 and a batch size of 32 images per source domain.
- Implementation: Sharing mixture weights across all layers substantially and consistently improves accuracy in all settings.The authors report slower convergence for local mixture weights in lower layers than in higher layers.
4.2 Comparison with Other Methods
DSON is compared with established domain generalization methods through PACS accuracy and normalization-layer ablations. The evaluation emphasizes comparisons across target domains and normalization variants.
- Normalization ablations: Table 4 compares DSON with normalization variants, including baseline BN, DSBN, SN, and IBN-Net, using domain generalization accuracy.The caption identifies Art. as the Art painting domain in PACS.
- PACS comparison: DSON outperforms the baseline and other state-of-the-art methods on PACS, particularly on the hard Sketch domain.The advantage remains when ResNet-50 is used, indicating scalability and stability in the reported experiments.
PACS
On Office-Home, DSON outperforms the baseline and recently proposed methods, achieving the best score on every target domain. Its advantage is especially pronounced on the hard Clipart domain.
- Office-Home: DSON outperforms JiGen, D-SAM, and the baseline on the Office-Home dataset.The comparison evaluates target-domain performance using the results presented in Table 3.
- Office-Home: DSON achieves the best score on all Office-Home target domains and is especially advantageous on Clipart.
4.3 Ablation Study
The ablation study examines domain-specific normalization and mixtures of normalization methods, while additional experiments test digit recognition and robustness to label noise. DSON consistently shows strong reported performance across these settings.
- 4.3 Ablation Study: The ablation study evaluates DSON against baseline BN, DSBN, SN, IBN-Net, and other normalization implementations on PACS and Office-Home.It assesses both domain-specific normalization and optimization of multiple normalization methods.
- 4.3 Ablation Study: Other normalization methods can degrade performance relative to baseline depending on the dataset, whereas DSON consistently reports superior results.
- Digits: 87.32% average accuracy is achieved on five digit-recognition datasets, outperforming all other baselines by large margins.
Digits Dataset
The paper evaluates DSON beyond standard domain generalization, including multi-source domain adaptation, with domain-specific normalization outperforming domain-agnostic alternatives.
- Multi-Source Domain Adaptation: DSON outperforms both the baseline and DSBN in multi-source domain adaptation on PACS.All compared normalization methods improve over the baseline, while DSON exceeds the domain-specific DSBN model.
- Multi-Source Domain Adaptation: Domain-specific models consistently outperform their domain-agnostic counterparts in multi-source domain adaptation.
- Multi-Source Domain Adaptation: DSON can be extended to multi-source domain adaptation when unlabeled target-domain data are available.
4.5 Analysis
The analysis shows that DSON increases instance-normalization usage when more source domains are available and benefits from domain-specific mixture weights.
- Mixture Weights: The average IN-to-BN mixture-weight ratios are 3:7 for PACS and 1:9 for Office-Home.
- Mixture Weights: Training with multiple source domains causes a large and consistent increase in instance-normalization usage across domains.The multi-source setting requires more domain-invariant representations than the single-source setting.
- Mixture Weights: Domain-specific mixture weights improve performance over weights shared across domains in multi-source domain generalization.
Effects of Instance Normalization
Combining instance and batch normalization trades some same-domain accuracy for substantial gains in most cross-domain settings.
- Effects of Instance Normalization: DSON significantly improves performance over BN in most cross-domain source-target scenarios.
- Effects of Instance Normalization: DSON marginally sacrifices accuracy compared with BN when source and target domains are the same.
- Effects of Instance Normalization: Integrating single-domain branches provides consistent performance gains on PACS.The individual branches differ only slightly in accuracy, while their integration improves results consistently.
5 Conclusion
The conclusion presents DSON as a domain-specific optimized-normalization method that achieves strong domain generalization and extends to adaptation and label-noise settings.
- 5 Conclusion: DSON combines multiple normalization methods with separate affine parameters and weighted normalization statistics for each domain.
- 5 Conclusion: Instance normalization helps learn domain-invariant representations by removing domain-specific style while preserving semantic category information.
- 5 Conclusion: DSON achieves state-of-the-art accuracy on multiple standard benchmarks, including under substantial label noise.
- 5 Conclusion: The algorithm is also well-suited for unsupervised domain adaptation.