Source-linked AI summary
Adversarially Adaptive Normalization for Single Domain Generalization
Xinjie Fan, Qifei Wang, Junjie Ke, Feng Yang, Boqing Gong, Mingyuan Zhou
TL;DR
Single domain generalization addresses generalization to unseen domains when only one source domain is available, while normalization statistics remain underinvestigated. ASR-Norm learns adaptive standardization and rescaling statistics and consistently improves ADA approaches across three benchmarks.
Problem
Single domain generalization seeks models that generalize to unseen domains using data from only one source domain, while normalization statistics remain underinvestigated.
Method
ASR-Norm uses neural networks to learn standardization and rescaling statistics adaptively for each input sample and complements adversarial domain augmentation.
Results
ASR-Norm consistently improves state-of-the-art ADA approaches by 1.6%, 2.7%, and 6.3% on Digits, CIFAR-10-C, and PACS, respectively.
Takeaways & Limitations
The improvement is agnostic to the ADA method and becomes more significant as the domain discrepancy increases.
Abstract
from arXiv · showhide
Single domain generalization aims to learn a model that performs well on many unseen domains with only one domain data for training. Existing works focus on studying the adversarial domain augmentation (ADA) to improve the model's generalization capability. The impact on domain generalization of the statistics of normalization layers is still underinvestigated. In this paper, we propose a generic normalization approach, adaptive standardization and rescaling normalization (ASR-Norm), to complement the missing part in previous works. ASR-Norm learns both the standardization and rescaling statistics via neural networks. This new form of normalization can be viewed as a generic form of the traditional normalizations. When trained with ADA, the statistics in ASR-Norm are learned to be adaptive to the data coming from different domains, and hence improves the model generalization performance across domains, especially on the target domain with large discrepancy from the source domain. The experimental results show that ASR-Norm can bring consistent improvement to the state-of-the-art ADA approaches by 1.6%, 2.7%, and 6.3% averagely on the Digits, CIFAR-10-C, and PACS benchmarks, respectively. As a generic tool, the improvement introduced by ASR-Norm is agnostic to the choice of ADA methods.
1. Introduction
Single domain generalization trains on one source domain while targeting unseen domains with different visual presentations. The paper identifies normalization statistics as an underexplored limitation of adversarial domain augmentation and proposes ASR-Norm to make them adaptive.
- Single domain generalization trains on one source domain and evaluates generalization to unseen domains with different visual presentations.
- Existing approaches use adversarial domain augmentation to synthesize challenging virtual domains and learn domain-invariant features.
- BN can mismatch training and testing computation because it applies exponential moving averages of training statistics during inference.
- On CIFAR-10-C, BN underperforms IN as domain discrepancy increases, while BN-Test(16) requires a batch size of 16 during inference.
- ASR-Norm learns adaptive standardization and rescaling statistics for each input and can be viewed as a generic form of traditional normalizations.
- ASR-Norm consistently improves state-of-the-art ADA approaches across three benchmarks, independently of the ADA method, with larger gains as domain discrepancy increases.
2. Related work
Related work distinguishes single domain generalization from domain adaptation and multi-source generalization, emphasizing the challenge of learning from one source domain without target-domain access. Existing approaches use adversarial augmentation and increasingly flexible normalization to address distributional shifts.
- Domain generalization seeks domain-invariant representations without access to target domains, unlike domain adaptation, which uses unlabeled target data.
- Single domain generalization is a more challenging setting than multi-source generalization because training uses one source domain.
- Adversarial domain augmentation improves robustness by synthesizing virtual images or challenging transformations during training.
- Adversarial training for this paper targets natural, perceptible domain shifts rather than imperceptible adversarial perturbations.
- Normalization research has generalized fixed statistics through combinations such as BIN and SN, but restrictive forms remain insufficiently adaptive.
3.1. Background
The background formulates single domain generalization as classification on an unavailable target distribution and motivates robust optimization around the source distribution. It then describes normalization as standardization followed by rescaling, with statistics determined differently across normalization methods.
- Single Domain Generalization Problem Setup: Single domain generalization trains on source data from Ps and aims to classify target-domain images from an unavailable distribution Pt.
- Single Domain Generalization Problem Setup: Empirical risk minimization on the source domain alone can be sub-optimal for generalization to unseen domains.
- Single Domain Generalization Problem Setup: The robust objective considers worst-case distributions within a distance ρ of Ps, but direct optimization is generally difficult.
- Adversarial Domain Augmentation: The robust objective is converted into a Lagrangian min-max problem with penalty parameter η for approximate optimization.
- Adversarial Domain Augmentation: Adversarial augmentation maximizes prediction loss while maintaining semantic closeness to the original image, then alternates with minimizing loss on original and augmented data.
- Standardization and Rescaling: Normalization standardizes activations using mean and standard deviation, then rescales them with γ and β.
- Standardization and Rescaling: BN computes statistics per batch, whereas IN, GN, and LN compute sample statistics over different channel groupings; SN combines normalization statistics with learnable weights.
3.2. ASR-Norm: Adaptive Standardization and Rescaling Normalization
ASR-Norm uses neural networks to learn adaptive standardization and rescaling statistics from sample statistics, providing a generic form that encompasses several traditional normalization methods. Its residual and sample-wise design supports stable training and consistent behavior between training and testing.
- ASR-Norm overview: ASR-Norm learns both standardization and rescaling statistics with auto-encoder structured neural networks for each input sample.The method maps channel-wise statistics to learned normalization statistics and constructs rescaling parameters from the original statistics.
- Generic normalization form: The learned normalization form generalizes BN, DSBN, GN, IN, LN, and SN by replacing their restrictive statistic functions with learned functions.Constant, domain-wise, group-wise, or weighted-combination functions recover the corresponding traditional normalization forms.
- Adaptive standardization: Neural networks for standardization learn channel-wise mean and standard deviation transformations through encoder-decoder mappings with a bottleneck.The encoders extract cross-channel information, while decoders produce per-channel statistics; the standard deviation output is constrained to be non-negative.
- Residual learning: Residual learning combines learned standardization statistics with original statistics, allowing gradual transition from stable initial behavior to learned normalization.Learnable weights are initialized near zero and increase the contribution of learned statistics during training.
- Adaptive domain behavior: ASR-Norm computes statistics independently for each sample, removes batch dependencies during standardization, and preserves the same computational graph during training and testing.When coupled with adversarial domain augmentation, its statistics adapt to data from different domains; the approach is agnostic to the ADA method.
- Adaptive rescaling: Adaptive rescaling predicts sample-dependent β and γ from the original mean and standard deviation statistics instead of applying uniform parameters to all samples.Bounded activations constrain the rescaling statistics, while learned biases are initialized like traditional rescaling parameters.
4. Experiments
Experiments evaluate ASR-Norm on Digits, CIFAR-10-C, and PACS under single- and multi-source domain-generalization settings. Across benchmarks, ASR-Norm improves ADA-based methods, with larger gains under more challenging domain shifts and with both adaptive standardization and rescaling contributing.
- Experimental settings: Experiments use Digits, CIFAR-10-C, and PACS to evaluate generalization across unseen domains and corruption levels.Digits uses MNIST as the source domain; CIFAR-10-C varies corruption types and intensities; PACS evaluates both single-source and multi-source settings.
- Digits: ASR-Norm improves performance on challenging Digits domains, including SVHN and SYN, while ADA-based methods are weaker on USPS because it closely resembles MNIST.The method outperforms baseline and state-of-the-art methods on average.
- CIFAR-10-C: ASR-Norm significantly improves ADA and other comparison methods on CIFAR-10-C, with larger gains at more intense corruption levels.The comparison includes improved ADA methods, self-supervised training, test-time training, and PGD training.
- PACS: ASR-Norm improves RSC on PACS, especially on challenging domains, and outperforms reported state-of-the-art methods in the multi-source setting without domain labels.The multi-source comparison also exceeds methods using domain labels, including DSON and MetaReg.
- Normalization analysis: ASR-Norm outperforms BN, SN, and IN on CIFAR-10-C, with gains increasing as corruption rises; adaptive standardization and rescaling both contribute.Learning statistics with neural networks performs better than learning only combination weights for standardization statistics.
- Normalization analysis: ASR-Norm improves uncertainty prediction over BN and IN, with the improvement increasing as domain discrepancy rises.The reported benefit concerns both class predictions and predictive distributions.
- Normalization analysis: BN performs better than IN and ASR-Norm on in-domain CIFAR-10, indicating a trade-off between in-domain performance and cross-domain generalization.The authors attribute BN’s in-domain advantage to dependencies between training samples.
5. Conclusion
ASR-Norm is an adaptive, general normalization method that learns standardization and rescaling statistics with auto-encoder structured neural networks. It complements ADA approaches and consistently improves single domain generalization, with gains that are agnostic to the ADA method and increase with domain discrepancy.
- ASR-Norm learns both standardization and rescaling statistics with auto-encoder structured neural networks.
- ASR-Norm complements diverse ADA approaches by adapting normalization statistics to data from different domains.
- ASR-Norm consistently improves state-of-the-art ADA approaches across three standard benchmarks.
- Performance gains are agnostic to the ADA method and become more significant as domain discrepancy increases.
A. Additional Experimental Results
Additional experiments examine normalization effects, learned statistics, corruption types, and statistical significance. They report stronger uncertainty and generalization behavior for ASR-Norm in settings with greater domain discrepancy, while learned statistics become increasingly favored during training and cluster by domain.
- On the Effect of Normalization: Adding BN or BN-test to ConvNet worsens Digits performance relative to the no-normalization baseline.
- On the Effect of Normalization: Table 9 reports Digits accuracies, standard deviations, and p-values for each testing domain after training on MNIST.
- Statistical significance: ASR-Norm improvements over M-ADA, SN, AR, and AS are statistically significant across different CIFAR-10-C corruption levels.
- Analysis of Residual Learning: Learned standardization and rescaling statistics receive increasing adaptive weights throughout PACS training.
- Visualization of Learned Statistics: ASR-Norm learns domain-specific patterns of standardization statistics, visible as clustering structures across PACS domains.
- CIFAR-10-C Results for Different Corruption Types: CIFAR-10-C evaluates performance across 19 corruption types grouped into noise, blur, weather, and digital categories.
B. Detailed Formulation of Adversarial Domain Augmentation
ADA approximately optimizes a robust objective by expanding training data with synthesized adversarial examples. Its distributional distance is defined through a Wasserstein formulation using a learned distance over the input-label space and semantic neural-network features, with gradients estimated approximately by a min-max algorithm.
- ADA approximately optimizes a robust objective by expanding the training set with synthesized adversarial examples during training.
- The distance between distributions P and Q is defined using the Wasserstein distance over couplings between the two distributions.
- The learned distance cθ operates over the joint input-label space X × Y.
- ADA measures cθ using semantic features learned by the neural network.
- A min-max algorithm estimates the gradients approximately under the formulation’s stated conditions.
C. Additional Experimental Settings
The Digits benchmark trains only on MNIST and evaluates generalization to other digit domains. SVHN, MNIST-M, and SYN represent larger distributional shifts from MNIST than USPS.
- SVHN, MNIST-M, and SYN are more challenging than USPS because they have larger distributional shifts from MNIST.
- Digits single domain generalization uses MNIST for training and evaluates on SVHN, MNIST-M, SYN, and USPS.