Source-linked AI summary
Learning to Learn Single Domain Generalization
Fengchun Qiao, Long Zhao, Xi Peng
TL;DR
The paper addresses single domain generalization, where a model trained on one source domain must generalize to many unseen target domains. It proposes adversarial domain augmentation with meta-learning and a Wasserstein Auto-Encoder relaxation, and reports marginally better performance than state-of-the-art methods on several benchmarks.
Problem
Single domain generalization asks how a model trained on one source domain can generalize well to many unseen target domains.
Method
The method uses adversarial training to create challenging fictitious domains, organizes their training through meta-learning, and uses a Wasserstein Auto-Encoder to relax the worst-case constraint.
Results
The method marginally outperforms state-of-the-art methods on Digits, CIFAR-10-C, and SYTHIA benchmark datasets.
Takeaways & Limitations
M-ADA provides a meta-learning-based approach for efficiently training on augmented fictitious domains that are out-of-distribution from the source domain.
Takeaways & Limitations
The theoretical smoothness analysis assumes sufficiently large α and Lipschitzian smoothness.
Abstract
from arXiv · showhide
We are concerned with a worst-case scenario in model generalization, in the sense that a model aims to perform well on many unseen domains while there is only one single domain available for training. We propose a new method named adversarial domain augmentation to solve this Out-of-Distribution (OOD) generalization problem. The key idea is to leverage adversarial training to create "fictitious" yet "challenging" populations, from which a model can learn to generalize with theoretical guarantees. To facilitate fast and desirable domain augmentation, we cast the model training in a meta-learning scheme and use a Wasserstein Auto-Encoder (WAE) to relax the widely used worst-case constraint. Detailed theoretical analysis is provided to testify our formulation, while extensive experiments on multiple benchmark datasets indicate its superior performance in tackling single domain generalization.
1. Introduction
Single domain generalization asks models trained on one source domain to perform well across many unseen target domains, a setting not addressed by standard adaptation or multi-domain generalization. The paper proposes adversarial domain augmentation, using meta-learning and a relaxed Wasserstein constraint to create challenging fictitious domains.
- Motivation: Single domain generalization seeks good performance on many unseen target domains when only one source domain is available for training.
- Motivation: Domain adaptation typically uses target-domain data, whereas domain generalization assumes multiple source domains, leaving single domain generalization insufficiently addressed.
- Proposed approach: Adversarial domain augmentation creates fictitious yet challenging populations from one source domain for learning models with theoretical guarantees.
- Proposed approach: A Wasserstein Auto-Encoder relaxes the worst-case constraint to facilitate larger domain transportation during domain augmentation.
- Contribution: A meta-learning-based scheme organizes training of augmented domains, and experiments marginally outperform state-of-the-art methods on Digits, CIFAR-10-C, and SYTHIA.
2. Related Work
Related work addresses domain discrepancy through adaptation and generalization methods, while adversarial training improves robustness and meta-learning supports rapid adaptation. These lines of work motivate combining adversarial perturbations with meta-learning for single domain generalization.
- Domain discrepancy: Domain adaptation reduces source–target discrepancy using unsupervised, supervised, or few-shot access to target-domain data.
- Domain generalization: Domain generalization learns from multiple source domains without target access, using domain-invariant representations, domain-specific modules, or domain perturbations.
- Adversarial training: Adversarial training was developed to improve model robustness against adversarial perturbations and attacks, including through distributionally robust optimization.
- Meta-learning: Meta-learning studies how to learn new concepts or tasks quickly from few training examples and has been applied to optimization and few-shot classification.
3. Method
M-ADA addresses single-domain generalization by generating challenging fictitious domains through adversarial augmentation, relaxing semantic constraints with a WAE, and organizing training through meta-learning.
- Adversarial Domain Augmentation: M-ADA uses adversarial training to generate fictitious, challenging domains from one source domain for worst-case generalization.The task model is trained to generalize from these augmentations to unseen target domains.
- Adversarial Domain Augmentation: The task model maps inputs into an embedding space with a feature extractor and predicts labels using a classifier.Latent representations are defined as z = F(x).
- Adversarial Domain Augmentation: Lconst enforces semantic consistency and bounds Wasserstein domain distance, while Lrelax encourages larger transportation beyond the source domain.Together, the terms push augmentations away in input space while preserving their embedding-space relationship to the source.
- Adversarial Domain Augmentation: A WAE is pretrained on the source domain, frozen during augmentation, and used through reconstruction error to implement Lrelax.The WAE uses an encoder, decoder, and divergence metric between the encoded distribution and a prior.
- Meta-Learning Single Domain Generalization: Meta-learning performs meta-training on the source domain and meta-testing on augmented domains before updating one shared model.The procedure uses K augmented domains and is intended to prepare the model for fast adaptation to new targets.
- Meta-Learning Single Domain Generalization: The method targets a single efficient model rather than an ensemble and supports fast adaptation with one or a small number of gradient steps.The paper reports use for few-shot domain adaptation.
4. Theoretical Understanding
The analysis formulates adversarial domain augmentation as a relaxed worst-case problem using Wasserstein distances and a Lagrangian surrogate. It establishes smoothness and identifies the adversarial perturbation used in the method.
- The overall loss in Eq. (3) is directly derived from a relaxed worst-case problem.
- The admissible augmented domains remain within ρ of S in embedding space while being at least η away in input space.
- For deep neural networks, the intractable constrained problem is transformed into a Lagrangian relaxation with fixed penalties α ≥0 and β ≥0.
- The relaxed objective uses αWc(S, S+) −βWd(S, S+) and minimizes the robust surrogate φα,β.
- Under sufficient α and Lipschitzian smoothness, φα,β remains smooth with respect to θ, enabling gradient computation.
- The maximizing x⋆ is exactly the adversarial perturbation defined in Eq. (3).
5. Experiments
Experiments evaluate M-ADA across single-domain generalization benchmarks, ablations, efficiency, and few-shot adaptation. Results support improved robustness, training stability, and accuracy across several unseen-domain settings.
- Datasets and settings: Digits, CIFAR-10-C, and SYTHIA evaluate generalization from one source domain to unseen domains, corruptions, or driving-scene conditions.Digits uses MNIST as source; CIFAR-10-C uses clean CIFAR-10 as source; SYTHIA evaluates location and environmental shifts.
- Ablation study: 58.9% improvement in Wasserstein distance follows introducing Lrelax, increasing the distance between source and augmented domains from 0.032 to 0.078.t-SNE visualization also shows that augmented domains with Lrelax cover a larger region and better overlap unseen domains.
- Ablation study: 0.94% and 1.37% average-accuracy gains on Digits and CIFAR-10-C, respectively, result from adding the meta-learning scheme.Meta-learning also reduces variance and improves performance across all severity levels of Impulse Noise and Shot Noise.
- Ablation study: K = 3 and β = 2.0 × 10^3 produce the best classification accuracy in hyper-parameter tuning.Increasing K beyond three raises instability, while larger β can place augmented domains too far from the source manifold.
- Evaluation of single domain generalization: M-ADA outperforms GUD on Digits with fewer model parameters and faster inference speed.The strongest gains occur on SVHN, MNIST-M, and SYN, while improvement on USPS is smaller because it resembles MNIST.
- Evaluation of single domain generalization: M-ADA improves CIFAR-10-C robustness across corruption severities and obtains the lowest mCE and RmCE values.It exceeds ERM by more than 10% on several severe corruptions, including Snow, Glass blur, Pixelate, and noise-related corruptions.
6. Conclusion
M-ADA addresses single domain generalization by efficiently organizing adversarially created fictitious domains that are out-of-distribution from the source domain.
- M-ADA uses meta-learning to organize training over adversarially created fictitious domains that are OOD from the source domain.
Appendix A. Experimental Details
The appendix details dataset-specific task models and WAE implementations for Digits, CIFAR-10-C, and SYTHIA, along with their training schedules.
- The three datasets use task models and training strategies tailored to their characteristics.
- Task models: Digits uses a convolutional classifier with two convolutional layers, two max-pooling layers, two 1024-unit fully connected layers, and a 10-class softmax.
- Task models: CIFAR-10-C uses a 16-layer Wide Residual Network with width 4 and convolutional channel groups of 64, 128, and 256.
- Task models: SYTHIA uses FCN-32s with a ResNet-50 backbone, a 14-channel prediction layer, and bilinear up-sampling.
- Wasserstein Auto-Encoders: WAEs are implemented with dataset-specific architectural modifications.
- Wasserstein Auto-Encoders: Digits uses fully connected encoder, decoder, and discriminator networks with latent size 20 and discriminator output size 1.
- Wasserstein Auto-Encoders: CIFAR-10-C uses convolutional and fully connected encoder layers, fully connected and deconvolutional decoder layers, batch normalization, and a two-layer discriminator.
- Wasserstein Auto-Encoders: SYTHIA uses convolutional and fully connected encoder layers, fully connected and deconvolutional decoder layers, batch normalization, and a three-layer discriminator.
B.1. Ablation Study
The ablation study finds that the meta-learning scheme improves M-ADA’s stability and accuracy across unseen corruption severities, while performance peaks at three augmented domains.
- Validation of meta-learning scheme: M-ADA significantly reduces variance and improves performance across five severity levels for four unseen corruption types.The evaluated corruptions are Gaussian, Speckle, Impulse, and Shot Noise.
- Validation of meta-learning scheme: The results identify meta-learning as important for training stability and classification accuracy during adversarial domain augmentation.
- Hyper-parameter tuning: Accuracy reaches its summit when the number of augmented domains is K = 3 and falls as K increases further.
B.2. Comparison of Different Lrelax
The study compares relaxation terms for domain augmentation and reports that Wasserstein-based reconstruction-space distances provide a more reliable relaxation effect and improved accuracy.
- Relaxation terms: The reconstruction error Lrelax = ∥x+ − V(x+)∥2 indicates whether an augmented sample lies in the same distribution as the source sample.
- Relaxation terms: WAE’s Wasserstein metric measures distribution distance between input and reconstruction, making WAE instead of vanilla AE a key design choice for domain augmentation.
- Relaxation terms: The distance ∥V(x) − V(x+)∥2 provides a better relaxation effect and improved accuracy than the alternative reconstruction-space formulation.The reconstruction space is described as more reliable because a Wasserstein prior is applied there.
- Relaxation terms: The comparison is reported in Table 7, which evaluates accuracy under different relaxation terms.
B.3. Comparison on CIFAR-10-C
M-ADA is evaluated by training on clean CIFAR-10 and testing on 19 unseen CIFAR-10-C corruptions at the severest level. It substantially outperforms competing methods on most corruptions and achieves the lowest mCE and relative mCE.
- M-ADA is trained on clean CIFAR-10 and evaluated across 19 unseen CIFAR-10-C corruption domains at severity level 5.The evaluation reports classification accuracy for each corruption, along with mCE and relative mCE.
- M-ADA substantially outperforms other methods on most corruptions.
- M-ADA exceeds ERM by more than 10% on Frost, Glass blur, Gaussian blur, Pixelate, and several noise-related corruptions.
- M-ADA has the lowest mCE and relative mCE, indicating strong robustness against image corruptions.For both metrics, lower values are better.
- The selected hyperparameters are K = 3, α = 1.0, and β = 2.0 × 10^3 according to the best accuracy.