Source-linked AI summary
Domain Generalization: A Survey
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, Chen Change Loy
TL;DR
Machine learning systems often rely on i.i.d. source and target data, leaving them vulnerable to domain shift and OOD performance degradation. This survey reviews domain generalization, which learns from source data without target access, covering its definitions, methods, theories, evaluations, applications, and future directions. It synthesizes a decade of progress while identifying limitations in current theory, meta-learning, and benchmarks.
Problem
Most learning algorithms rely on the i.i.d. assumption and can suffer significant performance drops on OOD target domains, while target data is not always accessible for adaptation.
Method
The paper conducts a comprehensive literature review covering DG definitions and related fields, existing methods and theories, evaluation practices, and research directions.
Results
The survey synthesizes a decade of DG progress across methods and application areas, including domain alignment, meta-learning, data augmentation, ensemble learning, and self-supervised learning.
Takeaways & Limitations
DG is a broad research area addressing OOD generalization through source-only learning across diverse methods, theories, evaluation settings, and applications.
Takeaways & Limitations
Current DG theory often depends on specific model classes or strong assumptions, while benchmarks mainly represent homogeneous domain shifts unlike unpredictable real-world shifts.
Abstract
from arXiv · showhide
Generalization to out-of-distribution (OOD) data is a capability natural to humans yet challenging for machines to reproduce. This is because most learning algorithms strongly rely on the i.i.d.~assumption on source/target data, which is often violated in practice due to domain shift. Domain generalization (DG) aims to achieve OOD generalization by using only source data for model learning. Over the last ten years, research in DG has made great progress, leading to a broad spectrum of methodologies, e.g., those based on domain alignment, meta-learning, data augmentation, or ensemble learning, to name a few; DG has also been studied in various application areas including computer vision, speech recognition, natural language processing, medical imaging, and reinforcement learning. In this paper, for the first time a comprehensive literature review in DG is provided to summarize the developments over the past decade. Specifically, we first cover the background by formally defining DG and relating it to other relevant fields like domain adaptation and transfer learning. Then, we conduct a thorough review into existing methods and theories. Finally, we conclude this survey with insights and discussions on future research directions.
1 INTRODUCTION
Domain generalization addresses the domain shift that undermines models trained under the i.i.d. assumption by learning from source data to generalize to inaccessible or unknown target domains. This survey reviews DG methods, theories, applications, and future directions.
- Motivation: Domain shift between training and test distributions causes machine learning models to face OOD generalization challenges.Deep learning models can degrade significantly even under small variations in the data-generating process.
- Motivation: Domain adaptation addresses shift by using target data, but that assumption fails when target data is difficult to obtain or unknown before deployment.This limitation arises in applications such as biomedical diagnosis, traffic-scene segmentation, and data streams.
- Domain Generalization: Domain generalization learns from one or more related source domains so models can generalize to OOD target domains without target data during training.DG was introduced to address both domain shift and the absence of accessible target data.
- Research Landscape: DG research has developed methods based on domain alignment, meta-learning, and domain-synthesis augmentation, alongside studies across multiple application areas.Applications include computer vision, speech recognition, natural language processing, medical imaging, and reinforcement learning.
- Survey Contribution: The survey provides a comprehensive technical review of DG learning algorithms developed over the last decade and discusses future research directions.It aims to summarize methodological developments and offer insights into potential directions for future work.
2 BACKGROUND
This section defines domain generalization as learning from one or more distinct source domains to perform well on unseen target domains, and surveys its settings, evaluation, and applications. It distinguishes multi-source from single-source DG and catalogs benchmarks across vision, speech, segmentation, re-identification, and face analysis.
- 2.2 Problem Definition: Domain generalization was formally introduced as learning from source domains while withholding target data during model learning.Its objective is to minimize prediction error on an unseen target domain.
- 2.2 Problem Definition: Multi-source DG uses multiple related but distinct domains, whereas single-source DG assumes homogeneous training data from one domain.Single-source DG is closely related to out-of-distribution robustness and does not require domain labels.
- 2.3 Datasets and Applications: DG benchmarks span applications including computer vision, speech recognition, medical imaging, semantic segmentation, person re-identification, and face recognition.Speech Command treats speakers as domains and contains 1,888 domains with around 65,000 samples.
- 2.3 Datasets and Applications: Common visual domain shifts include changes in font style, color, background, environment, viewpoint, image style, and weather conditions.Examples include digit datasets, VLCS and Office-31, PACS and related style benchmarks, and synthetic-to-real segmentation datasets.
- 2.4 Evaluation: Evaluation commonly reports average performance across held-out domains or worst-case performance among them.Worst-case performance is often used for subpopulation shift, while average performance is common in most domain-shift scenarios.
3 METHODOLOGIES: A SURVEY
DG methodologies are organized around learning from source domains without target data, with domain alignment as a central strategy. Alignment methods differ in what distributions they align and which statistical or adversarial techniques they use, while source-only alignment raises unresolved generalization concerns.
- Methodology Overview: Existing DG methods are categorized by methodology and motivation, with domain-label requirements distinguishing multi-source from single- and multi-source applicability.The survey notes that most methods target multi-source DG, although some do not explicitly require domain labels.
- Domain Alignment: Alignment can operate in feature space, classifier output, or both, and requires choices about which distributions to align and how to measure their differences.The survey discusses statistical distances such as ℓ2, f-divergences, and Wasserstein distance.
- Domain Alignment: Domain alignment minimizes differences among source domains to learn domain-invariant representations expected to generalize to unseen target shifts.It is applied across applications including object recognition, action recognition, face antispoofing, and medical imaging.
- What to Align: A common DG assumption is that only P(X) shifts while P(Y|X) remains stable, motivating marginal alignment; alternative methods align P(X|Y) or posteriors under different causal assumptions.Some approaches also allow P(Y) to change, addressing heterogeneous DG.
- How to Align: Distribution alignment uses moments, contrastive loss, KL divergence, MMD, adversarial learning, multi-task learning, and combinations of explicit distances with regularization.Contrastive objectives pull together same-class samples from different domains and push apart different-class samples.
- Limitations: Because DG cannot access target data, source-domain alignment does not guarantee robustness to unseen shifts, creating a need for theories explaining when alignment improves generalization.The survey identifies this as a central concern for alignment-based DG.
3.2 Meta-Learning
Meta-learning for DG simulates domain shift during training by separating source domains into meta-source and meta-target sets. It offers a way to optimize models for cross-domain performance, but remains constrained by source diversity and computational demands.
- Computational Considerations: Second-order differentiation through base-model updates creates efficiency and memory-consumption issues for large neural networks.MAML-style training uses this differentiation to learn an initialization close to a target-task solution.
- Motivation: Meta-learning exposes DG models to simulated domain shifts so they can better handle shifts in unseen domains, but existing methods require multi-source data with domain labels.Its motivation is to train models across related episodes rather than only on pooled source data.
- Design Components: Meta-learning DG methods carefully design episodes and meta-representations, respectively determining how samples are organized and what model components are learned.Meta-representations may include entire networks, regularization parameters, stochastic networks, or task-specific models and losses.
- Episodes Construction: The common episode construction divides source domains into nonoverlapping meta-source and meta-target sets, then uses bi-level optimization to reduce meta-target test error after meta-source updates.This paradigm simulates domain shift using held-out source domains.
- Limitations and Directions: Meta-learning remains limited by robustness to only simulated source shifts, especially when source domains lack diversity, and recent work reports lower performance than direct source-data augmentation.The survey suggests combining meta-learning with augmentation or improving meta-representations, meta-optimizers, and meta-objectives.
3.3 Data Augmentation
Data augmentation expands source training data through label-preserving transformations intended to simulate domain shift. Existing methods are organized by whether transformations use hand-engineered operations, adversarial gradients, neural networks, or intermediate features.
- Data augmentation adds transformed pairs (A(x), y) to original (x, y) pairs, typically using label-preserving transformations.It is used to regularize training, reduce overfitting, and improve generalization.
- Image Transformations: Methods based on traditional image transformations use operations such as flips, rotations, and color changes, but transformation choices are usually problem-specific.They can simulate device-related color and geometry changes in medical images, while some operations may cause label shift in digit or character recognition.
- Adversarial Gradients: Task-adversarial gradients perturb input images using gradients from the task classifier, expanding the original data distribution to encourage more generalizable features.Domain-adversarial gradients instead use a domain classifier in multi-source DG to synthesize domain-agnostic images.
- Method Taxonomy: The figure categorizes augmentation by transformation formulation into hand-engineered image transformations, adversarial gradients, neural-network models, and intermediate-feature perturbations.Neural-network approaches include random CNNs, off-the-shelf style transfer, and learnable image generators.
- Adversarial Gradients: Adversarial-gradient perturbations are visually imperceptible and computationally expensive, limiting their ability to simulate complex real-world domain shifts.These methods often require two sets of forward and backward passes, potentially doubling computational cost.
- Model-Based Augmentation: Model-based augmentation uses random CNNs, style-transfer models, learnable augmentation networks, or feature-statistics mixing to generate more diverse domain variation.Feature-based methods such as MixStyle mix CNN feature statistics between instances from different domains, avoiding heavy image-to-image translation models.
3.4 Ensemble Learning
Ensemble learning improves domain generalization by combining multiple predictors, including exemplar classifiers and domain-specific neural networks. Domain-specific networks may share shallow layers that capture generic features while specializing elsewhere.
- Ensemble learning trains multiple model copies with different initializations or training-data splits and combines them for prediction.The survey describes this as a broadly effective technique for boosting single-model performance across applications.
- Exemplar Classifiers: Exemplar-SVM-based DG selects the top-K classifiers with the highest prediction scores for ensemble prediction on each test sample.Each exemplar classifier is trained using one positive instance and all negative instances.
- Domain-Specific Neural Networks: Domain-specific neural-network ensembles assign networks to source domains, while sharing shallow layers can capture generic features more efficiently.This shared-layer design avoids learning a fully independent CNN for every source domain.
3.5 Self-Supervised Learning
Self-supervised DG uses labels generated from the data itself, often by predicting image transformations or solving reconstruction-based pretext tasks. Multiple pretext tasks outperform a single pretext task in the reported evidence, but current evaluations remain limited to object recognition.
- Self-supervised learning generates supervision from the data itself, including predicting patch-shuffling orders or image rotation degrees.Jigsaw puzzles and rotation prediction are common pretext tasks in DG.
- Motivation: Pretext tasks can encourage generic features that are less over-fitted to domain-specific biases and require no domain labels in single- or multi-source settings.This makes the approach applicable when domain annotations are unavailable.
- Single Pretext Task: Single-pretext-task methods combine standard classification with tasks such as Jigsaw solving, metric learning, or reconstruction.These auxiliary tasks provide intrinsic supervision alongside the main task objective.
- Multiple Pretext Tasks: Multiple pretext tasks, including Jigsaw solving and rotation prediction, achieve better performance than a single pretext task in the reported study.Other combinations include Gabor-response reconstruction, rotation prediction, and feature-cluster assignment prediction.
- Limitations: Current self-supervised DG methods have been evaluated only on object recognition, leaving their effectiveness across broader OOD tasks unclear.Pretext-task selection is also problem-specific because no existing task is universal.
- Future Direction: Contrastive learning offers a related direction by learning transformation-invariant representations rather than predicting transformations directly.It pulls together different transformed views of the same instance and separates different instances.
3.6 Learning Disentangled Representations
Disentangled representation learning relaxes the requirement that all features be domain-invariant by allowing some components to remain domain-specific. Existing approaches use decomposition or generative modeling and require domain labels for feature disentanglement.
- Disentangled representations allow some model components to remain domain-specific instead of enforcing domain invariance everywhere.The surveyed approaches are based on either decomposition or generative modeling.
3.7 Regularization Strategies
Regularization strategies in DG can discourage reliance on local patches or textures, encouraging representations that capture global object structure.
- Regularization Strategies: Regularization methods can target global object structure by suppressing the predictive power of auxiliary patch-wise CNNs.The proposed implementation uses a stack of 1×1 convolution layers and maximizes the auxiliary CNNs’ classification errors.
3.8 Reinforcement Learning
In reinforcement learning, domain shift affects visual appearance as well as environment dynamics and rewards. DG methods for visual shifts include augmentation and self-supervision, while dynamics and reward shifts require more problem-specific designs.
- Reinforcement Learning: RL domain shift can involve visual appearance, transition dynamics, or rewards such as gravity and friction changes.Visual shifts include color and style changes, whereas dynamics and reward changes alter other aspects of the environment.
- Reinforcement Learning: Visual generalization methods for RL include label-preserving transformations, domain randomization, and feature-space domain synthesis.Examples include color jittering, Cutout, simulator-based randomization, random neural networks, and MixStyle.
- Reinforcement Learning: Self-supervised learning has also been combined with RL because it does not require manual labels.The surveyed pipeline augments an RL model with auxiliary self-supervised learning.
4 THEORIES
DG theory seeks risk and generalization-gap bounds despite lacking target data, but many existing analyses depend on restricted model classes or strong assumptions. Emerging work develops more generic bounds using feature properties or model complexity.
- THEORIES: Bounding DG risk is difficult because target data is absent, unlike domain adaptation, which has many theoretically guaranteed learning bounds.The survey reviews theoretical attempts to address this challenge.
- THEORIES: Many DG theories assume specific model classes or structural conditions that do not readily extend to broader DG methods.Examples include kernel methods, linear dependencies among domains, convex-hull source domains, online games, and proxy measures.
- THEORIES: Emerging generic bounds quantify feature distributions through variation and informativeness terms, bounding unseen-domain error with an expansion function.Variation measures cross-domain feature stability, while informativeness measures class discriminativeness.
- THEORIES: Lower model complexity with strong regularization can improve unseen-domain generalization according to Rademacher-complexity bounds.The survey connects this result with properly regularized ERM using leave-one-domain-out cross-validation as a strong DG baseline.
5 FUTURE RESEARCH DIRECTIONS
The survey concludes that DG remains far from solved and identifies future directions spanning model design, learning settings, and benchmark construction. These directions address adaptability, missing labels, synthetic diversity, causal features, side information, continual domains, and heterogeneous shifts.
- Future Research Directions: DG remains a challenging problem far from being solved, motivating future directions in models, learning, and benchmarks.The survey frames its future-work discussion around these three perspectives.
- Model: Dynamic architectures could improve adaptation because fixed CNN weights may generalize poorly when unseen-domain image statistics differ substantially.One proposed direction conditions weights on the input.
- Model: Normalization parameters learned from source data are not guaranteed to work under domain shift, motivating adaptive normalization for unseen domains.This concerns both distribution statistics and affine scaling and shift parameters.
- Learning: Future learning settings include domain-label-free DG, novel-domain synthesis, shortcut avoidance, causal representation learning, side information, transfer learning, and semi-supervised DG.These directions target arbitrary domain characteristics, limited domain coverage, unstable representations, auxiliary information, synthetic-to-real transfer, and unlabeled source data.
- Learning: Semi-supervised DG addresses settings where labeled source data are limited but abundant unlabeled data are cheaper to collect.Reviewed approaches use pseudo-labels, style transfer, and mixed feature statistics for augmentation.
- Benchmarks: Benchmark directions include open-domain and incremental DG, while heterogeneous domain shift remains a critical underexplored setting.Current datasets mainly contain correlated source-source and source-target shifts, unlike unpredictable real-world target shifts.
6 CONCLUSION
This survey organizes a decade of domain generalization research and clarifies its relationship to neighboring fields, evaluation, and progress. It also aims to inspire future work advancing the field.
- The survey addresses how domain generalization relates to neighboring fields, how it is evaluated and benchmarked, and what progress it has made.
- It provides a timely, up-to-date synthesis intended to support continued advances in domain generalization.