PaperlayerView exact revision on arXiv

Source-linked AI summary

Generalizing to Unseen Domains: A Survey on Domain Generalization

Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, Philip S. Yu

arXiv:2103.03097v7cs.LGcs.AIcs.CV

TL;DR

Machine learning often faces different training and testing distributions, motivating domain generalization for unseen domains. This paper surveys DG through its definitions, theories, algorithms, datasets, applications, evaluation resources, and future challenges, concluding with analyses and directions for further research.

  • Problem

    Domain generalization addresses the need to generalize from related source domains to unseen target distributions when collecting data from all possible domains is prohibitively difficult.

  • Method

    The paper synthesizes DG formulations, related theories, algorithms organized into data manipulation, representation learning, and learning strategies, plus datasets, applications, evaluation standards, and DeepDG.

  • Results

    The survey analyzes existing theories and methods, reports benchmark observations on PACS and Office-Home, and identifies research challenges and future directions.

  • Takeaways & Limitations

    The review provides a consolidated account of DG methods, theoretical analysis, benchmarks, applications, and open-source evaluation resources for researchers.

  • Takeaways & Limitations

    Data-manipulation methods lack a theoretical guarantee for the unbound risk of generalization, motivating further theory to guide data-generation designs.

Abstract

from arXiv · show

Machine learning systems generally assume that the training and testing distributions are the same. To this end, a key requirement is to develop models that can generalize to unseen distributions. Domain generalization (DG), i.e., out-of-distribution generalization, has attracted increasing interests in recent years. Domain generalization deals with a challenging setting where one or several different but related domain(s) are given, and the goal is to learn a model that can generalize to an unseen test domain. Great progress has been made in the area of domain generalization for years. This paper presents the first review of recent advances in this area. First, we provide a formal definition of domain generalization and discuss several related fields. We then thoroughly review the theories related to domain generalization and carefully analyze the theory behind generalization. We categorize recent algorithms into three classes: data manipulation, representation learning, and learning strategy, and present several popular algorithms in detail for each category. Third, we introduce the commonly used datasets, applications, and our open-sourced codebase for fair evaluation. Finally, we summarize existing literature and present some potential research topics for the future.

1 INTRODUCTION

Domain generalization addresses performance loss when training and testing distributions differ by learning from related source domains for an unseen target. This survey reviews the field’s formulations, theories, methods, datasets, applications, evaluation standards, and future directions.

  • Problem: Domain generalization learns from one or several related domains to generalize to an unseen testing domain with a different distribution.The PACS example uses sketches, cartoons, and paintings for training and photos as the unseen target domain.
  • Scope: The survey presents a comprehensive review of domain generalization’s formulations, theories, algorithms, research areas, datasets, applications, and future research directions.It is presented as the first survey focused on these aspects of domain generalization.
  • Extensions: This extended version adds theory analysis covering domain generalization and related domain adaptation.
  • Extensions: The methodology is broadened with categories including causality-inspired methods, generative modeling for feature disentanglement, invariant risk minimization, gradient operation-based methods, and other learning strategies.
  • Extensions: The survey expands method analysis with more related algorithms, comparisons, discussion, and over 30% new work.
  • Resources and evaluation: It extends datasets and applications, explores evaluation standards, and introduces the open-sourced DeepDG codebase with analyses on public datasets.

2 BACKGROUND

This section formalizes domain generalization as learning from multiple distinct source domains without access to an unseen target domain, and distinguishes it from related paradigms.

  • 2.1 Formalization of Domain Generalization: A domain consists of data sampled from a joint input-label distribution P_XY.The input lies in X, labels lie in Y, and P_XY denotes their joint distribution.
  • 2.1 Formalization of Domain Generalization: Domain generalization uses M different source domains to learn a predictive function that minimizes error on an inaccessible test domain.The test domain cannot be accessed during training.
  • 2.2 Related Research Areas: The paper distinguishes domain generalization from transfer learning, domain adaptation, meta-learning, lifelong learning, and zero-shot learning.These paradigms differ in target-domain access, task relationships, sequential learning requirements, or whether testing classes are unseen.
  • 2.2 Related Research Areas: Multi-task learning optimizes related tasks jointly, whereas domain generalization typically retains the task and targets distribution shifts across domains.Multi-domain learning instead trains models for the original domains rather than new test domains.
  • 2.2 Related Research Areas: Unlike domain adaptation, domain generalization cannot use target-domain data during training, making it more challenging but more realistic for practical applications.Domain adaptation optimizes performance on a given target domain using source domains.

3 THEORY

The theory section develops bounds for domain adaptation and domain generalization, covering unknown-target average risk and specific-target settings under covariate shift. These bounds motivate domain-invariant representation methods and show why retaining domain-wise information can improve estimation.

  • Domain adaptation: Domain adaptation bounds target risk using source risk, distribution difference, and differences between source and target labeling functions.The target risk is inaccessible directly because the target labeling function is unknown.
  • Domain adaptation: Total variation can loosen adaptation bounds and is difficult to estimate from finite samples, motivating H∆H-divergence and non-asymptotic alternatives.The H∆H-divergence incorporates the hypothesis class and provides a better finite-sample guarantee.
  • Unknown-target generalization: Unknown-target DG theory models source and target domains as draws from a hyperdistribution and evaluates average risk across possible target domains.The classifier may take both the domain distribution and an input instance as arguments.
  • Average risk estimation error bound: Using domain-wise datasets yields a smaller estimation bound in general than replacing M domains of size n with one pooled dataset of size Mn.The result indicates that domain information plays a role in estimation.
  • Specific-target generalization: For a specific target under covariate shift, DG bounds measure the target distribution’s distance from the convex hull of source distributions and the associated ideal joint risk.The covariate-shift assumption keeps the labeling function or PY|X the same across domains.
  • Specific-target generalization: The resulting multi-source bound extends domain-adaptation bounds and motivates minimizing source risks and representation distribution differences.These objectives aim to reduce the target-to-source-hull distance and related representation-space terms.

4 METHODOLOGY

The survey organizes domain generalization methods into data manipulation, representation learning, and learning strategy. The taxonomy distinguishes methods by whether they modify inputs, learn domain-robust features, or alter the learning procedure.

  • Data manipulation: Data manipulation methods alter inputs through data augmentation or data generation to assist learning general representations.Augmentation uses transformations or randomization, whereas generation creates diverse samples.
  • Representation learning: Representation learning methods include domain-invariant representation learning and feature disentanglement.They align or otherwise constrain representations across domains, or separate domain-shared from domain-specific features.
  • Learning strategy: Learning-strategy methods promote generalization through ensemble learning, meta-learning, gradient operation, distributionally robust optimization, and self-supervised learning.These methods modify how models learn rather than focusing primarily on input manipulation.

4.1 Data Manipulation

Data manipulation-based DG increases training diversity by transforming existing samples or generating new ones. The section covers conventional and adversarial augmentation, generative models, and Mixup, while noting optimization and heuristic limitations.

  • Overview: Data manipulation is a relatively cheap way to generate samples when training data are limited, although NLP models may rely on trivial syntactic heuristics.The cited experiments concern predictions on both out-of-distribution and in-distribution NLP samples.
  • Overview: The data-manipulation objective applies a transformation x′ = M(x), with methods divided into data augmentation and data generation.The distinction is based on the function used to manipulate or create the data.
  • Data augmentation: Standard augmentation uses operations such as flipping, rotation, scaling, cropping, and noise addition to reduce overfitting and improve generalization.These operations can be adopted for domain generalization through M(·).
  • Data augmentation: Domain randomization increases sample diversity by changing object properties, illumination, camera views, scene structure, and noise in simulated data.Some approaches also augment labels, and scene-aware placement can encourage context use for detection.
  • Data augmentation: Adversarial augmentation optimizes transformations for generalization while preserving reliability, including perturbations that change domains while minimally changing class labels.CrossGrad models dependencies among labels, domains, and input instances; related procedures iteratively augment source data.
  • Data generation: Data generation uses VAEs, GANs, or Mixup to create diverse samples, with Mixup linearly interpolating instances and labels without training a generative model.DG methods apply Mixup in either the original space or feature space.
  • Data generation: Generative approaches can combine synthetic samples with distribution-discrepancy minimization or adversarial training to preserve semantics and create challenging populations.Examples use ComboGAN with MMD or WAE-based generation with large domain transportation.

4.2 Representation Learning

Representation learning for domain generalization seeks features that transfer across domains, using domain-invariant representations or disentangling shared and domain-specific factors. The survey organizes major approaches including kernels, adversarial learning, explicit alignment, invariant risk minimization, and disentanglement.

  • Representation learning: Representation learning decomposes h into a representation function g and classifier f, with regularization guiding feature extraction.The survey categorizes methods as domain-invariant representation learning and feature disentanglement.
  • Domain-invariant representation learning: Kernel-based methods learn domain-invariant kernels or components to reduce distances among multiple source domains.Examples include positive semi-definite kernel learning, transfer component analysis, and Domain-Invariant Component Analysis.
  • Domain-invariant representation learning: Domain-adversarial methods train a discriminator to distinguish domains while the generator learns features that fool it.The survey describes DANN and subsequent adaptations of adversarial training for domain generalization.
  • Domain-invariant representation learning: Explicit alignment methods reduce cross-domain feature discrepancies through distribution alignment, normalization, contrastive losses, or restitution of discriminative information.Style normalization and restitution combines instance normalization with recovery of task-relevant residual features.
  • Invariant risk minimization and feature disentanglement: IRM enforces a shared optimal representation-level classifier across domains, whereas disentanglement separates domain-shared and domain-specific features with regularization and reconstruction losses.IRM may fail when representations capture fully informative invariant features, especially in classification, motivating information-bottleneck regularization.

4.3 Learning Strategy

Learning-strategy approaches extend domain generalization beyond data manipulation and representation learning. The survey covers ensembles, meta-learning, gradient-based methods, distributionally robust optimization, self-supervision, and other strategies.

  • Overview: Learning-strategy methods include ensemble, meta-learning, gradient operation, distributionally robust optimization, self-supervised learning, and other approaches.These categories apply general machine-learning paradigms to domain generalization.
  • Ensemble learning-based DG: Ensemble DG combines source-specific models or experts, using domain-dependent predictions, normalization statistics, or classifier heads to form test-time predictions.DAEL uses a shared CNN extractor and multiple domain-specific classifier heads trained collaboratively.
  • Meta-learning-based DG: Meta-learning splits multisource data into meta-train and meta-test sets to simulate domain shift while learning general representations.The resulting procedures form a bi-level optimization problem, with outer and inner learning rates η and β.
  • Gradient operation-based DG: Gradient-based DG manipulates or regularizes gradients to encourage generalized representations and closer input-output correspondence across domains.Methods include self-challenging feature removal, gradient-invariance losses, gradient similarity regularization, and gradient filtering.
  • DRO and self-supervised learning: DRO optimizes for worst-case distribution scenarios, while self-supervised DG uses pretext or contrastive tasks and can support unsupervised DG.The survey notes that SSL-based DG may require substantial computation and resources.

5 OTHER DOMAIN GENERALIZATION RESEARCH AREAS

The survey describes extensions of domain generalization to settings with fewer labels, privacy constraints, differing label spaces, or no accessible training labels. These settings broaden the problem but also increase its difficulty.

  • Single-source DG: Single-source DG is more challenging than traditional multi-source DG because training provides less domain diversity.Data-generation techniques are used to create novel domains and increase training diversity and informativeness.
  • Semi-supervised DG: Semi-supervised DG uses unlabeled training samples alongside labeled data, often applying pseudo-labeling or consistency regularization.The survey characterizes this setting as more general than traditional DG.
  • Federated DG: Federated DG aggregates parameters from different clients without accessing raw training data, with aggregation schemes central to the setting.Federated DG is identified as particularly important in healthcare, while decentralized training may still involve privacy risks during updates.
  • Open and unsupervised DG: Open DG allows training and test label spaces to differ, whereas unsupervised DG assumes training-domain labels are inaccessible.The survey presents these as increasingly general and challenging DG settings.

6 APPLICATIONS

Domain generalization is applied across vision, language, control, scientific, industrial, and time-series problems where distributions differ across domains. The survey also highlights evaluation without access to the real test distribution as an open issue.

  • Vision: Vision applications include classification, segmentation, action recognition, face anti-spoofing, person re-identification, video understanding, image compression, and medical analysis.Medical analysis is important because of data scarcity and domain gaps.
  • Other applications: DG has also been applied to robot-control reinforcement learning, speech recognition, fault diagnosis, physics, and brain-computer interfaces.These applications target generalization to unseen environments or domains.
  • Natural language processing: Natural-language applications include sentiment classification, semantic parsing, and web-page classification across differently distributed domains.Domain-invariant representations are used to predict well on new domains.
  • Time series and other applications: Potential applications include financial analysis, weather prediction, logistics, and time-series modeling under temporal covariate shift.AdaRNN aligns hidden representations between pairs of training data from different domains.
  • Future directions: A major open problem is evaluating DG algorithms without accessing the real test distribution in practical deployment.The survey suggests meta-train and meta-test splits on original multisource data as one possible approach.

7 DATASETS, EVALUATION, AND BENCHMARK

The survey reviews datasets, evaluation strategies, and benchmark infrastructure for domain generalization, including the DeepDG codebase and experiments on PACS and Office-Home.

  • Datasets: PACS, VLCS, and Office-Home are among the most popular domain generalization datasets, while DomainNet and WILDS support large-scale evaluation.
  • Datasets: Domain generalization benchmarks also cover flow cytometry, semantic segmentation, and subject-level Parkinson’s disease prediction.
  • Datasets: Domain adaptation datasets overlap substantially with domain generalization benchmarks, including Office-31, Office-Caltech, Office-Home, DomainNet, and Amazon Review.
  • Evaluation: Three model-selection strategies are test-domain validation, leave-one-domain-out cross-validation, and training-domain validation, each with distinct access or stability constraints.Test-domain validation uses target data and is therefore unavailable when targets cannot be accessed; leave-one-domain-out validation is unavailable with a single source and can produce unstable results across source selections.
  • Evaluation: Current evaluation commonly uses training-domain validation, but validation distributions may differ from new training data, and accuracy alone may be insufficient.
  • Benchmark: DeepDG provides a unified, modular codebase for fair comparisons, supporting image classification datasets and nine methods across data manipulation, representation learning, and learning strategy.Its public version supports Office-31, PACS, VLCS, and Office-Home, with methods including Mixup, DDC, DANN, CORAL, MLDG, RSC, GroupDRO, and ANDMask.
  • Benchmark: RSC achieved the best performance on PACS and Office-Home, but its improvements over ERM were unremarkable; DANN and ANDMask sometimes performed worse than ERM.The experiments used ResNet-18 and training-domain validation, with 20% of source data reserved for validation.

8 DISCUSSION

The discussion groups domain generalization methods into data manipulation, representation learning, and learning strategy, while identifying theoretical and methodological gaps.

  • Data manipulation: Training-data quantity and diversity are critical to generalization, motivating data manipulation methods that enrich training data.
  • Data manipulation: Data manipulation methods lack theoretical guarantees for the unbounded risk of generalization, motivating theory that can guide data generation without violating ethical standards.
  • Representation learning: Representation learning has broader theoretical support, but adversarial methods have not shown significant improvements in domain generalization despite their success in domain adaptation.
  • Learning strategy: Learning-strategy research increasingly uses meta-learning to exploit information across domains, while broader method diversity remains an open direction.Existing work also applies random forests, not only deep networks, to domain generalization.

8.2 Future Research Challenges

The survey identifies future challenges spanning continuous and task generalization, interpretability, large-scale pre-training, test-time adaptation, and evaluation realism.

  • Continuous domain generalization: Continuous domain generalization must update models on streaming data with non-stationary statistics while overcoming catastrophic forgetting, yet few studies address it.
  • Domain and task generalization: Future DG settings should generalize to new categories as well as new domains, extending beyond the usual shared-label-space assumption.
  • Interpretability: Disentanglement provides some interpretability by separating domain-invariant and domain-specific features, but other DG methods lack deep semantic understanding of learned representations.
  • Large-scale pre-training: Designing efficient DG methods for large-scale pre-training and self-learning is an open research direction because pre-training can learn general representations.
  • Test-time generalization: Test-time generalization can use unlabeled inference data for adaptation, offering flexibility but requiring low computation and high efficiency under limited resources.
  • Performance evaluation: On several datasets, some DG methods perform almost the same as ERM, potentially reflecting inappropriate evaluation schemes or small domain gaps rather than the full value of DG.The survey notes dramatic improvement in more realistic settings such as person re-identification, where domain gaps are obvious.

9 CONCLUSION

The survey synthesizes theories, methods, datasets, benchmarks, and applications in domain generalization, then identifies challenges and future research directions.

  • The paper provides in-depth analysis of domain generalization theories, existing methods, datasets, benchmarks, and applications.
  • It derives potential research challenges from this analysis and aims to provide useful insights and inspire further progress.
Loading 2103.03097v7…