PaperlayerView exact revision on arXiv

Source-linked AI summary

In Search of Lost Domain Generalization

Ishaan Gulrajani, David Lopez-Paz

arXiv:2007.01434v1cs.LGstat.ML

TL;DR

Domain generalization seeks reliable prediction on distributions unseen during training, but inconsistent benchmarks and nontrivial model selection hinder fair comparisons. The paper studies model selection, builds DomainBed, and finds that carefully implemented ERM achieves state-of-the-art performance across the benchmarked datasets. It releases DomainBed to support rigorous and reproducible experimentation.

  • Problem

    Inconsistent datasets, architectures, and model selection criteria make it difficult to compare domain generalization algorithms fairly and realistically.

  • Method

    The paper studies model selection and implements DomainBed with nine algorithms, seven multi-domain datasets, and three model selection criteria.

  • Results

    Empirical risk minimization achieves state-of-the-art domain generalization performance when equipped with modern neural network architectures and data augmentation.

  • Takeaways & Limitations

    Domain generalization algorithms should include a model selection strategy, while DomainBed supports rigorous and reproducible experimentation.

  • Takeaways & Limitations

    The study leaves open whether modern ERM performance is close to the best achievable out-of-distribution performance and how to establish such upper bounds.

Abstract

from arXiv · show

The goal of domain generalization algorithms is to predict well on distributions different from those seen during training. While a myriad of domain generalization algorithms exist, inconsistencies in experimental conditions -- datasets, architectures, and model selection criteria -- render fair and realistic comparisons difficult. In this paper, we are interested in understanding how useful domain generalization algorithms are in realistic settings. As a first step, we realize that model selection is non-trivial for domain generalization tasks. Contrary to prior work, we argue that domain generalization algorithms without a model selection strategy should be regarded as incomplete. Next, we implement DomainBed, a testbed for domain generalization including seven multi-domain datasets, nine baseline algorithms, and three model selection criteria. We conduct extensive experiments using DomainBed and find that, when carefully implemented, empirical risk minimization shows state-of-the-art performance across all datasets. Looking forward, we hope that the release of DomainBed, along with contributions from fellow researchers, will streamline reproducible and rigorous research in domain generalization.

1 Introduction

Domain generalization targets prediction on unseen domains, but fragmented evaluation makes realistic comparisons difficult. The paper responds by requiring model selection strategies, benchmarking nine algorithms across seven datasets, and releasing DOMAINBED.

  • Domain generalization seeks predictors that perform well on unseen test domains using invariances learned from multiple training domains.
  • The literature is difficult to compare because algorithms use different datasets and model selection criteria.
  • Domain generalization algorithms should specify a model selection method rather than omitting this part of the learning procedure.
  • The benchmark implements nine domain generalization algorithms on seven multi-domain datasets using three model selection criteria.
  • Empirical risk minimization achieves state-of-the-art domain generalization performance when equipped with modern architectures and data augmentation.
  • DOMAINBED streamlines reproducible experimentation by supporting algorithm and dataset additions, automated experiments, model selection, and table generation.

2 The problem of domain generalization

Domain generalization extends supervised learning to multiple training domains and evaluates predictors on an unseen domain without test-domain data. It therefore relies on assumptions about invariances across domains and differs from domain adaptation, which uses unlabeled target data during training.

  • Supervised learning predicts targets from inputs with a predictor f, commonly decomposed into a featurizer and classifier.
  • Training data consist of iid examples from a joint distribution, while empirical risk minimization selects a predictor using the available dataset.
  • Domain generalization provides multiple training domains and aims to perform well on an unseen test domain.
  • Because test-domain data are unavailable during training, domain generalization assumes statistical invariances across training and testing domains.
  • Unlike unsupervised domain adaptation, domain generalization does not use unlabeled data from the test domain during training.

3 Model selection as part of the learning problem

Domain generalization model selection is a learning problem because no test-distributed validation set is available and different criteria encode different assumptions. The paper therefore treats algorithms without hyperparameter-selection strategies as incomplete and recommends explicit policies for oracle evaluation.

  • Motivation: Model selection is difficult because domain generalization lacks a validation set identically distributed to the test data.Prior work uses heuristics or omits hyperparameter-selection procedures, complicating fair assessment.
  • Model selection as learning: Selecting hyperparameters requires assumptions about how test data relates to training data, and the appropriate assumptions are unclear beforehand.The paper frames model selection as a learning problem at least as difficult as fitting the model.
  • Recommendations: A domain generalization algorithm without a strategy for choosing hyperparameters is incomplete.
  • Oracle selection: Algorithms lacking justified model-selection methods can be evaluated with oracle selection, but oracle results must be separated from ordinary baselines.The authors warn that unrestricted test-domain access can produce meaningless results, including direct supervised training on test-domain data.
  • Recommendations: Researchers should disclose oracle-selection results and specify policies limiting access to test-domain data.
  • Selection methods: Training-domain validation, leave-one-domain-out cross-validation, and test-domain validation represent distinct model-selection strategies with different assumptions.The first two strategies use training domains, whereas the oracle strategy validates on data following the test-domain distribution.

4 DOMAINBED: A PyTorch testbed for domain generalization

DOMAINBED is a PyTorch testbed designed to make domain generalization experiments reproducible and rigorous. Its initial release combines seven multi-domain datasets, nine baseline algorithms, and three model-selection methods with infrastructure for automated experimentation.

  • Testbed: DOMAINBED streamlines reproducible domain generalization research through a PyTorch testbed and automated experiment infrastructure.
  • Scope: The initial release contains nine algorithms, seven datasets, and three model-selection methods, with new components addable in a few lines of code.
  • Datasets: The seven datasets cover Colored MNIST, Rotated MNIST, PACS, VLCS, Office-Home, Terra Incognita, and DomainNet.
  • Dataset considerations: DOMAINBED distinguishes synthetic datasets, where prior knowledge about generalizing features is restricted, from natural datasets where such prior knowledge is sensible.
  • Algorithms: The benchmark includes ERM, DRO, Mixup, MLDG, IRM, DANN variants, CORAL, and MMD as baseline algorithm implementations.These methods respectively cover empirical-risk, robustness, interpolation, meta-learning, invariant-representation, adversarial, covariance, and distribution-matching approaches.
  • Implementation choices: The evaluation uses larger ResNet-50 models, extensive data augmentation, and all available Rotated MNIST digits to establish realistic implementation conditions.Smaller CNNs are used for the two MNIST datasets, while non-MNIST datasets receive image augmentations.

5 Experiments

The experiments evaluate all DOMAINBED dataset, algorithm, and model-selection configurations under repeated randomized hyperparameter searches. Across these evaluations, ERM is a strong baseline, no method substantially surpasses it under equal conditions, and model-selection criteria materially affect performance.

  • Protocol: The study evaluates every DOMAINBED algorithm, dataset, and model-selection configuration while holding one domain out for testing.
  • Protocol: Each algorithm and test environment receives 20 random hyperparameter-search trials, with smaller data splits reserved for hyperparameter selection.
  • Uncertainty: Three complete repetitions randomize hyperparameters, weight initializations, and dataset splits, reporting means with estimated standard errors.The protocol trains a total of 45,900 neural networks.
  • Results: ERM outperforms all previously published results when model selection uses a training-domain validation set.The authors attribute this strong performance to factors including ResNet-50, augmentations, careful tuning, and fuller Rotated MNIST data use.
  • Results: When conditions are equal, no algorithm significantly outperforms ERM, and substantial improvements over ERM remain challenging on these datasets.
  • Model selection: Training-domain validation outperforms leave-one-domain-out cross-validation across multiple datasets and algorithms, while oracle selection shows +2% headroom.The authors do not conclude that training-domain validation is universally correct.

6 Outlook

The evaluation concludes that ERM performs strongly relative to domain-generalization alternatives, while raising open questions about augmentation, dataset realism, assumptions, and achievable out-of-distribution performance.

  • ERM achieves state-of-the-art performance when compared with eight popular domain-generalization alternatives.
  • Model selection significantly affects domain-generalization results and should be treated as an integral part of proposed methods.
  • Data augmentation can improve out-of-distribution generalization without harming in-distribution generalization, but suitable pipelines depend on the domain.
  • The paper questions whether current datasets and neural networks adequately support the intended domain-generalization tasks.
  • Domain-generalization algorithms rely on different untestable invariance assumptions, so performance depends on the problem at hand.
  • Benchmark designers should balance modern techniques with realistic, well-motivated rules of evaluation.

Broader impact

Domain generalization targets failures caused by distribution shifts, and the literature has pursued several strategies for learning predictors that remain useful across domains.

  • The literature classifies methods into invariant features, parameter sharing, meta-learning, and data augmentation.
  • Invariant-feature methods seek representations whose distributions align across domains while preserving information relevant to targets.
  • Other approaches search for features yielding the same optimal classifier across training domains, linking invariance to causal structure.
  • Domain-specific classifiers, multitask autoencoders, and meta-learning methods provide alternative mechanisms for learning or adapting across domains.

A.4 Augmenting data

Data augmentation is presented as an effective domain-generalization strategy, while the benchmark materials organize comparisons across algorithms, datasets, and model-selection methods.

  • Data augmentation is an effective strategy for addressing domain generalization.
  • Designing efficient augmentation routines depends on the data type and requires substantial human-expert effort.
  • Table 5 compiles the best out-of-distribution test accuracies reported across a decade of domain-generalization research.
  • The benchmark includes training-domain validation, leave-one-domain-out cross-validation, and test-domain validation as model-selection methods.

C Dataset details

DOMAINBED provides loaders for seven multi-domain image-classification tasks and uses randomized train, validation, and test splits for each seed.

  • DOMAINBED includes downloaders and loaders for seven multi-domain image-classification tasks.
  • Colored MNIST uses domains with color-label correlations of 0.1, 0.3, and 0.9, while digit-label correlation is 0.75.
  • For each random seed, raw training, validation, and testing images are pooled before randomized splits are instantiated.
  • The experiments describe the model architectures and hyperparameter search spaces used for evaluation.

D.1 Architectures

DomainBed specifies dataset-specific neural network architectures and a reproducible random-search setup for training its models.

  • Architectures: Table 6 lists the neural network architecture used for each dataset.The MNIST network is specified separately in Table 7.
  • Architectures: The MNIST ConvNet uses 3 × 3 convolutions with “same” padding.
  • Architectures: ResNet-50 models replace the pretrained ImageNet classifier, freeze batch-normalization layers, and add dropout before the final linear layer.The modified network is fine-tuned.
  • Hyperparameters: Hyperparameters are selected through random sweeps whose distributions and default values are listed in Table 8.The study optimizes all models with Adam.

E Adding new datasets and algorithms to our framework

DomainBed exposes a simple interface for implementing algorithms and datasets, then supports automated experiments across the framework’s configurable components.

  • Algorithms: Algorithms implement update(minibatches) and predict(x), with one minibatch supplied for each training domain.This interface is illustrated using group DRO.
  • Algorithms: A group DRO implementation computes per-domain cross-entropy losses, updates domain weights, backpropagates the aggregate loss, and steps the optimizer.The update method returns the loss value.
  • Algorithms: New algorithms inherit default network, optimizer, and prediction functionality from ERM and register their hyperparameters in the framework registry.The example adds a dro_eta search distribution.
  • Datasets: New image datasets use the path /path/MyDataset/domain/class/image.jpg and a dataset class extending MultipleEnvironmentImageFolder.Dataset classes can specify training steps and checkpoint frequency.
  • Experiments: Experiments can be launched from the command line and fully automated across datasets, algorithms, test domains, and model-selection criteria.The automated sweep generates the manuscript’s result tables.
  • Extensions: Extending update to accept unlabeled test-domain minibatches allows DomainBed to support unsupervised domain adaptation experiments.
Loading 2007.01434v1…