PaperlayerView exact revision on arXiv

Source-linked AI summary

Generalizing to unseen domains via distribution matching

Isabela Albuquerque, João Monteiro, Mohammad Darvishi, Tiago H. Falk, Ioannis Mitliagkas

arXiv:1911.00804v6cs.LGstat.ML

TL;DR

The paper addresses domain generalization when test data may come from unseen distributions, beyond the often-violated i.i.d. assumption. It proves that matching pairwise source-domain discrepancies controls mixtures, then uses adversarial invariant representation learning; empirically, G2DM improves over recent domain-label-based methods and standard i.i.d. settings.

  • Problem

    Supervised learning commonly assumes i.i.d. data, but practical distribution shifts and inaccessible unseen test domains make ordinary target-risk estimation insufficient.

  • Method

    The paper proves a mixture-discrepancy lemma and bound, then trains an encoder with multiple domain discriminators to minimize pairwise divergences while preserving task information.

  • Results

    G2DM outperformed recent methods leveraging domain labels and improved performance relative to standard settings relying on i.i.d. requirements.

  • Takeaways & Limitations

    Generalization is supported for unseen domains near mixtures of observed training distributions, using domain information to learn invariant representations without test-domain data.

  • Takeaways & Limitations

    The guarantees concern domains near mixtures of source distributions, and the bound requires source-domain labels that ERM does not require.

Abstract

from arXiv · show

Supervised learning results typically rely on assumptions of i.i.d. data. Unfortunately, those assumptions are commonly violated in practice. In this work, we tackle such problem by focusing on domain generalization: a formalization where the data generating process at test time may yield samples from never-before-seen domains (distributions). Our work relies on the following lemma: by minimizing a notion of discrepancy between all pairs from a set of given domains, we also minimize the discrepancy between any pairs of mixtures of domains. Using this result, we derive a generalization bound for our setting. We then show that low risk over unseen domains can be achieved by representing the data in a space where (i) the training distributions are indistinguishable, and (ii) relevant information for the task at hand is preserved. Minimizing the terms in our bound yields an adversarial formulation which estimates and minimizes pairwise discrepancies. We validate our proposed strategy on standard domain generalization benchmarks, outperforming a number of recently introduced methods. Notably, we tackle a real-world application where the underlying data corresponds to multi-channel electroencephalography time series from different subjects, each considered as a distinct domain.

1 Introduction

Standard supervised learning assumes i.i.d. training and testing data, but practical distribution shifts and unseen domains violate this assumption. The paper develops domain-generalization guarantees and an invariant representation-learning strategy that uses source domains without observing test distributions.

  • 1 Introduction: i.i.d. assumptions are unverifiable and do not account for distribution shifts that commonly occur in practice.Examples include changing data-acquisition conditions and new speakers in speech recognition.
  • 1 Introduction: Domain generalization allows observed and unseen novel domains to appear at test time while seeking low risk regardless of the underlying domain.Unlike domain adaptation, it does not align training distributions with one specific target domain.
  • 1 Introduction: Pairwise small distances among training distributions imply small distances between mixtures of those distributions, supporting a risk bound for domains near their convex hull.The paper uses this result to characterize where generalization can be expected.
  • 1 Introduction: The proposed representation filters domain-dependent cues while preserving task-relevant information, without observing data from test distributions.This distinguishes the approach from standard domain adaptation strategies.
  • 1 Introduction: The paper introduces broader-than-i.i.d. assumptions, proves an unseen-domain bound, minimizes pairwise divergences adversarially, and reports improvements across varied domain-shift scenarios.The empirical evaluation includes settings where labeling functions might shift.

2 Background

The background formalizes domains, labeling functions, hypotheses, and risk, then reviews invariant-representation and self-supervised approaches. It also introduces ERM and domain-adaptation guarantees based on distribution divergence, while noting that ERM can remain a strong baseline.

  • 2.1 Notation: A domain is a data distribution paired with a deterministic labeling function, and a hypothesis maps inputs to predicted labels.Risk measures the loss between the hypothesis output and the domain's labeling function.
  • 2.2 Related work: Prior domain-generalization methods use causal-inference tools, self-supervised tasks, invariant representations, data augmentation, or domain-specific parameter components.Other methods regularize empirical loss by minimizing divergences between source domains.
  • 2.2 Related work: ERM can be extended by minimizing empirical risk across all source domains and using the resulting predictor for target-data generalization.The paper notes that this straightforward strategy often yields strong baselines.
  • 2.3 Generalization guarantees for domain adaptation: Domain-adaptation theory bounds target risk using source risk, a joint minimal-risk term, and an H-divergence between source and target distributions.The H-divergence can be estimated from the error of a binary classifier trained to distinguish domains.

3 Learning domain agnostic representations for domain generalization

The paper formalizes domain generalization through a meta-distribution over domains and derives guarantees for unseen domains near mixtures of observed sources. It then learns source-only representations that preserve class information while reducing domain distinguishability, using adversarial distribution matching with stabilization strategies.

  • 3.1 Formalizing domain generalization: Domain generalization models training and test examples as drawn from domains sampled from a meta-distribution, including unseen domains at test time.The framework expects low risk on observed or novel domains, unlike domain adaptation's alignment to a specific target.
  • 3.1 Formalizing domain generalization: Estimating meta-risk is impractical without test-domain information, so the theory restricts guarantees to unseen domains near mixtures of source distributions.The bound concerns domains whose H-divergence from the closest source mixture is γ.
  • 3.2 Matching distributions in the convex hull: Pairwise source-domain H-divergence bounds the H-divergence between any two distributions in the sources' convex hull.Thus, reducing the maximum pairwise divergence also reduces divergence among source mixtures.
  • 3.3 Generalizing to unseen domains: The unseen-domain risk bound combines source risk, distance γ to the closest source mixture, maximum pairwise divergence ϵ, and a labeling-function mismatch term.Under covariate shift, the labeling functions are shared across domains, making the mismatch term zero.
  • 3.3 Generalizing to unseen domains: The method learns an encoder that preserves class-relevant information while removing domain-specific cues, using source data alone to make domains harder to distinguish.Source-domain labels are needed to estimate ϵ, and the approach can work even when its assumptions are unlikely to hold.
  • 3.4 Practical contributions: G2DM stabilizes adversarial training with random projections for domain discriminators and a negative log-hypervolume objective that favors uniform reduction of pairwise divergences.One-vs-all classifiers estimate pairwise source divergences more efficiently than naively using O(N_S^2) estimators.

4 Experimental Setup and Results

G2DM is evaluated on object-recognition benchmarks under covariate shift and on EEG affective-state prediction where labeling-function mismatch is non-negligible. Across these settings, it improves unseen-domain performance, reduces estimated cross-domain discrepancies, and remains competitive under source-only model selection.

  • Evaluation under covariate shift: G2DM outperforms ERM in average accuracy on the VLCS and PACS unseen domains under leave-one-domain-out validation.It also exceeds the reported IRM and other previously proposed domain-generalization methods, with improvements highlighted on LabelMe and Sketch.
  • Distribution matching: G2DM reduces estimated pairwise H-divergences on PACS for most source distributions, except when photo is the unseen test domain.The encoder appears to filter domain information, which the authors connect to smaller source discrepancy terms and lower divergence involving unseen domains.
  • Access methods to test data: Source-only stopping with training loss lets G2DM outperform CIDDG on almost all PACS domains, while ERM degrades severely when sketch is unseen.Using ResNet-18 instead of AlexNet produces more stable average performance across stopping criteria.
  • Evaluation beyond covariate shift: On SEED EEG affective-state prediction, G2DM improves accuracy over ERM by more than 3.4% under both considered domain-generalization access scenarios.The experiment leaves one of 15 subjects for testing, uses 10 subjects for training, and reserves four for validation.
  • Comparison with domain adaptation: G2DM performs on par with or better than some privileged domain-adaptation strategies despite using domain-generalization models that target domain-agnostic representations.The privileged baselines use unseen-domain data during training, whereas G2DM does not target a specific test distribution.

5 Conclusion

The paper addresses domain generalization by learning invariant representations through pairwise source-domain discrepancy minimization. Its results support improved performance over i.i.d.-based ERM and recent domain-label methods, including a realistic EEG setting comparable to privileged systems.

  • Conclusion: G2DM uses multiple one-vs-all domain discriminators to estimate and minimize pairwise divergences between source distributions.The method is designed to yield invariant representations across unseen distributions.
  • Conclusion: The paper shows that generalization can be achieved near the set of mixtures of distributions observed during training.This conclusion follows from the proposed domain-generalization bound.
  • Conclusion: G2DM improves performance relative to standard i.i.d.-based settings and outperforms recent methods that also leverage domain labels.The paper reports strong results in a realistic setting, with performance comparable to privileged systems tailored to test distributions.

A Proof of Lemma 1

The proof extends pairwise H-divergence control from observed source domains to distributions in their convex-hull neighborhood. It uses the triangle inequality and sub-additivity of the supremum to establish the bound.

  • Proof of Lemma 1: The H-divergence between the relevant domains is expanded into terms connecting unseen and source-domain distributions.This decomposition is the basis for applying standard divergence inequalities.
  • Proof of Lemma 1: The lemma considers unseen domains and a distribution in the convex hull of the source domains, all sharing support Ω.The proof frames the target relationship through H-divergence comparisons among these distributions.
  • Proof of Lemma 1: Applying the triangle inequality and then sub-additivity of the supremum yields the stated upper bound.These two steps control the divergence between unseen-domain mixtures using pairwise source-domain discrepancies.

B Proof of Theorem 1

Theorem 1 bounds risk on an unseen domain by risk on a convex-hull projection, its divergence from that domain, and labeling-function mismatch. Pairwise source-domain divergence controls the divergence term through the highest within-source pairwise divergence.

  • Theorem 1 bounds unseen-domain risk using the projected-domain risk, divergence from the projection, and labeling-function mismatch.The projection is formed by choosing a mixture of source domains that minimizes divergence to the unseen domain.
  • The source projection uses mixture weights π_i to combine source labeling functions into f_Sπ.The resulting labeling function is defined on the support of the projected mixture domain.
  • The unseen domain’s divergence from each source is bounded through the projection distance γ and the maximum pairwise source divergence ϵ.This step uses the triangle inequality together with the pairwise-mixture lemma.
  • The final bound adds γ, ϵ, and the labeling-function mismatch term to the projected-domain risk.The bound therefore relates unseen-domain performance to quantities associated with observed source domains.

C Proof of Corollary 1

The corollary specializes the unseen-domain bound to domains in the convex hull of the source domains when all domains share the same labeling function. In that case, both the projection distance and labeling-function mismatch vanish.

  • For an unseen domain inside the source convex hull, Corollary 1 provides an upper bound on its risk.The result assumes that all domains in the meta-distribution share labeling function f.
  • Shared labeling functions eliminate the mismatch term between the unseen domain and its convex-hull projection.The proof explicitly sets this term to zero under the common-labeling-function assumption.
  • Membership in the source convex hull makes the unseen domain equal to its projection, so γ = 0.The remaining bound is consequently determined by the source-domain risk and divergence terms.
  • One-vs-all discriminators estimate pairwise H-divergences by assigning separate loss terms to examples from each source domain.With three sources, discriminator D1 estimates divergences between D1 and both D2 and D3.

F Extra experiments

Extra experiments examine how the number of source domains and the random projection layer affect domain-generalization performance. Removing a source generally hurts classification, while projection size affects convergence and target accuracy.

  • Reducing the VLCS training set from three source domains to two hurt classification performance for almost all source combinations.Some source exclusions caused severe decreases, and ERM was included as a reference baseline.
  • The random projection layer affected generalization on PACS when Sketch was treated as the unseen target domain.The experiment compared seven projection sizes with using the feature-extractor output directly.
  • The best PACS Sketch target accuracy was achieved with a random projection size of 1000.The comparison used the best target accuracy obtained across the tested projection sizes.

G Domain generalization benchmarks

The benchmarks use multi-domain image datasets with shared object classes and domain-specific datasets. VLCS uses four datasets with five common classes, while PACS uses four datasets with seven common classes.

  • VLCS contains four datasets sharing five classes: bird, car, chair, dog, and person.Each dataset is split into 80% training and 20% test partitions.
  • PACS contains four datasets sharing seven classes: dog, elephant, giraffe, guitar, horse, house, and person.The experiments use the benchmark’s original train/validation partitions.
  • The PACS benchmark includes Cartoon with 2344 examples and Sketch with 3929 examples.

H.1 VLCS and PACS benchmarks

Experiments use pre-trained AlexNet and ResNet-18 feature extractors for benchmark comparisons, with hyperparameters selected through grid searches. The SEED experiments use a SyncNet encoder and domain-specific classifier architectures.

  • Feature extraction: Pre-trained AlexNet and ResNet-18 initialize the feature extractors, whose 4096- and 512-dimensional representations feed the task classifier and domain discriminators.The final layer is discarded for both networks.
  • SEED preprocessing and training: SEED preprocessing clips artifacts above five times channel-mean amplitude, windows signals into 60-second chunks, and normalizes each window to zero mean and unit variance.Models are trained for 100 epochs with SGD and Polyak’s acceleration.
  • SEED architectures: The SEED encoder uses a two-filter convolutional layer producing a 602-dimensional output for four-layer domain discriminators and a two-layer task classifier.The discriminator includes a random projection layer, while the task classifier maps 602 → 100 → number of classes.
  • Domain-discriminator evaluation: Tree-ensemble domain discriminators use 100 estimators, five-fold cross-validation, independent domain-pair evaluation, and random samples of 500 per domain.
Loading 1911.00804v6…