PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain Generalization by Marginal Transfer Learning

Gilles Blanchard, Aniket Anand Deshmukh, Urun Dogan, Gyemin Lee, Clayton Scott

arXiv:1711.07910v3stat.ML

TL;DR

Domain generalization asks how to predict accurately on unseen unlabeled tasks when labeled data come from related but varying tasks. The paper reframes this problem through marginal transfer learning, develops distribution-free theory and a universally consistent kernel method, and compares it with pooling. Across one synthetic and three real-world data sets, marginal transfer learning consistently outperforms pooling, although the difference can be small when tasks are similar.

  • Problem

    Domain generalization requires accurate prediction on future unlabeled tasks whose distributions may vary because of environmental, technical, or biological factors.

  • Method

    The paper augments the feature space with the feature marginal distribution, develops two formal statistical frameworks and distribution-free kernel-based generalization analyses.

  • Results

    Marginal transfer learning consistently outperforms a pooling baseline on one synthetic and three real-world data sets, though differences are small on some data sets.

  • Takeaways & Limitations

    The framework provides learning-theoretic foundations for domain generalization, including a universally consistent procedure and an efficient implementation.

  • Takeaways & Limitations

    Without assumptions linking task marginals to conditional label distributions, domain generalization may perform no better than naïve pooling.

Abstract

from arXiv · show

In the problem of domain generalization (DG), there are labeled training data sets from several related prediction problems, and the goal is to make accurate predictions on future unlabeled data sets that are not known to the learner. This problem arises in several applications where data distributions fluctuate because of environmental, technical, or other sources of variation. We introduce a formal framework for DG, and argue that it can be viewed as a kind of supervised learning problem by augmenting the original feature space with the marginal distribution of feature vectors. While our framework has several connections to conventional analysis of supervised learning algorithms, several unique aspects of DG require new methods of analysis. This work lays the learning theoretic foundations of domain generalization, building on our earlier conference paper where the problem of DG was introduced (Blanchard et al., 2011). We present two formal models of data generation, corresponding notions of risk, and distribution-free generalization error analysis. By focusing our attention on kernel methods, we also provide more quantitative results and a universally consistent algorithm. An efficient implementation is provided for this algorithm, which is experimentally compared to a pooling strategy on one synthetic and three real-world data sets.

1. Introduction

Domain generalization learns from labeled data sets across related tasks to predict on an unseen unlabeled task. The paper reframes DG through marginal transfer learning, develops theory and algorithms, and evaluates them empirically.

  • Domain generalization uses labeled data from several related prediction problems to generalize to a future task without labeled data.
  • DG arises when environmental, technical, or biological variation makes distributions differ across applications such as precision medicine.
  • Marginal transfer learning augments the feature space with the marginal distribution generating features, treating DG as supervised learning.
  • The paper develops two statistical frameworks, including one allowing arbitrary within-data-set dependencies and another assuming i.i.d. observations for refined risk analysis.
  • A kernel-based procedure is universally consistent, computationally efficient, freely available, and evaluated against pooling on one synthetic and three real-world data sets.

2. Motivating Application: Automatic Gating of Flow Cytometry Data

Flow cytometry provides patient-specific cell measurements for diagnosing blood-related pathologies, but patient variation complicates automated classification. The motivating application is automating expert gating of unwanted cells.

  • Flow cytometry quantitatively measures individual cells using d-dimensional attribute vectors from patient-specific distributions.
  • A new patient’s data must be purified by identifying desired cell types, such as lymphocytes, before diagnosis.
  • Clinical gating manually removes unwanted cells through sequential two-dimensional scatter plots with drawn lines or polygons.
  • Manual gating is difficult to capture with a small set of simple rules because it requires domain-specific knowledge and iterative refinement.
  • Historical expert-gated patients provide training data, while biological and technical variation creates differing distributions alongside shared trends that may enable automation.

3. Formal Setting and General Results

The paper formalizes domain generalization through two data-generation models and an augmented feature space that combines a task’s marginal distribution with individual inputs. It derives generalization and consistency results, while showing that asymptotic Bayes performance depends on how strongly posteriors are determined by marginals.

  • 3. Formal Setting and General Results: The formal setting assumes Radon observation spaces and represents each training sample through its empirical joint and unlabeled marginal distributions.
  • 3.1 Data Generation Models: Two data-generation models are introduced: the agnostic model is more general, while the 2-stage model is a special case with latent task distributions generating within-task observations.
  • 3.2 Decision Functions and Augmented Feature Space: Domain generalization is formalized as learning a rule from labeled samples that predicts for an unseen task using its unlabeled data set.
  • 3.2 Decision Functions and Augmented Feature Space: The decision function operates on the augmented space PX × X, conveying information about a new task entirely through its marginal feature distribution.
  • 3.3 Risk and Generalization Error Bound under the Agnostic Generative Model: Under the agnostic model, the paper gives a distribution-free generalization bound whose main term is the conventional Rademacher complexity on the augmented space.
  • 3.4 Idealized Risk under the 2-stage Generative Model: The idealized 2-stage risk treats the true test marginal as available and becomes a standard supervised-learning risk over the augmented feature and label pair.
  • 3.4 Idealized Risk under the 2-stage Generative Model: For binary classification with 0/1 loss, the optimal predictor is expressed through the conditional label expectation on the augmented feature space, and classification-calibrated losses transfer consistency to 0/1 loss.
  • 3.4 Idealized Risk under the 2-stage Generative Model: The proposed algorithm is E∞-consistent when both the number of training tasks and average sample size grow, but without assumptions on µ its Bayes risk may exceed single-task Bayes risk.

4. Related Work

Related work distinguishes domain generalization from learning to learn, multitask learning, and domain adaptation by its focus on unseen tasks without labeled target data. It also situates the paper within prior DG methods and its earlier formulation.

  • Multitask Learning: Multitask learning improves predictors for known training tasks, while DG targets generalization to a new task.
  • Domain Adaptation: Domain adaptation typically optimizes performance for a fixed target distribution, whereas DG designs one predictor f(PX, x) across future tasks.
  • Domain Adaptation: Unlike most DG methods, unsupervised domain adaptation usually retrains when a new unlabeled target data set arrives.
  • Learning to Learn: Learning to learn transfers knowledge to future tasks with labeled target data, whereas DG receives only unlabeled data and requires no additional learning.LTL can achieve the new task’s Bayes risk; DG instead must generalize without target labels.
  • Prior DG Work: The paper extends the authors’ earlier DG work with an agnostic generative model, broader theory, regression results, literature review, and experiments.
  • Prior DG Work: Many DG methods learn a common feature space for all tasks, and these approaches are described as complementary to the paper’s method.

5. Learning Algorithm

The learning algorithm augments each input with its task’s empirical marginal distribution and applies a kernel on this extended space. Kernel choices interpolate between independent task models, multitask learning, and pooling.

  • Kernel Construction: A kernel on PX × X induces an RKHS, enabling norm-regularized empirical-risk methods analogous to support vector machines and kernel ridge regression.
  • Extended Representation: The algorithm represents each example as (empirical marginal distribution, feature vector) and learns over the extended space PX × X.The empirical marginal distribution plays a role analogous to the task index in multitask learning.
  • Kernel Construction: The method uses product-form kernels combining kernels on probability distributions and feature vectors, including Gaussian-like and polynomial-like choices.
  • Kernel Construction: Kernel mean embeddings map empirical marginal distributions into an RKHS; universal base kernels make this mapping injective.
  • Relation to Other Kernel Methods: τ = 0 trains independent kernel machines with no generalization to a new data set, whereas τ = 1 gives a pooling strategy independent of the marginal.
  • Relation to Other Kernel Methods: Intermediate 0 < τ < 1 yields a multitask kernel, but it was outperformed by pooling for generalization to a new unlabeled task.
  • Special Cases: When labels are constant within each task and kX is constant, the method reduces to distributional classification or regression.

6. Learning Theoretic Study

The paper develops distribution-free error analysis for the kernel method under agnostic and two-stage generative models. Under universality and boundedness conditions, it proves consistency, with different roles for task count and within-task sample sizes.

  • Generalization Analysis: The generalization analysis is nontrivial because the agnostic and two-stage models impose distinctive sampling structures and risks.
  • Generalization Analysis: Theorem 11 provides uniform estimation-error control over RKHS balls under the agnostic generative model.
  • Universal Kernels: Under compactness and universal-kernel assumptions, the product kernel on PX × X is universal.
  • Universal Consistency: With λ(N) → 0 and λ(N)N/log N → ∞, the learned risk converges almost surely to the best possible risk as N → ∞.
  • Universal Consistency: Consistency under the agnostic model requires only the number of tasks N to diverge; individual sample sizes ni may remain bounded.
  • Two-Stage Model: Under the two-stage model, convergence to the idealized risk requires within-task sample sizes to grow alongside the number of tasks.
  • Sample Sizes and Subsampling: The paper’s non-asymptotic analysis supports subsampling as an efficiency/accuracy tradeoff, while sharper convergence rates and refined N–n balancing remain future work.

7. Implementation

The implementation adapts standard kernel optimization to marginal transfer learning and offers approximate feature mappings for scalability. It extends random Fourier features to Gaussian kernels and compares them with Nyström approximations.

  • Kernel Optimization: For hinge loss, the optimization is equivalent to a cost-sensitive support vector machine with example-dependent costs and no offset.
  • Kernel Optimization: Existing SVM software can learn the weights using a user-specified kernel matrix and example-dependent costs.
  • Scalability: Standard nonlinear SVM solvers have complexity between O(N^2n^2) and O(N^3n^3) when all task sample sizes equal n.
  • Scalability: Approximate feature mappings replace kernel evaluations with finite-dimensional inner products, enabling efficient linear solvers.
  • Approximate Feature Mappings: The implementation examines Nyström approximations and random Fourier features, introducing a random-Fourier extension for marginal transfer learning with performance analysis.
  • Approximate Feature Mappings: Nyström approximations apply to any positive definite kernel, whereas random Fourier features require shift-invariant kernels and are easier to implement.

8. Experiments

Experiments compare marginal transfer learning with pooling across synthetic, regression, satellite, and flow-cytometry settings. The proposed method generally benefits from more tasks and examples, while pooling can remain ineffective on synthetic data.

  • Experimental design: The experiments compare marginal transfer learning with pooling using held-out tasks and empirical risk under classification and regression losses.The evaluation includes three classification problems with hinge loss and one regression problem with epsilon-insensitive loss.
  • Synthetic data: Synthetic tasks vary across 16, 64, or 256 tasks and 8, 16, 32, or 256 examples per task, with rotated ellipses defining task-specific labels.The ellipse’s major-axis orientation is randomly rotated for each task, and points are labeled by their position relative to that axis.
  • Synthetic data: Marginal transfer learning significantly outperforms pooling on the synthetic data, improving as the number of tasks and examples increases, while pooling performs no better than random guessing.The synthetic test set contains 10 tasks, each with one million data points.
  • Parkinson’s telemonitoring: On Parkinson’s telemonitoring data, marginal transfer learning clearly outperforms pooling, especially as the numbers of training users and recordings per user increase.The data contain voice measurements from 42 people with early-stage Parkinson’s, with approximately 200 recordings per patient.
  • Real-world classification: Across satellite and flow-cytometry experiments, the proposed method usually has lower classification error than pooling, with error decreasing as the number of training tasks increases.In flow cytometry, increasing examples per task has little effect once n ≥ 10^3, unlike the earlier experiments where n was at most 256.

9. Discussion

The discussion presents marginal transfer learning as regularized learning on an augmented feature space, supported by universal consistency and approximate implementations. Empirically, it consistently beats pooling but can offer only small gains when task transfer is weak.

  • Method and theory: The method minimizes regularized empirical loss in an RKHS over the augmented domain P_X × X and establishes universal consistency under two sampling plans.The paper also constructs a universal kernel and gives approximate feature mappings for efficient implementation.
  • Empirical findings: On one synthetic and three real-world data sets, marginal transfer learning consistently outperforms pooling, although the difference is small on some data sets.The discussion attributes varying gains to differences in the utility of transfer across domain-generalization problems.
  • Applications: Adaptive classifiers are motivated for biomedical applications involving biological or technical variation, including brain-computer interfaces and patient monitoring.The discussion gives continuous electrocardiogram monitoring and heartbeat classification as an example.
  • Future directions: Open theoretical questions include conditions under which domain-generalization risk approaches the test distribution’s expected Bayes risk and faster learning rates under distributional assumptions.The paper also identifies extensions to multiclass classification, class-probability estimation, and robustness to noise.
  • Future directions: A proposed extension considers semi-supervised domain generalization when a small number of test labels accompany many unlabeled test examples.The training-derived marginal-adaptive function could act as a prior or informed guess for learning on the test data.

Appendix A. Proofs, Technical Details, and Experimental Details

The appendix contains the paper’s remaining proofs and additional technical and experimental details.

  • The appendix provides the remaining proofs together with additional technical and experimental details.

A.1 Proof of Proposition 7

The proof establishes convergence by controlling the test marginal and integrating the resulting bound over the distribution of test tasks.

  • The proof uses inner regularity of the test marginal to restrict attention to a compact set and continuity to obtain uniform control.
  • The law of large numbers makes the empirical test marginal converge in probability, yielding the required probability bound for sufficiently large test samples.
  • Dominated convergence then integrates the bound over the test-distribution meta-distribution to obtain the desired conclusion.

A.2 Proof of Corollary 14

The proof establishes consistency by approximating the optimal risk with a function in the universal-kernel RKHS and controlling the learned minimizer probabilistically.

  • Universality and Lipschitz loss provide an RKHS function whose risk is arbitrarily close to the optimal risk.
  • Comparing the regularized objective at the minimizer and the null function bounds the minimizer’s RKHS norm by a quantity proportional to 1/λ.
  • Choosing δ = 1/N^2 yields probability controls that hold with probability at least 1 − 1/N^2.
  • For sufficiently large N, the approximating function lies within the relevant RKHS ball, enabling the high-probability consistency argument.
  • The result follows for every ε > 0 after selecting N sufficiently large and applying the Borel-Cantelli lemma.

A.3 Proof of Theorem 15

The proof of Theorem 15 begins by decomposing the difference between training loss and conditional risk at infinite sample size.

  • The analysis controls the difference between training loss and conditional risk through an explicit decomposition.

A.3.1 Control of Term (I)

The proof controls Term (I) through Lipschitz loss bounds, empirical-distribution concentration, kernel regularity, and approximation arguments for the augmented feature space.

  • Lipschitzness of the loss bounds the first component of Term (I).
  • Empirical distributions concentrate around their population distributions under bounded-kernel assumptions.
  • The reproducing property and kernel regularity convert distributional deviations into bounds over RKHS balls.
  • Twice differentiable kernels can yield a Lipschitz canonical feature mapping, corresponding to regularity order α = 1.
  • The proof uses mean-value arguments and derivative bounds to establish the required regularity for distance- and inner-product-based kernels.
  • Universality follows by representing product-space features in a tensor-product RKHS and applying Stone-Weierstrass density.
  • Random Fourier features approximate the extended-space kernel by sampling from the spectral distribution and constructing finite-dimensional features.
  • The approximation reaches any prescribed error with very high probability as the feature-sampling dimensions increase.
Loading 1711.07910v3…