PaperlayerView exact revision on arXiv

Source-linked AI summary

An introduction to domain adaptation and transfer learning

Wouter M. Kouw, Marco Loog

arXiv:1812.11806v2cs.LGcs.CVstat.ML

TL;DR

Training data may differ from test data, limiting standard classifiers' ability to generalize across domains. This report introduces transfer learning and domain adaptation through risk minimization, data-shift types, generalization bounds, and categorized adaptation approaches, concluding that guarantees depend on domain dissimilarity and remain unavailable in the unconstrained case.

  • Problem

    Training and test data can have different distributions, raising the question of when and how a classifier can generalize from a source domain to a target domain.

  • Method

    The report extends risk-minimization analysis with generalization bounds, data-shift categories, and a taxonomy of approaches including importance-weighting, subspace mapping, domain-invariant spaces, feature augmentation, minimax estimators, and robust algorithms.

  • Results

    The report concludes that unconstrained source-target differences provide no guarantees above chance, while guarantees in particular cases depend on domain dissimilarity.

  • Takeaways & Limitations

    Successful adaptation is more likely when source and target populations are similar, while highly dissimilar domains remain difficult to adapt between.

  • Takeaways & Limitations

    The covariate-shift assumption can be invalid because equal class-posterior distributions across the whole feature space are difficult to satisfy.

Abstract

from arXiv · show

In machine learning, if the training data is an unbiased sample of an underlying distribution, then the learned classification function will make accurate predictions for new samples. However, if the training data is not an unbiased sample, then there will be differences between how the training data is distributed and how the test data is distributed. Standard classifiers cannot cope with changes in data distributions between training and test phases, and will not perform well. Domain adaptation and transfer learning are sub-fields within machine learning that are concerned with accounting for these types of changes. Here, we present an introduction to these fields, guided by the question: when and how can a classifier generalize from a source to a target domain? We will start with a brief introduction into risk minimization, and how transfer learning and domain adaptation expand upon this framework. Following that, we discuss three special cases of data set shift, namely prior, covariate and concept shift. For more complex domain shifts, there are a wide variety of approaches. These are categorized into: importance-weighting, subspace mapping, domain-invariant spaces, feature augmentation, minimax estimators and robust algorithms. A number of points will arise, which we will discuss in the last section. We conclude with the remark that many open questions will have to be addressed before transfer learners and domain-adaptive classifiers become practical.

1 Introduction

The introduction motivates domain adaptation and transfer learning as responses to biased training data and differing source–target distributions. It frames the report around when and how classifiers can generalize across domains, with applications and categorized approaches.

  • Biased training data can make classifiers misjudge outcome frequencies and generalize poorly to broader populations.The introduction contrasts data collected from older patients with the total human population and notes resulting risks for younger patients.
  • Domain adaptation and transfer learning address settings where target labels are unavailable but similar source-population data can provide additional information.The central challenge is overcoming domain differences so a source-trained classifier generalizes to the target domain.
  • Clinical imaging illustrates the practical motivation: scanner differences can impair transferred systems, while adaptation can avoid annotating images from a new scanner.The differences arise from scanner mechanics, calibration, vendor, and acquisition protocol.
  • Applications include bioinformatics, predictive maintenance, personalized spam filtering, speaker recognition, face recognition, and activity recognition.These settings involve changing organisms, institutes, machines, or user and speaker populations.
  • The report organizes its treatment around when and how classifiers generalize, covering error bounds, data shifts, and adaptation approaches.Its outline introduces classification, risk minimization, domain definitions, simple distribution shifts, and methods for more complex shifts.

2 Classification

This section introduces statistical classification and risk minimization: classifiers map feature-based objects to classes, and training selects a low-risk function from a hypothesis space. Generalization depends on empirical estimation, sample size, hypothesis complexity, and regularization.

  • A statistical classifier maps feature-encoded objects to one of finitely many classes, producing predictions that can incur unavoidable errors.The classifier is represented as h, with h(x) denoting its prediction for input x.
  • Risk minimization: Classification risk is expected loss under the data distribution, and risk minimization selects the classifier with the smallest value.The choice of loss function affects classifier behavior, while convex surrogate losses are easier to optimize but may produce different solutions.
  • Risk minimization: Training minimizes empirical risk over a hypothesis space using finite labeled samples, while generalization concerns performance on future samples.Empirical risk approximates expected risk through a sample average, and its minimization is called training.
  • Generalization: More independently and identically distributed samples make empirical risk converge toward true risk and the learned classifier toward the optimum.The difference between true and empirical error is the generalization error.
  • Generalization: Generalization bounds trade classifier complexity against sample size, motivating simpler hypothesis spaces when data are scarce.Flexible models can fit training data exactly yet overfit, while regularization penalizes flexibility and excessive regularization can underfit.

3 Domain adaptation and transfer learning

Domain adaptation studies how a classifier trained on a source domain can generalize to a different target domain. The section formalizes domains and explains that cross-domain guarantees depend on joint error, domain divergence, and restrictions on domain differences.

  • A domain combines an input space X, an output space Y, and an associated probability distribution p; domains differ when at least one component differs.The source and target domains are denoted S and T, with domain-specific joint, marginal, and class-conditional distributions.
  • The source domain supplies labeled samples for training, whereas target labels are generally unavailable and the learned classifier must generalize across distributions.Source and target samples use distinct notation for observations and labels.
  • Cross-domain generalization error: Cross-domain generalization bounds require assumptions about the relationship between source and target domains, including low error for an ideal joint hypothesis.If the ideal joint error is too large, a source-trained classifier cannot be approximately correct in the target domain.
  • Cross-domain generalization error: The H∆H-divergence measures domain discrepancy by comparing classifier disagreement across the source and target distributions.It uses the largest difference in disagreement probabilities over pairs of classifiers.
  • Cross-domain generalization error: A non-adaptive bound indicates that larger ideal joint error or domain divergence reduces expected source-to-target generalization, motivating restricted adaptation strategies.Tighter adaptation bounds are possible for simplified shift settings, but not for the unrestricted general case.

4 Common data shifts

Data set shift describes domain changes where some distribution components differ while others remain constant. Prior, covariate, and concept shift support different adaptation strategies, whereas general shifts may be too difficult without exploitable domain relationships.

  • Covariate shift: Covariate shift changes data distributions while keeping posterior distributions equivalent.For covariate shift, pS(x) and pT(x) differ, but the posterior distributions remain equal.
  • Prior shift: Prior shift changes class proportions while keeping class-conditional distributions equivalent.For prior shift, pS(y) ≠ pT(y) but pS(x|y) = pT(x|y).
  • Adaptation implications: For prior and covariate shift, adaptation does not require labeled data from both domains.Prior shift can use target labels without unlabeled target samples, while covariate shift corrects source samples using pT(x)/pS(x).
  • Concept shift: Concept shift keeps data distributions constant while changing posterior distributions and shifting the decision boundary.Adaptation in this setting requires labeled target data because conditional distributions require simultaneous observations of both variables.
  • General domain shift: General domain shifts are hardest because learning may be impossible unless the domains have another exploitable relationship.Examples include a shared predictor, constrained worst-case labelings, or low data divergence.

5 Approaches

Domain adaptation approaches use weighting and distributional corrections to make source data informative for the target domain. Their reliability depends on estimating weights well, controlling variance, and keeping source and target domains sufficiently close.

  • Importance-weighting: Importance-weighting adapts source examples by assigning greater influence to samples that better represent the target domain.Generalization depends on hypothesis complexity, sample size, and the divergence between source and target distributions.
  • Importance-weighting: A valid importance-weighting bound requires finite hypothesis complexity and domain divergence, with nonzero weights.When the domains are too far apart, the Rényi divergence diverges and the bound no longer holds.
  • Importance-weighting: As domain divergence increases, the sample size must increase at a corresponding rate to maintain the same probability of approximate correctness.This relationship is stated for a fixed hypothesis space, such as linear classifiers.
  • Weight estimation: Importance weights can be estimated by modeling each density separately or estimating the density ratio directly.Direct methods minimize discrepancies between weighted source and target distributions while constraining weights to avoid unstable scaling.
  • Weight estimation: Gaussian density-ratio estimation can produce pathological classifiers when target variance makes weight variance diverge.If the target variance exceeds 2 in the stated example, one sample may receive overwhelming weight while others are nearly ignored.
  • Weight estimation: KDE controls ratio variance through kernel bandwidth, but selecting that hyperparameter remains unclear.KDE estimates target and source densities before dividing the target density by the source density.
  • Weight estimation: Kernel Mean Matching minimizes Maximum Mean Discrepancy between reweighted source samples and target samples.The discrepancy uses kernel functions, and convergence rates can be computed under weight-bounding conditions.
  • Weight estimation: KLIEP and LSIF provide alternative direct ratio-estimation objectives based on logarithmic and squared-error criteria.KLIEP approximates a target expectation using unlabeled target samples, while LSIF replaces expectations with sample averages.

5.2 Subspace mappings

Subspace mappings adapt domains by aligning representations, typically through transformations between domain-specific subspaces. Subspace Alignment provides a simple, robust implementation that has been extended in several directions.

  • Domains can occupy different subspaces, requiring mappings such as rotations, affine transformations, or nonlinear transformations.
  • Subspace Alignment rotates source principal components to align them with target components using a linear transformation matrix.The matrix is formed from transposed source components multiplied by target components.
  • Subspace Alignment selects an optimal subspace dimensionality so noise components do not influence domain matching.
  • Subspace Alignment is computationally inexpensive, easy to implement, intuitive to explain, and robust to unusual problem settings.
  • Extensions include landmark-based kernelized alignment, subspace distribution alignment, and a semi-supervised variant.

Metrics

Metric-based approaches transform cross-domain distances so same-class samples remain similar while different-class samples remain separated. These methods include Mahalanobis metrics, Fisher-based criteria, and asymmetric mappings for feature spaces of different dimensionalities.

  • Information-Theoretic Metric Learning learns a Mahalanobis metric for a subsequent nearest-neighbour classifier.Metrics define how distances between vector-space points are computed.
  • Transforming a space before Euclidean distance measurement is equivalent to measuring Mahalanobis distances weighted by the transformation matrix.
  • Correspondence constraints keep same-class samples from different domains close and different-class samples separated when target labels are available.The constraints impose an upper bound for same-label pairs and a lower bound for different-label pairs.
  • ITML requires source and target domains to share feature-space dimensionality, whereas ARC-t supports mappings between different-dimensional feature spaces.
  • Fisher-based criteria preserve class separability, and FIDOS additionally minimizes domain separability while incorporating multiple source domains.
  • Other methods align class margins for subsequent maximum-margin classifiers.

Manifolds

Manifold methods assume that source and target domains are connected through a structured space of transformations. They model intermediate domains or probability distributions to construct transfer representations and measure domain discrepancy.

  • Transformation-manifold methods assume a parameterized family of transformations connects the source and target domains.Each point on the manifold represents a possible domain generated by transformation parameters.
  • Geodesic Flow Kernel integrates classifiers across intermediate subspaces along a transformation path between source and target domains.Source data are projected onto intermediate subspaces, and classifiers propagate labels along the path.
  • Statistical manifolds represent probability distributions, with geodesic length measured by the Hellinger distance as a domain-discrepancy measure.

5.3 Finding domain-invariant spaces

Domain-invariant-space methods seek representations that remove domain-specific variation while retaining useful structure. They use shared components, discrepancy minimization, nonlinear projections, autoencoders, and related transfer subspaces.

  • Domain-invariant representations address the limitation that transformations otherwise leave classifiers in domain-specific spaces.
  • Common-component methods project both domains onto components selected by minimizing their Maximum Mean Discrepancy.
  • Distribution Matching Embedding learns an orthonormal projection that minimizes distribution discrepancy, with regularization available through the joint-kernel formulation.
  • Nonlinear distribution-matching methods add within-class regularization and relax the limitations of linear projection matrices.
  • Kernel learning can select basis functions whose distributions are similar by minimizing distributional variance.
  • Transfer Subspace Learning seeks a lower-dimensional subspace that discriminates target samples using source samples while minimizing Bregman divergence to both domains.
  • Autoencoders minimize reconstruction error through neural-network mappings, with deep variants stacking nonlinear layers for flexible transformations.
  • Kernel methods can recover nonlinear mappings but scale poorly, whereas neural networks provide better scaling with sample count.

5.4 Feature augmentation

Feature augmentation adapts across domains by identifying corresponding features and adding target-domain pivot words to a source bag-of-words representation. The approach ranges from linear correlation methods to nonlinear kernel and co-training methods, while correspondence discovery remains unresolved.

  • In NLP, differing word frequencies and context-specific vocabulary prevent document or sentence systems from generalizing across domains.
  • Feature augmentation adds target-domain pivot words to the bag-of-words representation, learns source–target correspondences, and trains in the augmented feature space.
  • Corresponding features can be extracted through joint principal components or cross-correlation, but these earliest methods model only linear relationships.
  • Later feature-coupling methods use kernelization or co-training to model more nonlinear relationships between features.
  • Finding corresponding features, or coupling subspaces more generally, remains an open question.

5.5 Minimax estimators

Minimax estimators address uncertain domain shifts by optimizing classifier risk against worst-case choices constrained by source-distribution information. These methods are conservative and can produce high-confidence predictions in source-supported regions while remaining uncertain elsewhere.

  • Minimax estimation maximizes an uncertain quantity before minimizing risk with respect to classifier parameters, producing conservative behavior.
  • The Robust Bias-Aware classifier treats the target posterior distribution as uncertain and constrains adversarial posteriors to match source feature-statistic moments.
  • First-order moment constraints equate the adversary’s posterior sample average with the first-order moment of the source label distribution.
  • Higher-order moments can also be included as constraints in the minimax estimator.
  • The estimator returns high-confidence predictions where source probability mass is high and uniform class predictions where source probability mass is low.
  • Robust adaptive algorithms can align source and target hinge loss, discouraging support vectors in regions with low target probability mass.

6 Discussion

The discussion identifies practical benefits and unresolved challenges of domain adaptation, including restrictive assumptions, task-specific discrepancy measures, and the risks and costs of sequential adaptation. It emphasizes that successful transfer depends on domain similarity and better understanding of domain differences.

  • 6.1 Validity of the covariate shift assumption: The covariate-shift assumption pT(y | x) = pS(y | x) may be too restrictive because equal posterior distributions are stronger than equal decision boundaries.
  • 6.1 Validity of the covariate shift assumption: Experiments suggest some robustness to covariate-shift violations, motivating perturbation analyses based on deviations between posterior distributions or decision boundaries.
  • 6.2 Measuring domain discrepancy: General domain-discrepancy measures yield looser generalization bounds, so more precise metrics incorporating task-specific prior knowledge are needed.
  • 6.2 Measuring domain discrepancy: Task- or data-specific discrepancy detection is difficult, and explicit adaptation descriptions could reveal which discrepancies matter for particular applications.
  • 6.3 Combining datasets: Batch effects from reagents, timing, or atmospheric conditions can make data from different research centers unsuitable for direct combination.
  • 6.3 Combining datasets: Modeling confounding variables could help correct batch effects and combine datasets across research centers, increasing the value of existing data and statistical power.
  • 6.4 Sequential adaptation: Successful adaptation means positive transfer, but domains that are too dissimilar remain difficult to adapt between.
  • 6.4 Sequential adaptation: Sequential adaptation could replace one large domain shift with several smaller transitions and need not rely on a domain-manifold mapping.

7 Conclusion

The conclusion asks when and how source-domain classifiers can generalize, finding that guarantees require constrained shifts and depend on domain dissimilarity. It emphasizes unresolved methodological questions and concludes that practical transfer learners and domain-adaptive classifiers remain a future goal.

  • Without constraints on source–target differences, no performance guarantee above chance is available.
  • Covariate shift is comparatively constrained, but applicability depends on whether equal class-posterior distributions hold and how strongly they are violated.
  • Importance-weight estimation requires checking domain separation, sample sizes, estimator complexity, denominator probabilities, and hyperparameter estimation feasibility.
  • Preprocessing choices may alter importance-weight estimation and classifier training, while separate normalization can conflict with the covariate-shift assumption.
  • General shifts are harder to study because multiple changes may be non-identifiable, making method success and estimation effects difficult to predict.
  • Many open questions remain before transfer learners and domain-adaptive classifiers become practical tools.
Loading 1812.11806v2…