Source-linked AI summary
Domain Generalization via Invariant Feature Representation
Krikamol Muandet, David Balduzzi, Bernhard Schölkopf
TL;DR
The paper addresses how to generalize knowledge from related domains to previously unseen domains when domain distributions differ. It proposes DICA, a kernel-based invariant representation method that reduces marginal dissimilarity while preserving the functional relationship, and reports theoretical and empirical support for improved generalization. The approach assumes that this functional relationship is stable or varies smoothly across domains.
Problem
Domain generalization seeks to apply knowledge from related domains to previously unseen domains, where standard learning assumptions fail under distributional variation.
Method
DICA learns a kernel-based transformation that minimizes differences between domain marginals while preserving the functional relationship P(Y |X).
Results
Theoretical analysis and experiments on synthetic and real-world datasets show that DICA learns invariant features and performs well for domain generalization.
Takeaways & Limitations
DICA provides an invariant-subspace approach for generalizing classifiers to new domains without retraining, while relating to KPCA, TCA, and COIR.
Takeaways & Limitations
The method assumes that the functional relationship is stable or varies smoothly across domains; its applicability when conditional distributions differ substantially remains unclear.
Abstract
from arXiv · showhide
This paper investigates domain generalization: How to take knowledge acquired from an arbitrary number of related domains and apply it to previously unseen domains? We propose Domain-Invariant Component Analysis (DICA), a kernel-based optimization algorithm that learns an invariant transformation by minimizing the dissimilarity across domains, whilst preserving the functional relationship between input and output variables. A learning-theoretic analysis shows that reducing dissimilarity improves the expected generalization ability of classifiers on new domains, motivating the proposed algorithm. Experimental results on synthetic and real-world datasets demonstrate that DICA successfully learns invariant features and improves classifier performance in practice.
1. Introduction
Domain generalization aims to transfer knowledge from related source domains to previously unseen domains without observing test domains during training. DICA learns features that reduce marginal-distribution differences while preserving the input–output relationship.
- Problem setting: Domain generalization applies knowledge from related domains to previously unseen domains, unlike domain adaptation, which observes the target domain during training.The classifier is intended to generalize to new domains without retraining.
- Problem setting: Covariate shift changes marginal distributions P(X) across domains while the functional relationship P(Y |X) remains largely stable.This mismatch violates the standard assumption that training and test data share a distribution.
- Motivation: Domain adaptation can require repeating classifier adaptation for every new patient, whereas this work uses across-domain information to generalize without retraining.The motivation is especially concrete for applications such as medical diagnosis.
- Approach: DICA is a kernel-based algorithm that minimizes differences between domain marginals while preserving the functional relationship P(Y |X).It extracts invariant features intended to transfer across domains.
- Related work: Prior shared-subspace methods reduce source–target mismatch, but previous work had not fully investigated whether a learned subspace generalizes to unseen domains.DICA is related to KPCA, TCA, and COIR.
2. Domain-Invariant Component Analysis
DICA learns a low-dimensional kernel transformation that reduces dissimilarity across domains while preserving the functional relationship between inputs and outputs. Its formulation uses distributional variance and central-subspace estimation, with a complexity tradeoff that tightens a generalization bound.
- DICA objective: DICA estimates a transformation that generalizes well to unseen domains by reducing dissimilarity between transformed training distributions while preserving the relationship between X and Y.The transformation targets Y ⊥ X | B(X) and is formulated using samples from multiple domains.
- Distributional variance: Distributional variance measures dissimilarity across domains through kernel mean embeddings and is zero exactly when all domain distributions are equal.A characteristic kernel makes the mean embedding injective, preserving distributional information.
- DICA objective: DICA finds an orthogonal transform B onto a low-dimensional subspace, m ≪ n, by minimizing empirical distributional variance while preserving the functional relationship between X and Y.The transformed-domain variance is estimated using block kernel and coefficient matrices.
- Central subspace: The central subspace is the minimal representation capturing the functional relationship between X and Y, and DICA extends its linear form to nonlinear kernel basis functions.Inverse regression estimates the relevant basis through covariance operators and generalized eigenvalue problems.
- Unsupervised DICA: When target information is unavailable, unsupervised DICA maximizes variance of X in feature space instead of preserving the central subspace.This variant is motivated by applications such as image denoising.
- Learning-theoretic bound: DICA balances reduced distributional variance against transform complexity, because more complicated transforms worsen the guarantee; its denominator therefore tightens the bound in Theorem 5.The optimization can be rewritten as a constrained problem whose stationary condition is a generalized eigenvalue problem.
3. Experiments
Experiments compare DICA with KPCA, UDICA, and COIR on synthetic, flow-cytometry, and Parkinson’s telemonitoring data. DICA learns more stable representations and generally improves classification and regression performance on unseen domains, while distributional models account for inter-domain variation.
- Experimental design: DICA was evaluated on synthetic data, flow-cytometry classification, and Parkinson’s telemonitoring regression.The experiments compare DICA with KPCA, UDICA, and COIR.
- Synthetic experiments: UDICA and DICA produced more stable projections across three unseen synthetic test domains than KPCA and COIR.COIR showed a substantial training–test difference, suggesting overfitting.
- Flow cytometry: With sufficiently many samples, DICA outperformed other methods in both pooling and distributional SVM settings on GvHD data.Pooling and distributional SVM achieved comparable accuracies in this experiment.
- Flow cytometry: Distributional SVM outperformed pooling SVM in leave-one-out evaluation, while DICA-based invariant features yielded higher accuracies than other approaches.The evaluation covered 30 subjects.
- Parkinson’s telemonitoring: DICA consistently, though not statistically significantly, outperformed other preprocessing approaches for Parkinson’s symptom prediction.Inter-patient variation worsened prediction on new patients, while reducing it with DICA improved accuracy; distributional GP regression further improved generalization.
4. Conclusion and Discussion
The paper concludes that DICA is theoretically and empirically significant for domain generalization, but its central assumption limits applicability. The authors also describe constrained and jointly optimized extensions.
- Conclusion: DICA learns an invariant transformation and is supported by both theoretical and empirical results for domain generalization.The conclusion also relates DICA to KPCA, COIR, and TCA.
- Discussion: DICA assumes that the functional relationship between inputs and outputs is stable or varies smoothly across domains.The authors motivate this assumption with flow-cytometry and Parkinson’s telemonitoring applications.
- Discussion: DICA can be extended with constraints for semi-supervised settings or with distributional-variance regularization for joint feature and classifier optimization.Examples include manifold or neighborhood-preserving constraints.
Appendix A. Domain Generalization and Related Frameworks
Domain generalization addresses transfer to unseen domains without observing the target domain during training. It differs from related frameworks in target availability, retraining requirements, and the type of distribution mismatch addressed.
- Background: Standard machine learning assumes independent and identically distributed observations, an assumption often violated in practice.The appendix frames distribution mismatch as a motivation for alternative learning settings.
- Related frameworks: Transfer learning uses knowledge from previous tasks to improve a target predictive function when the target has limited training data.Domain generalization is described as transfer learning when target-task knowledge is unavailable.
- Related frameworks: Multitask learning trains multiple related tasks simultaneously, whereas domain generalization targets previously unseen domains without retraining on the new task.Retraining is often required when learning a new task in multitask learning.
- Related frameworks: Domain adaptation observes the target domain during training, while domain generalization does not.Domain adaptation primarily addresses mismatch between training and test distributions; domain generalization includes multiple source domains.
- Comparison: Table 4 summarizes the main differences among these learning frameworks.
Appendix B. Proof of Theorem 1
The appendix proves that distributional variance under a characteristic kernel is zero exactly when all distributions in the set are identical. The proof uses the mean distribution and the metric property of the kernel-induced distance.
- Proof: The proof defines the mean distribution as the average of the distributions in P and uses linearity of expectation to average their mean embeddings.
- Proof: Expanding the distributional-variance expression completes the algebraic proof.
- Theorem 1: The distributional variance VH(P) equals zero if and only if P1 = P2 = · · · = PN for a characteristic kernel.This is the statement of Theorem 1.
- Proof: The kernel-induced distance is zero exactly when two distributions are equal, allowing zero variance to imply Pi = P̄ for every i.The converse follows when all distributions equal the mean distribution.
Appendix C. Proof of Theorem 2
The appendix establishes that the empirical quantity bVH(S) consistently estimates the population quantity VH(P). The result follows from convergence of the estimated mean embedding and Gram-related operator.
- Fast convergence of the estimated mean embedding implies convergence of the estimated Gram-related quantity to its population counterpart.
- bVH(S) is a consistent estimator of VH(P).
Appendix D. Derivation of Eq. (8)
The derivation expresses the covariance of the inverse regressor through conditional covariance operators in RKHSs, then estimates it from finite samples using covariance operators and kernel matrices.
- DICA represents the covariance of the inverse regressor E[φ(X)|Y] using conditional covariance operators.
- Under a stated regularity condition, the conditional covariance operator Σxx|y equals the expected conditional variance of φ(X) given Y.
- The covariance decomposition uses V(φ(X)) = Σxx and subtracts the conditional covariance term from the total covariance.
- The population covariance is estimated from finite samples through empirical covariance operators and kernel matrices K and L.
- The finite-sample construction uses feature maps Φx and Φy together with the corresponding input and output kernel matrices.
Appendix E. Derivation of Lagrangian (10)
The appendix shows that the optimization is unchanged when the transformation matrix B is rescaled by a nonzero scalar, allowing an equivalent Lagrangian formulation.
- Optimization is invariant to rescaling B 7→α · B.
- Because of this scale invariance, the original optimization is equivalent to a reformulated optimization.
Appendix F. Proof of Theorem 5
The proof establishes a finite-sample generalization result for classifiers by controlling the discrepancy between expected and empirical loss after applying transformation B.
- Domains are sampled from a distribution P over PX ×Y, while only samples from the domains are observed.
- The theorem concerns the difference between a classifier’s expected and empirical loss.
- The bound requires a Lipschitz and bounded loss, a bounded input kernel, bounded domain embeddings, and regularity of the domain kernel.
- The proof assumes equal sample counts across domains, with n representing the total number of samples.
- The proof adapts an earlier strategy to account for preprocessing by transformation B and squared errors.
- The proof controls multiple terms using Hilbert-space concentration, union bounds, and norm properties of tr(B⊤KB).
Appendix G. Leave-one-out accuracy
Figure 4 compares leave-one-out accuracy across subjects in the GvHD dataset, contrasting pooling and distributional settings. DICA’s invariant features achieve higher accuracy than other approaches.
- DICA’s invariant features give higher leave-one-out accuracies than other approaches across GvHD subjects.
- Figure 4 evaluates different methods separately on each subject rather than reporting only an aggregate result.
- The top figure shows the pooling setting, while the bottom figure shows the distributional setting.