Source-linked AI summary
Domain Generalization via Conditional Invariant Representation
Ya Li, Mingming Gong, Xinmei Tian, Tongliang Liu, Dacheng Tao
TL;DR
Domain generalization must handle heterogeneous source domains and unseen targets even when both feature distributions and labeling relationships change. The paper proposes conditional-invariant representations with regularization for class-conditional alignment and discrimination. Experiments on synthetic and real datasets demonstrate effectiveness, while invariant joint-distribution guarantees require unchanged class priors.
Problem
Existing methods often assume only P(X) changes while P(Y|X) remains stable, but this assumption can be violated in real-world applications.
Method
The CIDG method learns a representation with invariant P(h(X)|Y) using two regularization terms and discriminative kernel Fisher distances.
Results
CIDG achieves the best performance on 9 of 10 tasks in both the VLCS and Office+Caltech evaluations, while reaching 86.67% accuracy on the synthetic dataset.
Takeaways & Limitations
When class priors match across source and target domains, conditional invariance can provide invariant joint distributions P(h(X),Y).
Takeaways & Limitations
The joint-distribution guarantee depends on unchanged class priors, and prior-based marginal-invariance claims fail when class priors change across domains.
Abstract
from arXiv · showhide
Domain generalization aims to apply knowledge gained from multiple labeled source domains to unseen target domains. The main difficulty comes from the dataset bias: training data and test data have different distributions, and the training set contains heterogeneous samples from different distributions. Let $X$ denote the features, and $Y$ be the class labels. Existing domain generalization methods address the dataset bias problem by learning a domain-invariant representation $h(X)$ that has the same marginal distribution $\mathbb{P}(h(X))$ across multiple source domains. The functional relationship encoded in $\mathbb{P}(Y|X)$ is usually assumed to be stable across domains such that $\mathbb{P}(Y|h(X))$ is also invariant. However, it is unclear whether this assumption holds in practical problems. In this paper, we consider the general situation where both $\mathbb{P}(X)$ and $\mathbb{P}(Y|X)$ can change across all domains. We propose to learn a feature representation which has domain-invariant class conditional distributions $\mathbb{P}(h(X)|Y)$. With the conditional invariant representation, the invariance of the joint distribution $\mathbb{P}(h(X),Y)$ can be guaranteed if the class prior $\mathbb{P}(Y)$ does not change across training and test domains. Extensive experiments on both synthetic and real data demonstrate the effectiveness of the proposed method.
Introduction
Domain generalization addresses dataset bias when heterogeneous source domains differ from unseen test domains. The paper challenges stable P(Y|X) assumptions and proposes conditional-invariant representations that support invariant joint distributions when class priors remain unchanged.
- Domain generalization trains on heterogeneous source domains to generalize to unseen test distributions affected by dataset bias.
- Existing methods learn representations with stable marginal distributions across source domains, assuming distribution shifts arise from removable common factors.
- Existing approaches assume only P(X) changes while P(Y|X) remains stable, but this assumption is unclear in practical situations.
- The proposed method enforces invariant class-conditional representations using two regularization terms and preserves discrimination through kernel Fisher discriminant distances.
- If test-set class priors match the combined source-training prior, the learned representation can have invariant joint distribution P(h(X),Y) across domains.
Related Work
Related work applies domain generalization to reduce bias across heterogeneous domains and improve generalization to unseen related domains. Existing approaches commonly assume only P(X) changes, an assumption that can be violated in real-world applications.
- Domain generalization reduces data bias across domains and targets generalization to unseen but related domains in classification tasks.
- Many related methods assume marginal P(X) changes while conditional P(Y|X) remains stable or unchanged across domains.
- The stable-conditional assumption simplifies domain generalization but is easily violated in real-world applications.
- Under that assumption, domain shifts can be addressed through importance reweighting or domain-invariant feature learning, including approaches using unlabeled source data.
Conditional Invariant Domain Generalization
CIDG learns representations whose class-conditional distributions remain invariant across source domains, addressing settings where both feature distributions and label mechanisms may change. It combines local and global distributional regularization with discriminative constraints, using kernel-based feature transformations.
- Conditional Invariant Domain Generalization: CIDG targets invariant class-conditional distributions P(h(X)|Y), rather than only invariant marginal distributions P(h(X)).This design addresses domain shifts where both P(X) and P(Y|X) change.
- Conditional Invariant Domain Generalization: If class priors match between training and test domains, invariant P(h(X)|Y) yields an invariant joint distribution P(h(X),Y).When class priors differ, CIDG still guarantees invariant class-conditional distributions but not invariant joint distributions.
- Conditional Invariant Domain Generalization: The objective combines class-conditional scatter and class-prior-normalized marginal scatter to enforce local and global distribution invariance.The global term adjusts for changing class priors by assigning equal prior probability to each class.
- Conditional Invariant Domain Generalization: Unlike prior marginal-distribution methods, CIDG measures domain scatter separately for each class and then sums the class-specific discrepancies.This uses source labels that previous domain-invariant representation methods overlooked.
- Conditional Invariant Domain Generalization: CIDG represents h(x) as a linear transformation of kernel features, with W expressed through the sample feature matrix and coefficient matrix B.The kernel formulation avoids explicitly operating in the potentially infinite-dimensional RKHS.
- Conditional Invariant Domain Generalization: Additional within-class and between-class scatter terms preserve the discriminative power of the learned representation.The objective therefore balances domain-invariance regularization with supervised dimension-reduction criteria.
Experiments
Experiments evaluate CIDG on synthetic data and two real-world image datasets, comparing it with baseline and domain-invariant representation methods. CIDG achieves the strongest reported performance across the synthetic setting and most real-world domain-generalization tasks.
- Experimental setup: Experiments use one synthetic dataset and the VLCS and Office+Caltech image-classification datasets.The synthetic data enable visualization of learned distributions, while the real datasets evaluate domain-generalization performance.
- Synthetic Dataset: Domain 1 and domain 2 serve as source domains, while domain 3 is the unseen target domain in the synthetic experiments.The compared methods are KNN, KPCA, DICA, and SCA; Undo-Bias is excluded because it does not explicitly learn a feature representation.
- Synthetic Dataset: 86.67% is the best accuracy achieved by CIDG on the synthetic dataset.CIDG also learns well-shaped clusters for samples from the same class, whereas competing representations show class mixing, poor clustering, or line-shaped distributions.
- VLCS Dataset: CIDG performs best on 9 of 10 VLCS domain-generalization tasks.KPCA is best for the L,S-to-V,C task, while some learned representations perform worse than 1NN on raw features because real-image features are complicated and noisy.
- Office+Caltech Dataset: CIDG achieves the best performance on 9 of 10 Office+Caltech domain-generalization tasks.Undo-Bias may outperform CIDG for D,W source domains and A,C target domains; results average five random 70% training selections per domain.
Conclusion
The paper addresses domain generalization when both feature distributions and label-conditionals change across domains by learning conditionally invariant representations. Under unchanged class priors, the method can produce invariant joint representations, and two regularization terms are validated on synthetic and real datasets.
- Conclusion: The proposed method addresses domain generalization when both P(X) and P(Y |X) change across domains.
- Conclusion: CIDG learns representations with invariant joint distribution P(h(X), Y ) when the class prior P(Y ) is unchanged between source and target domains.
- Conclusion: Two regularization terms enforce class-conditional distribution invariance across domains and are validated on synthetic and real datasets.