PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain Generalization using Causal Matching

Divyat Mahajan, Shruti Tople, Amit Sharma

arXiv:2006.07500v3cs.LGcs.AIstat.ML

TL;DR

The paper shows that class-conditional domain invariance is insufficient when causal features vary within classes across domains. It uses a structural causal model to motivate object-conditional matching, proposes MatchDG and related matching methods, and reports competitive out-of-domain performance while recovering many ground-truth matches.

  • Problem

    Class-conditional domain invariance can fail to generalize when stable-feature distributions vary within class across domains.

  • Method

    A structural causal model motivates invariance across inputs sharing an object, implemented through perfect matching when objects are known and two-phase iterative MatchDG when they are not.

  • Results

    Matching methods are competitive across rotated MNIST, Fashion-MNIST, PACS, and Chest X-ray, while MatchDG obtains about 50% top-10 overlap with ground-truth matches on MNIST and Fashion-MNIST.

  • Takeaways & Limitations

    Domain generalization should model within-class causal variation and enforce object-level rather than only class-level invariance.

  • Takeaways & Limitations

    Causal features are unidentifiable from the observed distribution, so MatchDG relies on assumptions when objects are unknown.

Abstract

from arXiv · show

In the domain generalization literature, a common objective is to learn representations independent of the domain after conditioning on the class label. We show that this objective is not sufficient: there exist counter-examples where a model fails to generalize to unseen domains even after satisfying class-conditional domain invariance. We formalize this observation through a structural causal model and show the importance of modeling within-class variations for generalization. Specifically, classes contain objects that characterize specific causal features, and domains can be interpreted as interventions on these objects that change non-causal features. We highlight an alternative condition: inputs across domains should have the same representation if they are derived from the same object. Based on this objective, we propose matching-based algorithms when base objects are observed (e.g., through data augmentation) and approximate the objective when objects are not observed (MatchDG). Our simple matching-based algorithms are competitive to prior work on out-of-domain accuracy for rotated MNIST, Fashion-MNIST, PACS, and Chest-Xray datasets. Our method MatchDG also recovers ground-truth object matches: on MNIST and Fashion-MNIST, top-10 matches from MatchDG have over 50% overlap with ground-truth matches.

1. Introduction

The paper argues that class-conditional domain invariance can fail when within-class causal variation differs across domains. It proposes object-conditional matching and evaluates matching-based algorithms across several benchmarks.

  • Domain generalization seeks models that generalize from multiple training distributions to unseen data distributions.
  • Class-conditional domain-invariant representations are insufficient when stable-feature distributions vary within class across domains.Such within-class variation occurs in handwriting shapes and medical-image body characteristics.
  • The causal model treats domains as interventions changing non-causal features while stable features cause the output.
  • The proposed objective makes representations invariant across domains for inputs derived from the same object.When objects are observed, the perfect-match regularizer minimizes representation distance between same-object inputs.
  • MatchDG approximates object matching by iteratively learning contrastive representations from same-class matches, while MDGHybrid also uses original–augmentation pairs.The two-phase design learns the representation independently of the ERM loss before classification training.
  • Across rotated MNIST, Fashion-MNIST, PACS, and Chest X-ray, matching methods are competitive for out-of-domain accuracy.On rotated MNIST and Fashion-MNIST, MatchDG’s top-10 matches overlap ground-truth matches by about 50%.

2. Related Work

Prior domain-generalization work learns domain-invariant representations, uses causal reasoning, or applies matching and contrastive regularization. This paper distinguishes its two-phase iterative matching approach from these methods and tests it against several alternatives.

  • Domain-invariance methods enforce representations to be independent of domain marginally or conditional on class using divergence, adversarial, and discriminant techniques.
  • Class-conditional methods have been reported superior to marginal domain-invariance when class-label distributions vary across domains.
  • The paper argues that class-conditional invariance is still insufficient for generalization to unseen domains.
  • Matching and Contrastive Loss: Prior matching and contrastive approaches regularize representations or ERM, whereas MatchDG learns a representation independently of the ERM objective in two phases.
  • Other work: The broader literature also includes meta-learning, dataset augmentation, parameter decomposition, and invariant-optimal-predictor methods.

3. Insufficiency of class-conditional invariance

Counter-examples show that class-conditional invariance can select non-causal features and fail under domain shifts. Conditioning representation invariance on stable features or object matches substantially improves generalization in the presented examples.

  • Simple counter-example: A representation can satisfy Φ(x) ⊥⊥D|Y while its classifier fails on a new domain.
  • Simple counter-example: In the constructed example, the representation x1 achieves 100% training accuracy but only 62.5% test accuracy, including 25% on the positive class.
  • Simple counter-example: If P(Xc|Y) remains constant across domains, class-conditional invariance yields a generalizable classifier under the stated setup.
  • Simple counter-example: When P(Xc|Y) changes, the additional constraint is domain–representation independence conditioned on the stable feature xc.The paper expresses this as H(d|xc) = H(d|Φ, xc).
  • Empirical study: Conditional and unconditional invariance methods can fail on the slab dataset by retaining a simpler spurious linear feature instead of the stable slab feature.The failure occurs despite very good source-domain performance.
  • Empirical study: PerfectMatch raises out-of-domain accuracy to 78% by minimizing representation distance for points from different domains sharing the same slab value.

4. A Causal View of Domain Generalization

The paper models domain generalization with a structural causal model in which objects determine causal features and domains alter non-causal features. This framework shows why common domain-invariance objectives can exclude the causal representation and motivates matching representations across domains for the same object.

  • Data-generating process: Images are generated from an object and a domain, with causal features XC causing the class label and domain-dependent features affecting the observed input.The object may be unobserved, while domain and object can be correlated.
  • Causal representation: The causal representation satisfies Y ⊥⊥D|XC, so the target classifier can be expressed as h(XC) after learning XC through a representation Φ.The classifier is composed as h(Φ(x)).
  • Identification: Causal features are unidentifiable from observed and interventional distributions because multiple values of XC can produce the same distributions.This non-identifiability holds even when object information is observed.
  • Perfect-match invariant: The proposed invariant requires representations of inputs from the same object to remain unchanged across domains while remaining informative about the object and its class.The matching function Ω identifies cross-domain pairs corresponding to the same object.
  • Theoretical result: Theorem 1 states that, with finite domains and infinite examples, the object-based condition contains the optimal causal representation XC under stated assumptions.The theorem also gives conditions under which the associated loss has the true classifier as a minimizer.
  • Failure of standard objectives: Domain-invariant and class-conditional domain-invariant conditions can exclude XC, so satisfying either condition alone does not guarantee recovery of the causal representation.The paper establishes this through d-separation analysis and Proposition 3.

5. MatchDG: Matching without objects

MatchDG approximates object matching when object identities are unavailable by iteratively learning representations and updating same-class cross-domain matches. Its two-phase design separates matching from classification, while known augmentations can provide perfect matches.

  • MatchDG: When objects are unavailable, MatchDG learns which same-class inputs share causal features and iteratively refines their matches with contrastive learning.It begins with random same-class matches and updates them using the learned representation.
  • Contrastive matching: The contrastive loss pulls same-class inputs from different domains together and separates them from different-class inputs.Positive pairs are same-class cross-domain inputs; negative pairs have different classes.
  • Iterative matching: Iterative matching accounts for within-class variation by updating positive pairs to nearest same-class cross-domain examples instead of treating all same-class data as one unit.The goal is to identify pairs more likely to share the same base object.
  • Two-phase method: MatchDG uses two phases: contrastive representation learning first, followed by classifier training with matches computed from the learned representation.The final representation determines a new match function for the second phase.
  • Empirical result: The two-phase method achieves better overlap with ground-truth perfect matches than simultaneous classification and matching optimization.The comparison is reported in supplementary experiments.
  • Known-object extension: Known objects from self-augmentations can be incorporated as perfect matches in MatchDG Phase II through the MDGHybrid variant.Examples include rotations, horizontal flips, and color jitter.

6. Evaluation

The evaluation tests matching-based methods across synthetic and real-world domain-generalization benchmarks, measuring both out-of-domain accuracy and learned-match quality. Results show competitive performance overall, while highlighting representation-level effects and dataset-dependent limitations.

  • Benchmark evaluation: MatchDG and MDGHybrid are competitive with state-of-the-art methods for out-of-domain accuracy across rotated MNIST, Fashion-MNIST, PACS, and Chest X-rays.The evaluation includes both synthetic benchmarks with known objects and real-world datasets.
  • Rotated MNIST and Fashion-MNIST: 43.8% accuracy with three rotFashionMNIST source domains exceeds ERM-RandMatch’s 38.4%.MatchDG outperforms all baselines in the reported Rotated MNIST and Fashion-MNIST setting, with the gap widening as training domains decrease.
  • Match quality: About 50% of inputs place their perfect match within MatchDG’s top-10 matches, while about 25% of learned matches are perfect.MatchDG’s representation is more consistent with ground-truth matches than ERM loss matching; oracle initialization performs better.
  • Regularization behavior: After zero training error, ERM cannot minimize the matching loss, whereas IRM-style loss comparisons can be satisfied by plain ERM.MatchDG remains sensitive to within-class representation geometry because its regularization compares last-layer representations.
  • PACS: On PACS, MDGHybrid achieves the highest average accuracy among the compared methods except DDEC and RSC, while test-domain validation changes the comparison.With ResNet-50, MDGHybrid improves DomainBed ERM from 85.7 to 87.5 and is close to RSC’s 87.83.
  • Chest X-rays: On Chest X-rays, MDGHybrid is 8% above ERM with RSNA as target, but matching-based methods are not effective with ChexPert as target.The authors attribute the varying trends to inherent source-image variability and identify real-world domain generalization as challenging.

7. Conclusion

The paper concludes that domain generalization benefits from an object-conditional causal objective rather than only class-conditional invariance. Its matching methods perform competitively, but MatchDG depends on assumptions and needs improvement for mixed real-world results.

  • Conclusion: The proposed causal view yields an object-conditional objective for domain generalization.The conclusion frames matching representations of the same object as the central invariance condition.
  • Conclusion: Simple matching-based methods perform competitively with state-of-the-art methods on PACS.The authors use this result to emphasize the importance of selecting the appropriate invariance.
  • Conclusion: MatchDG relies on assumptions when the underlying objects are unknown, and mixed Chest-Xray results motivate better matching methods.The conclusion identifies assumption dependence and real-world performance variability as boundaries of the current approach.

A.1. Implementation Details for the Slab Dataset

The slab-dataset implementation constructs controlled spurious and stable features, then compares regularized classifiers and matching methods under a shared training procedure. MatchDG is evaluated under an explicit same-class causal-similarity assumption, with the causal model accommodating alternative label-generation viewpoints.

  • Dataset construction: The synthetic slab dataset uses binary labels and two-dimensional features: a linear feature and a more complex slab feature.Feature simplicity is characterized by the number of linear pieces in the optimal decision curve.
  • Dataset construction: Domain-dependent noise makes the linear feature spurious while preserving the slab feature’s relationship with the label across domains.Source domains use noise probabilities 0.0 and 0.1, while the target uses p = 1.0; experiments set k = 7, m = 0.1, and p_s = 0.1.
  • Training objectives: All methods use cross-entropy classification loss, with regularization penalties applied to the representation-network output.The overall objective is ERM + λ*Regularization Penalty.
  • Matching methods: RandMatch pairs same-class inputs randomly across domains, whereas PerfMatch accepts pairs sharing the same slab id as valid matches.Both methods use l2 distance for the matching loss in Eq. 3.
  • MatchDG assumption: MatchDG assumes same-class inputs across domains are closer in causal features than different-class inputs.The assumption is not satisfied by the Section 3.1 example, although related variants can satisfy it.
  • Causal formulation: The causal graph includes both Ytrue and observed Y, so reported d-separation results hold under either label-generation viewpoint.The paper uses Y in the main text because it corresponds to settings where observed labels may result from noisy labeling.

B.3. Proof of Proposition 1

Proposition 1 shows that class-conditional domain invariance is sufficient only when stable-feature distributions remain unchanged across domains. When within-class stable-feature distributions vary, the causal representation may fail the objective, motivating independence conditioned on the stable feature.

  • Class-conditional domain invariance yields a generalizable representation when P(Xc|Y) remains constant across domains.Under this condition, the learned representation is independent of domain given Xc.
  • When P(Xc|Y) changes across domains, the ground-truth representation Xc need not satisfy class-conditional domain invariance.The relevant entropy equality can fail in this setting.
  • The proposed alternative constraint requires domain and representation to be independent conditional on Xc: H(d|xc) = H(d|Φ, xc).This constraint directly targets learning the stable feature representation.
  • In the slab counterexample, class-conditional methods can learn x1 instead of causal feature x2 because x1 correlates linearly with the class label.The alternative constraint can also be satisfied by shifting x1 slightly, so the objective does not uniquely identify x2.
  • The causal feature is non-identifiable because multiple values of Xc can generate identical observational and interventional distributions.Different choices of the structural mapping gxc can preserve the same structural equations over the observed variables.

B.5. Proof of Theorem 1

Theorem 1 establishes that matching representations of the same object across domains excludes domain-generated variation, while ERM selects accurate functions of the causal feature. Its guarantee requires stochastic domain effects on relevant non-causal features and can fail for deterministic interventions.

  • The matching condition assigns zero regularizer to Φ(x) = xc because causal features are identical for inputs from the same object.The condition compares representations of matched cross-domain inputs using a non-negative distance.
  • Any nonconstant optimal representation depending on a domain-generated feature Xa would differ across matched inputs with nonzero probability.Therefore, under stochastic domain-dependent variation, optimal representations cannot depend on such Xa features.
  • The matching regularizer therefore admits functions of Xc, including constant functions, while ERM retains those that also maximize predictive accuracy.Constant functions can minimize the regularizer but have low accuracy.
  • For P-admissible losses, E[Y|Xc] minimizes the weighted loss across training domains as domain sample sizes grow.The same conditional expectation minimizes each domain’s asymptotic loss and thus their positive weighted sum.
  • The guarantee does not apply when domain effects are deterministic, as in fixed-angle Rotated-MNIST domains.A representation can memorize angle-specific transformations, match each object across training domains, and still fail at unseen rotation angles.

B.6. Proof of Proposition 3

Proposition 3 shows that marginal and class-conditional domain-invariance constraints generally exclude the causal representation Xc because object-mediated paths keep Xc associated with domain. Thus, additional assumptions are required even with infinite data.

  • Neither domain invariance nor class-conditional domain invariance is generally satisfied by the causal representation Xc.The proposition states that the corresponding solution sets need not contain Xc without additional assumptions.
  • Marginal domain invariance fails because the object O leaves a path connecting Xc and domain D.D-separation on the structural causal model gives Xc not independent of D.
  • Class-conditional domain invariance also fails because conditioning on Y does not block the object-mediated path between Xc and D.The same conclusion holds even when the true label Ytrue is observed.
  • Optimizing either invariance condition need not retrieve Xc, even as the number of samples tends to infinity.This follows because Xc is excluded from the set of optimal representations under those conditions.

C.1. Implementation details

The implementation evaluates matching-based methods across rotated image, natural-image, and Chest-X-ray settings. MatchDG initializes same-class cross-domain matches, refines them by representation-based nearest neighbors, and achieves competitive accuracy while match quality tracks generalization.

  • Implementation and evaluation: The evaluation covers rotated MNIST, Fashion-MNIST, PACS, and Chest X-ray datasets under multiple domain-generalization setups.Rotated datasets use rotation angles as domains, PACS uses four visual domains, and Chest X-ray uses NIH, ChexPert, and RSNA sources.
  • Implementation and evaluation: MatchDG begins with random same-class matches across domains and periodically replaces them with same-class nearest neighbors in representation space.Updates use l2 distances relative to a base domain selected per class.
  • Implementation and evaluation: Phase 1 trains contrastive representations using matched cross-domain inputs as positives and different-class inputs as negatives.Batches are sampled from the matched data matrix across source domains.
  • Results: On the six-domain rotated-MNIST setup, RandMatch and MatchDG outperform prior work except at test domain 0, where MatchDG is competitive with DIVA.For target angles 15°–60°, both methods approach oracle PerfMatch accuracy.
  • Results: MatchDG average accuracy is only 0.5% behind the best-performing approaches in another rotated-MNIST setup.This comparison uses a custom CNN architecture and all 70,000 images per domain.
  • Results: Higher fractions of perfect matches produce better target-domain accuracy, showing a monotonic relationship between match quality and generalization performance.This pattern is reported for both Rotated-MNIST and Fashion-MNIST.

D.5. Matching metrics for Fashion-MNIST dataset with 2000 training samples per domain

The evaluation examines MatchDG’s matching quality on Fashion-MNIST with 2,000 samples per domain, comparing random, oracle, and ERM-based matches and iterative updating. Iterative matching improves overlap with perfect matches, while sample-size effects complicate direct overlap comparisons.

  • Experimental setup: The experiment evaluates MatchDG Phase 1 on Fashion-MNIST using 2,000 samples per domain and compares results with the 10,000-sample setting.The smaller dataset is used to reproduce the matching metrics from the main-text evaluation.
  • Matching comparisons: MatchDG with random initialization is compared against oracle initialization with perfect matches and ERM-derived matches.The comparisons assess both the effect of initialization and the effectiveness of ERM representations for generating matches.
  • Sample-size effects: The smaller dataset may have fewer alternative matches to a perfect match, although larger sample sizes can still improve classifier accuracy.Thus, lower overlap at larger sample sizes does not necessarily imply worse downstream classification.
  • Iterative refinement: Without iterative updates, matches remain equal to the random initialization rather than being refined during Phase 1.The non-iterative comparison isolates the contribution of updating inferred matches.
  • Iterative refinement: The iterative approach provides a 2× improvement in overlap with perfect matches on rotated MNIST and Fashion-MNIST.Higher inferred-match overlap is associated with better Phase 2 classification accuracy in the cited evaluation.

E.2. AlexNet Results

The AlexNet evaluation compares RandMatch and MatchDG with ERM and prior methods on PACS, while t-SNE visualizations examine train–test feature overlap for unseen domains. Both matching methods outperform ERM across test domains, but MatchDG’s average accuracy remains below some stronger prior methods and depends on architecture.

  • Experimental setup: The PACS AlexNet experiment evaluates generalization to each test domain after training on the remaining three domains.Results are reported for standard AlexNet and AlexNet with test-domain validation.
  • Accuracy results: For all test domains, RandMatch and MatchDG outperform the baseline ERM method.This comparison is reported directly for the AlexNet evaluation.
  • Accuracy results: MDGHybrid improves over MatchDG in average accuracy, reaching 70.46 versus 69.91 across test domains.The comparison is an average over the PACS test domains.
  • Comparison with prior work: MatchDG and MDGHybrid exceed several prior methods on average, while MASF, DGER, and RSC achieve higher accuracy than MatchDG.The reported prior-method comparison indicates that performance varies across methods and architectures.
  • Representation visualization: Phase 1 t-SNE plots show greater train–test overlap for Photo than Sketch, although training-domain classes are well separated for both.The visualization highlights more difficult generalization to Sketch despite class separation in training domains.
Loading 2006.07500v3…