PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Causal Semantic Representation for Out-of-Distribution Prediction

Chang Liu, Xinwei Sun, Jindong Wang, Haoyue Tang, Tao Li, Tao Qin, Wei Chen, Tie-Yan Liu

arXiv:2011.01681v5stat.MLcs.AIcs.LG

TL;DR

OOD prediction is difficult because representations can mix semantic causes of the output with correlated variation that shifts across domains. This paper proposes CSG, separating those factors and using causal invariance for single-domain OOD generalization and adaptation. It reports theoretical guarantees under certain conditions and improved performance over prevailing baselines.

  • Problem

    OOD prediction is challenging because test distributions can differ from training, and representations may mix semantic factors with correlated variation that can mislead predictions.

    Source: §1 · p. 1

  • Method

    CSG separately models semantic and variation factors, using causal invariance and variational Bayes methods for single-domain OOD generalization and domain adaptation.

    Source: §1 · p. 2, §3 · p. 4, §3.1 · p. 4

  • Results

    The authors report semantic-factor identifiability under certain conditions, bounded OOD generalization error, successful adaptation, and improved performance over prevailing baselines.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The paper presents semantic identification from one training domain as a route to theoretical OOD prediction benefits and improved empirical performance.

    Source: §1 · p. 2

    The causal invariance principle assumes domain change affects the prior p(s, v), while the mechanisms p(x|s, v) and p(y|s) remain invariant.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Conventional supervised learning methods, especially deep ones, are found to be sensitive to out-of-distribution (OOD) examples, largely because the learned representation mixes the semantic factor with the variation factor due to their domain-specific correlation, while only the semantic factor causes the output. To address the problem, we propose a Causal Semantic Generative model (CSG) based on a causal reasoning so that the two factors are modeled separately, and develop methods for OOD prediction from a single training domain, which is common and challenging. The methods are based on the causal invariance principle, with a novel design in variational Bayes for both efficient learning and easy prediction. Theoretically, we prove that under certain conditions, CSG can identify the semantic factor by fitting training data, and this semantic-identification guarantees the boundedness of OOD generalization error and the success of adaptation. Empirical study shows improved OOD performance over prevailing baselines.

1 Introduction

OOD prediction is challenging because test data can differ from training data, while learned representations may mix semantic causes of the output with domain-correlated variation. The paper proposes CSG and single-domain prediction methods, with theoretical guarantees and improved empirical performance.

1 Introduction

  • OOD shifts can make supervised predictors fail even under irrelevant changes such as image position or background.The introduction connects this sensitivity to robustness concerns in risk-sensitive applications.Source: §1 · p. 1
  • Deep representations may mix semantic factors s with correlated variation factors v, letting spurious domain-specific associations mislead prediction.Only s causes y, while the s-v correlation may change across domains.Source: §1 · p. 1
  • CSG separately models semantic and variation factors and develops single-training-domain methods for OOD generalization and domain adaptation.The methods use causal invariance and reformulated variational Bayes objectives; adaptation additionally uses unlabeled test-domain data.Source: §1 · p. 2
  • The authors report mostly better performance than prevailing methods on real-world image classification tasks.They also claim theoretical guarantees for identifying the semantic factor under appropriate conditions and for the resulting OOD prediction benefits.Source: §1 · p. 2

2 Related Work

Prior causal and domain-invariant approaches address OOD prediction but often rely on observed causal relations, invariant representations, or multiple domains. CSG instead models latent semantic and variation factors under causal invariance and targets single-domain semantic identification.

2 Related Work

  • Causal OOD methods reweight inputs or samples, enforce inference invariance, or augment images, with some approaches limited in capacity for invariant causal mechanisms.Other methods use learned causal relations or directly observed variables, which may not suit image data whose causal structure is latent.Source: §2 · p. 2
  • Domain-invariant latent representations have achieved notable results, but are neither sufficient nor necessary for identifying semantics or lowering adaptation error.These approaches use inference invariance, which the paper considers less reliable than causal invariance.Source: §2 · p. 2
  • Prior generative latent-factor methods variously assume independence, do not adapt the prior, or lack guarantees for identifying their factors.CSG instead uses causal invariance and claims semantic-factor identification from one training domain, even when semantics and variation are correlated.Source: §2 · pp. 2–3
  • Unsupervised disentanglement lacks general guarantees, while identifiable VAE approaches use cause-variable data and do not model domain change for OOD prediction.The paper focuses on identifying the semantic factor and its OOD prediction benefits rather than disentangling all latent factors.Source: §2 · p. 3

3 The Causal Semantic Generative Model

The model treats latent factors as causes of both input and output, separates the semantic cause of prediction from variation, and allows those factors to be correlated. Its causal invariance principle keeps generative mechanisms fixed across domains while the latent prior changes.

3 The Causal Semantic Generative Model

  • CSG introduces latent causes z for x and y, excludes direct x-y edges, and separates z into semantic s, which causes y, and variation v, which does not.The model retains an s-v relation because the factors may be correlated; it defines CSG through p(s, v), p(x|s, v), and p(y|s).Source: §3 · pp. 3–4

3.1 The Causal Invariance Principle

  • Causal invariance holds the mechanisms p(x|s, v) and p(y|s) fixed across domains while allowing the prior p(s, v) to change.The text frames domain change as an intervention on s or v or both.Source: §3.1 · p. 4

3.2 Comparison with Inference Invariance

  • Inference invariance can fail when noisy or degenerate mechanisms let multiple latent values generate the same observation, making posterior inference depend on the domain-specific prior.The paper contrasts this ambiguity with causal invariance, which it describes as more fundamental and reliable.Source: §3.2 · p. 4, Figure 2 · p. 4

4 Method

The methods fit CSG with variational Bayes and use test-domain priors to make OOD predictions. Reformulated objectives support prediction and learning with one inference model per method, while adaptation additionally fits a new prior from unlabeled test data.

4.1 Method for OOD Generalization

  • Variational Bayes provides tractable surrogates for fitting CSG, and an auxiliary inference model supports sampling labels for prediction.The reformulated objective combines a cross-entropy term with an expected ELBO that drives the inference model and CSG toward their targets.Source: §4.1 · p. 5
  • CSG-ind predicts with the independent prior p⊥⊥(s, v) = p(s)p(v), discarding the training-domain s-v correlation.The paper says this prior has larger entropy than the training prior and yields a smaller generalization error bound.Source: §4.1 · p. 5

4.2 Method for Domain Adaptation

  • CSG-DA fits a new test-domain prior ˜p(s, v) from unlabeled test data while keeping the causal mechanisms shared.The method simultaneously solves the training- and test-domain optimization problems and uses the resulting inference model for prediction.Source: §4.2 · p. 6

4.3 Implementation and Model Selection

  • Each CSG method uses one inference model; model selection uses a training-domain validation set, with CSG-ind/DA predictions normalized for validation accuracy.The second unsupervised objective term is downscaled in practice because x often has much higher dimension than y.Source: §4.3 · p. 6

5 Theory

The theory establishes that a well-learned CSG can identify the semantic factor on one training domain under stated conditions. It then bounds OOD generalization error and characterizes domain adaptation when the learned test-domain prior is well learned.

5 Theory

  • The analysis assumes infinite data and additive-noise mechanisms with bijective f and injective g; the identifiability result also requires bounded log priors and one of two noise conditions.The noise conditions are sufficiently large causal-mechanism intensity or a noise distribution with an almost-everywhere non-zero characteristic function.Source: §5 · p. 6, §5.1 · p. 7

5.1 Identifiability Theory

  • Under these conditions, matching the training joint distribution lets CSG identify the semantic factor up to a semantic-preserving reparameterization.The learned semantic representation remains constant with respect to the ground-truth variation factor; bounded log p*(s, v) excludes deterministic s-v relations.Source: §5.1 · p. 7

5.2 OOD Generalization Theory

  • For a semantic-identified CSG, OOD prediction error is bounded; the bound reflects test-prior shift and becomes small when the causal mechanism is strong.The paper describes prior shift through a Fisher-divergence term and says an independent prior generally yields a smaller bound because it is less likely to miss test-prior support.Source: §5.2 · p. 8

5.3 Domain Adaptation Theory

  • With a semantic-identified CSG, fitting the test-domain marginal x identifies the corresponding test-domain prior under the learned parameterization.The adaptation result assumes the new prior is well learned; the bounds allow different inference models across domains, beyond inference invariance.Source: §5.3 · p. 8

6 Experiments

Across the reported image benchmarks, CSG variants generally compare favorably with prevailing methods, with the strongest reported results varying by task and dataset. The experiments also compare separate versus joint latent modeling and visualize the learned representations.

Shifted-MNIST

  • On Shifted-MNIST, CSG makes more genuine predictions in unseen domains, CSG-ind performs even better, and CSG-DA achieves the best adaptation results.The passage reports that existing adaptation methods can worsen results as misleading position representations strengthen on unsupervised test data.Source: §6 · p. 9

Results

  • CSG-ind achieves the best OOD generalization on ImageCLEF-DA, while CSG-DA leads most ImageCLEF adaptation cases and is best or on par with the best on PACS.On PACS, CSG-DA also outperforms most domain-generalization methods reported in [39], which use more information; VLCS shows a similar pattern.Source: §6 · p. 9

Visualization

  • LIME visualizations show the methods focusing more on semantic regions and shapes, which the authors take as an indication of causal representation learning.Source: §6 · p. 9

Dataset analysis

  • Performance varies by dataset: the authors report stronger results on shifted-MNIST and PACS (and VLCS) than on ImageCLEF-DA.They relate this pattern to greater variation-factor diversity for a given semantic factor in random-position or pooled training data.Source: §6 · p. 9

Ablation study

  • Separately modeling s and v outperforms treating them as a whole in all reported cases.The authors interpret this ablation as evidence that separate modeling directs the semantic representation into s.Source: §6 · p. 9

7 Conclusion and Discussion

The paper concludes that CSG separates semantic and variation factors for single-domain OOD prediction, with theoretical guarantees and improved empirical performance. It also connects this separation to fairness and causal representation learning, while identifying broader connections as future work.

7 Conclusion and Discussion

  • CSG separately models semantic and variation factors for single-domain OOD prediction, with proofs of semantic-identifiability, bounded generalization error, and successful adaptation.Experiments show improved performance over prevailing baselines.Source: §7 · p. 9
  • Separating semantics from variation also matters when sensitive variation factors may affect predictions, including in fairness settings.The discussion also relates the separation to texture sensitivity and adversarial noise correlated with semantics [15], [107] [38] [67], [50].Source: §7 · p. 10
  • The paper connects causal representation learning, which recovers variables given a graph, with causal discovery, which infers links from observed variables.Its theory links an identifiability condition for causal discovery, the additive noise assumption, to causal representation identifiability; studying the broader connection is left for future work.Source: §7 · pp. 9–10

A Proofs

The appendix defines reparameterizations and semantic-preserving equivalence for CSGs, then establishes properties used in the identifiability proof. It also gives mathematical tools concerning change of variables and convolution.

A Proofs

  • A reparameterization maps one CSG’s latent prior to another while preserving the conditional models for x and y.Such a mapping implies that the two CSGs induce the same observed-data distribution p(x, y).Source: A Proofs · p. 17
  • A reparameterization is semantic-preserving when its learned semantic component is constant with respect to the variation factor.Semantic-equivalence is defined through a semantic-preserving reparameterization, allowing different latent parameterizations while preserving the same observed distribution and semantic information.Source: A Proofs · p. 17
  • Under proper model assumptions, every reparameterization is semantic-preserving, so such CSGs cannot be transformed into one another by mixing semantic and variation factors.Lemma 11 gives a sufficient condition: an injective statistic of p′(y|s) forces the semantic component of any reparameterization to be constant in v.Source: A Proofs · p. 17
  • The change-of-variables rule gives the density after a homeomorphic transformation using the inverse map’s Jacobian determinant.The appendix also states a convolution uniqueness result: if a distribution’s characteristic function is almost everywhere nonzero, equal convolutions imply the underlying functions are equal almost everywhere.Source: A Proofs · p. 18

A.1 Proof of the Equivalence Relation

The proof establishes that semantic-equivalence is an equivalence relation when the variation space is connected and either open or closed. It decomposes a semantic-preserving map into a semantic-only component and a variation component, then verifies the relation’s three properties.

A.1 Proof of the Equivalence Relation

  • The proof assumes that V is connected and either open or closed in R^dV to establish semantic-equivalence as an equivalence relation.The semantic-preserving map is written as Φ(s, v) = (φ(s), ψ_s(v)), with its semantic output independent of v.Source: §A.1 · p. 18
  • Connectedness and openness or closedness of V ensure φ is injective, while bijectivity of Φ yields bijective variation maps ψ_s.The proof rules out multiple semantic preimages by considering the disjoint images ψ_s(V), which would partition V.Source: §A.1 · p. 18
  • The identity map establishes reflexivity, the inverse map establishes symmetry, and composition establishes transitivity.The inverse and composition remain semantic-preserving reparameterizations, completing the equivalence-relation proof.Source: §A.1 · p. 19

A.2 Proof of the Semantic-Identifiability Thm. 5

The appendix proves semantic-identifiability by showing that matching observed data distributions implies semantic equivalence under specified assumptions. It handles three noise-related conditions, with different degrees of exactness in the resulting conclusions.

A.2 Proof of the Semantic-Identifiability Thm. 5

  • The general theorem assumes additive noise structure, bounded derivatives, absolutely continuous latent priors, and log-densities bounded through second order.Under one of three conditions, two CSGs inducing the same data distribution are semantic-equivalent.Source: §A.2 · pp. 19–20
  • The proof constructs the mapping as Φ = f′^-1 ◦ f and shows it is a homeomorphism and a reparameterization between the CSGs.This connects equality of the observed distributions to the required latent-space relation.Source: §A.2 · p. 20
  • The theorem’s conclusions hold almost everywhere under condition (i), asymptotically under condition (ii), and up to a negligible quantity under condition (iii).For condition (ii), the proof takes the noise second moment toward zero; for condition (iii), it bounds residual terms and shows they can be neglected.Source: §A.2 · pp. 20, 22
  • The argument requires the two models to share the noise distribution p_μ, unless the ground-truth p_μ is known during learning.The appendix notes that p_μ may be easier to model, specify, or learn than f.Source: §A.2 · p. 20
  • The proof uses matching p(x) and E[y|x] to recover the transformed latent prior and semantic conditional, yielding a semantic-preserving reparameterization.The statistic g(s) uniquely determines p(y|s), which lets the proof establish equality of the semantic conditionals under the mapping.Source: §A.2 · pp. 20–22

A.3 Proof of the OOD Generalization Error Bound Thm. 6

Theorem 6′ bounds OOD prediction error when two CSGs share their generative mechanisms but differ in their latent priors. The proof also covers semantic-equivalent learned CSGs and the use of unlabeled target-domain data for adaptation.

A.3 Proof of the OOD Generalization Error Bound Thm. 6

  • Theorem 6′ bounds OOD generalization error for CSGs with shared generative mechanisms and different latent priors.Its second case applies the bound to a CSG semantic-equivalent to the reference CSG.Source: §A.3 · p. 26
  • The bound becomes negligible on shared input support when the stated noise and bound conditions hold.The theorem gives the condition as 1/σ²µ ≫ B′; the bound also relates error to the difference between latent priors.Source: §A.3 · p. 26
  • A semantic-equivalent learned CSG can express the error bound using its own structures, although recovering the exact ground-truth CSG is impractical.The proof identifies the relevant prior difference with a Fisher divergence.Source: §A.3 · p. 26

A.4 Proof of the Domain Adaptation Error Thm. 7

  • Fitting the new latent prior to unlabeled target-domain data provides the adaptation step used in the proof.Under the stated conditions, matching the target image distribution identifies the corresponding pushed-forward latent distribution; semantic equivalence preserves the predictor.Source: §A.4 · p. 28

B Alternative Identifiability Theory for CSG

The appendix gives a finite-variance, approximate form of semantic identifiability, explains the model’s latent causal interpretation, and places its contribution alongside prior latent-variable work.

B Alternative Identifiability Theory for CSG

  • The alternative theory defines δ-semantic-dependency by allowing bounded error in semantic conversion, and δ = 0 recovers semantic-equivalence under the stated assumption.The paper reports that the softer relation can be achieved at finite variance, trading off softness against variance.Source: B Alternative Identifiability Theory… · p. 28
  • Under the theorem’s assumptions, two CSGs inducing the same data distribution are δ-semantic-dependent for δ at least σ²µ.This provides approximate rather than exact semantic identification at finite variance.Source: B Alternative Identifiability Theory… · pp. 28–29

C More Explanations on the Model

  • The model treats conceptual latent factors as jointly generating the image and label, with the label produced from the semantic factor.The paper’s photographer example distinguishes image formation from the labeling process.Source: C More Explanations on the Model · p. 30

D More Related Work

  • The authors distinguish their contribution from latent-variable work that identifies latent variables without separating them into semantic and variation factors or showing OOD benefits.They describe their work as extending this literature toward semantic/variation identification and its OOD-prediction benefit.Source: D More Related Work · p. 31

E Relation to Existing Domain Adaptation Theory

This section relates CSG to domain-adaptation theory and describes its factorized inference and prediction procedure. It covers how fitting the full data distribution and using ELBO and variational-EM objectives support identifiability and target-domain guarantees, then outlines prediction by inferring latent factors from x and generating y through the invariant mechanism p(y|s).

E Relation to Existing Domain Adaptation Theory

  • DA-DIR can fail despite achieving domain invariance and low source risk: the representation-level labelers may differ across domains, and matching representations can harm target risk when p(y) differs.The cited examples include disjoint source and target supports; the paper also notes that domain-invariant representations are not necessary for learning a shared p(y|x).Source: E Relation to Existing Domain… · pp. 32–33
  • CSG uses causal invariance, allowing domain differences in p(s), p(s|x), and p(y|x), and uses different target-domain inference and prediction rules with OOD guarantees.The paper argues that invariance in the causal generative direction p(x|s, v) is more fundamental and reliable than covariate shift or inference invariance.Source: E Relation to Existing Domain… · p. 34
  • Fitting p(x, y) gives CSG an identifiability guarantee, whereas DA-DIR’s simultaneous domain-invariant representations and low source risk need not identify the correct predictor.DA-DIR seeks domain-invariant representations but does not fit p(x); the paper identifies this as an identifiability problem even under covariate shift and a strong existence assumption.Source: E Relation to Existing Domain… · p. 34
  • After adaptation, CSG-DA achieves the optimal target-domain labeling function; its bound relates the target risks of the source-optimal and target-oracle labeling functions.The bound measures the target-domain risk gap of the best source labeling function.Source: E Relation to Existing Domain… · p. 34

F.1.1 The Evidence Lower BOund (ELBO).

  • Variational EM uses a tractable posterior approximation to derive an ELBO, a lower bound on likelihood that avoids the intractable latent-variable integration.Optimizing the ELBO with respect to the approximation drives it toward the true posterior; optimizing with respect to the generative model effectively maximizes likelihood.Source: §F.1.1 · p. 35

F.1.2 Variational EM for learning CSG.

  • For supervised CSG learning, the expected ELBO combines cross-entropy training of q(y|x) with a generative term that fits p(x) and aligns the inference model with the generative model.The cross-entropy term drives q(y|x) toward p*(y|x), while the second term fits the input distribution and aligns q(s, v, y|x) with p(s, v, y|x).Source: §F.1.2 · pp. 35–36
  • CSG factorizes p(s, v, y|x) as p(s, v|x)p(y|s), using the known invariant mechanism p(y|s) and an inference model q(s, v|x) for the intractable p(s, v|x).This factorization follows from y being conditionally independent of x and v given s.Source: §F.1.2 · p. 36
  • For prediction, CSG approximates q(y|x) with an expectation over q(s, v|x), then samples (s, v) followed by y from p(y|s).The expectation can be estimated and optimized using reparameterization.Source: §F.1.2 · p. 36

F.1.3 Variational EM for learning CSG with test-domain inference model (Learning CSG-ind and CSG-DA on the training domain).

The appendix derives training objectives for CSG-ind and CSG-DA, extends the approach to an unsplit-latent CSGz ablation, and describes inference-model implementation and model selection.

F.1.3 Variational EM for learning CSG with test-domain inference model (Learning CSG-ind and CSG-DA on the training domain).

  • CSG-ind predicts by sampling latents from its test-domain inference model and then sampling y from p(y|s); CSG-DA instead uses a standalone test-domain prior.The CSG-DA objective follows by replacing the independent test-domain prior with a standalone prior model.Source: §F.1.3 · p. 37

F.1.4 Methods for CSGz for ablation study.

  • CSGz merges semantic and variation latents into z and retains a z-to-y link, unlike CSG’s separate factors.CSGz-DA is possible with a standalone test-domain prior, but there is no CSGz-ind version.Source: §F.1.4 · p. 38

F.2 Instantiating the Inference Model

  • A discriminative model can implement CSG by parsing hidden nodes into s and v, with q(y|s) implementing the generative mechanism p(y|s).The computational arrows constructing q(s, v|x) need not represent causal directions.Source: §F.2 · p. 38

F.3 Model Selection Details

  • Model selection uses a training-domain validation split; the authors report that their approach needs neither extra hyperparameters nor multiple training domains.The validation split is 80%-20% of training-domain data.Source: §F.3 · pp. 39–40

G Experiment Details

The appendix documents benchmark and implementation details for the experiments, along with ablation findings and a dataset-specific qualification for domain adaptation.

G Experiment Details

  • CSG outperforms the unsplit-latent CSGz methods on all tasks, while CSGz usually has greater run-to-run variance.The authors associate the variance with possible lack of semantic identifiability; CSGz still often exceeds discriminative methods.Source: G Experiment Details · p. 40, Table 2 · p. 41, Table 3 · p. 41
  • On ImageCLEF-DA, CDAN and BNM are comparable to or sometimes better than CSG-DA, unlike its stronger advantage on the other datasets.The authors attribute the weaker advantage to the absence of severe spurious correlation in ImageCLEF-DA.Source: G Experiment Details · p. 40

G.1 Shifted-MNIST

  • Shifted-MNIST correlates digit identity with horizontal position in training, while its test sets use zero shift or random shifts and balanced classes.The training shifts are sampled around -5 pixels for 0 and +5 pixels for 1.Source: §G.1 · p. 42

G.2 ImageCLEF-DA

  • ImageCLEF-DA has three domains, each with 12 classes and 600 images; the experiments use a pretrained ResNet50 backbone.The domains are Caltech-256, ImageNet, and Pascal VOC 2012.Source: §G.2 · pp. 42–43

G.3 PACS

The appendix reports PACS results for single-domain training, describes the VLCS benchmark, and compares learned image regions across methods.

G.3 PACS

  • PACS has seven classes across Photo, Art, Cartoon, and Sketch domains, and the standard setup pools three domains for training while holding one out.The dataset contains 9,991 images.Source: §G.3 · p. 43
  • With single-domain PACS training, CSG’s advantage is less significant than with pooled training, consistent with a need for diverse variation to identify factors.The table reports OOD generalization and domain adaptation accuracy, averaged over 10 runs.Source: §G.3 · p. 44, Table 4 · p. 44

G.4 VLCS

  • VLCS comprises five classes across VOC2007, LabelMe, Caltech101, and SUN09, with results reported for OOD generalization and domain adaptation.The dataset contains 10,729 images; its setup follows the PACS experiment.Source: §G.4 · p. 44

G.5 Visualization of the Learned Representation

  • LIME visualizations show CSG methods focusing more on semantic regions and object shapes, while baselines include more background regions.The authors interpret this as support for learning a causal semantic factor for prediction.Source: §G.5 · p. 44, Figure 5 · p. 45

Related papers

Loading 2011.01681v5…