PaperlayerView exact revision on arXiv

Source-linked AI summary

Invariant and Transportable Representations for Anti-Causal Domain Shifts

Yibo Jiang, Victor Veitch

arXiv:2207.01603v1cs.LGcs.AI

TL;DR

Domain shift can undermine performance in new domains, and no single method dominates across varied contexts. This paper formalizes an anti-causal shift setting and learns invariant and transportable representations; experiments report invariant and fast-adapting structures, while the work remains a proof of concept.

  • Problem

    Domain shifts can make predictors that perform well in training domains perform poorly in novel test domains, and benchmark methods show no single dominant approach across contexts.

    Source: §1 · p. 1

  • Method

    The paper formalizes an anti-causal domain-shift assumption and uses it to learn invariant predictors and transportable, domain-adaptive representations.

    Source: §1 · p. 2

  • Results

    The empirical studies report learning invariant structures, fast-adapting structures, and disentangling factors with domain-varying relationships from invariant factors.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper presents a proof of concept that translates its anti-causal domain-shift notion into useful learning principles for domain adaptation.

    Source: §7 · p. 13

    The setup covers a specific class of anti-causal shifts; the practical training procedure may be refined, and formal guarantees for robust models remain future work.

    Source: §7 · p. 13

Abstract

from arXiv · show

Real-world classification problems must contend with domain shift, the (potential) mismatch between the domain where a model is deployed and the domain(s) where the training data was gathered. Methods to handle such problems must specify what structure is common between the domains and what varies. A natural assumption is that causal (structural) relationships are invariant in all domains. Then, it is tempting to learn a predictor for label $Y$ that depends only on its causal parents. However, many real-world problems are "anti-causal" in the sense that $Y$ is a cause of the covariates $X$ -- in this case, $Y$ has no causal parents and the naive causal invariance is useless. In this paper, we study representation learning under a particular notion of domain shift that both respects causal invariance and that naturally handles the "anti-causal" structure. We show how to leverage the shared causal structure of the domains to learn a representation that both admits an invariant predictor and that also allows fast adaptation in new domains. The key is to translate causal assumptions into learning principles that disentangle "invariant" and "non-stable" features. Experiments on both synthetic and real-world data demonstrate the effectiveness of the proposed learning algorithm. Code is available at https://github.com/ybjiaang/ACTIR.

1 Introduction

Domain-shift learning seeks both predictors that work across domains and representations that enable rapid target-specific learning. The paper formalizes an anti-causal shift setting and uses its shared causal structure to support invariant and adaptive learning.

  • Domain-shift learning has two targets: an invariant predictor that performs well across domains and a transportable representation enabling quick target-specific learning from few examples.Source: §1 · p. 1
  • Because methods can fail across contexts and no predictor can withstand every shift, progress requires specifying shared versus varying structure.Source: §1 · pp. 1–2
  • The usual causal-parent strategy becomes vacuous when observed covariates are all caused by Y, leaving Y with no causal parents.Source: §1 · p. 2
  • The paper formalizes an alternative causal notion of domain shift for anti-causal problems while retaining invariant structural causal relationships.Source: §1 · p. 2
  • It leverages the causal assumption to learn invariant predictors and transportable, domain-adaptive representations.The introduction reports experiments that learn invariant and fast-adapting structures and disentangle stable from varying factors.Source: §1 · p. 2

2 Causal Setup

The setup allows domains to differ through the distribution of an unobserved common cause while preserving their causal structure. It separates stable and domain-varying information in X to support invariant prediction and rapid adaptation.

2 Causal Setup

  • Across compatible domains, the causal graph and conditional distribution given U stay fixed, while the distribution of U may vary.The setup considers discrete-label classification with finite datasets from multiple training domains and predictions in unseen domains.Source: §2 · pp. 2–3

2.1 Invariant Prediction

  • A predictor based only on parts of X not causally influenced by Z is counterfactually invariant to Z.Under equal label priors across domains, the lowest-risk counterfactually invariant predictor in one training domain is also lowest-risk among such predictors in all domains [Vei+21, Thm. 4.2].Source: §2.1 · p. 3

2.2 Causal Decomposition of X

  • The causal decomposition splits X into a part unaffected by Z and a Z-dependent part, and assumes the former does not cause the latter.This assumption permits changing other parts of X without affecting its Z-specific part; the paper gives changing an image’s object without changing its background as an example.Source: §2.2 · pp. 3–4

2.3 Rapid Adaptation

  • Because the invariant predictor excludes potentially predictive Z-dependent information, rapid adaptation adds a domain-specific correction based on a representation of that information.The correction is logit-linear in h(X), which the paper motivates as sample-efficient to learn once h is known.Source: §2.3 · p. 4

2.4 Learning Goals

  • The learning goals are an invariant, low-risk g(X) based on the Z-unaffected part and an h(X) based on the Z-dependent part, yielding low-risk domain-specific predictors.Z and U are unobserved, and the decomposition of X is unknown, motivating an observable-data relaxation in the next section.Source: §2.4 · p. 4

3 Observable Signature

Because the latent variables and causal decomposition are unavailable from observations, the paper uses observable implications to constrain learned representations. It enforces a necessary conditional-independence condition through a computable regularizer, while recognizing that these constraints do not guarantee recovery of the latent parts.

3 Observable Signature

  • With Z unobserved, the latent causal parts cannot be learned precisely from observational data, so the method instead requires observable implications of the causal structure.The observable signature constrains candidate representations but does not guarantee that they depend only on their intended latent parts.Source: §3 · p. 5
  • If g(X) and h(X) depend only on the Z-unaffected and Z-dependent parts, respectively, the causal graph implies g(X) and h(X) are conditionally independent given Y.This conditional independence can be measured from data and enforced during training.Source: §3 · p. 5
  • Jointly minimizing domain-specific risk while enforcing the observable signature can yield robust and fast-adapting predictors, but does not guarantee exact latent-part recovery.The signature is necessary, not sufficient, and multiple g,h pairs may parameterize the domain-specific predictor.Source: §3 · p. 5

3.1 Causal Regularization

  • The causal regularizer uses a conditional residual moment that must be zero under conditional independence, but this condition is not sufficient to establish independence.The conditional variable D is assumed discrete, as in this paper’s use case.Source: §3.1 · pp. 5–6

4 Learning Algorithm

Section 4 presents ACTIR, an algorithm that learns invariant and domain-specific predictors through a shared representation, then supports prediction in training domains and adaptation to new ones.

4 Learning Algorithm

  • ACTIR translates causal assumptions into a neural-network algorithm using reparameterization, bi-level optimization, and a practical solver.The paper names the method Anti-Causal Transportable and Invariant Representation.Source: §4 · p. 6

4.1 Reparameterization

  • A shared representation Φ feeds a fixed linear map W_b for the invariant predictor and a domain-specific map W_e for the domain-specific component.The shared representation can reuse features across predictors.Source: §4.1 · p. 6

4.2 Bi-Level Optimization

  • The bi-level objective balances low domain-specific and invariant risks while requiring each domain-specific perturbation to be optimal under the conditional-independence constraint.The learning goals include low training-domain risk for g and domain-optimal predictors of the form g(X) + M_e h(X).Source: §4 · p. 6, §4.2 · p. 7

4.3 Practical Algorithm

  • The practical algorithm replaces the inner optimality condition with an ℓ2 penalty on the gradient of domain-specific risk with respect to W_e.This makes the bi-level problem more tractable.Source: §4.3 · p. 7

4.4 Invariant and Adaptive Prediction

  • After training, the representation yields an invariant predictor, while a few labeled examples from a new domain support a domain-specific predictor by fine-tuning W_b.The method describes both invariant prediction and adaptation to a new domain.Source: §4.4 · p. 7

5 Related Work

The related work contrasts ACTIR’s restricted causal shift assumption with approaches to causal robustness, invariant representations, anti-causal learning, and adaptation.

Causal Prediction

  • Unlike causal robustness settings allowing interventions on any non-label node, this paper restricts shifts to unobserved confounders, permitting invariant predictors to use descendants of Y [e.g., PBM16; HDPM18; Arj+19; Lu+21].The narrower shift set enlarges the possible invariant predictors.Source: §5 · p. 7
  • Compared with Invariant Risk Minimization [Arj+19], ACTIR uses a different notion of invariance and learns non-stable factors alongside its representation.Both approaches seek a representation whose fixed linear map yields an invariant predictor.Source: §5 · p. 7
  • Unlike related anti-causal approaches that use generative models for latent stable and non-stable factors [Liu+21; Mit+21; Ils+20], this paper is fully nonparametric.Anti-causal prediction has also been studied in semi-supervised and transfer learning.Source: §5 · p. 8
  • In causal representation learning, ACTIR seeks to disentangle factors whose relationships with Y are domain-stable from those whose relationships vary [Bes+18] [Loc+20] [Sch+21] [WJ21].Source: §5 · p. 8
  • Veitch et al. [Vei+21] assume a known, observed Z and use one domain, whereas this paper assumes Z exists without requiring it to be known or measured and uses several domains.ACTIR also treats transportable representations, which Veitch et al. do not address.Source: §5 · p. 8

Domain Adaptation and Meta Learning

  • ACTIR combines invariant or robust learning with adaptation, connecting causal domain shift work to domain adaptation and meta-learning.Related theory shows shared structure can let meta-learning reduce sample complexity in unseen domains [TJJ21] [Du+20].Source: §5 · p. 8

6 Experiments

Experiments evaluate ACTIR’s invariant prediction, rapid adaptation, and feature disentanglement on synthetic and real-world shifts. Results show gains in several settings, while PACS sketch performance highlights a limitation when training domains may not be diverse enough.

6 Experiments

  • The experiments test invariant prediction, rapid linear-layer adaptation, and disentanglement, and include a counterexample showing ACTIR can fail when its causal assumptions do not hold.The studies use synthetic and real-world data; the authors suggest the anti-causal structure is appropriate for many image-classification problems.Source: §6 · pp. 8–9

6.1 Synthetic Dataset

  • On the synthetic shift, the invariant predictor reaches 75% accuracy, while the adaptive predictor reaches 90% and ERM reaches 10% in the test domain.ACTIR learns good invariant predictors and can adapt with very little data, whereas IRM’s performance stays the same after fine-tuning.Source: §6.1 · p. 9, Table 1 · p. 10

6.2 Color MNIST

  • On Color MNIST, ACTIR learns invariant and adaptive structures better than reference baselines, with representation coordinates separating features independent of color from color-dependent features.This separation is achieved without prior knowledge of Z or observations of it.Source: Figure 2 · p. 9, §6.2 · p. 10

6.3 Camelyon17

  • ACTIR has the best invariant accuracy on Camelyon17 and improves with small fine-tuning datasets, while other models need more examples for significant gains.Figure 3 plots accuracy against the number of fine-tuning examples; Table 1 reports accuracy across invariant and adaptive settings.Source: Figure 3 · p. 10, Table 1 · p. 10, §6.3 · p. 11

6.4 PACS

  • On PACS, ACTIR has competitive invariant performance and adapts with small fine-tuning datasets, but performs poorly on Sketch for both prediction types.The authors suspect that the art, cartoon, and photo training domains lack enough diversity for disentangling invariant and adaptive features.Source: §6.4 · p. 11, Table 2 · p. 11, Figure 4 · p. 12

6.5 VLCS

  • On VLCS, ACTIR has competitive invariant performance and adapts with small fine-tuning datasets; on LabelMe, it leads after fine-tuning despite similar pre-fine-tuning accuracy.Figure 5 reports accuracy across domains and fine-tuning set sizes; Table 3 reports invariant accuracy.Source: §6.5 · p. 12, Table 3 · p. 12, Figure 5 · p. 13

7 Discussion

The paper studies a specific class of anti-causal shifts, where domains share causal structure and differ in distributions of unobserved confounders. It presents a proof of concept and identifies training refinement and formal guarantees as future work.

  • Domains share a common anti-causal structure and differ only in the distribution of certain unobserved confounders.Source: §7 · p. 13
  • The work is a proof of concept; the practical training procedure may be refined, and formal guarantees for robust models remain future work.The authors suggest that such guarantees might rely on diversity of training domains.Source: §7 · p. 13

A Proofs

The proofs use d-separation in the causal graph to establish conditional independence, and derive a zero-expectation identity from conditional independence.

  • Under the causal graph, if g depends on X_z and h(X) depends only on X_z, then g(X) and h(X) are conditionally independent given Y.Source: A Proofs · p. 17
  • The proof reads d-separation from the causal graph and uses the fact that functions of independent variables remain independent.Source: A Proofs · p. 17
  • If A and B are conditionally independent given D, then E[A · (B − E[B|D])] = 0.Source: A Proofs · p. 17

B Counterexample: When the Data Generating Process Does not Fit Causal Assumptions

When the conditional-independence condition implied by the anti-causal structure fails, ACTIR loses competitive invariant performance against IRM, though it still outperforms ERM and adapts with a small fine-tuning set. The counterexample uses synthetic data in which X_y directly influences X_z.

Related papers

Loading 2207.01603v1…