Causality Inspired Representation Learning for Domain Generalization

Fangrui Lv; Jian Liang; Shuang Li; Bin Zang; Chi Harold Liu; Ziteng Wang; Di Liu

Source-linked AI summary

Causality Inspired Representation Learning for Domain Generalization

Fangrui Lv, Jian Liang, Shuang Li, Bin Zang, Chi Harold Liu, Ziteng Wang, Di Liu

arXiv:2203.14237v1Mar 27, 2022cs.LGcs.CV

TL;DR

Domain generalization must handle target distributions that differ from source domains, while statistical dependence may not capture invariant causal structure. The paper formalizes DG with a structural causal model and proposes CIRL to learn representations that mimic unobserved causal factors. CIRL outperforms compared baselines across experiments, including a 1.0% improvement over FACT on Digits-DG.

Problem
Domain generalization methods often model statistical dependence between inputs and labels, although that dependence can change across domains and fail to represent underlying causal mechanisms.
Method
CIRL learns representations that mimic causal factors by enforcing separation from non-causal factors, joint independence, and causal sufficiency.
Results
CIRL beats compared baselines in average accuracy on Digits-DG, surpassing FACT by 1.0% and CCSA and MMD-AAE by 8.0% and 7.9%, respectively.
Takeaways & Limitations
The experiments support using causal-factor-inspired representations to improve domain generalization beyond approaches based only on statistical dependence.
Takeaways & Limitations
The causal and non-causal factors are unobserved, and which factors can be extracted depends on distribution shifts, supervision signals, and available interventions.

Abstract

from arXiv · show

Domain generalization (DG) is essentially an out-of-distribution problem, aiming to generalize the knowledge learned from multiple source domains to an unseen target domain. The mainstream is to leverage statistical models to model the dependence between data and labels, intending to learn representations independent of domain. Nevertheless, the statistical models are superficial descriptions of reality since they are only required to model dependence instead of the intrinsic causal mechanism. When the dependence changes with the target distribution, the statistic models may fail to generalize. In this regard, we introduce a general structural causal model to formalize the DG problem. Specifically, we assume that each input is constructed from a mix of causal factors (whose relationship with the label is invariant across domains) and non-causal factors (category-independent), and only the former cause the classification judgments. Our goal is to extract the causal factors from inputs and then reconstruct the invariant causal mechanisms. However, the theoretical idea is far from practical of DG since the required causal/non-causal factors are unobserved. We highlight that ideal causal factors should meet three basic properties: separated from the non-causal ones, jointly independent, and causally sufficient for the classification. Based on that, we propose a Causality Inspired Representation Learning (CIRL) algorithm that enforces the representations to satisfy the above properties and then uses them to simulate the causal factors, which yields improved generalization ability. Extensive experimental results on several widely used datasets verify the effectiveness of our approach.

1. Introduction

The paper frames domain generalization as a causal problem: statistical dependence may change across domains, so robust representations should recover invariant causal factors. CIRL approximates these unobserved factors by enforcing separation, joint independence, and causal sufficiency, with experiments showing improved generalization.

Domain generalization seeks to transfer knowledge from multiple source domains to an unseen target domain with a different distribution.
Existing DG methods mainly model statistical dependence between data and labels, which may not capture the underlying causal mechanisms needed beyond i.i.d. settings.
The proposed causal model separates each input into domain-invariant causal factors S and category-independent non-causal factors U, with only S causing the label.
Ideal causal factors should be separated from non-causal factors, jointly independent, and causally sufficient for classification.
CIRL uses causal intervention, factorization, and adversarial masking modules to learn representations that mimic these causal-factor properties.
Extensive experiments on widely used datasets demonstrate the effectiveness and superiority of CIRL.

2. Related Work

Related work addresses domain generalization through invariant representations, augmentation, meta-learning, and causal approaches. The paper situates causal mechanisms as a route to knowledge that remains robust beyond observed data distributions.

Domain generalization methods align domain distributions through invariant representations and also use data augmentation to enrich source diversity.
Causal mechanisms distinguish statistical dependence from reasoning that supports predictions for counterfactual or far-out-of-distribution situations.
Causality-based DG methods seek invariant causal mechanisms or causal features to improve out-of-distribution generalization.

3. Method

The paper formalizes domain generalization through a structural causal model and learns representations that mimic causal factors using three coordinated modules. CIRL separates non-causal information, factorizes representation dimensions, and promotes causal sufficiency for classification.

DG from the Causal View: The structural causal model separates inputs into causal factors S, which influence labels, and non-causal factors U, which influence inputs but are domain-related.Given S, the model posits an invariant conditional mechanism P(Y|S) across distributions.
DG from the Causal View: Because causal factors are unobservable, CIRL learns representations that mimic their required properties rather than reconstructing them directly.The target properties are separation from U, joint independence among factors, and causal sufficiency for classification.
Causal Intervention Module: CIRL first intervenes on domain-related non-causal factors to generate data with altered U but unchanged S, enforcing representation invariance to that intervention.The method perturbs amplitude information and combines it with the original phase to create augmented images.
Causal Factorization Module: The factorization module uses representations from original and augmented images to preserve corresponding dimensions while reducing dependence between different dimensions.This implements separation from U and joint independence through correlation-based representation constraints.
Adversarial Mask Module: The adversarial mask module detects dimensions with less causal information and trains them to acquire novel causal information, increasing the representation’s causal sufficiency.Joint independence ensures the added information is not redundant with other dimensions.

4. Experiment

Experiments evaluate CIRL under leave-one-domain-out domain generalization on Digits-DG, PACS, and Office-Home, with ablations and analytical studies examining its learned representations and robustness. CIRL achieves the strongest reported average performance across the main benchmarks and shows evidence of causal-representation properties.

Datasets: Digits-DG includes four visually different digit domains, with 600 images per class per domain and an 80%/20% train-validation split.The domains are MNIST, MNIST-M, SVHN, and SYN, differing in font style, background, and stroke color.
Datasets: PACS contains 9,991 images across four stylistically different domains and seven shared categories, using its original training-validation split.The domains are Art-Painting, Cartoon, Photo, and Sketch; the categories include dog, elephant, giraffe, guitar, house, horse, and person.
Datasets: Office-Home contains 15,500 images from 65 categories across four domains that differ in viewpoint and image style, with each domain split 90%/10% for training and validation.The domains are Art, Clipart, Product, and Real-World.
Experimental Results: 1.0% improvement over FACT on Digits-DG accompanies larger gains of 8.0% and 7.9% over CCSA and MMD-AAE, while CIRL achieves the best average accuracy among compared baselines.These results are reported for average accuracy and are presented as evidence favoring causal mechanisms over superficial statistical dependence.
Experimental Results: 1.76% and 4.01% are CIRL’s reported margins over MatchDG on PACS with ResNet-18 and ResNet-50, respectively, while CIRL obtains the highest average accuracy on both backbones.The paper also reports poorer relative performance on saturated tasks or noisy samples, while retaining second-best performance on those tasks.
Experimental Results: 67.12% average performance on Office-Home surpasses FACT by 0.56%, and combining Causal Intervention with Causal Factorization performs better than either separated or independent representation variant alone.The Office-Home result is described as the best average performance on that benchmark; the ablation is conducted on PACS with ResNet-18.
Analytical Experiments: CIRL representations become more independent with training and achieve category-related attention patterns, while remaining competitive across broad ranges of τ and κ.The reported robust ranges are 5.0 ≤τ ≤10.0 and 0.5 ≤κ ≤0.6 for both ResNet-18 and ResNet-50.

5. Conclusion

The conclusion frames domain generalization causally: CIRL reconstructs causal factors and intrinsic causal mechanisms, and experiments support its effectiveness and superiority.

Conclusion: CIRL learns representations that mimic causal factors by reconstructing causal factors and excavating intrinsic causal mechanisms.The framework is motivated as an alternative to approaches based primarily on statistical dependence.
Conclusion: Comprehensive experiments demonstrate the effectiveness and superiority of CIRL within the paper’s domain-generalization evaluation.

Supplementary Materials

The paper discusses potential benefits and risks of applying causal domain generalization beyond observed data distributions, emphasizing annotation savings and reliability concerns.

Potential Negative Societal Impacts: Causal domain generalization may save annotation cost and time, improve knowledge reusability across domains, and enhance generalization on out-of-distribution data.
Potential Negative Societal Impacts: Systems used far from observed distributions may be misleading, and broader automation could cancel some classification or target-detection jobs.The paper identifies reliability in conditions very far from observed distributions as requiring caution and further research.

B. Implementation Details

The implementation normalizes representations before intervention comparison, uses Gumbel-softmax for approximate k-hot sampling, and specifies backbones and feature dimensions across datasets.

Causal intervention: Z-score normalization standardizes representation columns before measuring pre- and post-intervention correlation.The normalized representations are R_o and R_a.
Gumbel-softmax trick: Gumbel-softmax approximately samples a k-hot vector from a probability vector, with k = ⌊κN⌋.The probability vector has nonnegative entries summing to one.
Network architecture: Digits-DG uses ConvNet, while PACS and Office-Home use ImageNet-pre-trained ResNet backbones with a three-layer MLP masker.Feature dimension N is 256 for ConvNet, 512 for ResNet-18, and 2048 for ResNet-50.

C. Additional Results

Additional experiments examine feature dimension, κ selection, and amplitude-mixing sampling strategies, finding stable performance and limited sensitivity to these choices.

Feature dimension: Model performance remains relatively stable across feature dimensions N = {128, 256, 512, 1024, 2048} with fixed κ = 60% on PACS.This experiment is summarized in Table 7.
κ selection: κ = 60% is a stable choice across feature dimensions N = 512 and N = 2048.The tested κ values are 50%, 60%, 70%, 80%, and 90%.
Amplitude mixing: CIRL is not sensitive to whether amplitude-spectrum image pairs are sampled intra-domain or inter-domain on PACS.Both sampling strategies produce good performance in the reported ResNet-18 experiments.

C.3. Experimental Results on mini-DomainNet

On the larger mini-DomainNet benchmark, CIRL is evaluated with leave-one-domain-out results and outperforms the state-of-the-art method by 3.26%.

Results: 3.26% is the reported margin by which CIRL outperforms the state-of-the-art method on mini-DomainNet.The benchmark contains 140K images and 126 classes.
Motivation: The mini-DomainNet experiment addresses the limitation that the main benchmarks are relatively small and may have saturated performance.The paper characterizes improvements on those benchmarks as incremental.
Visualization: Figure 6 compares baseline DeepAll and CIRL visualizations with Cartoon, Photo, and Sketch as unseen PACS target domains.The figure provides additional visualization examples for domain-generalization behavior.

C.5. Experimental Results with Error Bars

The paper reports repeated-run averages with mean±std error bars and provides CIRL results across several leave-one-domain-out experiment tables.

Evaluation protocol: All experiments are run multiple times with random seeds, and results are reported as mean±std alongside main-body averages.Complete error-bar results are provided in Tables 11–14.
Digits-DG: CIRL reports 86.08±0.32, 80.59±0.19, 95.93±0.03, 82.67±0.47, and 86.32 in one leave-one-domain-out result row.The corresponding table is identified as a Digits-DG results table.
PACS: CIRL reports 90.67±0.21, 84.30±0.17, 97.84±0.08, 87.68±0.40, and 90.12 in another leave-one-domain-out result row.The corresponding table is identified as a PACS ResNet-50 results table.
Office-Home: CIRL reports 61.48±0.17, 55.28±0.29, 75.06±0.24, 76.64±0.09, and 67.12 in another leave-one-domain-out result row.The corresponding table is identified as an Office-Home ResNet-18 results table.