PaperlayerView exact revision on arXiv

Source-linked AI summary

CausalGAN: Learning Causal Implicit Generative Models with Adversarial Training

Murat Kocaoglu, Christopher Snyder, Alexandros G. Dimakis, Sriram Vishwanath

arXiv:1709.02023v2cs.LGcs.AIcs.ITstat.ML

TL;DR

The paper addresses causal conditional image generation when labels depend on one another and ordinary conditional GANs do not represent those dependencies. It trains a graph-structured label generator and new conditional GAN architectures, showing that the resulting models support observational and interventional sampling, including unseen label combinations.

  • Problem

    Existing conditional-generation architectures choose labels independently, so they do not capture label dependencies or support sampling other labels and images after conditioning on one label.

  • Method

    The paper uses a two-stage adversarial procedure: WassersteinGAN learns graph-structured binary labels, then CausalGAN or CausalBEGAN generates images conditioned on those labels.

  • Results

    The optimal CausalGAN generator samples the correct class-conditional image distribution, and the combined models capture observational and interventional image-label distributions, including unseen combinations.

  • Takeaways & Limitations

    Causal graph structure enables image generators to sample interventional distributions that differ from the dataset and can include label combinations absent during training.

  • Takeaways & Limitations

    The framework assumes the causal graph is given and causal sufficiency holds, and CausalBEGAN can occasionally produce poor-quality or anomalous images when label guidance conflicts with image quality.

Abstract

from arXiv · show

We propose an adversarial training procedure for learning a causal implicit generative model for a given causal graph. We show that adversarial training can be used to learn a generative model with true observational and interventional distributions if the generator architecture is consistent with the given causal graph. We consider the application of generating faces based on given binary labels where the dependency structure between the labels is preserved with a causal graph. This problem can be seen as learning a causal implicit generative model for the image and labels. We devise a two-stage procedure for this problem. First we train a causal implicit generative model over binary labels using a neural network consistent with a causal graph as the generator. We empirically show that WassersteinGAN can be used to output discrete labels. Later, we propose two new conditional GAN architectures, which we call CausalGAN and CausalBEGAN. We show that the optimal generator of the CausalGAN, given the labels, samples from the image distributions conditioned on these labels. The conditional GAN combined with a trained causal implicit generative model for the labels is then a causal implicit generative model over the labels and the generated image. We show that the proposed architectures can be used to sample from observational and interventional image distributions, even for interventions which do not naturally occur in the dataset.

1 Introduction

The paper extends conditional image generation with causal graphs so labels retain dependencies and interventions can generate distributions absent from the dataset. It proposes causal implicit generative models trained adversarially, with guarantees for conditional and interventional sampling.

  • Motivation: Independent-label conditional GANs cannot model dependencies among labels or sample other labels and images after conditioning on one label.The paper motivates causal structure using examples such as species and color in bird images.
  • Motivation: Causal image generation aims to capture both dependence and causal effects between labels and the image.The proposed graph represents labels as causes of the image while also modeling causal relations among labels.
  • Causal sampling: Interventions differ from conditioning because they fix a variable while preserving its causal mechanism and affect descendants without changing ancestor distributions.With Gender → Mustache, intervening on Gender = Female yields females without mustaches, unlike an empty graph.
  • Causal sampling: Causal implicit generative models sample probability, conditional, and interventional distributions but do not provide likelihoods for data points.The paper trains them by structuring generator connections according to the causal graph.
  • Method: The two-stage procedure trains a graph-structured label model with WassersteinGAN, then combines it with the proposed CausalGAN conditional generator.The resulting model covers labels and images jointly.
  • Results: The optimal CausalGAN generator samples the data image distribution conditioned on labels, while the combined model supports correct conditional and interventional sampling.The framework is evaluated on CelebA, including label combinations absent during training such as woman with mustache.

2 Related Work

Related work develops conditional, information-based, joint, and identity-aware generative models, while this paper focuses on causal dependence among labels and causal image sampling.

  • Conditional generation: Conditional GANs feed image labels to both the generator and discriminator, while InfoGAN maximizes a variational lower bound on label-image mutual information.These approaches address label-aware generation without the causal framework developed here.
  • Causal deep learning: Prior causal deep-learning work connects conditional GAN layers with structural equation models and uses neural networks to discover causal relations between image labels.The paper instead assumes a given causal graph and learns a causal generative model.
  • Other generative models: BiGAN and ALI learn mappings from image space to latent space, while CoGAN learns joint distributions from marginals through weight sharing.The paper notes that CoGAN’s compatibility with graph-structured generators is unclear.
  • Other generative models: Identity-observation latent factorization and age-conditioned generation provide additional ways to control generated face attributes.These methods address identity preservation or age modification rather than causal interventions over label relationships.

3 Background

The paper uses Pearl’s structural causal-model framework, in which directed acyclic graphs and structural equations define causal mechanisms and observable distributions. Interventions modify those mechanisms by removing a variable’s incoming connections, unlike conditioning.

  • Structural causal models: Pearl’s framework represents causal models with structural equations and directed acyclic graphs between random variables.The graph structure encodes causal relationships among variables.
  • Structural causal models: A structural causal model contains observable and exogenous variables, functional relations, and a distribution over exogenous variables that induces the observable joint distribution.Under causal sufficiency, exogenous variables are mutually independent.
  • Interventions: An intervention do(X_i = x_i) changes the causal mechanism by removing X_i’s connections to its parents, whereas conditioning leaves the causal graph unchanged.The post-interventional distribution is computed using the modified assignments for intervened nodes.
  • Assumptions: Causal sufficiency assumes that no unobserved variable affects more than one observable variable.The paper explicitly adopts this assumption for its causal model.
  • Assumptions: The paper assumes the causal graph is given and learns its functions and exogenous-variable distributions rather than identifying the graph from observational data.Multiple causal graphs can produce the same joint distribution without experiments or additional assumptions.

4 Causal Implicit Generative Models

The paper defines causal implicit generative models by requiring a feedforward generator to match both a causal graph and its observational distribution. Adversarial training can learn such models, whose observational distribution determines interventional distributions when the graph is correct.

  • Interventional validity: If two causal models share the true causal graph and observational distribution, they have identical interventional distributions under any intervention.The proof uses the fact that causal Bayesian-network interventions are determined by the graph and conditional probabilities.
  • Scope: The learned functions and exogenous-noise distributions are not unique, even when the causal graph and observed joint distribution are fixed.The procedure learns one causal model compatible with the observed distribution and graph.
  • Architecture: Figure 2 illustrates the correspondence between feedforward neural-network components and structural-equation functions in a causal graph.The figure contrasts the causal graph implied by a standard generator with a network implementation of X → Z ← Y.
  • Definition: A causal implicit generative model uses a feedforward network consistent with graph D whose output distribution matches the causal model’s observational distribution.Consistency constrains each output to depend on its graph parents and a separate subset of independent noise variables.
  • Adversarial training: Adversarial training arranges generator connections to mirror causal structure, with independent noise terms implementing the exogenous variables in structural equations.For X → Z ← Y, separate networks represent fX and fY, while a downstream network represents fZ(X, Y, NZ).

5 Causal Generative Adversarial Networks

The paper decomposes causal image generation into learning a causal label model and then generating images conditioned on those labels. CausalGAN is designed to recover class-conditional image distributions, while CausalBEGAN extends BEGAN with label-aware training and convergence monitoring.

  • Two-stage causal generation: The proposed two-step procedure first learns a causal implicit generative model over labels, then trains an image generator conditioned on those labels.The label model controls which image distribution is sampled under conditioning or intervention.
  • CausalGAN architecture: Combining a pretrained causal label model with CausalGAN yields a causal implicit generative model for labels and images, under a strictly positive joint label distribution assumption.That assumption does not hold for CelebA, although the trained model can extrapolate to certain interventional distributions when CausalGAN is not trained too long.
  • Causal label generation: WassersteinGAN is used for the causal label generator because standard GAN training is not suited to discrete distributions.The authors report that adding noise to make labels continuous also works and yielded better empirical convergence.
  • CausalGAN architecture: CausalGAN combines labeler, anti-labeler, discriminator, and label-loss terms so its optimal generator samples from class-conditional image distributions.The anti-labeler discourages generating only a few typical faces for a fixed label combination.
  • CausalBEGAN architecture: CausalBEGAN extends BEGAN by feeding labels to the generator and introducing label-aware margins based on the observation that label gradients are most informative when image quality is high.The generator jointly considers image quality and label losses, while the additional margin terms regulate when label loss contributes.
  • CausalBEGAN architecture: The CausalBEGAN extension preserves BEGAN’s monotonically decreasing scalar for tracking convergence during gradient descent.The paper reports that Mcomplete decreases progressively during optimization.

6 Theoretical Guarantees for CausalGAN

Theoretical analysis establishes when CausalGAN recovers class-conditional image distributions and when its two-stage construction forms a causal implicit generative model.

  • The optimal CausalGAN generator outputs class-conditional image distributions when the Causal Controller matches the real label distribution and labelers are optimal.The guarantee is stated first for one binary label and extends to multiple binary labels.
  • The optimal Labeler estimates Pr(l = 1|x), while the optimal Anti-Labeler estimates Pg(l = 1|x) for generated samples.
  • Minimizing the generator criterion under optimal discriminator, Labeler, and Anti-Labeler yields class-conditional image distributions.
  • Combining a causal label model with a class-conditional GAN yields a causal implicit generative model for graphs in which Image is a sink node.
  • With d binary labels, a 2^d-output Labeler and Anti-Labeler can provide the class-conditional guarantee; for image-determined labels, the joint label posterior is also correct.

7 Implementation

The implementation combines causal label generation, conditional image generation, and adversarial objectives, with practical adjustments for discrete outputs and training stability.

  • Wasserstein training produces almost discrete labels from continuous uniform noise, although outputs are rounded before entering the image generator.
  • CausalGAN extends DCGAN with Labeler networks, a pretrained Causal Controller, and modified loss functions.
  • The labeler implementation averages separate loss terms across the d labels rather than estimating probabilities for all 2^d label combinations.This implementation lacks the unrestricted-data guarantee, but is sufficient when labels are completely determined by images.
  • The Anti-Labeler is initially maximized to counter label-conditioned mode collapse, such as repeatedly generating one face for a fixed label.
  • Later training can make Anti-Labeler maximization conflict with Labeler minimization, weakening the generator's conditional image-generation property.
  • The Anti-Labeler loss decays exponentially with time constant T = 3000, approximately one training epoch in the experiments.

8 Results

Experiments show that causal-graph structure affects convergence, Wasserstein training yields near-discrete labels, and generated images distinguish intervention from conditioning.

  • 8.1 Dependence of GAN Behavior on Causal Graph: Across synthetic line, collider, and complete graphs, convergence depends on whether the generator architecture can represent the true joint distribution.
  • 8.1 Dependence of GAN Behavior on Causal Graph: For line-graph data, the line architecture converges best; complete and shallow fully connected networks also perform well, while deeper networks and the wrong collider graph perform worse.
  • 8.1 Dependence of GAN Behavior on Causal Graph: For collider data, fully connected 3- and 5-layer generators perform best, whereas the line graph performs worse; 10 layers show the worst convergence behavior.
  • 8.1 Dependence of GAN Behavior on Causal Graph: For complete-graph data, fully connected 3 performs best, while line and collider architectures do not show convergence.
  • 8.2 Wasserstein Causal Controller on CelebA Labels: 96% of unrounded Causal Controller samples fall within 0.05 of 0 or 1, demonstrating an almost discrete output distribution.
  • 8.2 Wasserstein Causal Controller on CelebA Labels: TVD decreases to 0 for cG1 and rcG1, while G1 asymptotes near 0.14 because of its incorrect conditional-independence assumptions.
  • 8.3 CausalGAN Results: Interventional and conditional sampling differ for mustache and bald labels: interventions can produce combinations absent during training, whereas conditioning follows observed label associations.
  • 8.4 CausalBEGAN Results: Removing the margin of margins deteriorates image quality for rare labels, while CausalBEGAN illustrates interventional-versus-conditional sampling for four labels.

9 Conclusion

The paper concludes that its label-conditioned generative models support both conditional and interventional sampling, backed by theoretical guarantees and illustrated on face labels.

  • The proposed generative model accepts labels and can sample both conditional and interventional distributions.
  • The theoretical analysis provides guarantees for correct sampling under interventions and conditionings, illustrated with CausalGAN and CausalBEGAN.

10 Appendix

The appendix derives the generator objective under optimal discriminator, labeler, and anti-labeler networks, concluding that the global optimum matches the relevant data distribution.

  • At the optimum, the generated distribution matches the data distribution, yielding generator criterion C(G) = −1.
  • The proof uses the non-negativity of KL divergence to establish that the generator criterion is bounded below by −1.
  • The equality condition identifies distribution matching as the condition for attaining the global minimum.

10.3 Proof of Corollary 2

For multiple binary labels, the appendix extends the CausalGAN analysis and proves that its optimal generator samples from the data’s class-conditional image distributions.

  • The multi-label construction extends the binary-label objective and retains the same optimum characterization.
  • For d binary labels, the labeler outputs a 2^d-dimensional distribution over label combinations and is optimized using the corresponding label assignments.
  • The optimal labeler recovers the posterior probability of each label combination given the image.
  • Theorem 3 states that the global minimum is achieved if and only if the generator matches every class-conditional image distribution Pdata(x|l).
  • The proof establishes equality between generated and data joint distributions, which implies equality of the corresponding conditional image distributions.

10.5 Alternate CausalGAN Architecture for d Labels

The alternate multiple-label CausalGAN architecture matches marginal image-label distributions at its optimum, but requires an image-determines-labels assumption to guarantee full joint conditional sampling.

  • The architecture’s optimal generator matches each individual label-conditioned image distribution and the overall image distribution.
  • This marginal matching does not generally imply that the generated image-label joint distribution matches the data joint distribution.
  • The required assumption is that the image determines all labels, making the label vector a deterministic function of the image.
  • Under this assumption, the conditional label marginals determine the joint conditional distribution through the stated factorization.
  • The resulting equalities establish that generated and data joint distributions, and therefore class-conditional image distributions, coincide.

10.6 Additional Simulations for Causal Controller

Additional simulations show that causal-graph choice affects synthetic distributions, causal controllers learn useful label marginals, and CausalGAN/CausalBEGAN exhibit conditional generation behavior with specific failures and sensitivities.

  • Synthetic data experiments: Using the correct causal graph produces the closest synthetic scatter plot to the original data, while a collider graph produces a substantially different distribution.
  • Synthetic data experiments: The experiments compare the original graph, completed and reverse orderings, and a fully connected feedforward generator as alternative causal specifications.
  • Causal Controller: The causal controllers never output the label combination {Female,Mustache}, despite graph inaccuracies in the experiments.
  • Causal Controller: Both tested causal graphs learn reasonable marginal distributions for all labels, with the worst label differing by no more than 0.03 from the dataset.
  • CausalGAN: CausalGAN label sweeps vary one label while holding other inputs fixed, and 256 samples are used to examine image diversity and mode collapse.
  • CausalGAN: Omitting CausalGAN’s third margin term worsens image quality for rare labels, especially under the mustache intervention.
  • CausalBEGAN: CausalBEGAN’s Mcomplete decreases monotonically during training, while its eyeglasses label is not properly captured in label-sweep experiments.
  • Joint training: Jointly training labels and images did not learn image generation after 20k steps in the reported experiment.
Loading 1709.02023v2…