PaperlayerView exact revision on arXiv

Source-linked AI summary

Counterfactual Generative Networks

Axel Sauer, Andreas Geiger

arXiv:2101.06046v1cs.LGcs.CV

TL;DR

The paper addresses shortcut learning, in which image classifiers exploit spurious texture, color, or background correlations instead of more robust factors. It decomposes image generation into independent causal mechanisms to create counterfactual images, which improve out-of-distribution robustness with a marginal drop in original-task accuracy.

  • Problem

    Image classifiers can rely on spurious correlations such as texture or background while ignoring object shape, limiting robustness when those correlations change.

  • Method

    The method structures image generation into independent mechanisms for factors such as shape, texture, and background, then generates counterfactual images by intervening on them.

  • Results

    Counterfactual images improve out-of-domain robustness with only marginally degraded overall classification accuracy on MNIST and ImageNet.

  • Takeaways & Limitations

    Causal generative structure can connect disentangled image generation with invariant classification and support classifiers less dependent on selected factors.

  • Takeaways & Limitations

    The approach assumes a known causal structure and a clear separation between class foreground and background, which may not hold in more complex scenes or medical images.

Abstract

from arXiv · show

Neural networks are prone to learning shortcuts -- they often model simple correlations, ignoring more complex ones that potentially generalize better. Prior works on image classification show that instead of learning a connection to object shape, deep classifiers tend to exploit spurious correlations with low-level texture or the background for solving the classification task. In this work, we take a step towards more robust and interpretable classifiers that explicitly expose the task's causal structure. Building on current advances in deep generative modeling, we propose to decompose the image generation process into independent causal mechanisms that we train without direct supervision. By exploiting appropriate inductive biases, these mechanisms disentangle object shape, object texture, and background; hence, they allow for generating counterfactual images. We demonstrate the ability of our model to generate such images on MNIST and ImageNet. Further, we show that the counterfactual images can improve out-of-distribution robustness with a marginal drop in performance on the original classification task, despite being synthetic. Lastly, our generative model can be trained efficiently on a single GPU, exploiting common pre-trained models as inductive biases.

1 INTRODUCTION

The paper targets shortcut learning, where classifiers exploit spurious image correlations instead of more robust factors such as object shape. It proposes causal generative mechanisms for counterfactual images and uses them to improve out-of-domain classification robustness.

  • Deep classifiers can learn spurious correlations, such as green grass texture for cows, that fail when the deployment distribution changes.
  • The approach decomposes image generation into independent mechanisms controlling object shape, texture, and background, enabling counterfactual combinations.
  • Counterfactual images train invariant classifiers that rely on selected factors while ignoring randomized factors of variation.
  • The method generates high-quality counterfactual images with direct control over shape, texture, and background using class labels and inductive biases rather than direct supervision.
  • Experiments on MNIST and ImageNet report improved out-of-domain robustness with only marginal degradation in overall accuracy.

2 STRUCTURAL CAUSAL MODELS FOR IMAGE GENERATION

The paper frames disentangled image generation as a structural causal model whose independent mechanisms control semantic factors. Intervening on these mechanisms produces counterfactual images for training classifiers invariant to selected factors.

  • 2.1 PROBLEM SETTING: Correlated factors can make classifiers rely on color or texture while ignoring shape, and standard disentangled VAEs cannot generate unseen factor combinations.
  • 2.2 STRUCTURAL CAUSAL MODELS: The proposed generative model decomposes image formation into autonomous functions so individual factors can be modified while other image aspects remain unchanged.
  • 2.2 STRUCTURAL CAUSAL MODELS: An SCM represents each variable as a function of its parents and an independent noise variable, with interventions replacing selected structural assignments.
  • 2.2 STRUCTURAL CAUSAL MODELS: Fixing the noise while changing mechanism inputs generates counterfactual images representing alternative versions of the same underlying instance.
  • 2.3 TRAINING AN INVARIANT CLASSIFIER: Invariant classifiers are trained by independently randomizing labels for all mechanisms and predicting the label supplied to one designated mechanism.

3 COUNTERFACTUAL GENERATIVE NETWORKS

The Counterfactual Generative Network decomposes image generation into separate shape, texture, and background mechanisms combined by a fixed composer. Its inductive biases and losses enable independently controlled counterfactual image generation through randomized mechanism inputs.

  • Independent mechanisms: The CGN models object shape, texture, and background with separate learned mechanisms, then combines them using a fixed composer.For ImageNet, the mechanisms are initialized from pre-trained BigGAN-deep-256 weights and optimized jointly end-to-end.
  • Composition mechanism: The composer forms images by alpha blending a generated mask, foreground texture, and background: xgen = C(m, f, b) = m ⊙f + (1 −m) ⊙b.This deterministic composition acts as an inductive bottleneck for generating realistic images.
  • Training and generation: During training, the mechanisms reconstruct conditional-GAN outputs using shared noise and labels, while inference randomizes inputs independently to produce counterfactuals.The cGAN supplies reconstruction supervision only during training; counterfactual generation independently varies mechanism labels and noise.
  • Mechanism-specific design: Shape, texture, and background mechanisms use distinct inductive biases and losses to isolate their intended image factors.Shape uses binary-mask constraints, texture uses architecture-specific structure, and background minimizes saliency to remove the object and inpaint its region.
  • Counterfactual generation: Randomizing labels across learned mechanism distributions yields a combinatorial space of counterfactual images, reaching 1000^3 label combinations for ImageNet.The model learns distributions rather than single outputs, allowing a nearly unlimited variety of shapes, textures, and backgrounds.
  • MNIST examples: On MNIST, the CGN generates high-quality counterfactual samples with varying class-conditional shapes, colors, and textures, including combinations absent from training data.The MNIST experiments require no network pre-training.

4 EXPERIMENTS

Experiments evaluate whether CGN learns disentangled mechanisms, which inductive biases support them, and whether counterfactual images improve classifier invariance across MNIST and ImageNet settings.

  • Disentangled mechanisms: The CGN generates high-fidelity counterfactuals on MNIST and ImageNet, including unseen combinations of shape, texture, and background.On ImageNet, one CGN covers all 1000 classes; on MNIST, counterfactual combinations absent from training are generated without pre-training.
  • Inductive biases: ImageNet scaling depends on powerful pre-trained models, enabling CGN training on one GTX 1080Ti within 12 hours.The paper contrasts this with BigGAN training on a 512-core TPU v3 Pod for up to 48 hours.
  • Inductive biases: Removing individual losses harms disentanglement quality, and jointly optimizing all losses is needed for high Inception Score without mask collapse.The unconstrained BigGAN reaches an IS of 202.9 as a natural upper bound; removing Lshape produces poor masks and low IS.
  • Invariant classifiers: Counterfactual training makes MNIST classifiers invariant to randomized colors and textures, while LNTL and IRM struggle to scale to more complex data.The test domain preserves only the digit shape as class-correlated, and adding environments can improve IRM but is impractical beyond simulated settings.
  • Invariant classifiers: Classifier shape and texture preferences can be controlled without significant degradation in ensemble performance.The cue-conflict experiments use separate classifier heads, each invariant to all but one field of variation.
  • Invariant classifiers: Training on generated mixed-background data reduces the BG-Gap while retaining high IN-9 accuracy, although a gap to full background invariance remains.The remaining gap is partly attributed to the domain difference between generated and real images.

5 RELATED WORK

The paper relates its approach to disentangled representation learning and invariant classification, addressing correlated factors and causal assumptions in prior methods.

  • Disentangled Representation Learning: Prior disentanglement methods aim to control image factors but struggle when underlying factors are highly correlated.
  • Disentangled Representation Learning: Related work disentangles background, shape, pose, and texture using object bounding boxes, but assumes images contain a single object category.
  • Disentangled Representation Learning: The paper scales its approach to all ImageNet classes, enabling inter-class counterfactuals that prior interpretable GAN directions cannot generate.
  • Invariant Classification: Invariant-classification methods use environment stability or removal of bias information rather than an explicit underlying causal model.

6 DISCUSSION

The discussion identifies reduced realism and strong assumptions as the main limitations, while noting that generated images do not yet improve ImageNet classification.

  • The method assumes images have a clear class foreground and background, limiting applicability to complex scenes and tasks without that distinction.
  • The additional CGN constraints reduce realism, and generated images are not yet high-quality enough to improve ImageNet performance.The paper notes that even higher-IS state-of-the-art generators are not yet sufficient for competitive ImageNet classifier training.
  • The experiments assume the causal structure is known, a stronger assumption than in general disentanglement frameworks.The paper suggests causal discovery as a possible extension.

7 CONCLUSION

The paper concludes that causal independent mechanisms can generate controllable counterfactuals for invariant-classifier training, with demonstrations spanning MNIST variants and ImageNet.

  • The method structures a generative network into independent mechanisms that generate counterfactual images for training invariant classifiers.
  • The approach is demonstrated on multiple MNIST variants and ImageNet, while its ideas remain compatible with advances in generative modeling.
  • A β-VAE on colored MNIST forms data clusters, but its latent axes do not separately represent color and shape, even with more dimensions.
  • Colored MNIST uses ten class-associated colors with test-time random assignment, evaluated in the most biased setting with σ = 0.02.
  • Double-Colored MNIST additionally encodes the class label in the background color, while Wildlife MNIST adds foreground and background texture biases.

A.3 ABLATION STUDIES

The ablations show that more counterfactual data and multiple counterfactuals per noise sample improve test accuracy, while the training procedure combines disentanglement losses and large gradient accumulations.

  • More counterfactual data improves test accuracy, and drawing several counterfactuals per sampled noise produces a more stable non-spurious signal.
  • For colored MNIST, the counterfactual ratio is capped at ten because each shape has only ten possible colors.
  • The MNIST and ImageNet SCMs generate masks, foregrounds, backgrounds, and composed images through independent mechanisms.
  • The shape loss combines binary-mask regularization with a mask constraint that forces the object to occupy between 10% and 90% of the image.
  • The texture loss samples masked 15 × 15 patches, arranges 36 into a 6 × 6 grid, and matches the foreground perceptually.
  • Training uses large gradient accumulations of 4000 steps, and the model uses pre-trained BigGAN models while omitting real data for CGN training.

C.4 CLASSIFIER TRAINING

Classifier training uses a shared CNN or ResNet-50 backbone with task-specific heads and evaluates disentangled image components through interpolation and generated outputs. The appendix visualizes outputs for multiple ImageNet classes and interpolations across shapes, textures, and backgrounds.

  • MNIST experiments use one CNN architecture across approaches and vary correlation environments for double-colored and Wildlife MNIST.
  • ImageNet classification shares a ResNet-50 backbone and adds separate shape, texture, and background heads, averaging their logits on real images.
  • Figures 20–22 visualize interpolation separately in shapes, textures, and backgrounds by interpolating between u and y pairs.

D.3 MORE COUNTERFACTUAL SAMPLES

Additional counterfactual examples are presented for MNIST variants and ImageNet, illustrating generated images alongside their labels where available.

  • Figure 23 presents counterfactuals for colored MNIST, double-colored MNIST, and Wildlife MNIST.
  • Figure 24 presents ImageNet counterfactual images above their ImageNet labels.

APPENDIX E IM OUTPUTS OVER THE COURSE OF TRAINING

The appendix tracks independent-mechanism outputs during training and documents several reconstruction and realism limitations. Small objects can retain background traces in texture maps, large-object inpainting can leave faint residues, and generated images have lower realism.

  • Training-progress figures track pre-masks, masks, texture maps, backgrounds, and composite images from the beginning to the end of training.
  • Texture maps for relatively small objects can retain traces of the background, with patch-size adaptation suggested as a possible remedy.
  • Large-object inpainting can leave faint background artifacts, although an object covers these residues in composite images.
  • Generated images have lower realism, as indicated by lower IS, because of enforced constraints and simplified composition; a refinement network is suggested as a remedy.

APPENDIX G COLLAPSING MASKS

The collapsing-mask analysis tests whether independently trained classifiers identify the invariant causal signal across environments. Shape is identified as causal because only the shape classifier remains stable across environments.

  • Disabling shape or texture losses causes mask collapse in the corresponding training comparison, while the full-loss condition is also shown.
  • The CGN is trained on double-colored MNIST under varying correlations, following an invariant-signal assumption across environments.
  • The shape classifier is designed to be invariant to object and background color.
  • The object color classifier is invariant to object shape and background color, while the background color classifier is invariant to object shape and object color.
  • Only the shape classifier has stable test accuracy across environments, identifying shape as the causal signal in double-colored MNIST.
Loading 2101.06046v1…