PaperlayerView exact revision on arXiv

Source-linked AI summary

To do($x$) or not to do($x$): Medical Image Counterfactuals for Dataset Augmentation

Yasin Ibrahim, Robin J. Evans, Konstantinos Kamnitsas

arXiv:2609.14124v1cs.LGcs.AI

TL;DR

Biased medical-image datasets can limit clinical applicability, and synthetic augmentation risks reproducing undesirable correlations. The paper compares causal and non-causal conditioning strategies for counterfactual generation and augmentation; results vary with graph correctness and generation setting, with benefits reported for causal and undirected augmentation on chest x-rays.

  • Problem

    The paper addresses limited systematic evidence on when causal counterfactual mechanisms provide practical bias-mitigation benefits over simpler generation, amid risks that synthetic data reproduce biased correlations.

    Source: §I · p. 1

  • Method

    The study compares deterministic, undirected, and causal conditioning, generating counterfactuals from real images and de novo images sampled from a generative model.

    Source: §I · p. 1, §VII · p. 9

  • Results

    On Morpho-MNIST, Causal performs consistently with a correct graph but loses its advantage when misspecified; on chest x-rays, Causal and Undirected augmentation improve disease prediction over no augmentation and Deterministic.

    Source: §VII · p. 9

  • Takeaways & Limitations

    When practitioners are uncertain about the underlying DAG, Deterministic or Undirected conditioning may be preferable to Causal modelling.

    Source: §VII · p. 9

Abstract

from arXiv · show

Medical image analysis is often hindered by biased datasets, which can lead to biased models and limited clinical applicability. A promising strategy for mitigating such biases is to augment training data with synthetic images. Counterfactual (CF) generation is one such strategy, though the term is used in two different senses: in some works, CFs are produced through causality-based interventions derived from structural causal models, whereas in others, they are produced by non-causal image edits or conventional conditional generative models, such as altering anatomy or adding pathologies. In this work, we study this distinction and evaluate its practical consequences for medical image augmentation. We compare three conditioning strategies: $\textit{Deterministic}$, which changes selected variables while holding the remaining variables fixed; $\textit{Undirected}$, which updates variables according to learned statistical associations without assigning causal directions; and $\textit{Causal}$, which propagates interventions along a directed causal graph. We analyse how these choices affect the resulting images, and explore when causally grounded methods improve dataset augmentation or bring limited benefit. In particular, we assess downstream performance and fairness, where fairness refers to reduced sensitivity to dataset biases across sensitive subgroups. Our experiments demonstrate that using a causal approach to synthetic training data generation can lead to tangible benefits, with these insights offering valuable guidance to machine learning practitioners for the effective design of data generation protocols.

I. INTRODUCTION

The paper distinguishes causal counterfactuals from broader non-causal image edits, then compares conditioning strategies to examine when their assumptions benefit medical-image augmentation. It situates this comparison in the challenge of mitigating dataset bias while preserving clinically relevant image properties.

I. INTRODUCTION

  • Biased datasets can yield biased models and hinder clinical deployment, while synthetic augmentation can counter undesirable correlations but may also reproduce or strengthen them.Prior work reports performance and subgroup-fairness benefits from synthetic augmentation, but its effects depend on the generation process.Source: §I · p. 1
  • Causal counterfactuals derive from interventions in a causal model, unlike broader image counterfactuals that need not encode causal relationships.The distinction helps compare methods according to the assumptions they make.Source: §I · pp. 1–2
  • The study compares deterministic, undirected, and causal conditioning to test when their assumptions improve augmentation performance and fairness.Causal interventions follow directed graph relationships; undirected methods use statistical associations, while deterministic methods hold other variables fixed.Source: §I · pp. 1–2

II. RELATED WORK

  • Synthetic medical images can help rebalance underrepresented groups, pathologies, acquisition conditions, or appearances when collecting or sharing additional real data is constrained.Collection may be expensive and time-consuming, while sharing can face privacy, regulatory, and governance constraints.Source: §II · p. 2
  • The paper categorizes counterfactual-style methods as deterministic, undirected, or causal and compares them in a common framework for downstream performance and fairness.Causal methods encode directed dependencies, whereas undirected methods use statistical associations without causal directions.Source: §II · pp. 2–3

III. BACKGROUND

The background introduces structural causal models as systems of variables and mechanisms, then describes interventions and sample-specific counterfactual generation. The abduction–action–prediction procedure adapts the model to an observed sample before applying an intervention and predicting counterfactual variables.

III. BACKGROUND

  • A Structural Causal Model is a tuple comprising endogenous variables, exogenous noise variables and their distribution, and structural mechanisms.Each endogenous variable is determined by its parents and corresponding noise variable.Source: §III · p. 3
  • The do-operator fixes an intervened variable to a chosen value and removes the influence of its parents.This changes the structural model to represent the intervention.Source: §III · p. 3
  • Sample-specific counterfactuals use abduction–action–prediction: infer exogenous noise, apply an intervention, then estimate the counterfactual variables.The paper uses x for the image and pa for parent variables such as age, sex, and race in chest x-rays.Source: §III · p. 3

IV. METHODS

The section asks when the extra cost of causal counterfactuals is worthwhile and distinguishes image edits from causal counterfactuals. It formalizes three conditioning approaches that make different assumptions about which variables change after intervention.

A. Model Conditioning

  • Image counterfactuals are identity-preserving image changes, while causal counterfactuals apply causal interventions to model variables before generating an altered image.The paper distinguishes the two terms to avoid confusion in medical imaging.Source: A. Model Conditioning · p. 3
  • Deterministic conditioning changes selected variables while forcing all others to retain their original values.This approach ignores relationships between variables.Source: A. Model Conditioning · p. 3
  • Undirected conditioning fixes selected variables and infers the remainder from their joint distribution, assuming known associations but no causal directions.It imposes an undirected graph over associated variables.Source: A. Model Conditioning · p. 3
  • Causal conditioning uses a directed graph so only variables causally downstream of an intervention are affected.It assumes known causal directions and uses abduction-action-prediction.Source: A. Model Conditioning · p. 3

B. Image Generation

The paper generates synthetic images either by altering variables for an input-image counterfactual or by sampling variables and latents for de novo generation. The conditioning approach determines how variables are inferred before decoding.

B. Image Generation

  • The paper compares counterfactual generation with direct latent-space sampling, which generates new images without conditioning on an input image.The earlier use of de novo sampling was limited to Deterministic conditioning.Source: B. Image Generation · p. 3
  • For de novo generation, variables are assigned or sampled, then a latent is sampled and the variables and latent are fed to the decoder.Undirected sampling conditions on connected variables; causal sampling conditions children on parents, not the reverse.Source: B. Image Generation · pp. 3–4
  • Counterfactual generation predicts missing variables from the input image, intervenes on a variable, infers the remaining variables, and decodes the altered values.The model outline contrasts training, where observed y values are used for decoding when available, with counterfactual generation.Source: B. Image Generation · p. 4, Figure 2 · p. 4

C. Architecture

The authors adapt a semi-supervised causal generative model to compare conditioning approaches while varying the availability of labelled endogenous variables. They implement causal, undirected, and deterministic variants through different conditioning modules.

C. Architecture

  • The semi-supervised model lets the authors vary labelled-data availability by treating some endogenous-variable values as missing.The conditioning module implements the three approaches described in the paper.Source: C. Architecture · p. 4
  • The causal model uses a hierarchical VAE, a specified DAG whose arrows are learned as normalising flows, and an encoder predictor for missing values.The encoder maps the image to multiple scales for decoding at multiple latent levels.Source: C. Architecture · p. 4
  • The Undirected variant adds reverse-direction flows to model bidirected relations, while the Deterministic variant removes the causal graph and predicts missing values with the encoder.For deterministic counterfactuals, the inferred latent and desired variable values condition the decoder.Source: C. Architecture · p. 4

V. ANALYSIS ON MORPHO-MNIST

On Morpho-MNIST, the authors compare three conditioning approaches across four specified causal graphs and evaluate their intervention outputs against known distributions. Causal conditioning behaves well across the tested settings, while the alternatives perform well only in particular intervention settings.

V. ANALYSIS ON MORPHO-MNIST

  • The experiments train 12 models: each of three conditioning approaches on datasets generated from four DAGs over digit, thickness, intensity, and foreground and background colours.Morpho-MNIST provides a specified generative process and distribution, allowing comparison with ground-truth log-likelihoods unavailable for real medical data.Source: §V · p. 4
  • Intervention outputs are assessed using predictors and true conditional distributions; digit is unchanged under Deterministic and Causal conditioning but can change under Undirected conditioning.The digit is never a child variable in the specified setup.Source: §V · p. 5
  • Across the tested DAGs, Causal conditioning behaves well, while Deterministic works in specific child-intervention settings and Undirected in parent-intervention settings.Deterministic can keep variables fixed when the assumed graph calls for changes; Undirected can change variables that should remain fixed.Source: §V · p. 5

A. Graph Misspecification

The misspecification experiment reverses the assumed causal directions and finds that Causal then matches the worse of the other methods. The clinical setup uses an assumed age–pleural-effusion–lung-volume graph, with sex outside that graph.

A. Graph Misspecification

  • The misspecification test keeps the data fixed and reverses DAG (b)'s assumed directions, affecting only Causal because the other methods do not model direction.Source: A. Graph Misspecification · p. 5
  • With deliberately reversed arrows, Causal matches the worse of the other two methods in each case, underscoring the importance of a correctly specified graph.The authors note that domain knowledge, data statistics, or causal-discovery techniques may narrow plausible graphs.Source: A. Graph Misspecification · p. 5

A. Setup

  • For CheXpert, the experiments assume age causes pleural effusion and both cause lung-volume decrease, while sex is conditional but disconnected from this graph.The age–effusion association is statistically significant, but the authors state that it does not prove causation; they assume it is causal for the experiment.Source: A. Setup · p. 5

B. Counterfactuals with Different Conditioning Methods

The study evaluates counterfactuals by intervening on variables and checking whether predicted relationships are preserved. Causal generally performs best or near-best, while the methods differ in how they handle interventions and graph misspecification.

B. Counterfactual evaluation

  • Because CheXpert lacks ground-truth causal effects, the evaluation uses predictors of parent variables from their children to estimate post-intervention parent error.Discrete-child predictors are trained on soft labels and calibrated on the validation set; intervened-on parent variables are checked for remaining unchanged.Source: B. Counterfactuals with Different… · pp. 5–6
  • Across Tables I–IV, Causal is always best or not statistically significantly worse than the best model on the reported intervention metrics.The tables evaluate interventions on four DAGs; log-likelihood is used for foreground colour, background colour, thickness, and intensity, and 1 - softmaxcorrect digit for digit.Source: B. Counterfactuals with Different… · p. 6, Table I · p. 6, Table II · p. 6
  • Causal and Deterministic better preserve parent variables than Undirected when changing a child, but both incorrectly flip pleural effusion about 30% of the time when lung size changes.Overall, Causal is best or close to best across the reported cases, while the other approaches fail on different intervention types.Source: Table VI · p. 6, B. Counterfactuals with Different… · p. 6

C. Dataset Augmentation

The augmentation experiments rebalance biased training data with synthetic images and evaluate classifiers on the unbiased test split. Causal and Undirected tend to help most when labels are scarce, while counterfactual realism and sampling diversity create a label-regime trade-off.

  • The experiment induces a spurious pleural-effusion–sex correlation, trains generators on the biased data, and rebalances synthetic samples before evaluating on the unbiased test split.The evaluation compares counterfactual generation with latent-space sampling for each conditioning approach.Source: C. Dataset Augmentation · pp. 6–7
  • Synthetic augmentation improves classifier performance at every tested label percentage and bias level; with fewer labels, Undirected and Causal tend to outperform Deterministic.With full labels, the three conditioning approaches perform similarly.Source: C. Dataset Augmentation · p. 7
  • Both Causal and Undirected improve performance over simpler conditioning for training-dataset augmentation.The reported comparison is summarized across synthetic augmentation approaches; Table VII reports pleural-effusion classifier AUC.Source: Table VII · p. 7, C. Dataset Augmentation · p. 8
  • At higher label percentages, counterfactuals offer a realism benefit over sampling, but at low label percentages sampling yields greater diversity and better downstream results.The authors attribute the low-label diversity gap to counterfactuals modifying existing labelled samples; Fig. 8 presents diversity metrics, with higher values indicating greater diversity.Source: C. Dataset Augmentation · pp. 7–8, Figure 8 · p. 8

D. Fairness

The section compares fairness across age and sex when synthetic data are used to counteract the PE–sex correlation. On MIMIC, synthetic augmentation improves PE prediction and slightly reduces the sex fairness gap, while age fairness differs by conditioning approach.

D. Fairness

  • Undirected augmentation is worst at improving age fairness, while non-augmented data often yields the least fair model across age groups.The analysis measures the difference in AUC between the best- and worst-performing age groups.Source: D. Fairness · p. 8, Figure 9 · p. 8
  • On MIMIC, synthetic augmentation improves PE-classifier performance, especially with Undirected and Causal methods, and slightly reduces the sex fairness gap across methods.Counterfactuals improve performance slightly more than unconditional sampling.Source: Table IX · p. 8, D. Fairness · p. 8
  • Age fairness gaps differ significantly across conditioning approaches; Deterministic and Causal reduce them, whereas Undirected offers less benefit.The authors suggest Undirected may over-amplify the anti-causal age–PE association by sampling age conditional on PE.Source: D. Fairness · pp. 8–9
  • Undirected counterfactuals can alter image regions beyond the lungs, whereas Causal interventions change almost exclusively the lungs when adding or removing PE.The authors link the broader changes to other variables, such as age, being changed.Source: D. Fairness · p. 8, Figure 11 · p. 9
  • In the Undirected method, age depends on disease status; Deterministic and Causal methods sample age unconditionally, producing greater overlap across PE groups.The age distributions are shown for samples with and without pleural effusion.Source: D. Fairness · p. 8, Figure 10 · p. 9

VII. CONCLUSION

The conclusion examines six medical-image generative approaches and when their conditioning assumptions and generation methods are useful. Clinical chest x-ray experiments find benefits and trade-offs, while Morpho-MNIST results show that causal conditioning depends on having the correct graph.

VII. CONCLUSION

  • The study investigates six generative approaches, three conditioning strategies, and both counterfactual generation from existing images and de novo sampling.Source: §VII · p. 9
  • On Morpho-MNIST, Causal conditioning performs consistently well across explored DAG types when the correct causal graph is identified; Deterministic is rigid and Undirected changes variables excessively.The data-generating process is controlled in this analysis.Source: §VII · p. 9
  • When the underlying DAG is uncertain or misspecified, Causal conditioning loses its advantage, and Deterministic or Undirected conditioning may be preferable.Source: §VII · p. 9
  • On clinical chest x-rays, Causal and Undirected augmentation improves disease prediction over no augmentation and Deterministic augmentation, with sampling favored at low-label levels and counterfactuals at high-label levels.Sampling offers greater diversity, while counterfactuals are typically more perceptually accurate because they are tied to real images; Undirected may exacerbate bias unless carefully controlled.Source: §VII · p. 9

VIII. FUTURE WORK

The authors identify model choice, dimensionality, and application scope as boundaries for validating their findings. They propose validating other causal models, extending analysis to 3D data, and studying uses beyond training-set augmentation.

VIII. FUTURE WORK

  • The findings use one variant of a causal hierarchical VAE; validation with other causal models, such as diffusion models, remains desirable.Source: §VIII · p. 9
  • The experiments use 2D images because sufficiently capable 3D causal generative models have not yet been developed; volumetric data could support richer analysis.The authors mention brain MRI with more meta-variables and structures as one possible setting.Source: §VIII · p. 9
  • The study only explores synthetic data for training-set augmentation, leaving applications such as stress-testing pretrained models for future work.The authors connect such testing to understanding model improvements and avoiding deployment in vulnerable settings when improvement is not possible.Source: §VIII · p. 9
Loading 2609.14124v1…