Causality matters in medical imaging

Daniel C. Castro; Ian Walker; Ben Glocker

Source-linked AI summary

Causality matters in medical imaging

Daniel C. Castro, Ian Walker, Ben Glocker

arXiv:1912.08142v1Dec 17, 2019eess.IVcs.AIcs.CVcs.LG

TL;DR

Medical-imaging machine learning faces scarce annotations and data mismatch that can undermine clinical generalization. The paper uses causal diagrams and causal analysis to scrutinize data-generation assumptions and learning strategies, finding that causal structure affects whether semi-supervision is appropriate and may make it theoretically futile for semantic segmentation.

Problem
Medical-imaging machine learning must address costly scarcity of labelled data and mismatches between development and deployment distributions that can impair generalizability.
Method
The paper applies causal reasoning and causal diagrams to analyse image–annotation relationships, data-generation assumptions, dataset shift, and selection bias.
Results
Causal structure determines the suitability of learning strategies: semi-supervision may be theoretically futile for causal semantic segmentation but can have a chance of success for anticausal tasks, while augmentation suits both.
Takeaways & Limitations
Making causal assumptions explicit supports more transparent scrutiny of data collection, annotation, and machine-learning choices in medical imaging.
Takeaways & Limitations
Selection bias and manifestation shift generally require assumptions about the selection mechanism or differences between domains for recovery or correction.

Abstract

from arXiv · show

This article discusses how the language of causality can shed new light on the major challenges in machine learning for medical imaging: 1) data scarcity, which is the limited availability of high-quality annotations, and 2) data mismatch, whereby a trained algorithm may fail to generalize in clinical practice. Looking at these challenges through the lens of causality allows decisions about data collection, annotation procedures, and learning strategies to be made (and scrutinized) more transparently. We discuss how causal relationships between images and annotations can not only have profound effects on the performance of predictive models, but may even dictate which learning strategies should be considered in the first place. For example, we conclude that semi-supervision may be unsuitable for image segmentation---one of the possibly surprising insights from our causal analysis, which is illustrated with representative real-world examples of computer-aided diagnosis (skin lesion classification in dermatology) and radiotherapy (automated contouring of tumours). We highlight that being aware of and accounting for the causal relationships in medical imaging data is important for the safe development of machine learning and essential for regulation and responsible reporting. To facilitate this we provide step-by-step recommendations for future studies.

Introduction

Medical imaging machine learning has achieved major predictive gains, but data scarcity and data mismatch still hinder translation into broader clinical practice. A causal analysis can expose these risks and guide data, annotation, and learning decisions.

Deep-learning methods have produced substantial accuracy improvements in computer-aided diagnosis and automated radiotherapy planning and monitoring.
Data mismatch occurs when models trained in laboratory settings fail to generalize to real-world clinical data.Retrospective study criteria, patient populations, and prospective multicentre settings can differ.
Data scarcity reflects insufficient high-quality image–annotation pairs for training predictive models.The shortage may arise from small samples, costly expert annotations, or expensive confirmatory laboratory tests.
Causal reasoning makes assumptions about data generation and collection explicit, allowing potential problems to be identified before deployment.This can inform data collection, annotation procedures, and machine-learning strategy selection.
The article uses causal analysis to examine whether common learning strategies are appropriate under specific medical-imaging data conditions.Its examples include diagnosis and automated tumour contouring.

Causality Matters

Medical-imaging prediction must be analysed in relation to data distributions, deployment conditions, and the causal direction between images and targets. These relationships determine how mismatch is understood and which learning strategies are applicable.

Medical-imaging predictive modelling estimates P(Y |X) from images X to annotations or other targets Y across tasks such as segmentation and disease classification.
Development data contain annotated image–target pairs used to train and test models, whereas deployment data may differ and lack ground-truth annotations.Consequently, development-test performance may not represent post-deployment clinical performance.
Data scarcity stems partly from the high cost of expert annotations and confirmatory laboratory tests, motivating methods such as semi-supervised learning and augmentation.
Causality Matters: Causal reasoning uses explicit assumptions about data-generating processes to identify dataset mismatch and guide principled mitigation strategies.The approach relies on relevant background and metadata about how data were acquired and collected.
Causality in medical imaging: Causal analysis distinguishes causal prediction, X →Y, from anticausal prediction, Y →X, according to whether images cause targets or targets cause image appearance.The direction affects the applicability of semi-supervised learning and the preference for generative or discriminative models.
Causality in medical imaging: Histopathology-based melanoma diagnosis is anticausal because disease status is treated as a cause of lesion appearance.

Skin lesion classification example.

The skin-lesion example shows that causal direction, selection mechanisms, and deployment conditions shape how imaging models should be interpreted and evaluated.

Routine visual labels differ causally from biopsy-confirmed diagnoses, because visual labels can be affected by digitally manipulated images whereas biopsy diagnosis follows the lesion’s appearance.
Biopsy-labeled datasets are subject to sample selection bias because patients are referred for biopsy only when dermoscopy raises suspicion.
Models trained on manual tumour contours learn to reproduce the annotation process rather than predict a true underlying anatomical layout.
Deployment across scanners and age distributions creates acquisition and population shifts that alter image quality, anatomy, and tumour prevalence.
Identifying causal direction can be difficult when prediction targets, such as radiology reports, incorporate blood tests or other diagnostic information beyond image findings.

Tackling data scarcity via semi-supervision.

Semi-supervision uses unlabelled inputs, but its causal value depends on whether input distributions inform labels; augmentation instead expands joint image–label information and applies broadly.

Semi-supervision: Semi-supervised learning requires the input distribution to carry relevant information about the prediction task.
Semi-supervision: For causal prediction, P(X) is uninformative about P(Y |X), making semi-supervision theoretically futile; semantic segmentation is therefore unlikely to fundamentally benefit from unlabelled data.
Semi-supervision: Unlabelled data may still regularize causal-task models by lowering variance or reducing the labelled-data requirement, but these gains need empirical validation.
Semi-supervision: A comprehensive empirical study found properly tuned supervised and transfer-learning models often competitive with or better than semi-supervised counterparts, while target shift can hurt classification.
Data augmentation: Data augmentation applies controlled perturbations, using invariant transformations for image-level tasks and equivariant transformations for segmentation or localization.
Data augmentation: By creating additional (x, y) pairs and enriching P(X, Y), augmentation is suitable for both causal and anticausal tasks.
Data augmentation: Emerging methods use unlabelled data to learn realistic transformations for augmentation, potentially combining semi-supervised data use with standard augmentation’s framework.

Data mismatch due to dataset shift.

Dataset shift describes differences between training and test distributions arising from changes in conditional relationships or marginal distributions. Causal diagrams, including unobserved anatomy, distinguish medically meaningful shift types and clarify when transport or correction is possible.

Dataset shift is defined by disagreement between training and test distributions caused by exogenous factors such as dissimilar cohorts or acquisition processes.
Causal analysis augments images and targets with unobserved anatomy, treating images as imperfect, potentially domain-dependent measurements of that anatomy.
The framework distinguishes population, annotation, prevalence, manifestation, and acquisition shifts by identifying changing or invariant factors in the joint distribution.
For causal prediction, population shift is directly transportable when only intrinsic population characteristics differ, although underfitting can introduce spurious dependencies.
Learning under missing training variation offers no guarantee of extrapolation to test modes absent from the training environment.
Annotation and manifestation shifts generally lack clear corrections without assumptions about label changes or how targets physically manifest across domains.
Acquisition shift commonly motivates spatial alignment, resampling, intensity normalization, and domain-adaptation methods for harmonizing imaging data.

Data mismatch due to sample selection bias.

Sample selection bias occurs when training samples are selectively observed from the same population as the test cohort. Its consequences depend on whether selection is random, image-dependent, target-dependent, or jointly dependent on image and target.

Sample selection defines training data as records observed under a selection process, whereas test data represent the unconditioned population.
Unlike standard dataset shift, sample selection changes the data-gathering process rather than the data-generating process.
Random selection introduces no bias when training data faithfully represent the target population, but preferential collection is expected to be common without safeguards.
Image-dependent selection can reflect anatomical features or image quality, with implications resembling population shift or acquisition shift.
Target-dependent selection preserves Pte(X|Y)=Ptr(X|Y) and can be treated as prevalence shift, but may arise from class rebalancing or lesion-containing patch selection.
When selection is a common effect of image and target, conditioning on observed samples can unblock spurious paths and create selection bias requiring additional control variables.

Discussion

The discussion presents causal reasoning as a transparent framework for understanding medical-imaging data scarcity and mismatch. It offers practical diagramming recommendations while emphasizing that causal reasoning itself does not solve these problems and that broader causal research remains preliminary.

Causal analysis suggests that semantic segmentation may not fundamentally benefit from unannotated images through semi-supervision.
Measurement bias and missingness are identified as related topics for future causal analysis, including semi-supervised settings where labels are observed for only a subset of data.
Causal inference and causal discovery remain largely unexplored in medical imaging, with possible applications including imaging biomarkers and personalized counterfactual predictions.
Causal reasoning supports responsible reporting and aligns with regulatory and machine-learning initiatives concerning safe development, monitoring, fairness, accountability, and transparency.
The article frames causal diagrams as a way to express assumptions clearly so other researchers can confirm or dispute them.
Table 4 recommendations and the generic scaffold diagram are intended to help analysts construct fuller causal accounts of medical-imaging datasets and identify biases.

Fundamentals of causal reasoning.

Causal reasoning distinguishes prediction, intervention, and counterfactual analysis, while emphasizing that causal direction matters even for predictive medical-imaging tasks. Observational data alone may not identify that direction, making domain knowledge or experiments important.

Prediction infers unobserved variables from observed data, interventions study forced changes, and counterfactuals ask what would have happened under an alternative treatment.
A direct cause A →B changes the likelihood of B when A is forced to different values, with B generated by a mechanism depending on A and other factors.
In the radiology example, the scan causes the referral decision because changing the scan changes the decision distribution, whereas changing the decision would not alter the scan.
Observational image and referral data may not distinguish A →B from B →A; interventions on either variable could help identify the correct direction.

Causal graphical models.

Causal graphical models represent variables and directed causal influences while supporting intervention reasoning beyond ordinary Bayesian-network factorization. Their joint distributions factor into conditional mechanisms associated with each variable’s parents.

A causal diagram is a directed acyclic graph whose nodes represent variables and whose edges represent postulated direct causal influences.
The joint probability distribution factorizes into conditional mechanisms, with pa(Vi) denoting the variables whose arrows point toward Vi.
Unlike Bayesian networks, causal models use independent mechanisms and support interventions that disconnect a changed node from its parents while leaving other mechanisms unaffected.

Building a causal diagram.

Constructing a causal diagram requires defining non-overlapping variables, specifying causal links and selection mechanisms, and iteratively checking the graph against domain knowledge. The graph’s abstraction level should balance accuracy with clarity.

Begin by eliciting well-defined variables that correspond unambiguously to real or postulated entities without overlapping meanings.
Determine causal links carefully because every arrow asserts causation, while every absent arrow asserts that no direct causal effect exists.
Selection diagrams use domain indicators to represent mechanisms by which environments such as hospitals or training and testing populations differ.
Selection variables capture preferential subsampling, with incoming arrows representing the criteria that influenced dataset collection.
Diagram construction is iterative: verify graph assumptions against domain knowledge and choose enough detail for accuracy without sacrificing clarity.

Interpreting causal diagrams.

Causal diagrams reveal how paths create or block associations among variables. They distinguish mediation, confounding, and collider structures, which clarify indirect effects and sources of bias.

A mediator lies on A →B →C, making B an indirect pathway through which A affects C and screening off their effect conditional on B.
A confounder is a common cause A ←B →C that creates an association between A and C, which controlling for B can remove.
A collider is a common effect A →B ←C; A and C are independent beforehand, but conditioning on B introduces an association between them.
Open paths can carry statistical association between endpoints, whereas paths containing a collider are blocked unless conditioning opens them.
Causal bias is spurious correlation from unblocked paths, with unadjusted confounding and conditioning on common effects as prototypical sources.