Source-linked AI summary
Diagnosing failures of fairness transfer across distribution shift in real-world medical settings
Jessica Schrouff, Natalie Harris, Oluwasanmi Koyejo, Ibrahim Alabdulmohsin, Eva Schnider, Krista Opsahl-Ong, Alex Brown, Subhrajit Roy, Diana Mincu, Christina Chen, Awa Dieng, Yuan Liu, Vivek Natarajan, Alan Karthikesalingam, Katherine Heller, Silvia Chiappa, Alexander D'Amour
TL;DR
Fairness properties satisfied during model development may not transfer across distribution shifts, and mitigation success depends on the structure of the shift. The paper uses conditional-independence tests in a causal framework to diagnose shifts in two healthcare applications, finding complex shifts that can complicate mitigation.
Problem
Fairness properties that hold on development data may fail after distribution shift, while few methods diagnose shift structure to guide mitigation.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The authors discuss potential remedies across the machine-learning pipeline, including collecting data from anticipated deployment environments to guide mitigation.
Source: §6 · p. 9Diagnosis power depends on sample number and quality, and the method cannot assess changes in unobserved variables shared across environments.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Diagnosing and mitigating changes in model fairness under distribution shift is an important component of the safe deployment of machine learning in healthcare settings. Importantly, the success of any mitigation strategy strongly depends on the structure of the shift. Despite this, there has been little discussion of how to empirically assess the structure of a distribution shift that one is encountering in practice. In this work, we adopt a causal framing to motivate conditional independence tests as a key tool for characterizing distribution shifts. Using our approach in two medical applications, we show that this knowledge can help diagnose failures of fairness transfer, including cases where real-world shifts are more complex than is often assumed in the literature. Based on these results, we discuss potential remedies at each step of the machine learning pipeline.
1 Introduction
Fairness achieved on development data may not hold after distribution shift, a concern in high-risk healthcare settings. Because mitigation success depends on shift structure, the paper proposes empirical tests to characterize shifts and uses two medical applications to diagnose failures of fairness transfer.
1 Introduction
- Fairness criteria satisfied during model development may fail after distribution shift, creating a concern for deployment across hospitals.Prior work has studied performance under shift and fairness transfer, but few methods diagnose shift structure to guide mitigation.Source: §1 · pp. 1–2
- The success of robust-fairness mitigation depends on the shift encountered, yet few methods diagnose shift structure and guide mitigation choice.The authors use statistical tests with a simplified causal graph and a small target dataset in dermatology and EHR applications.Source: §1 · p. 2
- The healthcare case studies show that clinically plausible shifts can exceed what current mitigation techniques handle, motivating remedies across the machine-learning pipeline.The paper discusses potential remedies at each step of the pipeline.Source: §1 · p. 2
2 Background
The background frames fairness transfer through conditional independences between model variables and an environment indicator. A joint causal graph represents which variable relationships may change across environments and how those changes affect applicable fairness metrics and mitigations.
2 Background
- The prediction model uses features X, sensitive attributes A, and outcome Y, and the paper considers subgroup accuracy gaps, demographic parity, and equalized odds.Depending on the setting, the predictor takes X and A or X alone as input.Source: §2 · p. 2
- A joint causal graph adds an environment variable S to represent source and target distributions; arrows from S indicate conditional relationships that may change across environments.Without an arrow to U, the conditional distribution of U given its causal parents remains the same across environments.Source: §2 · p. 2
- Distribution shifts can affect sensitive attributes A, features X, or labels Y; shifts with multiple environment arrows are compound.Examples include age-distribution changes, different image-acquisition devices, and changes in disease prevalence or labeling strategies.Source: §2 · p. 2
- Fairness transfer depends on conditional independences with respect to S: classification error, demographic parity, and equalized odds each have stated invariance conditions.The conditions respectively involve Y given V; V given A; and V given A and Y.Source: §2 · p. 3
- In JCI, the stated fairness-transfer conditions imply that certain arrows from S to A, X, and Y must be absent, which also constrains mitigation choices.The paper links shift structure to which mitigation strategies are applicable.Source: §2 · p. 3
3 Empirical tests for assessing shift structure
The proposed tests assess whether the environment has a direct effect on each variable in a simplified causal graph. They balance causal parents across source and target before testing whether reweighted variable distributions match; validation experiments test false positives and detection of engineered shifts.
3 Empirical tests for assessing shift structure
- The tests assess whether S directly affects each variable U by testing whether P(U | pa(U), S = 0) equals P(U | pa(U), S = 1).Rejecting this null implies an arrow from S to U in the joint causal graph.Source: §3 · p. 3
- To isolate S's direct effect, the procedure balances U's causal parents across environments, then tests equality of reweighted means, using lower-dimensional functions for high-dimensional U.The tests use source and target data and a causal graph; the paper describes the procedure in Algorithm 1.Source: §3 · p. 3
- In validation, random splits produced a false-positive rate similar to the 5% threshold, with greater variance at smaller sample sizes.The tests also identified affected dimensions in engineered shifts on A and Y, and in a separate shift on Y alone.Source: §3 · p. 4
4 Distribution shifts in real-world healthcare applications
The two healthcare case studies show fairness failures under shifts with multiple affected variables. Tests of the shift structure identify changes in demographics and other parts of the data, while the EHR case also links compound shift to fairness-transfer failure.
4.1 Predicting skin conditions in dermatology
- In dermatology, the maximum age-group performance gap increased from 1.05% on source data to 12.86% on target data.The source model performed similarly across subgroups; the assessment covered 10 model replicates.Source: §4.1 · p. 5
- In dermatology, the target population was older than the source, while sex distributions were well matched, supporting a direct effect of S on demographics A.Median age was 40 in source and 44 in target.Source: §4.1 · p. 5
- Dermatology tests found label-distribution differences for 3 of 27 conditions after conditioning on A, and image-feature tests also found a significant source-target difference.For older females with SK/ISK, source cases had a median of 3 images and target cases a median of 5.Source: §4.1 · pp. 5–6
- Dermatology tests found environment effects on demographics, labels, and image features, so none of the stated conditional-independence conditions for fairness transfer held overall.Some conditions might be invariant after controlling for demographics, suggesting possible mitigations.Source: §4.1 · p. 6
4.2 Clinical risk prediction from Electronic Health Records
- After equalized-odds and demographic-parity fairness were assessed across ICU types, fairness did not transfer; age unfairness appeared in both, while the sex performance gap rose from 1% to 2.7%.For sex, demographic parity was 0.002±0.002 in source and 0.016±0.003 in target.Source: §4.2 · p. 6
- In EHR data, source and target differed in age and sex, and five comorbidities differed after adjustment for age.The target had a higher prevalence of peripheral-vascular comorbidities; weighted tests were Bonferroni corrected.Source: §4.2 · p. 7
- The EHR case tested direct effects on treatments, labs and vitals, and the prolonged-stay label; tests found a treatment difference but no significant difference for labs/vitals or the label.Figure 3(d) shows prolonged-stay prevalence for older males with solid tumor comorbidity receiving vasopressors or inotropes.Source: §4.2 · p. 7
- The EHR shift is compound: ICU type is associated with changes across multiple variables, and the authors link multiple paths from S and A to covariates and labels with fairness-transfer failure.They note that ICU type correlates with the unobserved reason for admission, a driver of multiple variables.Source: §4.2 · p. 7
5 Mitigation and related work
The section reviews fairness-mitigation and shift-detection approaches, then shows that source-domain fairness gains can fail to transfer under a compound EHR shift.
Mitigation and related work
- Source-domain fairness mitigation may aid transfer by cutting causal-graph edges, but provable transfer is limited to specific settings, mostly demographic shifts.The section notes that fairness metrics and mitigations are typically evaluated on a single task or distribution.Source: §5 · p. 8
- In the EHR task, mitigating demographic parity or equalized odds on the source produced no improvement or worsened fairness on the target under compound shift.Target demographic parity increased to 0.078 ± 0.025; equalized odds increased from 0.05 ± 0.008 to 0.16 ± 0.04, and the age-group accuracy gap increased to 11.4%.Source: §5 · p. 8
- Robustly fair methods either provide a priori guarantees or bounds, or adapt to the new distribution; experimental robustness-mitigation studies do not establish provably robustly fair models..Source: §5 · p. 8
- Singh et al.'s feature-selection strategy with guarantees cannot be used under label shift and can yield a trivial predictor under compound shifts.Feature selection is better suited to semantic features than pixels; image masking requires a mask for every training image.Source: §5 · p. 8
- Adaptation methods can require labeled target data or multiple tasks, produce one tuned model per environment, and have unclear suitability for sparse EHR time-series prediction.The cited approaches include transfer learning, meta-learning, and fine-tuning.Source: §5 · p. 9
- Shift detectors often assume other variables are unaffected, while causal graph inspection is limited to low-dimensional datasets and fairness warnings do not guide mitigation choice.The section describes these as limitations of existing detection approaches.Source: §5 · p. 9
6 Discussion
The paper uses statistical tests in a joint causal framework to characterize shifts and guide mitigation, then discusses pipeline-level responses and the limits of its analysis. Its recommendations depend on observed data and assumptions about causal relationships, while some shift sources and fairness considerations remain outside its scope.
6 Discussion
- Tests within a joint causal framework characterize direct environmental effects and can guide mitigation when applicable.The authors recommend looking beyond purely algorithmic solutions and considering the full machine-learning pipeline.Source: §6 · p. 9
- Problem selection, deployment-environment data collection, outcome definition, algorithm development, and post-deployment safeguards are proposed as pipeline responses.The authors also note that they did not assess model architecture or different training strategies.Source: §6 · pp. 9–10
- The analysis is bounded by sample quality, proxy variables, and unobserved factors; its diagnoses indicate potentially problematic data aspects rather than proving invariance.Causal graphs may not represent the true data-generating process, and demographic features only approximate sensitive attributes.Source: §6 · p. 10
- The discussion focuses mostly on demographic parity and equalized odds, which may not capture health equity or social determinants of health.The authors note that different metrics and mitigation strategies may respond differently to distribution shift, and none provides guarantees under shift.Source: §6 · p. 10
A.1 Fairness metrics
- Demographic parity and equalized odds assess statistical independence, respectively, unconditionally and conditional on the outcome; subgroup accuracy gaps are also measured.Experiments use binary tasks, and lower values are better for all fairness metrics.Source: §A.1 · p. 17
A.2 Causal framework
- A causal Bayesian network represents causal influences with directed links and assumes each node’s conditional probability distribution is invariant across distributions.The framework uses d-separation to express conditional independence.Source: §A.2 · p. 17
A.3.1 General Strategy
- The general test controls for variables that block indirect paths from shift to a focal variable, then compares conditional distributions across environments.Importance weighting adjusts background-variable distributions so the weighted outcome means can be compared; bootstrap testing is used to reduce t-statistic variance.Source: §A.3.1 · pp. 17–18
A.3.2 Testing with High-Dimensional U
- For high-dimensional variables, testing summaries can lose power to detect differences compressed out of those summaries.A summary can be based on a model’s output for a variable of interest.Source: §A.3.2 · p. 18
A.3.3 Sanity checks with engineered shifts
Sanity checks show that the tests detect engineered changes while keeping false positives near the expected level, with more weighting samples reducing Type I error variance. The related-work review emphasizes that fairness-transfer methods are tailored to restricted shift structures and can fail under compound shifts.
A.3.3 Sanity checks with engineered shifts
- False positives ranged from 2–8.5% for comparing P(Y | A, S = 0) with P(Y | A, S = 1), and from 2–7% for comparing P(X | Y, A, S = 0) with P(X | Y, A, S = 1).In source-versus-source tests, the classifier for the first comparison performed at chance level (50%).Source: §A.3.3 · p. 19, Figure 5 · p. 19
- Increasing the sample count from 100 to 5,000 to fit P(S = s | V) reduced the variance of Type I error across conditions.The tests estimate weights from a classifier of environment membership.Source: §A.3.3 · p. 19
- In the age-and-Acne shift, tests detected changes in age and Y but not in P(X | Y, A), recovering the engineered causal graph.The absence of detected feature-distribution changes held with both logistic regression and gradient boosted trees.Source: §A.3.3 · pp. 19–20
- After labels were shuffled, tests detected changes in P(Y | A) for 19 of 27 conditions and none in P(X | Y, A).The label comparisons used Bonferroni correction; images were unchanged.Source: §A.3.3 · p. 20
- Overall, the tests detected changes on affected dimensions without an excessive number of false positives.This summarizes the engineered-shift validation results.Source: §A.3.3 · pp. 19–20
B Causal framing of related works
- The review concludes that most transferable-fairness methods target restricted shift structures, and their useful properties can break down under compound shifts.The review considers causal and anti-causal prediction tasks under an assumption of shared sensitive attributes across environments.Source: B Causal framing of related works · p. 20
B.1 Anti-causal prediction tasks
For anti-causal prediction, the review compares fairness-transfer strategies across demographic, covariate, label, and compound shifts. The first three have specific mitigation options, while compound shifts can make existing methods ineffective or trivial.
B.1 Anti-causal prediction tasks
- The analysis assumes Y causes X, while A directly causes both X and Y, as an approximation of healthcare prediction settings.Examples include sex-related differences in skin images and condition prevalence varying across demographic groups.Source: §B.1 · p. 20
- Under exclusive demographic shift, equalized odds or counterfactual invariance with respect to A can yield a robustly fair model based on X.The causal graph places the shift’s effects on X and Y through A.Source: §B.1 · p. 20, Figure 6 · p. 21
- Under exclusive covariate shift, separate independence regularizers can address A → X and S → X when labeled target data are available.With fully unlabeled target data, one feature-selection approach may return only A, producing a predictor based solely on demographics.Source: §B.1 · p. 20, Figure 6 · p. 21
- For exclusive label shift, a method detects and mitigates the shift by reweighting using unlabeled target data, alongside a fairness mitigation technique.The discussed method is attributed to Lipton et al..Source: §B.1 · p. 21, Figure 6 · p. 21
- Under compound shifts, current techniques may be ineffective or yield a trivial predictor; intersection-based mitigation can become intractable as attribute or shift values multiply.Feature selection may return an empty set, while adaptation techniques may provide partial mitigation.Source: §B.1 · p. 21, Figure 6 · p. 21
B.2 Causal prediction tasks
For causal prediction, the review finds distinct fairness-transfer implications across demographic, covariate, label, and compound shifts. It also notes that feature-selection methods can exclude useful signal under some shift structures.
B.2 Causal prediction tasks
- Under exclusive demographic shift, training on {X, A} while imposing invariance of Y on A can yield a robustly fair model.The relationship between X and Y is not affected by S under this condition.Source: §B.2 · p. 21
- Under exclusive covariate shift, (Y, A) ⊥⊥ S | X, so fairness and shift-robustness guarantees can be obtained independently.Distribution matching or invariant representation learning are described as options that require only unlabeled target data.Source: §B.2 · pp. 21–22
- Causal-task label shift is understudied, and the review identifies no method with formal fairness guarantees for exclusive label shift.Some mitigation techniques assume that there is no direct path S → Y.Source: §B.2 · p. 22
- As in the anti-causal case, compound shifts can lead to insufficient or trivial predictors.The review states this for the simplest causal prediction task considered.Source: §B.2 · p. 22
B.3 Feature selection mitigation strategies
- Feature selection cannot mitigate some exclusive covariate and label shifts without excluding signal, producing an empty input set in simplified cases.The review summarizes coverage of the method in.Source: §B.3 · p. 22, Table 1 · p. 22
C Data and method availability
Across dermatology datasets, subgroup performance is relatively similar in source data but can diverge in target data, with patterns varying by age, condition, and skin type. The section also reports the study’s data, model, and computational setup.
C Data and method availability
- Dermatology data are not public; de-identified EHR data are available under a PhysioNet user agreement, and code for extracting comorbidity scores is available.The dermatology images and metadata were de-identified under HIPAA Safe Harbor, and the protocol was deemed exempt from further IRB review.Source: C Data and method availability · p. 22, §E.1 · p. 23
D Computational resources
- Testing used logistic regression or boosted trees for environment prediction; summarizing high-dimensional U required 28 TPU resources for 24–30 hours in dermatology or 2 CPUs for about an hour in EHR.The study also trained 10 model replicates from different random seeds for the dermatology and EHR applications.Source: D Computational resources · p. 22
E.2.1 Model architecture
- The dermatology model used averaged image embeddings plus age and sex to predict 26 conditions and ‘other’; a 419-condition head was used during training only.The model used a wide ResNet-101×3 feature extractor and cases with 1–6 images.Source: §E.2.1 · p. 23
E.2.2 Fairness properties are affected by the environment
- Subgroup performance was relatively similar in source data, while differences appeared in target data, especially by age, across both dermatology target datasets.The reported comparisons use Top-1 accuracy; the figures show source and target performance across sex and age.Source: §E.2.2 · p. 23, Figure 8 · p. 24, §E.3.1 · p. 25
E.2.3 Per condition model performance
- Condition-specific patterns differed: SK/ISK performance increased with age only in the target, while ‘other’ conditions were more similar to source and Melanocytic Nevus declined with age in both.The authors note relatively few samples per subgroup and conclude that fairness-pattern changes do not result only from condition prevalence.Source: §E.2.3 · p. 24, Figure 9 · p. 24
E.2.4 Fitzpatrick’s skin type
- Skin-type proportions differed between datasets (t-test: p < 0.001), but grouped Top-3 subgroup performance differed by at most 6% in source and 5.5% in target.Group 5/6 remained small (source n = 53, target n = 131), limiting confidence and single-condition analysis.Source: §E.2.4 · p. 25, Figure 10 · p. 25
E.3 Second target dataset: skin cancer clinics in Australia
- The second target dataset comprised 21,661 partially labelled cases from skin cancer clinics in Australia and New Zealand.Source: §E.3 · p. 25
E.3.2 Per condition model performance
- For the second target dataset, Melanocytic Nevus age gaps reached 24.47% in source and 39.00% in target, while SK/ISK gaps were 4.00% and 31.78%, respectively.The SK/ISK pattern increased with age in the target but was similar across ages in source; the authors caution that some source comparisons had few test samples.Source: §E.3.2 · pp. 25–26
E.3.3 A compound shift
In the dermatology target, the environment affects demographics, labels, and image features in the simplified causal graph. The analyses also cannot assess whether label quality differs between datasets.
E.3.3 A compound shift
- The source population was younger and included more female patients: median ages were 40 versus 58 years, and female proportions were 62% versus 47%.Only age and sex were available in both datasets.Source: §E.3.3 · p. 26
- Labels differed across environments for 25 conditions after Bonferroni correction, including more eczema and psoriasis in the source and more basal cell carcinoma and melanoma in the target.The analysis measures label distributions, not label quality; source labels came from multiple clinicians, while some target labels were biopsy-verified.Source: §E.3.3 · p. 26
- Weighted tests found image-feature differences on 21 corrected dimensions, while older females with SK/ISK had a median of 3 images per source case versus 2 in the target.The image-count comparison included 61 source cases and 341 target cases.Source: §E.3.3 · pp. 26–27
- The analyses conclude that the environment affects every variable in the simplified causal graph.The tests found differences in demographic attributes, labels conditional on demographics, and image features.Source: §E.3.3 · pp. 26–27
E.4 Joint training
Joint training on the original source and a second dermatology dataset reduced the overall source–target performance gap, but fairness gaps remained. The joint source also differed from the target along shift dimensions not detected in the original-source comparison.
E.4 Joint training
- The training set combined the original source with an Australian skin-cancer-clinic dataset, sampling batches from each with equal probability and thereby oversampling the smaller original source.The added dataset approximately doubled training-set size and included more cancer-related cases.Source: §E.4 · pp. 27–28
- The joint source differed from the target in A, Y conditional on A for 5 conditions, and X conditional on A and Y for 2 conditions after correction.A cancer-condition distribution difference appeared in this comparison but had not been noticeable between the original source and target.Source: §E.4 · p. 27
- Joint training reduced the overall source–target performance gap, but the age fairness gap remained 4.33% in the joint source versus 11.4% in the target.The authors conclude that joint training did not introduce environmental invariances here and therefore did not guarantee improved fairness transfer.Source: §E.4 · p. 28
F.1 Dataset
The EHR analysis compares ICU cohorts using patient features and fairness metrics, then tests whether comorbidities, treatments, length-of-stay labels, and labs or vitals vary by ICU unit. It finds evidence of unit-related differences in comorbidities and treatments, but not in the tested length-of-stay labels or labs and vitals.
F.1 Dataset
- The cohort contained 28,083 adult patients with ICU stays longer than one day, split by first care unit into 17,641 source patients and 10,442 target patients.The target was prolonged ICU length of stay, defined as remaining longer than three days; features included 31,303 binary and 28,048 numeric variables.Source: §F.1 · pp. 28–29
F.2 Model and performance
- Source accuracy was 78.6 ± 0.7% versus 79.7 ± 0.9% on the target; demographic parity increased from 0.002 ± 0.002 to 0.016 ± 0.003 for sex and from 0.05 ± 0.006 to 0.066 ± 0.010 for age.Equalized odds showed no significant difference between environments for age or sex.Source: §F.2 · p. 29, Table 6 · p. 30
F.3.1 The ICU unit S affects the comorbidities M
- Five comorbidities showed significant environment effects after Bonferroni correction: pulmonary circulation, peripheral vascular, liver disease, metastatic cancer, and solid tumor.The analysis was constrained by few patients with recorded prior comorbidities, leaving many tests invalid.Source: §F.3.1 · p. 30
F.3.2 The ICU unit S affects the treatments T
- Patients received different treatments across ICU units after weighting for demographics and comorbidities (p = 0.015).Comorbidities were not reported for every patient, limiting this analysis.Source: §F.3.2 · p. 30
F.3.3 The ICU unit S does not affect the length of stay Y
- The tests found no evidence that ICU unit affected length-of-stay prevalence after controlling for demographics, comorbidities, and treatments (p = 0.18).The authors caution that high dimensionality could make the weighting test under-powered.Source: §F.3.3 · p. 30
F.3.4 The ICU unit S does not affect the labs and vitals X
- Labs and vitals showed no evidence of being affected by ICU unit (p = 0.45).Source: §F.3.4 · p. 31