PaperlayerView exact revision on arXiv

Source-linked AI summary

Detecting Shortcut Learning for Fair Medical AI using Shortcut Testing

Alexander Brown, Nenad Tomasev, Jan Freyberg, Yuan Liu, Alan Karthikesalingam, Jessica Schrouff

arXiv:2207.10384v2cs.LG

TL;DR

Sensitive attributes can be clinically relevant, making shortcut learning difficult to diagnose when those attributes are linked to disease. The paper tests whether fairness depends on attribute encoding, finding that encoding itself is a poor measure and that unfairness may have other sources.

  • Problem

    Diagnosing shortcut learning is difficult when sensitive attributes may be causally related to disease, despite the risk that shortcuts worsen unfairness and disparities.

    Source: Introduction · pp. 2–3

  • Method

    The method varies sensitive-attribute encoding through an added demographic prediction head with variable gradient scaling and tests how model fairness responds.

    Source: Intervening on attribute encoding… · p. 4, Discussion · p. 11

  • Results

    The degree to which a model encodes a sensitive attribute is a poor measure of its use, and dermatology results showed no clear shortcut-learning pattern despite unequal age performance.

    Source: Discussion · pp. 10, 13

  • Takeaways & Limitations

    Fairness assessment should consider how prediction fairness depends on attribute encoding and the broader clinical and social context, not model behavior alone.

    Source: Discussion · pp. 10, 14

    The study treats disease presence as a binary outcome; extending the framework to account for attribute-linked disease severity or subtype distributions requires further work.

    Source: Discussion · p. 13

Abstract

from arXiv · show

Machine learning (ML) holds great promise for improving healthcare, but it is critical to ensure that its use will not propagate or amplify health disparities. An important step is to characterize the (un)fairness of ML models - their tendency to perform differently across subgroups of the population - and to understand its underlying mechanisms. One potential driver of algorithmic unfairness, shortcut learning, arises when ML models base predictions on improper correlations in the training data. However, diagnosing this phenomenon is difficult, especially when sensitive attributes are causally linked with disease. Using multi-task learning, we propose the first method to assess and mitigate shortcut learning as a part of the fairness assessment of clinical ML systems, and demonstrate its application to clinical tasks in radiology and dermatology. Finally, our approach reveals instances when shortcutting is not responsible for unfairness, highlighting the need for a holistic approach to fairness mitigation in medical AI.

Introduction

The introduction frames subgroup performance differences as a fairness concern and asks whether sensitive-attribute information reflects harmful shortcutting or clinically relevant signal. It presents a framework for assessing this distinction when attributes may be linked to disease.

Introduction

  • Sensitive attributes can improve predictions when they correlate with disease risk or presentation, so ignoring or ablating them may reduce clinical performance.Source: Introduction · p. 2
  • Shortcut-based rules may generalize poorly to new populations or settings and amplify errors on atypical cases, potentially worsening unfairness and health disparities.Source: Introduction · p. 2
  • Encoding sensitive-attribute information does not by itself establish that a model uses it for clinical predictions or that such use constitutes shortcut learning.Source: Introduction · p. 3
  • The paper defines shortcut learning as sensitive-attribute effects that affect fairness without considerably improving performance, distinguishing them from potentially beneficial biological effects.Source: Introduction · p. 3
  • The proposed framework studies and mitigates shortcut learning in clinical models, with radiology and dermatology examples focused on age and an investigation of race encoding.Source: Introduction · p. 3

Results

Across chest X-ray and dermatology tasks, models encoded age and showed subgroup performance disparities, but ShorT found encoding-related unfairness in some settings and not others. Interventions linked shortcutting more strongly to induced age bias and race encoding than to Acne fairness.

Prediction models encode age and are unfair

  • CXR models encoded age despite not being trained to, and all three tasks showed age-based separation of 0.01–0.02, equivalent to an 11–22% performance difference per decade.Age prediction transfer models performed significantly better than chance for Effusion, Atelectasis, and Abnormal.Source: Prediction models encode age and are… · p. 4

Intervening on attribute encoding affects fairness metrics

  • Gradient-scaled age-head multitask learning varied age encoding across experimental bounds; reducing encoding slightly improved fairness but lowered overall performance.The analysis varied age-head gradient scaling and assessed resulting fairness and performance.Source: Intervening on attribute encoding… · pp. 4–5, Figure 2 · p. 5

ShorT efficiently detects shortcut learning

  • In biased Effusion data, fairness worsened; balanced-data AUC remained similar to original (0.883 versus 0.882) while fairness improved.Similar patterns appeared for biased Atelectasis and Abnormal datasets, and Effusion results replicated with other fairness metrics.Source: Figure 3 · p. 6, ShorT efficiently detects shortcut… · pp. 6–7
  • ρ=-0.668 (p=8.11e-17) in biased data, versus ρ=-0.224 (p=0.0156) in original and nonsignificant ρ=0.116 (p=0.218) in balanced data, showed dataset-dependent shortcutting.Correlations quantify the association between age encoding and unfairness.Source: Figure 4 · p. 7, ShorT efficiently detects shortcut… · p. 7

Shortcut learning cannot be identified by attribute encoding alone

  • At age MAE 8.5–9.5 years, balanced models averaged separation 0.0016 versus 0.0384 for biased models, showing age-transfer performance alone did not predict fairness across datasets.The reported separations corresponded to average disparities of 1.6% and 47% per decade, respectively.Source: Shortcut learning cannot be identified… · p. 8

ShorT detects shortcutting by race in cardiomegaly predictions

Beyond shortcut learning: Acne prediction in a Dermatology model

  • Despite an 18.6-year age gap and baseline separation of 0.0576–0.0755, changing age encoding did not systematically change Acne fairness (ρ=-0.177, p=0.0746).The authors conclude the specific cause of older patients’ lower Acne sensitivity does not appear to be shortcut learning.Source: Figure 6 · p. 9, Beyond shortcut learning: Acne… · p. 9

Discussion

Shortcut learning is difficult to diagnose when sensitive attributes may be causally linked to disease, so the discussion argues for testing whether model fairness depends on attribute encoding. Across applications, encoding strength and attribute–outcome correlation alone did not reliably identify shortcut-driven unfairness.

Discussion

  • Fairness criteria require clinical and societal context; separation was selected because age may be causally linked to CXR findings, and the framework can also use other metrics.Source: Discussion · p. 10
  • Correlation alone was insufficient: shortcutting increased with age–Effusion correlation in subsamples, yet occurred in the original CXR data and was absent for Acne despite stronger correlation.Source: Discussion · p. 11
  • The degree of sensitive-attribute encoding did not predict model fairness in either the CXR or dermatology tasks.Source: Discussion · p. 11
  • ShorT varies attribute encoding with an auxiliary demographic head and gradient scaling, then tests whether fairness changes systematically; it tests shortcut presence, not its strength.Source: Discussion · p. 11
  • ShorT may be unreliable if encoding interventions span too narrow a range or fairness estimates are under-powered or highly variable.Source: Discussion · p. 12
  • Gradient reversal ameliorated but did not eliminate biased-data effects, while balancing yielded fair CXR performance without overall performance cost; absent shortcutting calls for other mitigation.Source: Discussion · pp. 10, 13
  • Dermatology showed unequal age-based performance without a clear shortcut pattern, suggesting attribute encoding was not the main source of unfairness there.Source: Discussion · p. 13
  • The study examines binary disease presence and age; extending the framework to severity, subtype, or unobserved demographics remains future work.Source: Discussion · pp. 13–14

Methods

The methods operationalize ShorT as testing whether changes in sensitive-attribute encoding track fairness, using multitask learning in chest X-ray and dermatology tasks.

Datasets, tasks and models

  • CXR experiments used NIH data to predict Effusion, Atelectasis, and Abnormal labels, while the study also applied ShorT to a multiclass dermatology task.CXR condition models used AUROC for performance; the dermatology analysis examined one class in binarized form.Source: Datasets, tasks and models · p. 14, ShorT on other datasets · p. 18

Assessing the encoding of demographic information

  • Age encoding was measured by transferring a frozen condition-model feature extractor to age prediction, with lower age MAE indicating more encoded age information.Direct age prediction supplied an empirical lower error bound, while predicting training-set mean age supplied an upper bound.Source: Assessing the encoding of demographic… · p. 15

Assessing fairness for age as a continuous variable

  • Fairness was evaluated with continuous-age independence, separation, and sufficiency metrics, with separation emphasized because age may relate to the clinical task.Separation summarizes absolute age effects on true- and false-positive rates; zero indicates no monotonic relationship between age and model performance.Source: Assessing fairness for age as a… · p. 16

ShorT: testing for shortcut learning

  • ShorT tests whether intervening on sensitive-attribute encoding consistently changes fairness beyond performance gains, rather than treating shortcutting as a binary signal.The method estimates the intervention’s efficacy through attribute-prediction performance and relates age encoding to fairness metrics.Source: ShorT: testing for shortcut learning · pp. 16–17
  • The method varies age representation between weak and strong encoding and uses attribute-prediction performance as a proxy for intervention efficacy.It expects a negative Spearman correlation between age MAE and separation to indicate shortcut learning.Source: ShorT: testing for shortcut learning · p. 17
  • A shared feature extractor and separate age and condition heads varied age encoding by scaling age-head gradients; negative scaling reverses gradients and reduces age representation.Positive scaling encourages stronger age representation, and model uncertainty was estimated by bootstrapping test examples 1,000 times.Source: ShorT: testing for shortcut learning · p. 17

Assessing the efficacy of ShorT to detect shortcut learning

  • Training-set resampling created biased data with about a decade of age difference between classes and balanced data with approximately matched age distributions.Resampling retained original examples and marginal disease probabilities while reducing the number of training examples.Source: Assessing the efficacy of ShorT to… · pp. 17–18

ShorT on other datasets

  • The approach was also applied to CheXpert chest X-rays and proprietary U.S. teledermatology images, adapting fairness metrics and hyperparameters to each task.The dermatology analysis used a multiclass prediction model.Source: ShorT on other datasets · p. 18

Code availability

The paper identifies the frameworks and code availability for its medical-imaging and simulated experiments.

Code availability

  • The detailed medical-imaging modeling code is proprietary, while pseudocode agnostic to the deep-learning framework is provided in the supplementary material.Source: Code availability · pp. 18–19
  • Code for the simulated MNIST experiment is available in the Google Research shortcut_testing repository.Source: Code availability · p. 19

Competing Interests Statement

The authors report Google funding and no competing interests.

Competing Interests Statement

Supplementary Figures

The supplementary figures compare model performance, age encoding, and fairness across CXR datasets and dermatology models, including settings where fairness depends on age representation and settings where balancing improves separation.

Supplementary Figure 2

  • Across other CXR labels, biased training increased unfairness, while gradient reversal ameliorated it; balancing produced fairer baselines, and gradient reversal degraded performance without further fairness gains.Source: Figure 2 · p. 24

Supplementary Figure 3

  • For Atelectasis and Abnormal, biased datasets showed significant fairness dependence on age representation, whereas balanced datasets showed no such dependence.In the original dataset, fairness for No Finding had a significant positive correlation with age representation; the figure notes an AUC exclusion threshold of 0.7.Source: Figure 3 · p. 25

Supplementary Figure 4

  • Effusion fairness metrics include independence, positive predictive value, and maximum subgroup performance gap, with age subgroups defined in four ranges.Correlation analyses excluded replicates below AUC 0.8.Source: Figure 4 · p. 26

Supplementary Figure 5

  • At the same age-encoding level, Balanced and Original Effusion models had similar performance, while Balanced performance fell at higher age-prediction errors and Biased data yielded spuriously higher AUC.Fairness differed across dataset types at matched age-encoding levels.Source: Figure 5 · p. 27

Supplementary Figure 6

  • Balanced-data models achieved similar performance to original-data models, while separation was considerably improved; biased-data in-distribution performance was highest but degraded out of distribution.Source: Figure 6 · p. 27, Supplementary Figures · p. 28

Supplementary Figure 7

  • Dermatology ShorT models spanned age-prediction errors, with a broad middle gradient-scaling plateau near baseline error.The caption attributes the plateau and wide error-bound range likely to richer soft labels and stronger age-condition dependence for many, but not all, dermatological problems.Source: Figure 7 · p. 28

Supplementary Methods

The supplementary methods describe the CXR datasets and model architectures used for clinical and demographic prediction. Models use image embeddings followed by prediction heads, with gradient reversal available to scale attribute-head gradients.

Model Architectures

  • Medical imaging models use modified ResNet 101x3 image embeddings followed by MLPs for clinical classification and age or race prediction.The dermatology task averages embeddings across the 1–6 images in each clinical case before passing them to an MLP.Source: Model Architectures · p. 30
  • The supplied architecture pseudocode uses gradient reversal on image embeddings, with lambda controlling the strength and direction of gradient scaling.The pseudocode also describes cross-entropy for clinical predictions and attribute losses, including mean squared error and binary cross-entropy.Source: Model Architectures · p. 30

Hyperparameter Tuning and model selection

The supplementary methods describe model selection, multitask training, data subsampling, and task-specific experiments. Comparisons show ShorT detects more shortcutting in biased CXR datasets than in original or balanced datasets.

Hyperparameter Tuning and model selection

  • Models were trained for 17,500 epochs, and validation performance selected the model while validation F1 selected its decision threshold.Batch size, learning rate, weight decay, and penultimate-layer dropout were tuned; the same parameters were used across CXR labels.Source: Hyperparameter Tuning and model… · p. 31

Multitask Prediction

  • Multitask models add a demographic head, down-weight age regression loss by 100, and sweep 25 gradient scales with five replicates each.For attribute transfer, the feature extractor is frozen and a linear demographic head is retrained; extra hidden layers made no material difference.Source: Multitask Prediction · p. 31

Subsampling of training data

  • Training-set subsampling uses an age-dependent logistic retention probability, with negative examples retained at probability 1 − pretain.The generated datasets do not precisely match the desired age shifts because of stochastic errors.Source: Subsampling of training data · p. 32

Significance testing when comparing ShorT across datasets

  • p = 1e-8 for original versus biased CXR datasets and p = 1e-4 for original versus balanced datasets; shortcutting was greater in biased and lower in balanced data.The comparison uses permutation tests of differences in ShorT correlations with a bootstrapped empirical null distribution.Source: Significance testing when comparing… · p. 33

Race in cardiomegaly models

  • Race experiments use CheXpert cardiomegaly, re-split training data because the public validation set has only nine Black individuals, and estimate fairness via equalized odds.Cardiomegaly prevalence was 11.5% for White patients and 19.8% for Black patients; the experiment sweeps race-head gradient scale from −0.1 to +0.1.Source: Race in cardiomegaly models · pp. 33–34

Dermatology Dataset and experiments

  • The dermatology task predicts 26 skin conditions plus an “other” category from a commercial teledermatology dataset using a more modern architecture and a smaller training set than prior results.Single-class performance uses binarized metrics; top-3 selection did not change results.Source: Dermatology Dataset and experiments · p. 34

Dermatology Dataset - Demographics

Supplementary Note 1

Simulated MNIST data let the authors control label–attribute correlations and test ShorT against known shortcutting. ShorT produced few significant results at low correlations and identified shortcutting in all high-correlation, asymmetric cases.

Simulated data

  • The simulation adds a red or green square at a random location in MNIST images, with color correlated to the label as the sensitive attribute.The square color can be switched to generate counterfactual samples.Source: Simulated data · p. 35
  • The simulation varied label–attribute correlations across 20 steps for each label and used a small three-layer MLP as its feature extractor.The attribute head included an additional dense layer; baseline accuracy was 0.8–0.86.Source: Simulated data · pp. 35–36
  • 3/50 low-correlation cases were significant, while ShorT identified shortcutting in 50/50 high-correlation asymmetric cases, including after Bonferroni correction.The high-correlation setting used different correlation ranges by label, because asymmetry was needed for equalized-odds unfairness.Source: Simulated data · p. 36, Figure S8 · p. 36

Supplementary Discussion

The analysis preserves age as continuous and uses logistic regression to characterize fairness, avoiding arbitrary age quantization. This choice assumes monotonic discrepancies and has limits for complex fairness patterns and some metrics.

Supplementary Discussion

  • The logistic-regression approach assumes that observed performance discrepancies vary monotonically toward older or younger patients.For bimodal or more complex fairness distributions, the authors suggest examining model outputs rather than relying on particular fairness-metric formulations.Source: Supplementary Discussion · p. 37
  • The logistic-regression model requires a binarized outcome per example, making it unsuitable for continuous prediction scores or AUC.Alternative methods may address some limitations, at the expense of interpretability.Source: Supplementary Discussion · p. 37
  • The framework can also use binary or discrete sensitive attributes by substituting conventional fairness definitions for the model-based metrics.The continuous-age analysis avoids arbitrary quantization but relies on its stated monotonicity assumption.Source: Supplementary Discussion · p. 37

Related papers

Loading 2207.10384v2…