PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Characterization of Fair Machine Learning For Clinical Risk Prediction

Stephen R. Pfohl, Agata Foryciarz, Nigam H. Shah

arXiv:2007.10306v3stat.MLcs.CYcs.LGstat.AP

TL;DR

The paper examines trade-offs associated with algorithmic fairness constraints in clinical predictive models using observational health data. It reports heterogeneous effects of imposing group-fairness measures and emphasizes limitations of the framework and the study’s generalizability.

  • Problem

    Trade-offs between model-performance measures under algorithmic fairness constraints remain insufficiently understood for clinical predictive models.

  • Method

    The study tunes a non-negative regularization parameter to explore trade-offs in clinical predictive modeling and examines data sources with biological sex recorded as a sensitive attribute.

  • Results

    The effects of imposing group-fairness measures show substantial heterogeneity across measures of model performance and group fairness.

  • Takeaways & Limitations

    These results quantify trade-offs among model-performance and group-fairness measures in clinical predictive models, where such trade-offs were under-explored.

  • Takeaways & Limitations

    The work has technical limitations that may limit the generalizability of its results.

Abstract

from arXiv · show

The use of machine learning to guide clinical decision making has the potential to worsen existing health disparities. Several recent works frame the problem as that of algorithmic fairness, a framework that has attracted considerable attention and criticism. However, the appropriateness of this framework is unclear due to both ethical as well as technical considerations, the latter of which include trade-offs between measures of fairness and model performance that are not well-understood for predictive models of clinical outcomes. To inform the ongoing debate, we conduct an empirical study to characterize the impact of penalizing group fairness violations on an array of measures of model performance and group fairness. We repeat the analyses across multiple observational healthcare databases, clinical outcomes, and sensitive attributes. We find that procedures that penalize differences between the distributions of predictions across groups induce nearly-universal degradation of multiple performance metrics within groups. On examining the secondary impact of these procedures, we observe heterogeneity of the effect of these procedures on measures of fairness in calibration and ranking across experimental conditions. Beyond the reported trade-offs, we emphasize that analyses of algorithmic fairness in healthcare lack the contextual grounding and causal awareness necessary to reason about the mechanisms that lead to health disparities, as well as about the potential of algorithmic fairness methods to counteract those mechanisms. In light of these limitations, we encourage researchers building predictive models for clinical use to step outside the algorithmic fairness frame and engage critically with the broader sociotechnical context surrounding the use of machine learning in healthcare.

1. Introduction

Clinical machine learning may exacerbate health disparities, while the role and consequences of algorithmic fairness methods in clinical prediction remain contested and insufficiently studied. The paper examines these trade-offs empirically across diverse healthcare settings and discusses limits of the fairness framework.

  • Clinical machine learning may introduce or worsen health disparities for disadvantaged and underrepresented populations.
  • Algorithmic fairness methods are mathematically defined criteria and procedures for auditing models and reducing fairness violations, but their clinical use remains relatively limited.
  • The paper argues that fairness methods alone are generally insufficient to mitigate disparities arising from broader sociotechnical and structural inequities.
  • Typical group-fairness criteria may be misleading because they rely on predictions, observed outcomes, demographic groups, and retrospectively derived cohorts without sufficient contextual grounding.
  • Fairness constraints create substantial trade-offs between model-performance measures and fairness-criterion satisfaction, depending on the dataset and learning algorithm.
  • The study characterizes these performance–fairness trade-offs by varying penalties for group-fairness violations across multiple databases, outcomes, and sensitive attributes.

2. Methods

The study trains predictive models across 25 database–outcome–sensitive-attribute combinations, varying penalties for conditional prediction parity and evaluating performance and multiple fairness criteria. It formalizes three fairness categories—conditional prediction parity, calibration, and cross-group ranking—and assesses them across several regularization strategies.

  • Fairness criteria: The fairness framework comprises conditional prediction parity, calibration, and cross-group ranking, each operationalized through one or more metrics.The study evaluates these metrics for models trained with six regularization strategies penalizing conditional prediction parity violations.
  • Experimental procedure: The experimental procedure extracts cohorts from databases, defines inclusion criteria, outcomes, and sensitive attributes, then performs feature extraction, model training, and evaluation.Figure 1 summarizes this workflow, including a cohort drawn from STARR.
  • Experimental procedure: 25 database–outcome–sensitive-attribute combinations are used to train models penalized to varying degrees for conditional prediction parity violations.The study reports model performance and group-fairness effects across these combinations.
  • Notation and problem formulation: For binary outcomes, the predictor estimates P(Y = 1 | X), optionally producing binarized predictions with threshold T.The group indicator A denotes a protected or sensitive attribute with K groups.
  • Conditional prediction parity: Conditional prediction parity metrics compare empirical prediction distributions across groups, with Earth Mover’s Distance used in the experiments.The framework includes demographic parity, equal opportunity, and equalized odds, including threshold-free and threshold-based variants.

distribution (M mean

The calibration analysis measures whether predicted risks correspond to observed event rates overall and whether this relationship differs across groups. It uses auxiliary estimators to define absolute and relative calibration errors, while noting estimator dependence and a limitation of signed measures.

  • Relative calibration: Relative calibration error measures whether observed event rates conditioned on predicted risk differ across groups, corresponding to violation of the matching conditional frequencies criterion.The analysis compares a population estimator gφ with a group-specific estimator gk.
  • Absolute calibration: Calibration assesses whether probabilistic predictions are faithful estimates of observed event rates, with perfect calibration satisfying P(Y = 1 | fθ = p) = p.Thus, among patients assigned risk p, the observed event rate should equal p.
  • Absolute calibration: Absolute calibration error measures deviations from perfect calibration using an auxiliary estimator of P(Y = 1 | fθ).The estimator maps predicted risk to an estimated outcome probability.
  • Signed calibration: The signed calibration measure indicates the direction of miscalibration but can be misleading when positive and negative deviations offset one another.A positive signed value corresponds to underprediction of risk.
  • Estimator dependence: The calibration measures depend on the forms of the estimators gφ and gk, which are implemented with logistic regression in the experiments.Alternative estimators include loess regression and kernel density estimates.

over P(fθ | Y ) combined with simple conditional probability rules, or binning estimators.

Cross-group ranking measures extend AUROC-style assessment by examining how positive and negative instances from different groups are ranked relative to one another. They address limitations of comparing AUROC values across groups alone.

  • AUROC and cross-group ranking: AUROC is the probability that predictions for positive examples rank above predictions for negative examples.It is a standard performance metric for clinical predictive models.
  • AUROC and cross-group ranking: Comparing AUROC across groups can be misleading because it does not account for positive examples in one group being ranked above negative examples from other groups.Cross-group ranking measures are introduced to provide insight into this phenomenon.
  • xAUC extensions: The xAUC framework is extended to multiple groups to evaluate cross-group ranking performance.These measures assess ranking relationships involving positive and negative instances from different groups.

Similarly, we define xAUC0

The xAUC0 measure evaluates the ranking of negative instances from one group relative to positive instances from all other groups.

  • xAUC0: xAUC0 is the probability that negative instances of group Ak are ranked below positive instances of all other groups.It is a cross-group ranking measure defined within the multi-group xAUC framework.

xAUC0

The study evaluates regularized learning objectives that penalize violations of threshold-free conditional prediction parity across multiple clinical databases, outcomes, and demographic groupings. It compares model performance and fairness as the regularization strength varies.

  • Regularized objectives: The authors focus on regularized learning objectives that penalize violations of threshold-free conditional prediction parity.The objective combines a predictive loss with a non-negative fairness-violation regularizer weighted by λ.
  • Fairness penalties: Six regularization strategies are evaluated, using MMD-based or mean-based penalties to compare prediction distributions across groups or outcome strata.The strategies target demographic parity, equalized odds, and equal opportunity.
  • Datasets and outcomes: The experiments span STARR, Optum CDM, and MIMIC-OMOP clinical databases with outcomes including mortality, prolonged stay, and readmission.The cohorts include 198,644 STARR admissions, 8,074,571 Optum CDM patients, and 26,170 MIMIC-OMOP patients.
  • Sensitive attributes: Fairness is evaluated across race and ethnicity, sex, and discretized age groups defined separately for each database.Race and ethnicity groups are aggregated when few observed outcomes are available, and race and ethnicity are unavailable in the Optum CDM version used.

3. Results

Baseline models show substantial differences in group-level performance and fairness-related behavior. Fairness-promoting regularization usually improves the targeted criterion, but its effects on other fairness measures and performance vary across conditions.

  • Cross-condition effects: The effects of fairness-promoting objectives are heterogeneous across models and experimental conditions.The reported analyses emphasize both trends that replicate across conditions and notable exceptions.
  • Baseline results: Baseline models exhibit substantial differences in group-level AUROC, average precision, and cross entropy loss.For STARR 30-day readmission, mean AUROC ranges from 0.66 for Black patients to 0.80 for the Other group.
  • Baseline results: STARR 30-day readmission models show large cross-group AUROC variability while signed absolute and relative calibration errors remain small.Mean AUROC is 0.78, 0.66, 0.77, 0.80, and 0.71 for Asian, Black, Hispanic, Other, and White groups, respectively.
  • Targeted fairness: Regularization typically improves satisfaction of the fairness criterion corresponding to the penalty used.An unconditional demographic-parity penalty typically minimizes EMD and differences in mean predictions across groups.
  • Targeted fairness: The targeted fairness criterion can nevertheless be violated more after regularization in some conditions.For sex-based equal opportunity in STARR 30-day readmission, MEqOpp is 0.0069 at λ = 10.

than at baseline (MEqOpp = 0.0056).

Increasing fairness-penalty strength generally reduced group-level predictive performance, while effects on calibration, ranking, and fairness varied across penalties and experimental conditions.

  • Fairness trade-offs: The experiments quantify trade-offs among model-performance measures and group-fairness measures across multiple penalty choices and clinical prediction settings.The reported analyses include performance, calibration, conditional prediction parity, and cross-group ranking measures relative to unpenalized training.
  • Fairness trade-offs: Fairness outcomes were not uniform: conditional penalties could further violate demographic parity, while unconditional penalties sometimes improved it.For example, conditional penalties in outcome-observed strata increased demographic-parity violation, whereas age-based demographic-parity penalization improved it in one STARR readmission setting.
  • Model performance: Conditional penalties targeting equalized odds or equal opportunity generally caused monotonic reductions in group-level model performance as λ increased.Unconditional demographic-parity penalties produced more heterogeneous performance changes, including little change or improvements for some groups.
  • Model performance: AUROC increased from 0.66 to 0.69 and Average Precision from 0.15 to 0.17 for the Black group at λ = 3.6 under demographic-parity penalization.This improvement was reported for 30-day readmission prediction in the STARR cohort.
  • Calibration: Conditional penalties generally worsened absolute group-level calibration, whereas unconditional penalties ranged from limited impact to effects similar to conditional penalties.Both penalty types could introduce relative calibration error across groups independently of absolute-calibration changes.
  • Cross-group ranking: Fairness-promoting objectives produced heterogeneous changes in cross-group ranking across regularizers, databases, outcomes, and sensitive attributes.Cross-group ranking accuracy often declined with λ, although some trajectories converged in ways that improved fairness while allowing performance gains for at least one group at another's expense.

4. Discussion

The study empirically examines trade-offs between group fairness and model performance in clinical prediction, while emphasizing technical, conceptual, and sociotechnical limits of the fairness framework.

  • Empirical trade-offs: The experiments evaluate how penalizing group fairness violations affects clinical predictive-model performance and fairness across datasets, outcomes, sensitive attributes, group definitions, and regularization strategies.The study aims to characterize these trade-offs empirically.
  • Empirical trade-offs: The results show substantial heterogeneity in how imposing group-fairness measures affects performance and fairness metrics across experimental conditions.The analyses quantify how trade-offs described by impossibility theorems manifest in clinical predictive modeling.
  • Technical limitations: The study’s one-vs-marginal metric construction can make conditional-prediction-parity violations appear smaller for over-represented groups.The regularization objectives may also exaggerate reported trade-offs relative to constrained Lagrangian formulations.
  • Technical limitations: The findings may have limited generalizability because constrained optimization could achieve fairness with less severe trade-offs than the penalized objectives used here.Guarantees from constrained approaches typically require convex objectives, although modified proxy-Lagrangian methods have shown empirical success for non-convex problems.
  • Conceptual limitations: Group fairness treats categorical attributes as homogeneous and interchangeable, which can marginalize intersectional identities and obscure contextual differences between race, sex, and other attributes.The paper notes that racial differences are primarily linked to structural racism, whereas sex differences may also reflect physiology and sociological factors.
  • Sociotechnical implications: Group-fairness evaluations overlook inequities in data generation and measurement and lack the causal and decision-theoretic context needed to connect predictions with downstream interventions and health disparities.The authors therefore argue that fairness constraints alone can be insufficient and may work against health-equity goals.
  • Sociotechnical implications: The paper recommends participatory design involving patients, advocacy groups, and civil society to identify how measurement error, bias, and historical inequities shape models and interventions.It also allows that responsible analysis may lead to abstaining from algorithm-aided decision making when responsible use is impractical.

5. Conclusion

The paper addresses the limited empirical characterization of algorithmic fairness methods in clinical prediction by measuring performance–fairness trade-offs across clinical modeling settings. It recommends contextualizing fairness and performance within the effects of model-guided interventions.

  • The healthcare fairness debate has proceeded with limited empirical characterization of how these techniques affect predictive models trained on large-scale clinical data.
  • The study measures trade-offs involving model performance and conditional prediction parity, relative calibration, and cross-group ranking in clinical predictive models.These fairness constructs are well understood theoretically but under-explored for clinical prediction.
  • The authors recommend using algorithmic fairness for monitoring and auditing only when performance and fairness measures can be contextualized within the complex effects of clinical model outputs.

Appendix A. Supplementary Cohort Tables

The supplementary appendix documents cohort characteristics, feature extraction, hyperparameter search, and selected model configurations for the study databases.

  • Cohort tables: Supplementary tables report cohort characteristics for patients drawn from Optum CDM and MIMIC-III, grouped by demographic attributes.
  • Cohort tables: The STARR cohort includes patient counts and outcome incidence for 30-day readmission, prolonged hospital stay, ICU stay, and mortality outcomes.
  • Feature extraction: STARR features are extracted from patient timelines using interval-based binary representations of OMOP concepts and laboratory results.Numeric laboratory values are represented using reference-range comparisons and empirical quintile bins, then concatenated across intervals.
  • Model selection: Hyperparameter tuning uses a Cartesian-product grid with random search evaluating fifty configurations, with selected settings tabulated separately for STARR, Optum CDM, and MIMIC-III.

Appendix D.1. STARR

The STARR appendix presents fairness and group-level performance measures as the penalty for fairness-criterion violations varies across sensitive attributes, outcomes, and fairness objectives.

  • Experimental settings: The STARR analyses examine hospital mortality and prolonged length of stay using race and ethnicity, sex, and age group as sensitive attributes.
  • Fairness measures: Fairness figures report conditional prediction parity and relative calibration metrics under penalties for Demographic Parity, Equalized Odds, and Equal Opportunity.
  • Fairness measures: Conditional prediction parity is assessed for the whole population and separately within outcome-observed and outcome-unobserved strata.
  • Baselines: Dashed lines indicate the mean result from the unpenalized training procedure, providing the baseline for interpreting penalty effects.
  • Performance measures: Performance figures report mean ± SD for AUROC, average precision, cross-entropy loss, calibration errors, and cross-group ranking performance.

Appendix D.2. Optum CDM

Appendix D.2 examines how increasing penalties for group-fairness violations relate to model performance and fairness metrics across Optum CDM prediction settings. The analyses vary sensitive attributes, outcomes, fairness criteria, and penalty formulations.

  • Experimental settings: Results cover prolonged length of stay and 30-day readmission predictions with sex or age as sensitive attributes in the Optum CDM database.The appendix also includes decomposed group-level conditional-parity and relative-calibration measures for these settings.
  • Performance measures: The appendix reports performance measures including AUROC, average precision, cross-entropy loss, calibration error, and cross-group ranking.These metrics are shown for group-level evaluation under penalized objectives.
  • Fairness objectives: The analyses evaluate threshold-free Demographic Parity, Equalized Odds, and Equal Opportunity using MMD- and mean-based penalties.Conditional prediction parity is assessed in the whole population and separately for outcome strata y=1 and y=0.
  • Baseline: Dashed lines represent the unpenalized training procedure used as the comparison baseline.This baseline is stated for the plotted performance and fairness results.
Loading 2007.10306v3…