PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Demographic Balance: Multi-Metric and Intersectional Evaluation of Fairness in MIMIC-IV Mortality Prediction

Abdullah Al Noman, Fahmid Al Rifat, Tahrima Hashem, Syed Muhammad Ibne Zulfiker, Rishov Paul, Tanzima HAshem

arXiv:2610.01645v1cs.LG

TL;DR

The paper examines whether fairness conclusions in ICU mortality prediction vary with the metrics and demographic resolution used. It evaluates demographic-only intersectional head adaptation alongside other interventions, finding that assessments can differ across metrics and that marginal summaries can conceal intersectional error profiles.

  • Problem

    The paper asks whether fairness interventions receive consistent assessments across predictive-utility and subgroup-error metrics, and whether marginal evaluations conceal intersectional error behavior.

    Source: §1 · p. 2

  • Method

    The study balances observed ethnicity–gender–insurance representation during classifier-head adaptation without conditioning the balancing rule on mortality outcomes.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Interventions have metric-dependent profiles, and marginal demographic summaries can conceal materially different three-way subgroup behavior, including among better-supported intersections.

    Source: §5 · p. 8, §7 · p. 9

  • Takeaways & Limitations

    Clinical fairness evaluations should report complementary utility and error metrics, match subgroup resolution to the intervention, and make support for fine-grained estimates explicit.

    Source: §7 · p. 9

    The study uses retrospective EHR data from a single health system, so generalizability to other populations, settings, and more recent contexts, and real-world clinical benefit, remain to be established.

    Source: §6 · p. 8

Abstract

from arXiv · show

Fairness conclusions in clinical prediction can depend strongly on both the metrics reported and the demographic resolution at which performance is evaluated. We revisit these evaluation choices for ICU mortality prediction on MIMIC-IV, comparing predictive-utility and subgroup-error metrics across several fairness interventions. As a complementary case study, we introduce a lightweight adaptation strategy that jointly balances ethnicity--gender--insurance representation without conditioning on mortality outcomes, allowing demographic representation balancing to be examined separately from outcome-conditioned or direct error-rate interventions. We evaluate its behavior at both marginal and corresponding three-way intersectional subgroup levels, while accounting for the statistical support of finer-grained estimates. The results show that interventions can receive substantially different assessments across accuracy/AUROC, sensitivity, and false-positive rate, and that marginal demographic summaries can conceal heterogeneous error profiles within their constituent intersections, including among larger subgroups. These findings highlight the importance of evaluating fairness interventions at both complementary metric and subgroup resolutions, while accounting for the intervention target and the reliability of subgroup estimates.

1 Introduction

Fairness assessments in clinical prediction depend on both the metrics used and the demographic resolution evaluated. The paper studies demographic-only intersectional head adaptation and asks how its evaluation compares across these choices.

1 Introduction

  • IBHFT balances observed ethnicity–gender–insurance representation during classifier-head adaptation without using mortality labels or directly optimizing subgroup error parity.This separates demographic representation balancing from outcome-conditioned and direct error-rate interventions.Source: §1 · p. 2
  • The paper evaluates this demographic-only strategy as a focused case study because conditioning on mortality would also change output-label composition.Source: §1 · p. 2
  • The study asks whether interventions assessed with accuracy/AUROC versus sensitivity and false-positive rate receive consistent conclusions, and whether marginal evaluations conceal intersectional errors.Source: §1 · p. 2
  • Fairness conclusions can differ across predictive-utility and subgroup-error metrics, and marginal summaries can conceal heterogeneous errors at ethnicity–gender–insurance intersections.The paper examines both dependencies in MIMIC-IV ICU mortality prediction.Source: §1 · p. 2

2 Related Work

Related work covers clinical fairness evaluation, interventions that target outcomes as well as demographics, intersectional analysis, and lightweight classifier adaptation. This study complements those approaches by balancing three demographic attributes without conditioning on mortality.

Fairness in clinical prediction

  • Clinical fairness studies differ in cohorts, endpoints, protected attributes, and evaluation criteria, making protocol relevant to interpreting results.Source: §2 · p. 3

Outcome prevalence and fairness criteria

  • Demographic balancing, demographic–outcome balancing, and direct error-rate optimization target different aspects of prediction; this intervention balances representation without conditioning on mortality.Source: §2 · p. 3

Multiple sensitive attributes and intersectional fairness

  • Prior work examines multiple sensitive attributes and cross-axis effects; this study jointly evaluates ethnicity, gender, and insurance at marginal and three-way intersectional resolutions.Source: §2 · p. 3

Lightweight classifier adaptation

  • Unlike final-classifier adaptation using sensitive-group–label balanced data or fairness penalties, this configuration balances three demographic attributes without mortality labels and adapts a replacement head.Source: §2 · p. 3

3 Methods

The methods use a MIMIC-IV ICU mortality prediction pipeline and adapt a classifier head using equal-count sampling across observed demographic intersections. Comparisons vary both the performance metrics and the subgroup resolution.

3.1 Prediction task and base model

  • The MIMIC-IV experiments use a time-series LSTM for ICU mortality prediction on processed training and held-out test stays.Source: §3.1 · p. 4

3.2 Intersectionally balanced head fine-tuning

  • IBHFT samples observed ethnicity–gender–insurance intersections to equal counts, excludes mortality labels from the grouping criterion, and optimizes a replacement head while freezing the learned representation.The intervention does not set intersectional mortality prevalence or directly constrain TPR or FPR.Source: §3.2 · p. 4

3.3 Comparators and evaluation protocol

  • The comparison includes an unmitigated baseline, suppression, reweighting, temperature scaling, and IBHFT; temperature scaling serves as a calibration control.Source: §3.3 · pp. 4–5
  • Evaluation compares accuracy and AUROC with threshold-dependent subgroup errors, and marginal evaluations with full ethnicity–gender–insurance intersections.Intersectional reporting includes subgroup support and focuses in the main text on N ≥100 and N+ ≥5; Appendix B also reports 95% Wilson intervals for sensitivity.Source: §3.3 · p. 5

4 Results

The interventions receive different assessments across predictive-utility and subgroup-error metrics, while marginal summaries can conceal heterogeneous errors across ethnicity–gender–insurance intersections. Intersectional estimates vary in statistical support, so their reliability must also be considered.

4.1 RQ1: metric choice changes the assessment of the interventions

  • Across ethnicity groups, reweighting’s median TPR is approximately 0.15–0.45 versus roughly 0.70–0.96 for baseline and other interventions, despite very low FPR.Judged by FPR alone, reweighting appears favorable; judged by sensitivity, the conclusion reverses.Source: §4.1 · p. 5, Figure 2 · p. 6
  • For Medicaid, Medicare, and Other insurance groups, IBHFT reduced median FPR from 0.23 to 0.18, 0.31 to 0.23, and 0.23 to 0.18, respectively, while TPRs remained around 0.8 or higher.Its TPR stayed closer to baseline than reweighting’s, illustrating a different error profile.Source: §4.1 · p. 5, Figure 2 · p. 6
  • Reweighting appears favorable on accuracy/AUROC but occupies a high-accuracy, low-sensitivity region, so predictive utility does not by itself establish acceptable error behavior.AUROC summarizes discrimination across thresholds, whereas TPR and FPR describe errors at the selected operating point.Source: §4.1 · pp. 5–6, Figure 3 · p. 6

4.2 RQ2: marginal evaluation can conceal intersectional error profiles

  • The fine-tuned predictions cover all 30 observed intersections, but sizes range from 9 to 817 patients and positive cases from 1 to 101; the main comparison uses N ≥100 and N+ ≥5.Figure 4 shows sensitivity for the better-supported intersections, with 95% Wilson intervals; Appendix B reports intersections with N+ ≥5.Source: §4.2 · pp. 6–7, Figure 4 · p. 7
  • Black patients’ marginal sensitivity was 0.805, while Black female Medicare and Other intersections had sensitivities of 0.944 and 0.400, respectively.The Other-insurance intersection had 106 patients but only five mortality-positive cases, making its sensitivity estimate imprecise.Source: §4.2 · p. 7
  • For Other ethnicity, marginal FPR was 0.282, while four larger intersections had FPRs of 0.211–0.371 despite 24–27 positive cases each.These intersections contained 117–207 patients apiece.Source: §4.2 · p. 7
  • Among four White Medicare/Other intersections (543–817 patients, 41–101 positives), sensitivity was 0.683–0.782 and FPR 0.153–0.230, versus marginal 0.747 and 0.206.The discrepancy is therefore not confined to very small demographic subgroups.Source: §4.2 · p. 7
  • Across all 30 intersections, sensitivity ranged 0.333–1.000, FPR 0.083–0.667, and AUROC 0.542–1.000; extremes partly reflect sparse subgroups.Marginal summaries provide population-level estimates but can mask heterogeneity among constituent intersections.Source: §4.2 · p. 7, Figure 4 · p. 7

5 Discussion

The discussion emphasizes that fairness conclusions depend on both the metric and subgroup resolution, and that demographic balancing is a distinct intervention target. Deployment-oriented rankings also require clinically meaningful thresholds or explicit error costs.

5 Discussion

  • The evaluated interventions show distinct fairness–utility profiles rather than a uniform ranking: reweighting combines strong accuracy/AUROC and very low FPR with substantially reduced sensitivity.IBHFT generally lowers FPR while retaining sensitivity closer to baseline; suppression and calibration show less pronounced trade-offs.Source: §5 · p. 8
  • IBHFT targets demographic representation during adaptation, unlike demographic–outcome balancing or direct error-rate constraints; these are distinct intervention targets, especially when subgroup base rates differ.The study’s demographic-only balancing excludes mortality labels from the sampling criterion.Source: §5 · p. 8
  • Marginal estimates can conceal substantially different intersectional error profiles, making subgroup support and uncertainty important alongside intersectional estimates.The discussion connects this pattern with prior intersectional and cross-axis analyses.Source: §5 · p. 8
  • A deployment-oriented conclusion requires a clinically meaningful decision threshold or explicit cost structure for false-negative and false-positive errors.Without these, the evidence supports comparing error profiles rather than universally ranking methods.Source: §5 · p. 8

6 Limitations

The study’s evidence comes from retrospectively collected EHR data at a single health system, leaving generalizability and real-world clinical benefit to be established.

6 Limitations

  • Because the study uses retrospective EHR data from one health system, generalizability to other populations, settings, and more recent contexts—and real-world clinical benefit—remains to be established.The authors also identify ethnicity and insurance as broad categories that do not capture the full patient heterogeneity; insurance should not be treated as a direct socioeconomic-status proxy.Source: §6 · p. 8

7 Conclusion

On MIMIC-IV ICU mortality prediction, the adaptation strategy balances ethnicity–gender–insurance representation without conditioning on mortality labels. Model assessments vary by metric, and marginal demographic summaries conceal heterogeneous three-way subgroup behavior.

7 Conclusion

  • The adaptation balances observed ethnicity–gender–insurance representation during classifier-head fine-tuning without conditioning on mortality labels.This separates demographic representation balancing from outcome balancing.Source: §7 · p. 8
  • Metric-dependent comparisons mean high accuracy/AUROC or low FPR can coexist with poor sensitivity.No single favorable metric is sufficient to characterize the clinical error profile.Source: §7 · pp. 8–9
  • Marginal ethnicity, gender, and insurance summaries conceal materially different three-way subgroup behavior, including among better-supported intersections.Source: §7 · p. 9

A Adaptation procedure

Algorithm 1 gives the adaptation procedure used in the experiments.

A Adaptation procedure

B Intersectional subgroup results

The intersectional audit considers ethnicity–gender–insurance subgroups with at least five mortality-positive cases. Results are presented alongside subgroup support, with TPR–FPR profiles and sensitivity estimates with Wilson intervals.

B Intersectional subgroup results

Related papers

Loading 2610.01645v1…