PaperlayerView exact revision on arXiv

Source-linked AI summary

Fair Conformal Predictors for Applications in Medical Imaging

Charles Lu, Andreanne Lemay, Ken Chang, Katharina Hoebel, Jayashree Kalpathy-Cramer

arXiv:2109.04392v3eess.IVcs.CVcs.LG

TL;DR

Medical AI deployment is limited by opaque uncertainty estimates, clinician trust concerns, and algorithmic bias. The paper combines clinician interviews with conformal prediction experiments and subgroup-specific calibration for skin-lesion classification. Group conformal predictors reduce coverage disparity and better represent uncertainty across differing skin-tone subgroups.

Abstract

from arXiv · show

Deep learning has the potential to automate many clinically useful tasks in medical imaging. However translation of deep learning into clinical practice has been hindered by issues such as lack of the transparency and interpretability in these "black box" algorithms compared to traditional statistical methods. Specifically, many clinical deep learning models lack rigorous and robust techniques for conveying certainty (or lack thereof) in their predictions -- ultimately limiting their appeal for extensive use in medical decision-making. Furthermore, numerous demonstrations of algorithmic bias have increased hesitancy towards deployment of deep learning for clinical applications. To this end, we explore how conformal predictions can complement existing deep learning approaches by providing an intuitive way of expressing uncertainty while facilitating greater transparency to clinical users. In this paper, we conduct field interviews with radiologists to assess possible use-cases for conformal predictors. Using insights gathered from these interviews, we devise two clinical use-cases and empirically evaluate several methods of conformal predictions on a dermatology photography dataset for skin lesion classification. We show how to modify conformal predictions to be more adaptive to subgroup differences in patient skin tones through equalized coverage. Finally, we compare conformal prediction against measures of epistemic uncertainty.

Introduction

Clinical deployment of medical AI is hindered by opaque uncertainty estimates, limited clinician trust, safety risks, and algorithmic bias. This paper explores conformal predictors as an intuitive, statistically grounded complement to deep learning, including subgroup-adaptive coverage for skin-lesion classification.

Problem

  • Clinical machine learning needs ways to identify low-confidence cases for review, while limited transparency reduces clinicians’ willingness to trust AI autonomously.Radiologists may value AI yet remain unwilling to use it for autonomous clinical decisions.Source: Introduction · p. 1

Risks

  • Deep learning models are difficult to interrogate and vulnerable to attacks, while medical imaging applications also face documented algorithmic bias.These concerns include skin-lesion classification and other applications involving protected or visually diverse populations.Source: Introduction · p. 1

Approach

  • Conformal predictors provide prediction sets that express uncertainty intuitively, can complement existing models, and offer formal statistical guarantees.Their set-based outputs resemble differential diagnoses and convey uncertainty through the number of included items.Source: Introduction · p. 1

Study design

  • The study uses clinician interviews to characterize rule-in and rule-out use-cases and evaluates subgroup-adaptive conformal prediction for skin-lesion classification.The evaluation uses Fitzpatrick skin type as the subgroup attribute and compares conformal uncertainty with epistemic uncertainty.Source: Introduction · pp. 1–2

Related Work

Prior work frames fairness as subgroup disparity and uncertainty as either model-parameter uncertainty or prediction calibration. Conformal prediction offers a model-agnostic, distribution-free alternative with formal coverage guarantees, though fairness criteria and clinical assumptions can be difficult to satisfy.

Fairness in Machine Learning

  • Fairness metrics target parity across demographic groups, but not all criteria are mutually satisfiable and many require measurable outcomes for every subgroup.These requirements may be impractical in clinical settings.Source: Fairness in Machine Learning · p. 2
  • Common bias-mitigation strategies may require training access or explicit distributional assumptions that proprietary medical-device software does not provide.This constrains the practical applicability of methods that modify model training or depend on known distributions.Source: Fairness in Machine Learning · p. 2

Uncertainty for Deep Learning

  • Epistemic uncertainty estimates uncertainty in model parameters using approaches such as Bayesian approximations, Monte-Carlo dropout, ensembles, or predictive entropy.Predictive entropy can be computed by averaging entropy across sampled predictions and classes.Source: Uncertainty for Deep Learning · p. 2
  • Post-hoc calibration aligns confidence with empirical correctness, but deep neural networks are often poorly calibrated and calibration methods may require model access or distributional assumptions.Platt Scaling calibrates softmax outputs using a learned parameter fitted on validation data.Source: Uncertainty for Deep Learning · p. 2, Conformal Predictions · p. 2

Conformal Predictions

  • Conformal prediction is model-agnostic, distribution-free uncertainty quantification with formal marginal coverage guarantees for prediction sets.At miscoverage level α, the method estimates a score quantile on a held-out IID set so the true class is included at the desired average rate.Source: Conformal Predictions · p. 2
  • A conformal predictor converts class scores into a set of candidate labels whose average coverage is controlled, rather than returning only the highest-scoring class.The framework applies to classifiers and has also been extended to regression and multi-label prediction.Source: Conformal Predictions · pp. 2–3

Field Interviews

Interviews identified two complementary clinical uses for conformal predictions: confidently ruling out critical conditions and ruling in patients needing urgent intervention. Clinicians also emphasized interpretable confidence displays and subgroup monitoring for bias.

Clinician preferences

  • Clinicians preferred semantically meaningful confidence categories or withholding low-confidence outputs over exposing raw percentages alone.Suggested interface labels included “highly confident,” “cautionary,” “uncertain,” and “non-diagnostic.”Source: Field Interviews · p. 3

Clinical use-cases

  • Conformal prediction can support ruling out hemorrhagic stroke before TPA, where sensitivity should take priority because treatment could worsen bleeding.The interview-derived use-case targets a contraindication to clot-breaking treatment.Source: Field Interviews · p. 3, Figure 1 · p. 3
  • For emergency triage, high-specificity identification of normal cases could help clinicians discharge low-risk patients without waiting for radiology.This use-case reflects limited emergency-department bed space and the need to expedite patients without severe conditions.Source: Field Interviews · p. 3, Figure 2 · p. 3
  • Conformal predictors are also suited to ruling in critical conditions because prediction sets can represent many conditions and their interactions.Some conditions may be low concern separately but more concerning together.Source: Field Interviews · pp. 3–4

Fairness and monitoring

  • Interview feedback also supported auditing dashboards that display demographic metadata and monitor changing disease prevalence to detect and recalibrate subgroup bias.Age and race were cited as clinically relevant demographics for monitoring.Source: Field Interviews · p. 4

Methods

The methods extend distribution-free conformal prediction to subgroup-specific coverage by estimating separate calibration thresholds for known groups. The resulting group adaptive prediction sets apply the relevant threshold at inference time.

Equalized coverage

  • Equalized coverage adapts conformal prediction to subgroup differences by calibrating prediction sets separately for each known group.This approach is intended to address differences between subgroups more effectively than aggregate conformal methods.Source: Methods · p. 4

Fairness motivation

  • The fairness rationale is to preserve clinically relevant prevalence differences while monitoring disparity across patient cohorts without relying on statistical parity alone.The paper frames prediction sets as providing patient- and group-level intuition for triage and screening.Source: Methods · p. 4

Group adaptive prediction sets

  • Group adaptive prediction sets use a scoring function that ranks classes by softmax score and accumulates scores through the true class.The method operates on examples containing features, group attributes, and target classes.Source: Methods · p. 4
  • Algorithm 1 computes a separate empirical quantile for each group from its held-out calibration examples, then applies the test example’s group-specific quantile.The procedure returns a conformal prediction set for the test case.Source: Methods · p. 4

Coverage guarantee

  • The coverage guarantee assumes samples are drawn interchangeably from a potentially unknown distribution and bounds subgroup marginal coverage near 1 − α.The supplied expression includes a finite-sample correction of 1/(n+1).Source: Methods · p. 4

Experiments

The experiments compare five prediction-set methods across rule-in and rule-out skin-lesion use cases, using Fitzpatrick skin type to assess subgroup behavior. They evaluate marginal coverage, set size, and disparities between subgroups on the Fitzpatrick17k dataset.

Experiments

  • Five methods are compared: Naive, APS, RAPS, GAPS, and GRAPS, across rule-in and rule-out malignant-lesion classification use cases.The experiments also examine how group conformal predictors compare with aggregate predictors and how prediction-set size relates to epistemic uncertainty.Source: Experiments · p. 4

Skin Lesion Dataset

  • Fitzpatrick17k contains 16,577 dermatology images covering 114 conditions, aggregated into non-neoplastic, benign, and malignant categories.Each image has an ordinal Fitzpatrick skin-type label, which is used as the subgroup attribute.Source: Skin Lesion Dataset · pp. 4–5, Figure 3 · p. 5, Table 2 · p. 5
  • The rule-in task evaluates whether critical malignant conditions are included in prediction sets, whereas rule-out evaluates whether non-critical cases exclude all 11 malignant conditions.The 11 malignant conditions are treated as critical and the other 103 conditions as non-critical.Source: Skin Lesion Dataset · p. 5

Evaluation metrics

  • Marginal coverage measures how often the true class appears in a prediction set, while set size measures the average number of predicted classes.Coverage disparity and set-size disparity summarize average pairwise differences between subgroups.Source: Evaluation metrics · p. 5

Implementation Details

  • A ResNet-18 classifier is averaged across five initializations, with Monte Carlo dropout and calibrated softmax scores used to estimate epistemic-uncertainty measures.The non-conformal baseline forms prediction sets by sorting softmax scores until reaching the 1 − α threshold.Source: Implementation Details · p. 5

Results

Group and aggregate conformal methods perform similarly in subgroup accuracy and set size at α = 0.1, while group methods reduce coverage disparity across skin types. Group methods also produce stronger subgroup separation in set size and lower correlation with epistemic uncertainty.

Subgroup accuracy

  • At α = 0.1, all four prediction-set methods perform similarly in subgroup accuracy and set size for the rule-in and rule-out use cases.GAPS and GRAPS perform slightly better for ruling out malignant conditions and worse for skin types 1 and 2 in the reported comparisons.Source: Results · p. 5, Figure 4 · p. 6

Coverage and set size

  • GAPS and GRAPS achieve lower subgroup coverage disparity but higher set-size disparity than APS and RAPS across Fitzpatrick skin types.Figure 5 compares coverage and set size across α values, with colors denoting Fitzpatrick skin types.Source: Results · p. 5, Figure 5 · p. 6

Epistemic uncertainty

  • Group conformal methods show greater separation between Fitzpatrick subgroups and lower Spearman correlation with epistemic uncertainty than Naive, APS, and RAPS.The separation is especially evident for skin types 1 and 6 across the reported α values.Source: Results · p. 5

Discussion

The study develops group conformal predictors for subgroup-aware uncertainty and evaluates their relationship to epistemic uncertainty in dermatology imaging. These methods reduce coverage disparity and better represent differences across Fitzpatrick skin types.

Discussion

  • The interviews identified transparency-oriented clinical uses for conformal predictions, including identifying possible clinical mimics and supporting two clinical decision-making use-cases.These use-cases motivated the subgroup-aware evaluation on a heterogeneous skin-lesion dataset.Source: Discussion · p. 6
  • Group conformal predictors have lower coverage disparity than aggregate conformal methods while showing greater subgroup-level variation in prediction-set size.Skin types 1 and 6 have larger sets; skin type 6 represents about 4% of the dataset, while lighter skin tones have higher malignancy rates.Source: Discussion · p. 6
  • Group conformal uncertainty is less correlated with maximum softmax probability and predictive entropy than aggregate conformal uncertainty.This pattern indicates greater adaptiveness to subgroup differences across task difficulty levels.Source: Discussion · p. 6, Table 3 · p. 7
  • Group methods show greater separation between Fitzpatrick subgroups than naive and aggregate conformal methods in Figure 6.The separation is especially visible for skin types 1 and 6 and for both malignant and non-malignant conditions.Source: Figure 6 · p. 7, Discussion · p. 7

Conclusion

The conclusion presents fair conformal predictors as a model-agnostic way to express uncertainty across clinically relevant subpopulations. The authors report lower disparity and improved subgroup sensitivity as promising directions for clinical usability and trustworthiness.

Conclusion

  • Fair conformal predictors provide meaningful uncertainty across clinically relevant subpopulations without modifying existing deep-learning training procedures.The authors describe these distribution-free methods as potentially increasing clinician trust and clinical usability.Source: Conclusion · p. 7
  • Group conformal predictors better describe subgroup uncertainty than regular conformal methods when subgroups come from different data distributions.The conclusion is supported by reduced correlation with epistemic uncertainty measures and increased separation across lightest and darkest skin tones.Source: Table 4 · p. 7, Conclusion · p. 7
Loading 2109.04392v3…