PaperlayerView exact revision on arXiv

Source-linked AI summary

RMExplorer: A Visual Analytics Approach to Explore the Performance and the Fairness of Disease Risk Models on Population Subgroups

Bum Chul Kwon, Uri Kartoun, Shaan Khurshid, Mikhail Yurochkin, Subha Maity, Deanna G Brockman, Amit V Khera, Patrick T Ellinor, Steven A Lubitz, Kenney Ng

arXiv:2209.06378v1cs.HC

TL;DR

Risk models may behave differently across datasets and patient subgroups, yet researchers face difficulty inspecting performance and fairness across groups. RMExplorer lets users define subgroups and explore model metrics and feature contributions; its atrial fibrillation case study demonstrates the usefulness of these visualizations for generating clinically relevant findings.

  • Problem

    Researchers lack an easy way to inspect risk-model performance and fairness across subgroups, while fixed-subgroup reports do not support applying models to users’ own datasets or defining custom groups.

    Source: §1 · p. 2

  • Method

    RMExplorer combines interactive visualizations and computational approaches so users can define patient subgroups, assess model performance and fairness, and inspect feature contributions.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    The atrial fibrillation case study demonstrates the usefulness of RMExplorer’s interactive visualizations for generating clinically relevant findings.

    Source: §4 · p. 4

  • Takeaways & Limitations

    Researchers can use RMExplorer to evaluate risk-model performance and biases on subpopulations of interest in their own data.

    Source: §1 · p. 2

    The demonstrated fairness views use two group metrics and an individual fairness metric, and future work could investigate other fairness metrics.

    Source: §4 · p. 4

Abstract

from arXiv · show

Disease risk models can identify high-risk patients and help clinicians provide more personalized care. However, risk models developed on one dataset may not generalize across diverse subpopulations of patients in different datasets and may have unexpected performance. It is challenging for clinical researchers to inspect risk models across different subgroups without any tools. Therefore, we developed an interactive visualization system called RMExplorer (Risk Model Explorer) to enable interactive risk model assessment. Specifically, the system allows users to define subgroups of patients by selecting clinical, demographic, or other characteristics, to explore the performance and fairness of risk models on the subgroups, and to understand the feature contributions to risk scores. To demonstrate the usefulness of the tool, we conduct a case study, where we use RMExplorer to explore three atrial fibrillation risk models by applying them to the UK Biobank dataset of 445,329 individuals. RMExplorer can help researchers to evaluate the performance and biases of risk models on subpopulations of interest in their data.

1 INTRODUCTION

Risk models may generalize poorly across datasets and subpopulations, while biases in training data can contribute to unfair outcomes. Existing approaches make subgroup assessment computationally and cognitively demanding, motivating RMExplorer’s interactive exploration of model performance and fairness.

1 INTRODUCTION

  • Models trained on one dataset may not generalize to new datasets with different observation periods, demographics, or healthcare policies, and may propagate systematic biases.The text notes that biased medical decisions may disproportionately worsen outcomes for racial, gender, and socioeconomic groups.Source: §1 · p. 1
  • Fairness toolkits provide model-agnostic metrics, but generating measures for every possible subgroup combination remains computationally challenging.Analyzing the resulting measures also places a cognitive burden on users.Source: §1 · pp. 1–2
  • Fixed-subgroup model report cards do not let users apply models to their own datasets, define custom subgroups, or explore subgroup behavior interactively.Computing measures across all possible subgroup combinations is computationally challenging, and analyzing the results is cognitively demanding.Source: §1 · p. 2
  • Prior visual analytics work addressed fairness in other domains, but had not extensively investigated risk-model performance and fairness on healthcare data.Examples include ranking decisions [1] and image classification [15].Source: §1 · p. 2
  • RMExplorer lets researchers interactively examine risk-model performance and fairness across user-defined subgroups in their own datasets.The system is demonstrated with three validated atrial fibrillation risk models on the UK Biobank dataset.Source: §1 · p. 2

2 VISUAL ANALYTICS SYSTEM DESIGN

RMExplorer supports an iterative workflow for assessing model performance, subgroup fairness, and model behavior. Its coordinated views let users define patient subgroups, inspect risk and fairness metrics, and examine feature contributions and distributions.

2.1 Users, Tasks, and Requirements

  • Researchers iteratively assess risk scores and prediction accuracy, inspect biases across subpopulations, and compare metrics across risk models.The requirements were characterized by study authors from cardiology, machine learning, and visualization.Source: §2.1 · p. 2

2.2 Subgroup Builder

  • Subgroup Builder displays patient distributions across selected variables and forms mutually exclusive groups from combinations of their levels.Empty groups are discarded; selecting one variable with five discrete levels produces five subgroups.Source: §2.2 · p. 2

2.3 Risk Score Distribution

  • Users supply previously trained models that output risk scores and disease probabilities over a follow-up window, and RMExplorer computes metrics for selected patients.Metrics are computed on the fly using the model covariates.Source: §2.3 · p. 2

2.4 Subgroup Summary: Performance

  • Performance views compare subgroup calibration slopes and concordance indices, and show Kaplan-Meier disease-free survival curves over time.The scatterplot maps calibration slope against concordance index; comparative mode displays multiple models across and within subgroups.Source: §2.4 · p. 2

2.5 Subgroup Summary: Fairness

  • Fairness views let users choose two metrics at a time, including group disparities and individual fairness violation rates across subgroups.Supported group metrics include statistical parity difference and true positive rate difference; zero indicates equal performance for the specified protected attribute.Source: §2.5 · pp. 2–3

2.6 Model Behavior Explanations

  • Model Behavior Explanations combine a SHAP beeswarm plot with feature distributions to help users inspect why predictions differ for a selected subgroup.The beeswarm encodes feature values by color and SHAP values by horizontal position; Parallel Trends summarizes feature central tendency and dispersion across subgroups.Source: §2.6 · p. 3

3 CASE STUDY: ATRIAL FIBRILLATION RISK MODELS

The case study applies three atrial fibrillation risk models to UK Biobank and uses RMExplorer to examine performance, fairness, feature contributions, and threshold changes across income subgroups. The analyses reveal differences both among income groups and among models.

3 CASE STUDY: ATRIAL FIBRILLATION RISK MODELS

  • The case study used 445,329 UK Biobank patients, including 7,407 incident AF cases within five years, and divided reported household income into five ranges.The cohort excluded people under 45, those with prevalent AF, and those missing baseline height, weight, or blood-pressure data; people who did not report income were also excluded.Source: §3 · p. 3
  • Lower-income groups had lower EHR-AF concordance and less ideal calibration; fairness differences also appeared across income subgroups.The fairness plots show negative sex disparities across all subgroups and greater race statistical-parity disparities in lower-income groups.Source: §3 · p. 3, Figure 2 · p. 3
  • C2HEST had lower performance across income subgroups and individual-fairness violation rates near 0.9, versus close to 0 for the other models.The comparison included EHR-AF, CHARGE-AF, and C2HEST.Source: §3 · p. 4, Figure 3 · p. 4
  • Age was the most influential EHR-AF risk-estimation factor across income subgroups, and lower-income groups had a higher mean age.Age, sex, weight, and height had nonzero SHAP values; lower-income groups also had lower height, while mean weight was consistent across groups.Source: §3 · p. 4
  • Changing the 5-year risk threshold from 0.05 to 0.08 altered fairness metrics differently across models and income subgroups.For EHR-AF, female true-positive-rate differences became higher than or equal to male rates in subgroups 1–4, and the race statistical-parity gap narrowed; CHARGE-AF showed both wider and narrower spreads depending on the metric.Source: §3 · p. 4, Figure 4 · p. 4

4 DISCUSSION AND FUTURE WORK

RMExplorer supports user-defined subgroup assessment of risk-model performance, fairness, and behavior. The discussion identifies current scope boundaries and directions for extending subgroup comparison, fairness measures, explanations, and evaluation.

4 DISCUSSION AND FUTURE WORK

  • RMExplorer supports customizable mutually exclusive subgroups and uses polygons in scatter plots to compare multiple risk models.The authors note that the polygon approach may need improvement for a greater number of subgroups.Source: §4 · p. 4
  • The demonstrated performance visualization uses a survival curve, while the authors say the same strategy can apply to other prediction models with metrics such as AUC.The current fairness plots show two group fairness metrics and an individual fairness metric; future work could examine other fairness metrics.Source: §4 · p. 4
  • RMExplorer uses SHAP plots and feature distributions to explain subgroup behavior; other explanation approaches, including LIME, could be added with additional coding.For attention-based deep-learning models, visualizations would need to be adapted to show attention scores for relevant input features.Source: §4 · p. 4
  • The authors identify evaluation against other visual analytics systems, such as FairSight, as future work.Source: §4 · p. 4

A APPENDIX

The appendix includes a diagram of dataset-column categories, an illustration of comparative scatter plots, and a table about UK Biobank population selection for incident atrial fibrillation.

A APPENDIX

  • Figure A.1 illustrates three dataset-column categories: predictor variables, subgroup variables, and protected attributes.Source: Figure A.1 · p. 6
  • Figure A.2 illustrates comparative scatter plots, where subgroup-colored polygons represent different risk models across two continuous measures.Each polygon vertex represents a different risk model.Source: Figure A.2 · p. 6
  • Table A.1 concerns population selection from the UK Biobank for incident atrial fibrillation.Source: Table A.1 · p. 6

A.1 Individual Fairness

The individual fairness audit searches for similar individuals whose predicted risk scores fall on opposite sides of a user-specified threshold. Similarity is measured using a fair distance that excludes differences in a sensitive subspace.

A.1 Individual Fairness

  • Fair distance treats individuals as similar when they are similar outside the sensitive subspace, whose directions are derived from logistic-regression coefficients predicting sensitive attributes.Source: §A.1 · p. 6
  • The audit seeks similar individuals with predicted risk scores on opposite sides of the user-specified threshold, and reports the fraction of individuals for whom such a fairness violation is found.The procedure aims to find similar individuals while maximizing prediction loss.Source: §A.1 · p. 6
  • For high-risk individuals the audit minimizes predicted risk plus a distance penalty; for others it maximizes predicted risk minus that penalty.Source: §A.1 · p. 6

A.2 Calculation of Cardiovascular Risk Models

The three cardiovascular risk models calculate patient scores from covariate values and model coefficients, then use baseline hazard and mean covariate estimates to compute 5-year estimated risk.

A.2 Calculation of Cardiovascular Risk Models

  • Each patient’s score is the sum of the products of predictor-covariate values and their coefficients.Table A.3 presents coefficients by predictor variable and model for EHR-AF, CHARGE-AF, and C2HEST; a dash indicates a variable was not used for that score.Source: §A.2 · p. 6, Table A.3 · p. 7
  • 5-year estimated risk is computed using the baseline hazard and mean covariate estimates; c represents average AF-free survival probability at 5 years.Source: §A.2 · p. 6
  • The predictor coefficients belong to EHR-AF, CHARGE-AF, and C2HEST, whose model details are described in their respective papers.Source: §A.2 · p. 7, Table A.3 · p. 7

Related papers

Loading 2209.06378v1…