PaperlayerView exact revision on arXiv

Source-linked AI summary

Translation of Black-Box Clinical Prediction Models into Standalone Transparent Nomograms: Temporal External Validation in Heart Transplantation

Henry Pigot, Paulo J. G. Lisboa, Sandra Ortega-Martorell, Ivan Olier, Joseph Mahon, Johan Nilsson

arXiv:2609.07610v1cs.LGstat.AP

TL;DR

Clinical prediction must contend with changing recipient profiles and treatment, while simplified nomograms may not retain a deployed model’s learned behavior or validation. PRiSM transfers source-model response shapes and lets outcomes select and weight them; in temporal external validation, all five baseline nomograms met the prespecified discrimination noninferiority criterion.

  • Problem

    Clinical decision support uses prediction for prognosis, treatment selection, and organ allocation, but temporal changes in recipient profiles and treatment complicate prediction, and simplified nomograms may carry over neither a deployed model’s learned behavior nor its validation.

    Source: Introduction · pp. 2–3

  • Method

    PRiSM transfers source-model response shapes while letting the observed outcome select and weight them in one instrument.

    Source: Discussion · p. 18

  • Results

    All five baseline PRiSM nomograms met the prespecified noninferiority criterion for discrimination in temporal external validation.

    Source: Discussion · pp. 15, 17

  • Takeaways & Limitations

    Two nomograms that met noninferiority were identified as candidates for deployment in their source models’ settings.

    Source: Discussion · p. 17

    Under aggressive sparsification, few Phase I responses may be retained, and those responses do not represent the source model’s higher-order behavior.

    Source: Discussion · p. 17

Abstract

from arXiv · show

We convert black-box clinical prediction models for tabular data into standalone nomograms that can be audited term by term. PRiSM (Partial Responses in Structured Models) takes the shape of each effect and interaction from the source model, not merely which variables mattered, and lets the outcome select and weight them. We tested this in 50,356 heart transplant recipients, with validation in a later era than training. Nomograms from all 5 source models - a public clinical risk score, logistic regression, neural networks, random forests and extreme gradient boosting - met a prespecified noninferiority criterion for discrimination before any further simplification, and generally preserved calibration and clinical net benefit. Those from the 3 machine-learning models showed no detectable difference in discrimination from de novo generalized additive and explainable boosting models, exceeded neural additive models, and carried fewer terms than the explainable boosting model. PRiSM is released as an open-source Python package.

Introduction

This study evaluates PRiSM, which translates trained tabular classifiers’ response shapes into standalone, auditable nomograms, in temporal external validation after heart transplantation. It assesses baseline translations across five source models alongside additive comparators and sparse refinement.

Introduction

  • Clinical prediction requires interpretability, calibration, and auditability, yet black-box models can learn nonlinear patterns and interactions at the cost of explainability.The introduction identifies temporal shifts in recipient profiles and treatment as an added challenge for predicting 1-year mortality.Source: Introduction · p. 2
  • PRiSM reads main-effect and pairwise-interaction shapes from any probabilistic black-box classifier, then uses the observed outcome to select and weight them in a standalone nomogram.It uses the source model's input-output behavior directly, without fitting an intermediate surrogate.Source: Introduction · pp. 3–4
  • An optional refinement relearns retained shapes from the outcome, preserving the selected variable-and-interaction scaffold rather than source shapes that no longer fit the shorter model.Source: Introduction · p. 4
  • All five PRiSM nomograms met the prespecified noninferiority criterion in temporal external validation before further simplification.Source: Introduction · p. 4

Methods

  • The temporal external-validation study assessed translation and optional PRN refinement across IMPACT, logistic regression, MLP, random forest, and XGBoost, with GAM, NAM, and EBM comparators.Source: Methods · pp. 4–5

Study design and population

  • The cohort included 50,356 adult recipients of isolated heart transplants from 2000–2022, split by transplant year into training (2000–2016), tuning (2017–2019), and external validation (2020–2022).The final temporal cohort was reserved for evaluation only.Source: Study design and population · p. 5

Predictor selection and preprocessing

  • 22 predictors were retained from 56 candidates after expert review of 1,293 registry variables and SHAP ranking; preprocessing was fitted on training data and applied unchanged to later cohorts.Continuous variables were scaled, categorical variables dummy encoded, and missing values imputed with MissForest to avoid information leakage.Source: Predictor selection and preprocessing · p. 6

Source model development

  • IMPACT was used as a pretrained score, while four other source models were trained and tuned for tuning-cohort AUROC; calibration was assessed secondarily without recalibration.Discrimination alone was the tuning objective to avoid confounding calibration comparisons.Source: Source model development · p. 6

PRiSM methodology

  • PRiSM uses functional ANOVA to extract univariate effects and bivariate interactions on the log-odds scale, averaging over observed distributions of other predictors.The resulting nomogram captures source-model behavior in one- or two-variable terms, not joint effects of three or more variables.Source: PRiSM methodology · p. 7
  • Outcome-guided LASSO selects and weights informative responses, distinguishing PRiSM from distillation; its standalone nomogram is not intended to reproduce source-model probabilities exactly.The decomposition uses source-model input-output behavior, while the observed outcome guides selection rather than matching source-model predictions.Source: PRiSM methodology · p. 7
  • Optional Phase II refinement re-estimates selected effects in a structured neural network restricted to the selected main effects and interactions, relearning their shapes from outcomes.The approach limits higher-order interaction complexity; the study also defined baseline and sparse LASSO penalties to trade source-model fidelity against term count.Source: PRiSM methodology · p. 8
  • The unreported Phase II baseline-lambda nomograms met noninferiority for all 5 source models, with term counts no greater than their corresponding baseline PRiSM nomograms.The paper says this variant was not reported because it neither retained the source model’s learned shapes nor provided the shortest nomogram.Source: PRiSM methodology · p. 9

Primary and secondary endpoints

  • Temporal external-validation AUROC noninferiority was defined as preserving at least 90% of source-model discrimination above chance, with success requiring the lower two-sided 95% confidence bound for T to exceed zero.Secondary endpoints included calibration slope and calibration-in-the-large; decision-curve analysis quantified net benefit across threshold probabilities.Source: Primary and secondary endpoints · p. 9

De novo additive comparators

  • Nomograms were compared with de novo GAM, NAM, and EBM models tuned for AUROC, using paired DeLong tests; without an equivalence margin, these comparisons can detect differences but not establish equivalence.Source: De novo additive comparators · p. 10

Cohort characteristics

  • The 50,356-recipient cohort was partitioned chronologically into training (2000–2016; n = 32,543), tuning (2017–2019; n = 8,558), and external validation (2020–2022; n = 9,255), with higher acuity and mechanical circulatory support use at transplantation alongside an allocation-policy revision.Pretransplant ICU admission increased from 29.5% in training to 55.1% in validation; intraaortic balloon pump use rose from 5.7% to 27.9%, and extracorporeal membrane oxygenation from 0.6% to 6.1%.Source: Cohort characteristics · p. 11

Predictive noninferiority and preservation of source model performance

  • All 5 baseline PRiSM nomograms met noninferiority, retaining more than 90% of source discrimination above chance, with 9–30 terms across models.Term counts were 9 for IMPACT, 21 for LR, 22 for RF, 26 for XGB, and 30 for MLP.Source: Predictive noninferiority and… · p. 11
  • Calibration was generally preserved, and decision-curve net benefit was similar to source models across 5% to 20% threshold probabilities.These calibration and decision-curve comparisons were descriptive, without formal between-model testing.Source: Predictive noninferiority and… · p. 12

Sparse PRN simplification

  • Sparse PRN nomograms had 5–11 terms and met noninferiority only for LR and XGB; Phase II recovered noninferiority lost through Phase I sparsification for 2 of 5 models.Phase II refinement changed retained-effect shapes, not which effects were retained.Source: Sparse PRN simplification · p. 12

Recovery of known model structure

  • LR response profiles were linear, whereas MLP, XGB, and RF profiles were nonlinear; IMPACT bilirubin and creatinine-clearance responses stepped at published thresholds.The thresholds were 17.1, 34.2, and 68.4 umol/L for bilirubin and 30 and 50 mL/min for creatinine clearance; selected categorical IMPACT predictors were recovered as defined groups.Source: Recovery of known model structure · p. 13

Comparator analysis

  • MLP-, XGB-, and RF-derived baseline nomograms were indistinguishable from EBM and GAM and all exceeded NAM, whose AUROCs were 0.647, 0.645, and 0.615, respectively.Sparse PRN MLP and XGB also exceeded NAM; the 8-term RF sparse PRN nomogram was 0.023 AUROC below EBM and 0.021 below GAM, and did not separate from NAM.Source: Comparator analysis · pp. 13–14

Sensitivity and subgroup analyses

  • Noninferiority conclusions were identical across seeds and agreed for 19 of 20 nomograms with and without transplant year; the exception missed zero by less than 0.001 AUROC.Source: Sensitivity and subgroup analyses · p. 14
  • Discrimination was lower in Black and Hispanic than White recipients across evaluated models, and PRiSM translation did not alter those differences; sex differences were at most 0.014 AUROC.No nomogram showed calibration heterogeneity across groups.Source: Sensitivity and subgroup analyses · p. 14

Discussion

  • All 5 baseline PRiSM nomograms met noninferiority, but no sparse PRN nomogram did; Phase II refinement recovered the criterion for 2 of 5 source models.The sparse refinements substantially reduced the number of terms.Source: Discussion · pp. 15, 17
  • Across the 5 source models, AUROC was 0.62 to 0.65; discrimination was lower in Black and Hispanic than White recipients in every evaluated model, unchanged by PRiSM translation.The authors note that missing biological, surgical, and institutional factors and changes in treatment practices may contribute to modest discrimination.Source: Discussion · pp. 15, 18
  • Baseline MLP, RF, and XGB nomograms captured nonlinear predictor effects, and the MLP nomogram exposed a creatinine-clearance interaction with recipient clinical status.Baseline nomograms retained 10 interactions for MLP, 8 for RF, and 7 for XGB; all sparse PRN nomograms retained none. These patterns are hypothesis-generating, not causal.Source: Discussion · pp. 16–17
  • Additive comparators discriminated in the same range, with no detectable difference between EBM and baseline or sparse MLP nomograms despite EBM’s 42 terms versus 30 and 10.The comparators were trained de novo and do not translate or replace an already trained source model.Source: Discussion · pp. 17–18
  • PRiSM combines response shapes from the source model with outcome-based selection and weighting in one standalone nomogram that can be inspected term by term.The study supplies temporal-validation evidence and an open-source Python implementation.Source: Discussion · pp. 15, 18
  • Limitations include single imputation, 57.0% missing serum albumin in the tuning cohort, a current restriction to tabular binary classification, and the need for international validation.Rare predictor combinations could distort term selection in principle, although retained source-model discrimination under temporal validation argues against a material effect here.Source: Discussion · p. 18
  • Prospective testing is needed to assess workflow integration, user behavior, and patient outcomes before clinical implementation is claimed.The registry's single binary sex field also means subgroup analysis by sex could not distinguish sex from gender.Source: Discussion · p. 19
  • Every baseline PRiSM nomogram met the prespecified noninferiority criterion across five source-model families in temporal external validation of 1-year mortality after heart transplantation.The criterion assessed fidelity to the source model, not absolute clinical adequacy.Source: Discussion · p. 19
  • Further simplification traded discrimination for brevity, and the regularization path makes that cost visible before a nomogram is selected.Source: Discussion · p. 19
  • Baseline nomograms retain source-model main effects and pairwise interactions but are fitted to observed outcomes rather than source-model predictions.A term-by-term readable, deployable model enables teams to investigate whether nomogram use changes their decisions.Source: Discussion · p. 19

Data and code availability

  • PRiSM is available as an open-source Python package, and all 20 study nomograms can be explored through an interactive web interface.The interface lets users adjust predictor values and inspect points and predicted 1-year mortality risk; clinical implementation was not evaluated.Source: Data and code availability · p. 20

Declaration of interests

The authors report no competing interests and describe using generative AI tools only to refine language and editorial consistency. They reviewed and edited the content and take responsibility for it.

Declaration of interests

process

  • ChatGPT, Gemini, and Claude were used to refine language and support editorial consistency.Source: process · p. 22
  • The authors reviewed and edited the content as needed and take full responsibility for the published article.Source: process · p. 22
  • Generative AI tools were not used to generate primary data, define outcomes, perform analyses, conduct independent inference, alter results, or generate figures.Source: process · p. 22

Figure legends

The figure legends describe PRiSM’s translation and refinement workflow and the assessment of its nomograms in temporal external validation. They also illustrate how Phase II refinement changes response shapes while retaining the Phase I variable-and-interaction scaffold.

Figure 1. Overview of the PRiSM framework

  • PRiSM decomposes a source model on the log-odds scale into predictor effects and selected pairwise interactions, then uses LASSO to select terms for a standalone nomogram.The selected effects define a Partial Response Network with one or two inputs per subnetwork; it is trained on cohort data and outcomes, then re-decomposed and reselected to produce a refined nomogram.Source: Figure 1. Overview of the PRiSM… · p. 27
  • Source and PRN models were trained in the training cohort (n = 32,543), decisions were made in the tuning cohort (n = 8,558), and evaluation used temporal external validation (n = 9,255).Source: Figure 1. Overview of the PRiSM… · p. 27
  • Validation used T = AUROC_nomogram - 0.90 × AUROC_source - 0.05; noninferiority required its 95% confidence interval entirely above zero, and decision curves covered 5%–20% thresholds.Decision-curve net benefit was plotted as a point estimate against treat-all and treat-none strategies.Source: Figure 1. Overview of the PRiSM… · p. 27

Figure 3. LASSO regularization paths for Phase I and Phase II

  • Phase I PRiSM bilirubin and creatinine-clearance responses step at IMPACT thresholds, recovering its published additive structure; Phase II yields a sparse MLP PRN nomogram.The nomogram scales predictor values into points and converts total points to predicted mortality risk.Source: Figure 1. Overview of the PRiSM… · p. 29

Tables

  • The MLP sparse PRN nomogram’s lower 95% confidence bound for retention contrast was -0.0003, although it rounded to 0.000 and did not exceed zero.Noninferiority was defined by a lower confidence bound for T above zero.Source: Tables · p. 32
  • The EBM reference AUROC was 0.647 (95% CI, 0.627 to 0.667; 42 terms, including 20 pairwise), and comparator differences cannot establish equivalence.No equivalence margin was set for these comparisons.Source: Table 3 · p. 32, Tables · p. 33
  • Figure 4 illustrates PRiSM and sparse PRN response curves for three continuous predictors, showing Phase II shape refinement within the Phase I-selected variable-and-interaction scaffold.The responses were derived from models fitted in the training cohort (n = 32,543); the predictors shown are total bilirubin, creatinine clearance, and ischemic time.Source: Figure 4

S1 Supplementary Methods

The supplementary methods describe a registry-based study with cohorts separated by transplant year, including a later external-validation period. They report cohort exclusions and leakage safeguards for model development and evaluation.

S1.1 Study Design Details

  • The study followed TRIPOD+AI reporting guidance, with adherence documented in a checklist.Source: S1.1 Study Design Details · p. 39
  • The final adult cohort included 50,356 recipients, or 60.5% of adult heart transplant recipients in the source dataset.Records with missing transplant dates or outcome status, and patients censored before 365 days, were excluded.Source: S1.1 Study Design Details · p. 39
  • Cohorts were assigned by transplant year: training in 2000–2016, tuning in 2017–2019, and external validation in 2020–2022.The cohorts were nonoverlapping, and the same assignments were used throughout the study.Source: S1.1 Study Design Details · p. 39
  • The leakage audit confirmed no patient overlap, reproducible year-based assignment, and no post-transplant predictors or future information.Scaling was based on training data, and model selection used tuning data while external validation remained held out until final evaluation.Source: S1.1 Study Design Details · p. 39

S1.2 Predictor Selection and Preprocessing

Predictors were selected through clinical review and training-cohort SHAP ranking, with 22 retained for trained models. Preprocessing included cohort-consistent imputation and transformations designed to allow category-specific effects.

S1.2 Predictor Selection and Preprocessing

  • 57 candidate variables were identified by expert clinical review from 1,293 registry variables, then one fully missing variable was removed, leaving 56 for imputation.Source: S1.2 Predictor Selection and… · p. 39
  • 22 predictors were retained for trained models after SHAP-based ranking on the training cohort.A separate XGBoost feature-selection model used grid search and five-fold cross-validation; transplant year was carried forward for split verification but not used as a predictor.Source: S1.2 Predictor Selection and… · p. 39
  • IMPACT was applied as a pretrained score with its own 12 predictor definitions; nine variables overlapped with the candidate set for trained models.IMPACT also used recipient sex, race or ethnic group, and intraaortic balloon pump, while trained models included other variables.Source: S1.2 Predictor Selection and… · pp. 39–40
  • Missing values were imputed with a single iterative procedure based on MissForest, trained on the training cohort and applied unchanged to later cohorts.The implementation used HistGradientBoostingRegressor instead of MissForest's original random-forest estimator.Source: S1.2 Predictor Selection and… · p. 40
  • Continuous predictors were median-centered and scaled, while binary variables remained 0/1 and multi-category variables were dummy encoded against reference categories.Dummy encoding allowed category-specific effects without assuming a monotonic ordering.Source: S1.2 Predictor Selection and… · p. 40

S1.3 Source Model Development

Five models for 1-year mortality were converted using PRiSM, with IMPACT as the pretrained clinical score and the other four trained on the study cohort. Hyperparameters were selected using tuning-cohort AUROC.

S1.3 Source Model Development

  • PRiSM conversion covered IMPACT, logistic regression, MLP, random forest, and XGBoost models for predicting 1-year mortality.IMPACT is a published clinical score; the other four models were trained on the training cohort.Source: S1.3 Source Model Development · p. 40
  • Logistic regression and MLP used PyTorch with Adam optimization and early stopping based on tuning-cohort loss.The MLP used one tanh hidden layer, with its size and other hyperparameters selected through Optuna.Source: S1.3 Source Model Development · p. 40
  • The tree models used XGBoost’s XGBClassifier for boosting and XGBRFClassifier for random forest, with model hyperparameters selected via Optuna.The random forest implementation used bagging with per-node column sampling.Source: S1.3 Source Model Development · p. 40
  • The four trained models used Optuna tuning, with trial counts scaled to each model’s search-space complexity and per-trial cost.The supplied passage specifies 256 trials for tree-based models, 128 for MLP, and 64 for logistic regression.Source: S1.3 Source Model Development · pp. 40–41

S1.4 PRiSM Methodology

PRiSM decomposes source-model log-odds into partial responses, selects and weights terms using outcomes, and can refine selected terms in a constrained network. The Lebesgue decomposition averages over the empirical distribution of remaining predictors, preserving their correlation structure.

S1.4.1 Mathematical Foundation

  • Functional ANOVA decomposes classifier log-odds into non-overlapping, generally nonlinear partial responses whose sum matches the model prediction.The retained terms are univariate main effects and bivariate interactions, trading interpretability against fidelity to higher-order interactions.Source: S1.4.1 Mathematical Foundation · p. 41

S1.4.2 Lebesgue ANOVA Decomposition

  • The Lebesgue method averages predictions over the empirical distribution of remaining predictors, preserving their correlation structure at quadratic computational scaling.The alternative Dirac method fixes other predictors at reference values, scales linearly with observations, and discards predictor correlations; all reported results used Lebesgue.Source: S1.4.2 Lebesgue ANOVA Decomposition · p. 41

S1.4.3 LASSO Partial Response Selection

  • LASSO selects partial responses from the univariate and bivariate terms, retaining terms with |β| > 0.1 as a practical near-zero coefficient filter.The λ regularization strength is the primary complexity control, with higher λ producing sparser models; categorical predictors are selected as a single feature and retain all category effects if selected.Source: S1.4.3 LASSO Partial Response Selection · p. 42

S1.4.4 Nomogram Prediction Calculation

  • The nomogram sums each selected partial response multiplied by its coefficient, then adds the intercept to obtain log-odds for conversion to probability.Individual contributions can be read from the nomogram and summed manually.Source: S1.4.4 Nomogram Prediction Calculation · p. 42

S1.4.5 Partial Response Network Refinement

  • Phase I preserves source-model response shapes while outcome labels determine selected terms and weights; Phase II re-learns shapes within the selected-term architecture.Phase II excludes higher-order interactions by construction, and its constrained re-estimation can remove contributions from unmodeled higher-order interactions present in Phase I.Source: S1.4.5 Partial Response Network… · p. 43
  • PRN refinement made tuning-cohort AUROC largely insensitive to λ, and refinement recovered noninferiority for 2 of the 5 source models.The PRN architecture prevents higher-order interactions beyond those selected in Phase I from entering; the sparse Phase I selection therefore retained a fixed term count during refinement.Source: S1.4.5 Partial Response Network… · p. 43

S1.5 Statistical Analysis

The statistical analysis assessed discrimination, calibration, overall accuracy, and clinical net benefit. Paired AUROC comparisons used DeLong methods, while calibration was assessed without validation-data recalibration.

S1.5 Statistical Analysis

  • Paired AUROC difference confidence intervals used DeLong covariance to account for the same patients being evaluated by a source model and its nomogram.The paired difference was defined as nomogram AUROC minus source-model AUROC.Source: S1.5 Statistical Analysis · p. 43
  • Calibration-in-the-large and calibration slope were assessed without recalibrating model probabilities on validation data.A calibration-in-the-large value of 0 indicates ideal level calibration; a slope of 1 indicates ideal spread, while values below 1 suggest prediction compression.Source: S1.5 Statistical Analysis · p. 44
  • The analysis also reported observed-to-expected ratios and Brier scores, with bootstrap percentile confidence intervals from 1,000 cohort resamples.An observed-to-expected ratio of 1.0 indicates perfect calibration-in-the-large; lower Brier scores indicate better overall accuracy.Source: S1.5 Statistical Analysis · p. 44
  • Decision curve analysis measured net benefit across 5–20% threshold probabilities for post-transplant mortality interventions.The analysis was used to assess whether noninferior discrimination translated to preserved clinical utility.Source: S1.5 Statistical Analysis · p. 44

S1.6 Software Implementation

PRiSM was released as an open-source Python package under the BSD 3-Clause licence, with support for cross-platform deployment.

S1.6 Software Implementation

  • PRiSM is an open-source Python package under the BSD 3-Clause licence, supporting Windows, macOS, and Linux.Batching, vectorization, and caching accelerate its computationally intensive Lebesgue ANOVA decomposition.Source: S1.6 Software Implementation · p. 44

S1.7 De Novo Additive Comparators

The de novo additive comparators used the same 22 predictors and additive function class, with differences in feature handling and ranking. They were tuned on the source-model cohort without using the external-validation cohort.

S1.7 De Novo Additive Comparators

  • All comparators used the same 22 predictors and spanned the same additive function class, while differing in feature handling and ranking.EBM and GAM represented each multi-category predictor as one feature; NAM used dummy-encoded columns, and EBM ranked 231 grouped-predictor pairs for interaction search.Source: S1.7 De Novo Additive Comparators · pp. 44–45
  • Comparator tuning used the source-model cohort and objective, while the external-validation cohort was never loaded during a search.GAM configurations were enumerated; EBM and NAM configurations were fitted across multiple seeds, and EBM's larger grid was sampled.Source: S1.7 De Novo Additive Comparators · p. 45
  • EBM included main effects and automatically detected pairwise interactions; sampling 150 configurations from a 4,050-point grid found no setting that separated from the selected configuration.The implementation used 128 bins (32 for interaction terms), 20 interactions, and 8 outer bags.Source: S1.7 De Novo Additive Comparators · p. 45
  • GAM used univariate smooths for continuous predictors and parametric terms for binary and categorical predictors; its 72-point grid displaced none of the selected settings.Smooths used cubic regression splines with basis dimension k = 5 and smoothing parameters estimated by REML.Source: S1.7 De Novo Additive Comparators · p. 45
  • NAM modeled main effects only, and its tuning selected the widest network searched before wider networks scored lower.Its 24-point grid varied per-feature layer width, additional hidden layers, and dropout.Source: S1.7 De Novo Additive Comparators · p. 45

S1.8 Sensitivity and Subgroup Analyses

Across seed repeats, PRiSM discrimination and noninferiority conclusions were stable, although some PRN term sets and calibration slopes varied. Transplant-year inclusion had little effect on discrimination but usually improved calibration, while exploratory subgroup analyses found limited evidence of calibration heterogeneity.

S1.8.1 Seed Stability of PRiSM Conversion

  • 0.003 was the maximum AUROC range across seeds for all 20 nomograms, far below the approximately 0.040 DeLong 95% confidence-interval width.Point estimates across seeds fell within one another's confidence intervals, and the three-seed ensemble AUROCs were consistent with individual seeds.Source: S1.8.1 Seed Stability of PRiSM… · p. 45
  • 20/20 noninferiority conclusions matched across three seeds and the ensemble; all 10 baseline nomograms passed, while 8 of 10 sparse variants failed.LR Sparse PRN and XGB Sparse PRN were the two sparse variants that consistently passed.Source: S1.8.1 Seed Stability of PRiSM… · p. 45
  • Phase I selected identical term sets across seeds for all five models, whereas Phase II PRN term selection varied modestly in three models.MLP PRN selected 19–26 terms, RF PRN 15–18, and LR PRN 17–19; borderline interactions explain MLP and RF variation, while LR varied at the selection boundary.Source: S1.8.1 Seed Stability of PRiSM… · p. 46
  • Sparse deployment candidates were especially stable: 8 of 10 had identical terms across seeds, and the other two differed by one term.The differing term was prior blood transfusions, included only in seed 42 for the two LR sparse nomograms.Source: S1.8.1 Seed Stability of PRiSM… · p. 46

S1.8.2 Sensitivity to Transplant Year Inclusion

  • Transplant-year inclusion had negligible discrimination effects: all 20 AUROC confidence intervals overlapped, with point estimates differing by at most 0.019.LR Sparse PRN changed from 0.625 to 0.644; IMPACT AUROCs were identical across steps.Source: S1.8.2 Sensitivity to Transplant Year… · p. 46
  • 19 of 20 noninferiority conclusions agreed with and without transplant year; the sole disagreement was borderline MLP Sparse PRN.Its lower 95% CI bound for T changed from −0.0003 without transplant year to +0.006 with it.Source: S1.8.2 Sensitivity to Transplant Year… · p. 46
  • Transplant year moved O:E ratios and CITL closer to target for most non-IMPACT nomograms, but calibration-slope changes had no consistent direction.The two sparse PRN exceptions were MLP and XGB; using transplant year also requires knowledge of the transplant era, constraining temporal transportability. Complexity usually changed by −1 to +2 terms, with PRiSM XGB gaining 9 and PRN XGB 6.Source: S1.8.2 Sensitivity to Transplant Year… · p. 46

S1.8.3 Exploratory Subgroup Analysis

  • Across subgroup analyses, mortality differed by less than 0.5 percentage points across White, Black, and Hispanic recipients; median CI half-width was 0.028 across 80 translation comparisons.Three of 112 unadjusted calibration-heterogeneity tests reached p < 0.05, none on a translated nomogram; the Other group had only 54 deaths and supported no finding of its own.Source: S1.8.3 Exploratory Subgroup Analysis · p. 47

S1.9 Supplementary Findings

In external validation, the IMPACT PRiSM nomogram had a calibration slope closer to one than its source model, while all models and translated nomograms overpredicted one-year mortality. The later validation era and absence of recalibration contextualize that overprediction.

S1.9 Supplementary Findings

  • 0.988 vs 0.602 calibration slope: the baseline PRiSM IMPACT nomogram was closer to one than the IMPACT source model.The decomposition re-expresses predictions as partial responses averaged over the training-cohort distribution, and LASSO reweighting implicitly adjusts prediction scale; this was not a separate validation-data calibration step.Source: S1.9 Supplementary Findings · p. 47
  • All external-validation O:E ratios were below 1.0 (0.642 to 0.917), indicating systematic overprediction of one-year mortality.The 2020–2022 validation cohort had lower observed mortality than the 2000–2016 training cohort; models were evaluated without recalibration.Source: S1.9 Supplementary Findings · p. 47

S1.10 Methodological Considerations

The methodological discussion identifies limits around imputation uncertainty, outcome-supervised translation, and marginal averaging with correlated predictors. It notes evidence bounding—but not quantifying—the practical concern from extrapolation.

S1.10 Methodological Considerations

  • Single iterative imputation was used for tractability, so uncertainty from alternative imputation draws was not evaluated.Serum albumin had 57.0% missingness in the tuning cohort and 25.9% in training; seed stability does not address imputation uncertainty because imputed data were held fixed.Source: S1.10 Methodological Considerations · p. 48
  • Outcome labels supervise both LASSO stages and PRN, so coefficients are not guaranteed to reproduce source-model risk estimates for individual patients.An outcome-independent alternative could preserve individual predictions but also propagate source-model miscalibration; comparing the approaches remains future work.Source: S1.10 Methodological Considerations · p. 48
  • Marginal averaging can evaluate correlated predictors at rare or absent combinations, where data do not constrain partial responses and term selection could be distorted.Density-weighted formulations were not implemented; they restrict decomposition to high-density regions but require estimating the joint predictor density.Source: S1.10 Methodological Considerations · p. 48
  • Fixing remaining predictors at reference values is not a remedy, because that single evaluation point may itself be unobserved.All five baseline PRiSM nomograms retained source-model discrimination under temporal external validation, but residual extrapolation effects were not quantified.Source: S1.10 Methodological Considerations · p. 48

S2 Supplementary Figures

The supplementary figures show PRiSM nomograms at baseline and after sparse refinement, alongside a worked example of converting predictor contributions into estimated 1-year mortality risk.

Sparse Phase I nomograms

  • Figure S1 shows sparse Phase I PRiSM nomograms for five model families, with predictor contributions and total points mapped to predicted 1-year mortality risk.The source models were fitted in 32,543 recipients and λ was selected in 8,558; points have no confidence intervals.Source: S2 Supplementary Figures · p. 50

Sparse PRN nomograms

  • Figure S2 presents sparse PRN nomograms for all five model families, with predictor contributions and a bottom scale mapping total points to predicted 1-year mortality risk.The nomograms were trained on 32,543 recipients and tuned on 8,558; estimates are point estimates without confidence intervals.Source: Figure S2 · p. 55

Baseline MLP nomogram

  • The complete baseline MLP nomogram displays all selected univariate and bivariate terms before sparse selection.Terms were selected at the baseline regularization strength preserving at least 99.8% of maximum tuning-cohort AUROC.Source: Figure S3 · p. 60

Worked risk calculation

  • A synthetic high-risk patient’s predictor contributions sum to +39.8 points, corresponding to an estimated 85.1% 1-year mortality risk.The example traces points from categorical markers or continuous response curves; it uses no cohort data.Source: Figure S4 · p. 65

S3 Supplementary Tables

The supplementary tables document reporting coverage, model configurations and implementation details, alongside discrimination and refinement analyses across cohorts. They also present robustness and subgroup analyses of noninferiority outcomes, including a rounded-boundary exception and stated limitations.

S3 Supplementary Tables

  • The TRIPOD+AI checklist maps reporting items to the main text or Document S1, while supplementary tables also cover sequential exclusions and discrimination analyses in training and tuning cohorts.The training-cohort table is related to Table 2; the tuning-cohort table is also related to Table 2.Source: Table S1 · p. 66, S3 Supplementary Tables · p. 66, Table S2 · p. 68
  • Supplementary tables specify standard training and model-specific hyperparameters, including a single-hidden-layer tanh MLP and an XGBoost binary-logistic classifier.The MLP used Adam, BCELoss, and Xavier uniform initialization; XGBoost used logloss as its evaluation metric.Source: Table S3 · p. 68, Table S4 · p. 68, Table S5 · p. 68
  • The pretrained IMPACT score was applied without tuning or training, and all VAD types were assigned 3 points because SRTR did not distinguish device subtypes.The original score assigns different values to pulsatile, continuous-flow, and HeartMate II devices.Source: S3 Supplementary Tables · p. 69, Table S8 · p. 69
  • PRN refinement used a LASSO-selected input mask, with hyperparameters tuned separately for each source model over 128 Optuna trials to maximize tuning-cohort AUROC.The accompanying tables report PRN configuration and tuned values by source model and sparse lambda.Source: S3 Supplementary Tables · p. 70, Table S10 · p. 70, Table S11 · p. 70
  • The training-cohort discrimination table defines T as the primary retention contrast but reports no noninferiority verdict, which was prespecified only for external validation.T = AUROC_nomogram − 0.90 × AUROC_source − 0.05; positive values indicate retention of more than 90% of source-model discrimination above chance.Source: S3 Supplementary Tables · p. 71, Table S12 · p. 71

Calibration and accuracy

  • Calibration tables define ideal CITL and O:E as 0 and 1, respectively, and ideal calibration slope as 1; lower Brier scores indicate better accuracy.The null-model Brier score, predicting cohort prevalence for all patients, was 0.084; categorical variables counted as one term in comparator and nomogram term counts.Source: S3 Supplementary Tables · p. 77, Table S16 · p. 77

Seed stability

Seed-specific results

  • The MLP Sparse PRN nomogram did not meet noninferiority in seed 123, although its lower confidence bound rounded to 0.000.Noninferiority required the lower bound for the retention contrast T to exceed 0.Source: S3 Supplementary Tables · p. 82, Table S22 · p. 82

Transplant year

Subgroup analyses

  • Subgroup analyses were exploratory, unadjusted for multiplicity, and evaluated models without recalibration; the Other race or ethnic group included 464 patients and 54 deaths.Recipient sex and race or ethnic group were not among the 22 predictors, except in the IMPACT score.Source: S3 Supplementary Tables · p. 84, Table S26 · p. 84

S4 Supplementary References

The supplementary references cover clinical prediction reporting, source-model methods, statistical evaluation, and interpretable machine-learning approaches.

Clinical prediction and reporting

Source-model and optimization methods

Statistical methods and evaluation

Black-box interpretability

Interpretable machine learning

Related papers

Loading 2609.07610v1…