Source-linked AI summary
Stability of clinical prediction models developed using statistical or machine learning methods
Richard D Riley, Gary S Collins
TL;DR
Clinical prediction models may be unstable because their estimated risks depend on the development sample and modelling strategy, especially when data are limited. The paper defines four stability levels and uses simulations, case studies, and bootstrap-based assessments to examine them, finding considerable instability that can manifest as miscalibration and undermine reliability for individuals.
Problem
Small development datasets and modelling choices can make prediction models and estimated risks unstable, threatening their reliability for clinical use.
Method
The paper uses simulations and case studies, defines four levels of prediction stability, and proposes bootstrap models with instability plots and measures for development-stage assessment.
Results
Instability in estimated risks is often considerable and can manifest as miscalibration of predictions in new data.
Takeaways & Limitations
Researchers should examine prediction instability during model development to assess whether predictions are likely to be reliable in new individuals.
Takeaways & Limitations
The bootstrap process only examines instability in the model development data and does not address all sources of uncertainty.
Abstract
from arXiv · showhide
Clinical prediction models estimate an individual's risk of a particular health outcome, conditional on their values of multiple predictors. A developed model is a consequence of the development dataset and the chosen model building strategy, including the sample size, number of predictors and analysis method (e.g., regression or machine learning). Here, we raise the concern that many models are developed using small datasets that lead to instability in the model and its predictions (estimated risks). We define four levels of model stability in estimated risks moving from the overall mean to the individual level. Then, through simulation and case studies of statistical and machine learning approaches, we show instability in a model's estimated risks is often considerable, and ultimately manifests itself as miscalibration of predictions in new data. Therefore, we recommend researchers should always examine instability at the model development stage and propose instability plots and measures to do so. This entails repeating the model building steps (those used in the development of the original prediction model) in each of multiple (e.g., 1000) bootstrap samples, to produce multiple bootstrap models, and then deriving (i) a prediction instability plot of bootstrap model predictions (y-axis) versus original model predictions (x-axis), (ii) a calibration instability plot showing calibration curves for the bootstrap models in the original sample; and (iii) the instability index, which is the mean absolute difference between individuals' original and bootstrap model predictions. A case study is used to illustrate how these instability assessments help reassure (or not) whether model predictions are likely to be reliable (or not), whilst also informing a model's critical appraisal (risk of bias rating), fairness assessment and further validation requirements.
1 Introduction
Clinical prediction models can become unstable when development samples are small or model complexity and analysis choices promote overfitting. The paper frames stability checks as a routine development-stage assessment because instability can make predictions unreliable and threaten internal and external validity.
- A developed prediction model depends on its development sample, predictors, modelling framework, penalization, and handling of missing values.
- Predictions are more likely to be unreliable when development data are small, model complexity is high relative to events, or overfitting is not adjusted.
- Many prediction models use such approaches, raising concern that their predictions may be unreliable and not fit for purpose.
- Model instability means that developing the model in another sample of the same size from the same population could produce a substantially different model.
- Instability in predictions makes estimated risks depend heavily on the development sample and modelling strategy, threatening internal and external validity.
- The paper uses simulations and case studies to demonstrate instability and explain development-stage stability checks, including instability plots and measures.
2 Defining stability of clinical prediction models
The paper defines four levels of stability in estimated risks, moving from population means and distributions to subgroups and individuals. Simulations show that instability increases as development samples shrink and can remain severe for individual predictions even when population-level measures appear stable.
- The four stability levels are the mean, distribution, subgroups, and individuals, with individual-level stability typically most important for individualized decisions.
- Level 1: Stability in a model’s mean estimated risk: With sample sizes of 50, 100, and 5000, the 95% range of models’ mean estimated risks was 0.36 to 0.64, 0.42 to 0.58, and 0.49 to 0.51, respectively.
- Level 2: Stability in a model’s distribution of estimated risks: Calibration-curve variability increased as sample size decreased, with substantial spread at small sizes and greater stability once at least 385 participants were used.
- Level 2: Stability in a model’s distribution of estimated risks: MAPE variability increased as sample size decreased and began stabilizing from 385 participants; average MAPE was higher for the full model, whereas instability was generally larger for LASSO.
- Level 4: Stability in a model’s predictions for individuals: For one individual, estimated risks ranged from 0.14 to 0.85 across example models, making the prediction hugely unstable and potentially altering clinical decisions and risk communication.
- Level 4: Stability in a model’s predictions for individuals: Minimum sample-size calculations may target level 1 and 2 stability without ensuring stability at subgroup and individual levels.
3 Quantifying instability in model development studies
Because true individual risks are unavailable in practice, the paper uses bootstrap refitting to quantify instability in predictions from a developed model. It recommends prediction, calibration, and MAPE-based assessments, while noting practical compromises and the limits of aggregate summaries.
- 3.1 Bootstrap assessment of instability: Bootstrap assessment replaces unavailable individual true risks with repeated model development on resampled versions of the development dataset.Each bootstrap sample should match the original sample size and reproduce the original model-building strategy as closely as practical.
- Practical considerations: Some model-building steps, including nonlinear-function selection or multiple-imputation handling, may require compromises to implement across bootstrap samples.The smoothing bandwidth for prediction ranges is also subjective and example-specific, with values between 0.2 and 0.8 generally found sensible.
- 3.2.1 Prediction instability plot: Prediction instability plots place bootstrap predictions on the y-axis against original predictions on the x-axis, optionally showing individual 95% ranges.These ranges use the 2.5th and 97.5th percentiles of bootstrap predictions and may be smoothed across individuals.
- 3.1 Bootstrap assessment of instability: The bootstrap process stores original and bootstrap-model predictions for each individual, then summarises instability with prediction plots, calibration plots, and MAPE.The suggested procedure uses at least 200 bootstrap iterations.
- 3.2.2 Calibration instability plot: Wider spreads among bootstrap calibration curves indicate greater instability and a greater threat of model miscalibration in the actual population.The curves are overlaid with the original model’s calibration curve in the original data.
- 3.2.3 Mean absolute predictor error (MAPE): MAPE is the mean absolute difference between bootstrap and original predictions, but average MAPE can mask larger or smaller instability across individuals or risk regions.The paper therefore recommends accompanying average MAPE with an individual-level MAPE instability plot.
- 3.2.3 Mean absolute predictor error (MAPE): Individual-level instability can be severe, and MAPE instability plots help identify where instability is most concerning across original predicted risks.The plot places individual MAPE values on the y-axis and original model risk on the x-axis.
4 Investigation of stability in various case studies
Case studies show that prediction instability increases with small samples, many candidate predictors, and some modelling approaches, threatening reliability in new data. Bootstrap instability assessments distinguish relatively stable models from models needing further validation.
- Study design: The case studies use bootstrap resampling to examine how prediction instability changes with sample size and modelling approach.They use the GUSTO-I dataset and repeatedly develop models in bootstrap samples.
- Sample size: 0.0027 average MAPE indicated very low individual-prediction instability in the large-sample logistic-regression scenario.Bootstrap calibration curves and individual risk predictions also showed low variability.
5 Further role of stability assessments
Stability assessments extend beyond individual risks to fairness, discrimination, decision curves, classification, and explainability. These assessments can reveal uncertainty that aggregate performance measures or single fitted models may conceal.
- Fairness: Instability assessments can support fairness evaluation by comparing prediction stability across subgroups such as males and females.In one example, average MAPE was 0.027 for females and 0.017 for males, although the stability difference was considered small.
- Fairness: Subgroups with substantially smaller sample sizes may show greater stability differences, creating a boundary for fairness assessment.The paper highlights under-represented or marginal groups as an important application context.
- Discrimination: Bootstrap C-statistics quantify discrimination instability, with greater variability indicating greater instability concern.For the example LASSO model, C-statistics across 1000 bootstrap samples ranged from about 0.74 to 0.79.
- Clinical utility: Decision-curve instability was tiny with a large sample but more pronounced in the small-sample setting.Net benefit remained above zero across all curves from 0 to 0.2, whereas curves spanned above and below zero from 0.3 to 0.5.
- Classification: The classification instability index measures the proportion of bootstrap models that classify an individual differently from the original model.Small samples may produce large classification instability even for predictions far from the threshold.
- Explainability: Unstable predictions make individual-level explanations such as LIME and SHAP unstable and therefore likely misleading.This links prediction stability to the interpretability of machine-learning models in healthcare.
6 Discussion
The discussion argues that clinical prediction models should undergo routine stability checks after development, because instability can undermine reliability for new individuals and complicate critical appraisal. Bootstrapping-based plots and measures can assess stability, but external validation remains necessary for other populations.
- Scope of validation: Bootstrapping examines instability within the population from which the development dataset was sampled, not performance in different populations.Evaluation in other populations requires external validation using sufficiently large samples from those populations.
- Levels of stability: Four stability levels move from overall or population-level stability to subgroup and individual-level stability, with at least levels 1 and 2 considered a minimum.Because models guide individual decisions, the discussion also argues for stability at levels 3 and 4, although this requires very large samples.
- Practical constraints: Very large samples may be needed for subgroup and individual stability, but rare outcomes and costly prospective studies can make this difficult.Data sharing and individual participant data meta-analysis may help, while stability checks remain recommended regardless of sample size.
- Clinical relevance: Stability requirements should reflect clinical decision-making, since instability in risk ranges that affect decisions may matter more than instability elsewhere.Focusing only on a narrow threshold range is subjective and incomplete because risk thresholds can vary across individuals and settings.
- Interpreting instability ranges: The proposed instability ranges are not confidence intervals for an individual's true risk, because no fitted model is itself true.They instead describe the range of predictions produced by bootstrap versions of the particular developed model.
- Routine stability assessment: Clinical prediction models should routinely report instability plots and measures after development to assess whether predictions are likely reliable in new individuals.These assessments can also help stakeholders judge model reliability and support systematic and peer reviewers' critical appraisal, including risk-of-bias assessment.