PaperlayerView exact revision on arXiv

Source-linked AI summary

cutpointr: Improved Estimation and Validation of Optimal Cutpoints in R

Christian Thiele, Gerrit Hirschfeld

arXiv:2002.09209v1stat.CO

TL;DR

Sample-based optimal cutpoints can be highly variable and overestimate out-of-sample performance. cutpointr addresses this with robust estimation and validation methods, whose effectiveness depends on data distribution, sample size, and separation.

  • Problem

    Sample-based cutpoint optimization can produce highly variable cutpoints and overestimate diagnostic utility and out-of-sample performance.

  • Method

    cutpointr combines bootstrap validation and smoothing methods with flexible metrics, cutpoint controls, scalable computation, and workflow integration.

  • Results

    The most efficient estimation method depends on distribution, sample size, and separation: Normal performs best for normal data, while bootstrapping or GAM perform best for non-normal data.

  • Takeaways & Limitations

    The package offers methods intended to produce cutpoints that are more stable and perform relatively well in the population rather than only in the specific sample.

  • Takeaways & Limitations

    LOESS can undersmooth, spline smoothing requires tuning, and midpoint-related bias is mainly relevant for small or sparse data and ROC-based methods.

Abstract

from arXiv · show

'Optimal cutpoints' for binary classification tasks are often established by testing which cutpoint yields the best discrimination, for example the Youden index, in a specific sample. This results in 'optimal' cutpoints that are highly variable and systematically overestimate the out-of-sample performance. To address these concerns, the cutpointr package offers robust methods for estimating optimal cutpoints and the out-of-sample performance. The robust methods include bootstrapping and smoothing based on kernel estimation, generalized additive models, smoothing splines, and local regression. These methods can be applied to a wide range of binary-classification and cost-based metrics. cutpointr also provides mechanisms to utilize user-defined metrics and estimation methods. The package has capabilities for parallelization of the bootstrapping, including reproducible random number generation. Furthermore, it is pipe-friendly, for example for compatibility with functions from tidyverse. Various functions for plotting receiver operating characteristic curves, precision recall graphs, bootstrap results and other representations of the data are included. The package contains example data from a study on psychological characteristics and suicide attempts suitable for applying binary classification algorithms.

1. Introduction

Optimal cutpoints are widely used to interpret continuous measurements, but sample-based selection can be unstable and overestimate diagnostic utility. cutpointr provides robust estimation, validation, and workflow features for addressing these problems.

  • Optimal cutpoints support intervention decisions in medicine and screening decisions in psychology by translating continuous measurements into binary classifications.
  • The usual procedure evaluates discriminative ability at every candidate cutpoint and selects the sample-optimal value.
  • Sample-derived cutpoints can be highly variable and overestimate diagnostic utility, while existing robust estimation methods had not been applied to out-of-sample cutpoint performance.
  • cutpointr targets scalable computation, pipe-friendly tidyverse workflows, and user control over direction and midpoint handling.
  • Existing R packages provide cutpoint or ROC functionality but generally lack robust estimation, out-of-sample validation, or comparable control over calculation details.

2. Estimating optimal cutpoints

cutpointr separates optimal-cutpoint estimation into choosing a discriminatory metric and selecting the cutpoint that optimizes it. It supports confusion-matrix metrics, constraints, ROC-based visualization, and several empirical or smoothed selection methods.

  • Optimal cutpoints combine a metric quantifying discriminatory ability with a method for selecting the cutpoint that optimizes that metric.
  • Metrics to quantify discriminatory ability: Sensitivity and specificity derive from the 2 × 2 confusion matrix, and the Youden index maximizes their joint discrimination as Se + Sp − 1.
  • Metrics to quantify discriminatory ability: Constrained metrics can maximize sensitivity subject to a minimum specificity, such as Sp > 0.9, or impose analogous restrictions for other metrics.
  • The ROC curve plots 1 − Sp against Se, with each point representing a sensitivity-specificity combination for a possible cutoff.
  • Methods to select optimal cutpoints: Empirical, LOESS, spline, GAM, and bootstrap methods can select cutpoints by optimizing the metric function m(c), whereas Normal and kernel methods specifically estimate the Youden index.

Nonparametric empirical method

The empirical method selects the in-sample metric maximum, but its sensitivity to sample randomness motivates bootstrap aggregation and smoothing alternatives such as GAM.

  • The nonparametric empirical method searches observed cutpoints and chooses the one maximizing m(c) in-sample.
  • Empirical optimization of the Youden index can produce highly variable cutpoints because it is sensitive to randomness in the sample.
  • Bootstrapping: Bootstrap aggregation draws multiple samples, fits a cutpoint model in each, and summarizes the resulting optimal cutpoints, by default with their mean.
  • GAM smoothing: GAM smoothing fits a nonparametric curve to metric values across cutpoints and estimates the maximally obtainable sum of sensitivity and specificity from that curve.
  • GAM smoothing: GAM smoothing uses thin plate regression splines and typically requires little tuning under its default settings.

Spline smoothing

Spline smoothing regularizes the metric-by-cutpoint function and can improve on empirical selection, but its tuning and performance depend on the data and may be inferior to GAM or bootstrapping.

  • Spline smoothing estimates the metric function using cubic splines, with implementation options for knots and other smoothing parameters.
  • The smoothing parameter λ penalizes large second derivatives, making spline smoothing suitable when the true metric function is expected to be smooth across many cutpoints.
  • No generally recommended λ exists, so tuning may require validation methods such as cross-validation or bootstrapping.
  • In simulations, spline smoothing improved on the empirical method but appeared inferior to GAM smoothing or bootstrapping.
  • LOESS method: Automatic LOESS smoothing can undersmooth metric functions, especially for large normal-distribution samples, because its behavior depends on the number of unique cutpoints.

Parametric method assuming normality

cutpointr includes a parametric Normal method for maximizing the Youden index under normally distributed predictors, alongside a nonparametric Kernel method that smooths empirical distributions with Gaussian kernels. The package uses plug-in bandwidth selection for the Kernel method and reports metrics on the unsmoothed ROC curve.

  • Parametric Normal method: The Normal method maximizes the Youden index, equivalently Se+Sp, assuming normally distributed predictors in negative and positive observations.
  • Parametric Normal method: The Normal method in cutpointr always assumes unequal standard deviations.When the standard deviations are equal, the optimal cutpoint expression can be simplified.
  • Nonparametric Kernel method: The Kernel method nonparametrically optimizes the Youden index by smoothing empirical distribution functions with Gaussian kernels.
  • Nonparametric Kernel method: Kernel bandwidths use Silverman’s direct plug-in rule, with sample standard deviation, interquartile range, and class-specific sample-size scaling.The bandwidth formulas are hy = 0.9 ∗min{sy, iqry/1.34} ∗n−0.2 and hx = 0.9 ∗min{sx, iqrx/1.34} ∗m−0.2.
  • Nonparametric Kernel method: cutpointr reports metrics by applying the Kernel-estimated cutpoint to the unsmoothed ROC curve.

Manual, mean and median methods

cutpointr provides mean and median cutpoint estimators mainly as bootstrap benchmarks and offers a manual function for fixed cutpoints.

  • Manual, mean and median methods: oc_mean and oc_median select the mean or median, respectively, as the optimal cutpoint.
  • Manual, mean and median methods: Estimating the mean or median within every bootstrap sample avoids bias from using full-sample values.
  • Manual, mean and median methods: oc_manual sets a fixed cutpoint.

3. Bootstrap estimates of performance and cutpoint variability

cutpointr uses bootstrap validation to estimate out-of-sample performance and cutpoint variability, favoring a conservative regular leave-one-out bootstrap. Each resample yields in-bag and out-of-bag metric estimates.

  • Bootstrap validation: Validation methods based on data splitting can have limitations, motivating bootstrap-based estimation of out-of-sample performance.
  • Bootstrap validation: cutpointr uses the regular leave-one-out bootstrap conservatively rather than correcting its pessimistic bias.Compared with k-fold cross-validation, bootstrap is usually more pessimistically biased but has lower variance.
  • Bootstrap procedure: A bootstrap sample matches the original size and is drawn with replacement, containing 63.2% of full-data observations on average.
  • Bootstrap procedure: The cutpoint is estimated on in-bag data, then applied to in-bag and out-of-bag observations to record TP, FP, TN, and FN for performance metrics.Bootstrap outputs distinguish in-bag metrics with _b from out-of-bag metrics with _oob.
  • Cutpoint variability: Bootstrap results quantify variability in estimated cutpoints, which can differ across estimation procedures and chosen metrics.

4. Performance of different cutpoint estimation methods

A simulation compared seven Youden-index estimation methods across distributions, separation levels, and sample sizes. Method efficiency depended on the underlying distribution, sample size, and separation: Normal and bootstrap performed best in some normal-data settings, whereas bootstrap or GAM performed best for non-normal data.

  • Simulation design: The simulation varied normal, log-normal, and gamma distributions, four separation levels, and sample sizes from 30 to 1,000 across 108 scenarios.Each scenario used 10,000 generated samples and held prevalence at 50%.
  • Simulation design: The comparison evaluated seven methods, excluding mean, median, and manual methods, to estimate the Youden-index cutpoint.
  • Simulation results: For normal distributions, the Normal method and bootstrapping for small samples yielded the best results.
  • Simulation results: The Normal method produced erroneous optimal cutpoints for lognormal and gamma distributions.
  • Simulation results: For non-normal distributions, bootstrapping or GAM performed best; bootstrap favored small separation and sample sizes, while GAM favored larger values of both.
  • Simulation results: The most efficient estimation method depended on distribution, sample size, and separation level.

5. Software

cutpointr provides a main interface for estimating cutpoints and returning organized classification outputs, while related functions support multiple predictors, ROC analysis, bootstrapping results, and included example data.

  • Main functions: The cutpointr function accepts data-frame inputs with named predictor, outcome, and grouping variables or raw vectors for programming use.Its returned object preserves a tibble/data-frame structure for readable output and downstream piping.
  • Main functions: multi_cutpointr wraps cutpointr across numeric predictor columns and returns one output row per predictor.It has a summary method, but plotting functions do not support multi_cutpointr objects.
  • ROC analysis: The package returns ROC curves with cutpoints, classification counts, rates, and metric values, including in- and out-of-bag curves when bootstrapping is used.ROC data are available through cutpointr, multi_cutpointr, and the exported roc function.
  • Output objects: Results include nested data frames for original data and ROC curves, with subgroup-specific rows when a subgroup is defined.Bootstrap results are stored in the boot column when bootstrapping is run.
  • Example data: The included suicide dataset contains psychological and clinical characteristics from 532 participants for binary classification analyses.The study assessed characteristics to identify persons at risk for attempting suicide.

6. Example: Suicide dataset

The suicide-dataset example demonstrates cutpointr’s basic cutpoint estimation, bootstrap validation, subgroup analysis, midpoint handling, and large-sample benchmarking. Its outputs compare in-sample with out-of-bag performance and show that computational advantages emerge for very large datasets.

  • Basic estimation: Using the suicide data, cutpointr selects cutpoint 2 for DSI-SS, classifying scores of at least 2 as positive and yielding Se + Sp = 1.75.The output also reports accuracy, sensitivity, specificity, AUC, prevalence, the ROC curve, and bootstrap-related objects.
  • Bootstrap validation: With 1,000 bootstrap samples, cutpoint 2 recurs in 777 samples, while out-of-bag Se + Sp has first and third quartiles of 1.67 and 1.78.The bootstrap output contains cutpoints and in-bag and out-of-bag metrics for each resample.
  • Bootstrap validation: The median in-bag Se + Sp is 1.76 versus 1.72 out of bag, compared with 1.75 for the full-sample cutpoint.This comparison provides an example of assessing optimism in the in-sample performance estimate.
  • Subgroups: In subgroup analyses, the female group has about 0.29 higher mean out-of-bag Se + Sp and less variable selected cutpoints than the male group.The larger AUC also emphasizes higher predictive performance for the predictor in the female group.
  • Subgroups: Kernel estimation produces less variable cutpoints and slightly higher out-of-bag performance; in males, mean Se + Sp rises from 1.48 to 1.54.The example attributes the expected lower variability to the Kernel method relative to simple maximization.
  • Midpoints: Midpoints address bias from selecting observed predictor values and provide an efficient equivalent to an infinitely dense candidate-cutpoint sequence.For the example, enabling midpoints produces an optimal cutpoint of 1.5; without midpoints, direction-dependent bias can matter especially in small, sparse data.
  • Benchmarks: For sample sizes ≥1e6, cutpointr is the fastest benchmarked solution, taking about 70% of ROCR and pROC runtime at 1e7 observations.It is slower in small samples, while OptimalCutpoints and ThresholdROC are excluded above 10000 observations because of high memory requirements.

7. Conclusion

The conclusion positions cutpointr as a response to empirical cutpoint procedures that can be variable and overestimate diagnostic utility. It provides more stable estimation methods, while leaving the choice of best method dependent on study and metric characteristics.

  • Existing empirical procedures can produce highly variable cutpoints and overestimate diagnostic utility.
  • Figure 6 benchmarks cutpoint estimation and ROC curve calculation across solutions and sample sizes from 1e+02 to 1e+07.
  • cutpointr incorporates methods intended to select cutpoints that perform relatively well in the source population and are more stable.
  • The manuscript does not determine a single best estimation method because suitability depends on study characteristics and the chosen metric.
  • The package may support simulation studies through scalability and applied research through estimating cutpoint variability and out-of-sample performance.

Affiliation:

The supplied material identifies the Journal of Statistical Software as the publication venue and provides its website and publisher information.

  • The paper is associated with the Journal of Statistical Software.
  • The journal website is listed as http://www.jstatsoft.org/.
  • The journal is published by the Foundation for Open Access Statistics.
Loading 2002.09209v1…