Source-linked AI summary
ROC and AUC with a Binary Predictor: a Potentially Misleading Metric
John Muschelli
TL;DR
ROC and AUC analyses are widely applied to binary and categorical predictors even though a binary predictor supplies only one threshold and ties are consequential. This paper derives binary-predictor AUC calculations, compares software implementations, and finds that interpolation and tie handling can make commonly reported AUCs misleading.
Problem
Binary and categorical predictors are widely used in medical analysis, but ROC/AUC analysis may not appropriately represent their limited thresholds and numerous tied risk scores.
Method
The paper derives AUC calculations for a single binary predictor, defines a strict tie-excluding AUC, and examines ROC/AUC implementations across statistical software.
Results
Common software implementations show inconsistencies, and the paper finds that their calculations may not reflect the discreteness of binary data.
Takeaways & Limitations
AUC reports for discrete predictors should state how ties and interpolation are handled; step-function interpolation and not counting ties may be more appropriate.
Takeaways & Limitations
Different tie methods create additional issues, including AUC values below 0.5 and tests that may lack the same theoretical properties or Wilcoxon rank-sum connections.
Abstract
from arXiv · showhide
In analysis of binary outcomes, the receiver operator characteristic (ROC) curve is heavily used to show the performance of a model or algorithm. The ROC curve is informative about the performance over a series of thresholds and can be summarized by the area under the curve (AUC), a single number. When a predictor is categorical, the ROC curve has one less than number of categories as potential thresholds; when the predictor is binary there is only one threshold. As the AUC may be used in decision-making processes on determining the best model, it important to discuss how it agrees with the intuition from the ROC curve. We discuss how the interpolation of the curve between thresholds with binary predictors can largely change the AUC. Overall, we show using a linear interpolation from the ROC curve with binary predictors corresponds to the estimated AUC, which is most commonly done in software, which we believe can lead to misleading results. We compare R, Python, Stata, and SAS software implementations. We recommend using reporting the interpolation used and discuss the merit of using the step function interpolator, also referred to as the "pessimistic" approach by Fawcett (2006).
1 Introduction
ROC analysis is widely used for threshold-based performance assessment, including with binary and categorical predictors. For binary predictors, ties and interpolation choices can make AUC calculations and ROC representations potentially misleading.
- 1 Introduction: Binary and categorical predictors are common in medical and other applications, often representing thresholded values or inherently discrete traits.Examples include medical tests, blood-pressure categories, and gene presence or absence.
- 1 Introduction: A binary predictor generated by thresholding a continuous variable corresponds to one point on the underlying ROC curve, rather than a full ROC curve.The paper notes that applying ROC analysis to such predictors is nevertheless common.
- 1 Introduction: ROC interpolation can treat ties pessimistically, optimistically, or by averaging their effects between points.The expected-performance approach averages the pessimistic and optimistic step segments and corresponds to assigning tied cases a half probability.
- 1 Introduction: The paper examines binary-predictor AUC calculations and software implementations, arguing that AUC alone may mislead unless tie handling and interpolation are reported explicitly.It proposes considering step-function interpolation when comparing AUC values.
2 Mathematical Proof of AUC for Single Binary Predictor
For a binary predictor, the strict AUC equals sensitivity multiplied by specificity, while accounting for ties produces a different value commonly reported by software. Geometric interpolation and Monte Carlo calculations show why the chosen AUC definition matters.
- Strict AUC: The strict AUC is the probability that a randomly selected positive receives a higher predictor value than a randomly selected negative.Tied predictor values contribute nothing under the strict inequality definition.
- Strict AUC: specificity × sensitivity (1) is the strict AUC for a binary predictor.The derivation identifies sensitivity as P(X_i = 1|Y_i = 1) and specificity as P(X_j = 0|Y_j = 0).
- Tie-adjusted AUC: AUCw/ties adds half the probability of tied positive-negative predictor values to AUCdefinition.This tie-adjusted definition corresponds to the common AUC definition reported by most software.
- Monte Carlo Estimation: 0.364517 and 0.603929 are the simulated estimates for AUCdefinition and AUCw/ties, respectively.The simulations agree with the corresponding calculations with negligible Monte Carlo error.
- Geometric Argument: The step-function ROC area represents AUCdefinition, whereas linear interpolation corresponds to AUCw/ties.The paper notes that linear interpolation can suggest sensitivity-specificity combinations unavailable from the binary predictor.
- Geometric Argument: AUCdefinition and AUCw/ties both reduce to simple forms involving only sensitivity and specificity for binary predictors.The paper compares these definitions geometrically and algebraically before examining software implementations.
3 AUC Calculation: Current Implementations
Common software implementations generally report the tie-adjusted AUC, but ROC plots may use linear or step interpolation, creating disagreements between the plotted curve, estimated AUC, and variability.
- Software implementations: Most R, Python, Stata, and SAS implementations report AUCw/ties, which adds the probability of tied predictions.The reported value is commonly aligned with linear interpolation rather than the strict AUC definition.
- fbroc tie strategies: 0.6036415 is reported by both fbroc tie strategies, despite their different ROC-curve interpolations.Strategy 2 is linear by default, whereas strategy 1 uses a step or constant representation.
- Stata: Stata's roctab reports 0.6037, while rocreg agrees with AUCdefinition and differs from roctab.The point estimate remains unchanged by bootstrap estimation, but the command choice changes the reported AUC.
- Interpolation and AUC: Linear interpolation coincides with AUCw/ties, whereas Stata and fbroc step curves coincide with AUCdefinition.The software comparison therefore yields different relationships between the visual ROC curve and the reported AUC.
- Confidence intervals: fbroc's confidence interval reflects AUCdefinition variability while its reported value reflects AUCw/ties.This combines a step-function-based variability estimate with a tie-adjusted point estimate.
- fbroc tie strategies: Different fbroc tie strategies can produce inconsistent combinations of AUC estimates, ROC representations, and variability estimates.The paper identifies strategy 2 as mismatching its plotted curve and strategy 1 as mismatching variability estimation.
4 Categorical Predictor Example
The example compares a 4-level categorical predictor with a binary predictor and shows that interpolation choice determines whether their AUCs appear nearly identical or different.
- 4 Categorical Predictor Example: The example uses a categorical predictor measured on a 4-point Likert scale and compares it with a binary predictor.The categorical predictor has multiple possible sensitivities and specificities, unlike the binary predictor.
- 4 Categorical Predictor Example: Under linear interpolation, the categorical and binary predictors produce nearly identical ROC curves and AUCs.The small AUC discrepancy results from integer-valued cells in the example table.
- 4 Categorical Predictor Example: The identical linear-interpolated AUC is counterintuitive because the categorical predictor has more gradation than the binary predictor.The binary predictor cannot take values beyond its two categories.
- 4 Categorical Predictor Example: Using pessimistic interpolation yields different AUC estimates for the categorical and binary predictors.Figure 4 represents these curves with blue dashed and red dotted lines, respectively.
5 Conclusion
The paper examines how common software plots and estimates ROC/AUC performance for binary predictors, emphasizing that interpolation and tie handling matter for discrete data. It concludes that reporting these choices explicitly, and often using step-function interpolation, can reduce misleading interpretation.
- 5 Conclusion: Common statistical software implementations estimate ROC curves and AUCs for univariate binary predictors, but results are inconsistent across platforms and packages.The paper specifically notes inconsistencies involving R, Stata, and fbroc.
- 5 Conclusion: For predictors that are not fundamentally discrete, linear interpolation may be reasonable when theoretically possible intermediate values are unobserved.This condition distinguishes such predictors from fundamentally discrete or categorical variables.
- 5 Conclusion: For fundamentally discrete predictors, the paper recommends step-function interpolation without counting ties as more appropriate.The authors also call for explicit tie-handling information and possible warnings for discrete data.
- 5 Conclusion: The paper argues that current AUC methodology can mislead users when applied to binary or categorical predictors.Different tie methods also raise issues such as AUC values below 0.5 and altered theoretical connections to Wilcoxon rank-sum tests.