Source-linked AI summary
Deep ROC Analysis and AUC as Balanced Average Accuracy to Improve Model Selection, Understanding and Interpretation
André M. Carrington, Douglas G. Manuel, Paul W. Fieguth, Tim Ramsay, Venet Osmani, Bernhard Wernly, Carol Bennett, Steven Hawken, Matthew McInnes, Olivia Magwood, Yusuf Sheikh, Andreas Holzinger
TL;DR
Existing performance measures can be too specific to one threshold or too general across operating points, limiting interpretation for different risk groups. The paper proposes deep ROC analysis, which evaluates explicitly defined risk groups and translates partial ROC measures into familiar accuracy concepts. It reports that the method can improve model selection in some cases and provide group-specific performance views, while noting complexity and small-dataset limitations.
Problem
Single-threshold and global performance measures provide limited information about performance across groups of predicted risk and ROC operating points.
Method
Deep ROC analysis evaluates performance in explicitly defined groups based on predicted risk, FPR or specificity, or TPR or sensitivity.
Results
Deep ROC analysis can improve model selection in some cases and provide an informed view of model performance by risk groups for patient assurance.
Takeaways & Limitations
Group-wise analysis can reveal models’ strengths and weaknesses that overall performance measures do not show.
Takeaways & Limitations
Additional information can complicate communication, and groups with fewer than 50 instances may produce downward-biased performance estimates.
Abstract
from arXiv · showhide
Optimal performance is critical for decision-making tasks from medicine to autonomous driving, however common performance measures may be too general or too specific. For binary classifiers, diagnostic tests or prognosis at a timepoint, measures such as the area under the receiver operating characteristic curve, or the area under the precision recall curve, are too general because they include unrealistic decision thresholds. On the other hand, measures such as accuracy, sensitivity or the F1 score are measures at a single threshold that reflect an individual single probability or predicted risk, rather than a range of individuals or risk. We propose a method in between, deep ROC analysis, that examines groups of probabilities or predicted risks for more insightful analysis. We translate esoteric measures into familiar terms: AUC and the normalized concordant partial AUC are balanced average accuracy (a new finding); the normalized partial AUC is average sensitivity; and the normalized horizontal partial AUC is average specificity. Along with post-test measures, we provide a method that can improve model selection in some cases and provide interpretation and assurance for patients in each risk group. We demonstrate deep ROC analysis in two case studies and provide a toolkit in Python.
1 INTRODUCTION
Common performance measures are either too specific to one threshold or too general across unrealistic operating points. Deep ROC analysis addresses this gap by evaluating explicitly defined risk groups and interpreting group-wise ROC measures in familiar accuracy terms.
- Single-threshold measures depend on a specific choice of misclassification costs and provide limited information about nearby operating points.
- Global measures such as AUC include operating points that may not be used in practice and obscure performance distribution along the ROC curve.
- ROC plots show performance distributions but do not provide quantitative group-wise measures such as average sensitivity, AUC, or positive predictive value.
- In an illustrative 30%-prevalence example, the high-risk group has 85% balanced average accuracy but 67% average sensitivity, showing how global AUC can obscure group-wise behavior.
- Deep ROC analysis quantitatively evaluates explicitly specified groups of probability or risk, potentially changing classifier or test selection decisions.
- AUC and normalized concordant partial AUC represent balanced average accuracy, while normalized partial AUC and normalized horizontal partial AUC represent average sensitivity and average specificity.
2 RELATED WORK
Related work includes group-wise ROC comparisons, partial-area measures, and guidance on discrimination, calibration, and clinical utility. The paper positions deep ROC analysis as a general method that combines group-wise measures with familiar interpretations and explainability-oriented use.
- The paper introduces deep ROC analysis and a new interpretation of AUC, extending familiar reporting of discrimination and calibration with deeper group-wise analysis.
- Prior literature contains many performance measures, but less work addresses how to use measures together as an overall method for greater insight.
- Provost et al. identify dominant classifiers by groups, but do not quantify how much better one classifier performs.
- The half-AUC divides the ROC area into two fixed groups, limiting analysis and lacking a demonstrated comparable meaning to AUC or the C statistic.
- Normalized partial AUC corresponds to group average sensitivity, while normalized concordant partial AUC corresponds to a group’s AUC.
- The proposed interpretations are presented as supporting causability for explainable medicine by helping human experts achieve measurable causal understanding from AI explanations.
3 BACKGROUND
The background distinguishes partial ROC measures by perspective and introduces normalized concordant partial AUC as an interpretable, comparable measure within ROC regions. It also identifies limitations of conventional pAUC and sPA that motivate this choice.
- Limitations of prior measures: The standard pAUC is limited because it considers the width of a specificity range but not the range’s values, potentially making substantially different tests appear equally effective.The cited example reports nearly identical sensitivity but specificity ranges of 78−95% versus 50−60%.
- Metric background: The AUC and C statistic are equal for empirical ROC curves with binary outcomes, but this binary-outcome C statistic differs from Harrell’s and Uno’s continuous-outcome C statistics.The distinction prevents conflating related metrics defined for different outcome types.
- Limitations of prior measures: Standardized Partial Area can produce negative values for ROC curves partly above and partly below the major diagonal, preventing interpretation as an AUC or C statistic.Such mixed-position ROC curves are reported to occur in real life.
- Partial ROC measures: Normalized pAUC measures average sensitivity within a selected ROC region, while normalized horizontal pAUC measures average specificity.The two measures provide vertical and horizontal perspectives over specified ROC ranges.
- Concordant partial AUC: Normalized cpAUC generalizes AUC to a ROC region and can be compared with AUC or cpAUC from other regions.Its normalized form is interpreted as the AUC within the selected part.
- Balanced average accuracy: AUC and normalized cpAUC are interpreted as balanced average accuracy, combining average sensitivity and average specificity.For partial ROC curves, the two component averages are generally different, but their balance is represented by cpAUC.
4 METHOD: DEEP ROC ANALYSIS
Deep ROC analysis evaluates discrimination within explicitly defined risk or probability groups, supplementing global and single-threshold measures with group-specific comparisons. The method combines pre-test and post-test measures and can support model selection, interpretation, and calibration comparisons.
- Deep ROC analysis: The method is intended to improve or confirm model selection, understanding, and explanations for binary classifiers, diagnostic tests, and binary prognosis at a time point.It complements calibration measures when the same groups are used, while clinical utility is evaluated separately.
- Post-test measures: Post-test evaluation uses PPV or LR+ for low prevalence and NPV or LR− for high prevalence, with predictive value or odds-ratio measures used to include both classes.PPV is an absolute measure of how often a positive result is correct, whereas likelihood ratios express relative evidence.
- Pre-test measures: AUC and normalized cpAUC serve as combined pre-test measures of balanced average accuracy, while normalized pAUC and horizontal pAUC represent average sensitivity and average specificity.The combined measures include both positive and negative class detection.
- Measure selection: Accuracy is unsuitable as a general alternative to AUC or sensitivity because low prevalence obscures outcomes, whereas high prevalence overweights them.The method therefore uses balanced measures when both positive and negative detection should be considered.
- Workflow: The workflow identifies the evaluation purpose, chooses group boundaries and group count, creates group-wise performance tables or plots, and compares relevant groups.A calibration plot is highly recommended, especially when its groups align with the deep ROC analysis.
- Calibrated scores: Probabilistic or calibrated scores are required when groups are formed by predicted risk or probability, and calibration can convert non-probabilistic scores into probabilities.The paper identifies isotonic regression and Platt’s method as calibration approaches.
- Case-study interpretation: In the case-study tables, LSTM performs consistently well in balanced average accuracy across FPR-defined risk groups, while SOFA performs best in the wrong group.Average sensitivity and specificity alone cannot be compared across risk groups because they change monotonically in opposite directions.
5 CASE STUDY: MORTALITY PREDICTION BASED ON ARTERIAL BLOOD GAS ANALYSIS OF SEPTIC PA-
The septic-patient case study compares four classifiers using overall and risk-region ROC measures. Deep ROC analysis reveals clinically relevant differences that overall AUC can obscure, especially in the high-risk region.
- LSTM was almost fully dominant over LR, Lactate, and SOFA in the ROC plot.
- LSTM achieved an AUC of 88%, compared with 82% for Logistic Regression, 80% for lactate, and 72% for SOFA.The paper interprets AUC as average balanced accuracy over the whole ROC curve.
- 0.67, 0.81, 0.85, and 0.89 were the high-risk-region cpAUCn values for SOFA, Lactate, LR, and LSTM, respectively.Relative to overall AUC, SOFA was 5% worse, LR 3% better, and LSTM and Lactate 1% better in this region.
- The high-risk sensitivity difference between LSTM and LR was 9%, despite a 6% overall AUC difference and a 4% high-risk cpAUCn difference.The paper argues that complete reporting of cpAUCn, AUC, sensitivity, and specificity gives a fuller comparison.
- High-risk average sensitivity was 76% for LSTM, 67% for LR, 58% for Lactate, and 39% for SOFA.The authors describe SOFA’s high-risk sensitivity as worse than chance.
- Normalized partial AUC can favor a rightmost bin regardless of classifier quality, whereas normalized concordant partial AUC is the proper analogue to AUC.
6 CASE STUDY 2: GERMAN BREAST CANCER STUDY GROUP
The German breast cancer case study evaluates selected models across six predicted-risk groups using deep ROC measures. Group-level analysis exposes edge-group performance declines and can change model-selection conclusions relative to overall AUC.
- The study focused on an SVM with a Mercer sigmoid kernel, penalized logistic regression, and a small-batch random forest after evaluating 14 models.
- SVM performance sagged by about 3% in the high-risk group and 9% in the low-risk group, while random-forest performance sagged approximately 5–6% in both groups.
- No significant AUC difference separated the best SVM and random-forest models over matched pairs across 10 folds.
- Deep ROC analysis nevertheless found a significant group-1 difference between SVM and random forests in cpAUCn and pAUCn, and also distinguished logistic regression from SVM there.
- Average group measures fell below the overall measure for every group in this case study.With 686 samples, 228 positives, five folds, and six groups, minority-class errors were exaggerated and weighted as much as majority-class errors.
- Optimizing group measures rather than AUC or AUPRC produced no significant difference in average performance across the four reported measures.
7 AUC IS BALANCED AVERAGE ACCURACY
The paper establishes that whole-curve AUC can be interpreted as balanced average accuracy. This interpretation links AUC to average sensitivity and average specificity while distinguishing it from pointwise balanced accuracy.
- AUC is interpreted as balanced average accuracy, an average of aggregate class-performance measures rather than average balanced accuracy at individual points.
- AUC equals average sensitivity when the ROC curve is viewed vertically.
- AUC equals average specificity when the ROC curve is viewed horizontally.
- Because average sensitivity and average specificity are equal over the whole ROC curve, their simple average equals AUC.
- Balanced average accuracy describes the balance of average accuracy in each class, whereas balanced accuracy combines sensitivity and specificity at one operating point.
8 THE NORMALIZED CONCORDANT PARTIAL AUC IS BALANCED AVERAGE ACCURACY
The normalized concordant partial AUC extends the balanced average accuracy interpretation from the whole ROC curve to a selected ROC segment or risk group.
- Within a selected ROC segment or risk group, normalized concordant partial AUC is the partial-curve analogue of AUC.
- The normalized concordant partial AUC is defined using the horizontal and vertical widths of the selected ROC region.
- Normalized concordant partial AUC is balanced average accuracy for that segment, averaging average sensitivity and average specificity there.
9 AUC IS NOT AVERAGE BALANCED ACCURACY
AUC is balanced average accuracy, not average balanced accuracy: it balances the average accuracy of each class across the ROC curve. The normalized concordant partial AUC extends this interpretation within risk groups.
- AUC balances average sensitivity and average specificity across ROC points, rather than averaging balanced accuracy at each point.The paper presents this distinction through an integral and a simple example.
- The normalized concordant partial AUC provides the within-group AUC needed to compare groups arranged along an ROC plot.Group-average sensitivity and specificity alone are directionally constrained, so a within-group AUC is required.
- Average balanced accuracy over a ROC range requires integrating the sensitivity and specificity vector along that range.The continuous index w and differential vector dw express this averaging operation.
- The discrete form is not proved because it is complex and meticulous, although the authors report it is evident in experimental results from their deepROC toolkit.
- The resulting interpretation is that AUC is balanced average accuracy, not average balanced accuracy.
10 LIMITATIONS
The method adds complexity to reporting, can bias group performance downward in small datasets, and applies only to binary outcomes at a time point.
- Additional information from deep ROC analysis may complicate communication of results, motivating guidance for uniform reporting.
- In datasets with fewer than 50 instances per group, errors were observed to bias group performance downward relative to overall performance measures.
- The method is limited to binary classifiers, diagnostic tests, and prognosis at a time point with binary outcomes.
11 DISCUSSION ON CONTINUOUS METHODS VERSUS PREDICTED RISK AND SUBGROUPS IN BINARY CLASSIFICATION
The paper argues that grouping predicted risk can improve understanding when categorical decisions are useful, while acknowledging that clinical reasoning may exceed predefined categories and tests.
- The authors note that clinicians may choose whether binning or categorization helps their decisions while retaining both categories and numbers when feasible.
- Categorizing predicted risk can summarize information when excessive continuous detail would reduce understanding and decision-making.
- Binary and categorical clinical prediction models assume a priori completeness of the available options and tests.
- Clinical diagnosis and treatment may involve unanticipated options, differential reasoning, and continuous-value Bayesian thinking beyond predefined protocols.
- The authors caution that categorical or subgroup methods might distract or blind clinicians and regulators in such dynamic decision-making contexts.
12 CONCLUSIONS AND FUTURE WORK
Deep ROC analysis reveals performance differences across risk groups and can improve model selection in some cases. The case studies show both useful group-level interpretation and no significant optimization gain in one setting.
- Models and tests can behave differently across risk groups, with within-group performance better or worse than overall averages.
- The normalized concordant partial AUC interprets performance within each group as balanced average accuracy, indicating where an algorithm is strong or weak.
- In the first case study, deep ROC analysis better distinguished LR from Lactate in the key high-risk group and exposed SOFA’s absolute inadequacy.
- Deep ROC analysis can improve model selection in some cases and provide group-wise performance information for assurance.
- In the second case study, performance was lower in the highest and lowest risk groups, while optimizing group measures produced no significant difference from AUC or AUPRC.