PaperlayerView exact revision on arXiv

Source-linked AI summary

Area under the ROC Curve has the Most Consistent Evaluation for Binary Classification

Jing Li

arXiv:2408.10193v2stat.MLcs.LG

TL;DR

The paper examines whether binary-classification metrics evaluate models consistently when prevalence changes, while sample size and relationships among variables are held constant. Across simulated scenarios and a broad set of models and metrics, less prevalence-sensitive metrics—especially AUC—provide more consistent evaluations and rankings.

  • Problem

    Prior research largely evaluates metrics on one dataset and does not sufficiently separate prevalence effects from model and sample-size effects.

    Source: §1 · p. 3

  • Method

    The study uses statistical simulation to vary prevalence while holding sample size and relationships among variables constant, evaluating 18 metrics across five machine-learning models and a naive random-guess model.

    Source: §1 · p. 3

  • Results

    Metrics less influenced by prevalence give more consistent model evaluations and rankings, and AUC has the smallest variance on both measures.

    Source: §1 · p. 3, §5.1 · pp. 20–21

  • Takeaways & Limitations

    For binary classification across the studied prevalence scenarios, AUC offers the most consistent model evaluation and ranking.

    Source: §5.1 · p. 21, §5.2 · p. 22

    The simulation uses one more-or-less balanced dataset and seven predictor features, and the study covers only binary classification.

    Source: §5.1 · pp. 21–22

Abstract

from arXiv · show

The proper use of model evaluation metrics is important for model evaluation and model selection in binary classification tasks. This study investigates how consistent different metrics are at evaluating models across data of different prevalence while the relationships between different variables and the sample size are kept constant. Analyzing 156 data scenarios, 18 model evaluation metrics and five commonly used machine learning models as well as a naive random guess model, I find that evaluation metrics that are less influenced by prevalence offer more consistent evaluation of individual models and more consistent ranking of a set of models. In particular, Area Under the ROC Curve (AUC) which takes all decision thresholds into account when evaluating models has the smallest variance in evaluating individual models and smallest variance in ranking of a set of models. A close threshold analysis using all possible thresholds for all metrics further supports the hypothesis that considering all decision thresholds helps reduce the variance in model evaluation with respect to prevalence change in data. The results have significant implications for model evaluation and model selection in binary classification tasks.

1 Introduction

The paper asks whether binary-classification metrics evaluate models consistently as prevalence changes, a question complicated by prior work’s focus on individual datasets and limited control of model and sample-size effects. Using simulation and broader threshold analysis, it finds that less prevalence-sensitive metrics—especially AUC—provide more consistent evaluation and model rankings.

1 Introduction

  • Prior studies often use one dataset and do not separate prevalence effects from model and sample-size effects; threshold effects for metrics other than AUC also remain underexamined.These gaps limit the evidence on evaluation consistency across datasets with varying prevalence.Source: §1 · p. 3
  • Across simulated prevalence scenarios, metrics less influenced by prevalence give more consistent evaluations of individual models and rankings of model sets, with AUC showing the smallest variance on both measures.The study assesses 18 metrics across five machine-learning models and a naive random-guess model.Source: §1 · p. 3
  • When prevalence changes, metric values can become artifacts of the metric rather than measures of intrinsic model performance, leaving practitioners unsure of a model’s predictive power.The paper connects this concern to consequences for model evaluation and selection across domains [47].Source: §1 · pp. 3–4
  • The study uses statistical simulation with a real-world dataset to generate richer data scenarios while keeping sample size and relationships among variables constant.It links prevalence, individual-model evaluation, and rankings across a larger metric set than prior work considered.Source: §1 · pp. 3–4
  • A threshold analysis covering all possible thresholds for all metrics supports the hypothesis that considering more thresholds improves consistency as prevalence changes.The paper identifies AUC’s use of all decision thresholds as a possible reason for its consistency.Source: §1 · pp. 3–4

2 Methods

The methods introduce binary-classification metrics through confusion-matrix quantities, explain what different metric families emphasize, and re-express metrics using sample size, prevalence, TPR, and TNR. These formulations make prevalence’s role explicit for metrics that depend on it.

2.1 Background and Basic Notations

  • Binary predictions are organized into true positives, false negatives, true negatives, and false positives, which together form the confusion-matrix basis for the metrics.The four counts sum to the sample size.Source: §2.1 · p. 4
  • F-scores emphasize TPR, while balanced accuracy, bookmaker informedness, geometric mean, and MCC balance performance on both classes.These balanced metrics are most useful when evaluation should weigh accuracy on both classes equally.Source: §2.1 · p. 6
  • The metrics have distinct application uses: bookmaker informedness measures performance above random guessing, while Jaccard index measures set similarity and F-scores are common in information retrieval and NLP tasks.The passage also describes markedness, diagnostic odds ratio, Fowlkes–Mallows index, and Cohen’s Kappa in their respective domains.Source: §2.1 · p. 6

2.2 Alternative Derivation of Confusion Matrix Metrics

  • The alternative derivation expresses confusion-matrix metrics using sample size, prevalence, TPR, and TNR, making prevalence dependence explicit in formulations such as PPV and NPV.F1 score is also rewritten in these quantities.Source: §2.2 · p. 7

3 Case Study: Crime Recidivism

The case study compares metric-based rankings of six classifiers on Broward County recidivism data, then motivates a simulation that varies prevalence while holding sample size and predictor relationships largely constant. The simulation expands the analysis beyond the single, approximately balanced case.

3 Case Study: Crime Recidivism

  • The case study uses 6,214 Broward County observations, seven defendant features, five classifiers, and a prevalence-based random-guess model, with an 80/20 train-test split and 10-fold cross-validation.The classifiers are logistic regression, random forest, KNN, LDA, and GBM.Source: §3 · p. 8
  • In the recidivism case study, all metrics rank naive random guess last, while GBM is most often ranked best and random forest second most often.AUC uses all possible thresholds, whereas the other metrics use the 0.5 cutoff.Source: §3 · p. 8

3.1 Simulation Study

  • Because the case study is one more-or-less balanced dataset with prevalence 0.452, the authors use simulation to examine metric behavior as prevalence becomes imbalanced while holding sample size constant.The simulation varies prevalence to study how changes in TPR and TNR affect metrics.Source: §3 · p. 9
  • The simulation upsamples one class while downsampling the other in 30-case increments, creating data with prevalence from about 0.08 to about 0.83 while largely preserving predictor relationships.The authors report minimal changes in correlations among the eight variables across simulated datasets.Source: §3.1 · pp. 9–10

4 Results

Across the simulations, metrics less influenced by prevalence yielded more consistent evaluations and model rankings. AUC was largely prevalence-independent, had the smallest ranking variance, and remained invariant to threshold changes; using more thresholds generally reduced variance for other metrics too.

4.1 Simulation Results: Evaluation Metrics

  • Across metrics, prevalence was associated with distinct patterns: several rates and F-scores tracked it monotonically, balanced-class metrics were concave, accuracy was convex, and AUC was largely flat.PPV and NPV fluctuated more than TPR and TNR; the paper attributes this to both their numerators and denominators changing with prevalence, whereas the denominators of TPR and TNR are known before prediction.Source: §4.1 · pp. 11, 13, Figure 1 · p. 12

4.2 Simulation Results: Comparisons of Models

  • Metrics more influenced by prevalence also differentiated models less clearly; with MCC and AUC, random forest and Gradient Boosted Machine outperformed the other trained models.F-test, Bartlett, and Levene results found a statistically significant difference in evaluation variance between more and less prevalence-influenced metrics.Source: §4.2 · pp. 13, 15, Figure 2 · p. 14

4.3 Simulation Results: Ranking of Different Models

  • AUC had the smallest variance in model rankings across prevalence scenarios; rankings from less prevalence-sensitive metrics fluctuated less than those from more sensitive metrics.The difference in ranking variance between the two metric groups was statistically significant in F-test, Bartlett, and Levene tests.Source: §4.3 · pp. 15–17, Figure 3 · p. 16

4.4 Threshold Analysis

  • For the random forest, threshold-dependent metrics showed prevalence-linked patterns moderated by decision threshold, while AUC was threshold-invariant and ranged from 0.7 to 0.79 across prevalence.AUC also had a narrower value range than the other metrics described in this analysis.Source: §4.4 · p. 17, Figure 4 · p. 18
  • As more thresholds were included, metric variance with respect to prevalence generally declined, supporting the proposed link between considering thresholds and evaluation consistency.Thresholds were added outward from 0.5, and the analysis averaged metric values as thresholds were added.Source: §4.4 · pp. 17, 19, Figure 5 · p. 19
  • In regressions of variance on threshold count, 104 of 114 metric-model combinations had a statistically significant negative estimate; the 10 exceptions included eight PPV/NPV cases, one Kappa case, and one AUC case.DOR had a larger negative estimate than the other metrics and was omitted from the figure for visualization.Source: §4.4 · p. 19, Figure 6 · p. 20

5 Discussions and Conclusion

Across changing-prevalence scenarios, AUC provides the most consistent model evaluations, while other metrics show distinct prevalence relationships. The discussion gives guidance for metric choice and identifies data and task boundaries for the findings.

5.1 Discussions

  • The analysis holds variable relationships and sample size constant while examining prevalence across 156 data sets, five machine-learning models, and a random-guess model.It also uses all possible thresholds to study how metric values change with threshold and prevalence.Source: §5.1 · p. 20
  • TPR, TNR, F-scores, Fowlkes Mallows, and Jaccard change monotonically with prevalence; several other metrics are U-shaped or lack a systematic relationship.Balanced accuracy, bookmaker informedness, geometric mean, and MCC are close to U-shaped; markedness and DOR show no systematic relationship.Source: §5.1 · p. 20
  • AUC is mostly prevalence-independent and offers the most consistent ratings and rankings; considering all thresholds offsets prevalence influence in the reported analysis.Metrics less influenced by prevalence also show lower variance when evaluating individual models and ranking models.Source: §5.1 · pp. 20–21
  • For a complete performance picture, examine TP, TN, FN, and FP; for consistent ratings or rankings across data scenarios, use AUC.The authors caution that no single metric generally summarizes all four basic quantities satisfactorily.Source: §5.1 · p. 21
  • The simulations use one roughly balanced-outcome data set and seven predictors; higher-dimensional data may add variation, and findings need confirmation on other data.The authors identify more imbalanced outcomes and larger predictor sets as settings for further analysis.Source: §5.1 · pp. 21–22
  • GBM and random forest have smaller ranking variance than GLM, KNN, and LDA; random guess is most often ranked least-performing.Evaluations of almost all metrics overlap on rank 6 for random guess.Source: §5.1 · p. 21
  • The study covers binary classification; the authors expect similar results for one-versus-rest multiclass comparisons, but one-versus-one results depend on other classes' relative sizes.Most metric formulations can be extended to multiclass settings, according to the discussion.Source: §5.1 · p. 22

5.2 Conclusion

  • Across scenarios from overwhelmingly positive to overwhelmingly negative, the conclusion reports clear patterns for 18 metrics and six models, including random guess.The authors suggest theory-driven simulation can yield findings unavailable from analyzing observed empirical data alone.Source: §5.2 · p. 22

Supporting information

The supporting information includes model-evaluation heat maps across decision thresholds and correlation heat maps for the original and selected simulated samples.

Supporting information

  • Heat maps show model evaluations at different decision thresholds and correlations among predictors and outcome for the original sample and 79 selected simulated samples.The selected simulated samples are every other sample from the full set.Source: Supporting information · p. 22

A Prediction Results for Crime Recidivism

The appendix presents crime-recidivism evaluation results for six classifiers and 18 metrics, using a Broward County cohort with seven defendant features.

A Prediction Results for Crime Recidivism

B F1 score as a result of prevalence change when only sample size n is constant

The appendix explains how F1 changes with prevalence when sample size is held constant, concluding that F1 decreases as prevalence decreases.

B F1 score as a result of prevalence change when only sample size n is constant

C Supplementary Information

Supplementary figures clarify that apparent AUC changes across thresholds are plotting artifacts, alongside model-specific threshold evaluations and correlation heat maps for different data samples.

C Supplementary Information

  • Apparent AUC changes across thresholds in small heat-map regions are plotting artifacts; AUC should be independent of the chosen threshold because it considers all possible thresholds.The apparent changes occur where multiple prevalence levels share an x-axis position and have different AUC values.Source: C Supplementary Information · p. 29
  • Supplementary plots show model evaluation across different prevalence levels and decision thresholds for GLM, KNN, LDA, GBM, and Random Guess.Source: Figure 7 · p. 29, Figure 8 · p. 30, Figure 9 · p. 30
  • Supplementary correlation heat maps cover different data samples, with Fig. 12 labeling panes by the random positive cases added or dropped from the original dataset.Source: Figure 12 · p. 32, Figure 13 · p. 33, Figure 14 · p. 34

Related papers

Loading 2408.10193v2…