A Closer Look at AUROC and AUPRC under Class Imbalance

Matthew B. A. McDermott; Haoran Zhang; Lasse Hyldig Hansen; Giovanni Angelotti; Jack Gallifant

Source-linked AI summary

A Closer Look at AUROC and AUPRC under Class Imbalance

Matthew B. A. McDermott, Haoran Zhang, Lasse Hyldig Hansen, Giovanni Angelotti, Jack Gallifant

arXiv:2401.06091v4Jan 11, 2024cs.LGstat.ME

TL;DR

The paper examines the widespread belief that AUPRC is superior to AUROC for imbalanced classification. It combines theoretical analysis, experiments, and a review of over 1.5 million papers, finding that AUPRC can favor higher-scoring and higher-prevalence subgroups rather than being universally superior.

Problem
The paper addresses the widespread claim that AUPRC is a better evaluation metric than AUROC for class-imbalanced classification.
Method
The authors compare the metrics theoretically, validate the analysis with synthetic and real-world fairness experiments, and review over 1.5 million scientific papers.
Results
AUROC treats positive-sample improvements uniformly, whereas AUPRC favors higher-score improvements and can favor higher-prevalence subgroups.
Takeaways & Limitations
Metric selection should be context-aware because AUPRC is not universally superior under class imbalance and can create fairness risks when subgroup prevalences differ.
Takeaways & Limitations
The theoretical findings could be generalized to account for task difficulty, uncalibrated models, and more than two subpopulations, while experiments could include more fairness datasets.

Abstract

from arXiv · show

In machine learning (ML), a widespread claim is that the area under the precision-recall curve (AUPRC) is a superior metric for model comparison to the area under the receiver operating characteristic (AUROC) for tasks with class imbalance. This paper refutes this notion on two fronts. First, we theoretically characterize the behavior of AUROC and AUPRC in the presence of model mistakes, establishing clearly that AUPRC is not generally superior in cases of class imbalance. We further show that AUPRC can be a harmful metric as it can unduly favor model improvements in subpopulations with more frequent positive labels, heightening algorithmic disparities. Next, we empirically support our theory using experiments on both semi-synthetic and real-world fairness datasets. Prompted by these insights, we conduct a review of over 1.5 million scientific papers to understand the origin of this invalid claim, finding that it is often made without citation, misattributed to papers that do not argue this point, and aggressively over-generalized from source arguments. Our findings represent a dual contribution: a significant technical advancement in understanding the relationship between AUROC and AUPRC and a stark warning about unchecked assumptions in the ML community.

1 Introduction

This paper challenges the widespread claim that AUPRC is generally more accurate or fairer than AUROC under class imbalance. It argues that metric choice should instead reflect model use, and shows that AUPRC can favor higher-prevalence subpopulations.

Motivation: AUROC and AUPRC are examined in both model-comparison settings without deployment details and application settings with estimated workflows and error costs.The paper emphasizes that appropriate metric selection matters in both contexts.
Central claim: The paper identifies as its central target the claim that AUPRC is better, more accurate, or fairer than AUROC when negative labels greatly outnumber positive labels.This claim is described as widely used across scientific, educational, and popular sources.
Theoretical distinction: AUROC weights false positives equally, whereas AUPRC weights false positives at a threshold by the inverse of the model’s probability of producing scores above that threshold.This difference motivates reasoning about whether reducing false positives at lower or higher thresholds matters more.
Theoretical distinction: AUROC favors improvements uniformly across positive samples, while AUPRC favors improvements for higher-scored samples rather than lower-scored samples.The paper identifies this score-dependent prioritization, not class imbalance itself, as the key distinction between the metrics.
Fairness concern: AUPRC can prioritize improvements for higher-prevalence subpopulations at the expense of lower-prevalence subpopulations, raising fairness concerns.The paper establishes this claim theoretically and evaluates it empirically on synthetic and real-world fairness datasets.
Evidence and review: The authors establish their claims theoretically and empirically, then review more than 1.5 million scientific papers to examine how the disputed claim spread.The review concerns improper uses of AUPRC relative to AUROC across high-stakes domains and established venues.

2 Theoretical Analyses

The theoretical analysis shows that AUROC and AUPRC differ in how they weight model errors: AUROC treats mistakes uniformly, while AUPRC emphasizes higher-score mistakes. Under sufficiently severe subgroup prevalence imbalance, this emphasis can favor the higher-prevalence subgroup.

Theorem 1: Theorem 1 characterizes AUROC as minimizing false-positive rates over positive samples uniformly, while AUPRC weights them by the inverse of the model’s firing rate.The firing rate is the probability that the model outputs a score above the relevant threshold.
Theorem 2: A model mistake is an incorrectly ranked adjacent pair in which a positive sample receives a lower score than a neighboring negative sample.The definition assumes distinct samples and scores, with the pair adjacent in score order.
Theorem 2: Theorem 2 states that correcting any given mistake improves AUROC by the same amount, whereas correcting a higher-score mistake improves AUPRC more than correcting a lower-score mistake.This follows because a corrected adjacent pair changes the false-positive rate by the same amount but affects AUPRC at the pair’s score threshold.
Theorem 3: When subgroup 0 has a sufficiently lower positive-label rate than subgroup 1, the mistake whose correction maximally improves AUPRC lies entirely in subgroup 1.Thus, under sufficiently severe prevalence imbalance, AUPRC favors the higher-prevalence subgroup.
Implication: The paper argues that this preference can introduce bias in fairness-sensitive settings and does not establish AUPRC as superior for distinguishing low-prevalence from high-risk subgroups.Instead, maximizing AUPRC is described as more likely to optimize within the high-risk subgroup than across subgroups under severe imbalance.

3 Experimental Validation

Synthetic and real-world experiments show that selecting or optimizing models with AUPRC can increase disparities between subpopulations, especially when their positive-label prevalences differ. AUROC-based optimization produces more similar subgroup improvements, while real-world effects strengthen with larger prevalence mismatches.

Synthetic experiments: AUPRC optimization introduced notable disparities across both synthetic optimization procedures by favoring the higher-prevalence subpopulation.AUROC optimization increased both groups’ AUROCs together, whereas AUPRC optimization produced divergent subgroup performance.
Synthetic experiments: In the random-permutation procedure, optimizing AUPRC decreased AUROC for the low-prevalence subgroup.This procedure gave both groups the same optimization capacity while allowing performance to worsen under some permutations.
Real-world validation: Real-world validation evaluated fairness gaps across four binary-classification datasets using sex and race as sensitive attributes.The study trained XGBoost models with diverse per-group weighting and compared validation metrics with test-set subgroup gaps across repeated random splits.
Real-world validation: In datasets with higher prevalence ratios, overall AUPRC was more positively correlated with the AUROC gap than overall AUROC.This indicates that AUPRC more aggressively favored the higher-prevalence group; lower prevalence ratios produced mixed results.
Real-world validation: The difference in correlations increased with prevalence mismatch, with Spearman’s ρ = 0.905 and p = 0.002.The authors note that limited prevalence disparities reduced the power to detect the effect in these real-world datasets.

4 When Should One Use AUPRC vs. AUROC?

The paper recommends choosing between AUROC and AUPRC according to deployment context rather than class imbalance alone. AUROC is preferred when improvements should be valued broadly or equitably, while AUPRC is reserved for high-cost false positives in single-group settings without equity concerns.

Guidance: For context-independent evaluation, AUROC impartially values improvements across the model’s entire output space when error costs and deployment thresholds are unknown.This setting provides no basis for preferring particular mistake corrections.
Guidance: For deployment scenarios with elevated false-negative costs, AUROC prioritizes reducing missed positives through improvements at lower score thresholds.The paper gives early screening for critical illnesses such as cancer as an example.
Guidance: For equitable resource distribution across diverse populations, AUROC avoids selectively favoring one subpopulation.AUPRC’s emphasis on high-score regions risks favoring higher-prevalence subpopulations.
Guidance: For single-group interventions where false positives are much costlier than false negatives and equity concerns are absent, AUPRC is recommended.The paper illustrates this setting with selecting candidate molecules for costly experimental validation.

5 Literature Review: Examining how Claim 1 Became so Widespread

A review of 1.5 million arXiv papers found that the claim that AUPRC is better than AUROC under class imbalance is widespread but poorly supported. It is frequently uncited, misattributed, and generalized beyond the settings supported by cited work.

Claim prevalence: The review identified 424 publications making the claim that AUPRC is better than AUROC in cases of class imbalance.The authors combined manual literature search with an automated search of over 1.5 million arXiv papers.
Citation practices: 167 of the 424 publications made the claim without an associated citation.These included papers from high-profile venues such as NeurIPS, ICML, and ICLR.
Citation practices: Among 257 publications citing a source, 135 cited no paper that actually made the claim.The claim was often attributed to, which discussed when precision-recall curves may be informative but did not assert that AUPRC should replace AUROC under imbalance.
Over-generalization: Valid arguments for preferring AUPRC in particular settings were often over-generalized into recommending it for every imbalanced problem.The paper distinguishes deployment-metric relevance and communication limitations from the separate question of comparing models using metric values.

6 Limitations and Future Works

The paper identifies several directions for strengthening its theory and empirical validation, including relaxing assumptions, expanding datasets and subgroup analyses, and studying additional metrics.

Theory: Theoretical results could be generalized to account for task difficulty, uncalibrated models, and more than two subpopulations.Theorem 3 in particular relies on restrictive assumptions, including perfect calibration.
Empirical scope: Future empirical work should include more fairness datasets and more nuanced analyses of how metric choice affects disparity measures.The authors also propose extending analyses to precision-recall-gain, net benefit, and single-threshold deployment metrics.

7 Conclusion

The study challenges the assumption that AUPRC is better than AUROC for imbalanced classification. Its analyses show that metric choice should be thoughtful and context-aware because AUPRC can favor higher-prevalence subgroups.

The study challenges the pervasive assumption that AUPRC is a better metric than AUROC in class-imbalanced settings.
AUPRC can act as an explicitly discriminatory metric by favoring higher-prevalence subgroups.
The authors advocate a more thoughtful and context-aware approach to selecting machine-learning evaluation metrics.

A Broader Impact and Ethical Considerations

The paper frames metric selection as an ethical issue because AUPRC may amplify algorithmic biases and affect decisions involving individuals and communities. It also uses a large-language-model-assisted literature analysis to re-examine established assumptions, while cautioning against interpreting COMPAS results as deployment guidance.

AUPRC may disproportionately favor models with higher positive-label prevalence, potentially amplifying algorithmic biases in high-stakes domains.
The authors use large language models to analyze scientific literature and re-evaluate long-standing machine-learning assumptions.
The paper highlights metric selection as an ethical concern when algorithmic decisions affect individuals and communities.
The study uses COMPAS because it is common in fairness research but does not advocate deploying recidivism-prediction models.

E Proof of Theorem 2

The proof analyzes an incorrectly ranked adjacent pair by swapping its assigned scores and tracking which thresholds change false positives. It then contrasts AUROC’s constant improvement with AUPRC’s score-dependent improvement, while noting a tie-specific limitation.

The proof constructs a corrected model by swapping the probabilities assigned to an incorrectly ranked adjacent pair.
Only the threshold at the lower score of the adjacent pair changes the false-positive count after the mistake is corrected.
AUROC improves by a constant amount for any corrected atomic mistake, whereas AUPRC favors mistakes involving higher scores and disfavors lower-score mistakes.
With strict threshold inequalities, separating tied positive and negative scores has no impact on AUROC or AUPRC.

F Proof of Theorem 3

The proof establishes that under sufficiently lower prevalence in one subgroup, the AUPRC-maximizing mistake lies entirely in the higher-prevalence subgroup. Calibration and vanishing score probabilities in the lower-prevalence subgroup drive this result.

The largest AUPRC improvement comes from correcting the mistake at the maximal score because this minimizes the firing-rate denominator.
The probability of a mistake in subgroup 1 remains at least δ, while mistakes involving subgroup 2 become vanishingly unlikely.
As subgroup 2 prevalence tends to zero, the probability of observing scores above the relevant threshold from that subgroup also tends to zero.
Therefore, the AUPRC-maximizing correctable mistake is purely within the higher-prevalence subgroup.

G Details for Synthetic Experiments

The synthetic experiments generate score-label samples with controlled AUROC and prevalence, then optimize AUROC or AUPRC through several score-perturbation procedures. The sampling procedure targets AUROC in expectation but does not guarantee the same target for the pooled population when subpopulations are generated independently.

The synthetic-score procedure samples positive scores, assigns negative-score windows using the target AUROC, and samples negatives within those windows.
Generated scores can be rescaled for prevalence expectations or calibrated without changing AUROC.
The procedure guarantees the target AUROC in expectation, but independently generated subpopulations need not preserve that target for the overall population.
M1 adds bounded random noise, evaluates the selection metric repeatedly, and returns the perturbation achieving the maximum value.
M2 fixes atomic mistakes sequentially, whereas M3 evaluates nearby score permutations and returns the best candidate at each step.
Figures 4 and 5 compare AUROC- and AUPRC-based optimization across groups with prevalences 0.05 and 0.01, including an initial AUROC of 0.65 in Figure 5.

H.1 Dataset Details

The real-world evaluation uses four binary-classification datasets with sex and race as protected attributes, balancing groups and using stratified train, validation, and test splits. Hyperparameter searches vary model complexity, learning settings, protected-attribute use, and group weighting.

The study evaluates Adult, COMPAS, LSAC, and MIMIC binary-classification datasets, using sex and race as protected attributes.
Groups are balanced by subsampling the majority group, then split into 50% training, 25% validation, and 25% test data stratified by group.
Table 1 reports sample counts after balancing and defines higher- and lower-prevalence subgroup statistics.
The hyperparameter grid varies tree depth, learning rate, estimator count, minimum child weight, protected-attribute inclusion, and higher-prevalence-group weighting.
Figures 7 and 8 summarize the raw AUC results from these experiments in alternative views.

I Literature Review Methodology

The literature review combines automated keyword and language-model screening with manual verification to identify papers making or referencing the AUPRC-over-AUROC claim. It reports the screening pipeline, final paper count, and associated review materials.

Corpus construction: The review assembled over 1.5 million ArXiv texts and additional conference papers from NeurIPS, ICML, ICLR, ACL, and CVPR.The ArXiv dataset was approximately 93.8 GB, while the NeurIPS scrape contained 9,680 texts.
Automated screening: Keyword scripts narrowed the ArXiv pool from 16,022 texts to 8,244 mentioning both AUPRC and AUROC, while 78 of 9,680 NeurIPS texts contained both keyword sets.The review began with separate keyword lists for the two metrics.
Manual verification: Researchers manually reviewed extracted claims, recorded quotations and citation status, and identified 424 papers making or referencing some version of the target claim.The final set included papers with claims that AUPRC is better than AUROC under class imbalance.
Materials and reproducibility: The review states that all identified papers, screening results, and extracted quotations will be made available upon publication.The paper also reports that experiments use synthetic or publicly available datasets and that code is fully released.