A Closer Look at AUROC and AUPRC under Class Imbalance

Motivation

• Metric choice is critical because inappropriate evaluation can hinder model comparison or cause substantial deployment costs, including misdiagnosis in healthcare.The paper distinguishes methodological comparison from application/deployment settings, where workflows and false-positive or false-negative costs may be known.Source: §1 · p. 1

Problem

• The widely repeated claim that AUPRC provides a better, more accurate, or fairer evaluation than AUROC under class imbalance is wrong.The claim appears across scientific literature, educational content, popular press, and AI assistants, but the paper identifies many supporting justifications as invalid or misapplied.Source: §1 · p. 2

Theoretical contributions

• AUROC weights all false positives equally, whereas AUPRC weights false positives according to the inverse likelihood of producing scores above threshold τ.Theorem 1 links metric suitability to whether reducing false positives above low or high thresholds matters more, rather than to class imbalance alone.Source: §1 · p. 2
• AUROC values improvements uniformly across positive samples, whereas AUPRC favors improvements for higher-scored samples and can prioritize higher-prevalence subpopulations over lower-prevalence ones.Theorems 2 and 3 show how this score sensitivity can amplify algorithmic bias and create fairness concerns in multi-population settings.Source: §1 · p. 2

Empirical and literature evidence

• The study validates its claims with synthetic experiments and public fairness datasets, then reviews over 1.5 million scientific papers to document widespread misuse of AUPRC relative to AUROC.The review covers high-stakes domains such as healthcare and venues including AAAI, NeurIPS, ICML, ICLR, Cancer Cell, Nature Journals, and PNAS.Source: §1 · p. 2

2.1 Relationship between AUROC and AUPRC

• Theorem 1 shows that AUROC minimizes unweighted expected false-positive rates across positive samples, while AUPRC weights them by inverse firing rate at each positive score.The distinction arises because precision depends on the model’s firing rate, whereas AUROC averages false-positive rates over the positive-score distribution without this model-dependent weighting.Source: §2.1 · p. 4

2.2 AUPRC prioritizes high-score mistakes, AUROC treats all mistakes equally

• Theorem 2 shows that correcting any incorrectly ranked adjacent pair improves AUROC equally, whereas AUPRC improves more when the corrected mistake occurs at a higher score.A single correction changes the false-positive rate by the same amount but affects AUPRC through the threshold at the mistake’s score.Source: Figure 1 · p. 3, §2.2 · p. 4

2.3 AUPRC is explicitly discriminatory in favor of high-scoring subpopulations

• Theorem 3 shows that, under sufficiently severe prevalence imbalance and calibration, the mistake whose correction most improves AUPRC can lie entirely in the higher-prevalence subgroup.Thus, AUPRC can prioritize optimization within high-risk groups rather than differentiation across subgroups, creating bias when fairness across populations matters.Source: §2.3 · p. 5

3.1 Synthetic optimization experiments demonstrate AUPRC-induced disparities

• The synthetic experiments compare two optimization procedures under matched subgroup optimization capacity, showing that disparity arises from the evaluation metric rather than unequal optimization opportunity.One procedure fixes individual mistakes, while the other samples a limited set of score permutations before selecting the best option.Source: §3.1 · p. 6
• AUPRC optimization consistently favors higher-prevalence groups, while AUROC improves both groups together; under realistic perturbations, low-prevalence AUROC can decrease.Figure 2 reports this pattern across mistake-fixing and constrained score-permutation procedures, with similar trends for per-group AUPRC.Source: Figure 2 · p. 6, §3.1 · pp. 6–7

3.2 Real-world experimental validation

• In four fairness datasets, overall AUPRC more strongly correlates with the AUROC gap than overall AUROC when prevalence ratios are higher, indicating stronger preference for high-prevalence groups.Datasets include adult, compas, lsac, and mimic, evaluated across sex and race attributes with XGBoost models and repeated random hyperparameter searches.Source: §3.2 · pp. 7–8
• Across fairness datasets and attributes, prevalence disparity was statistically significantly and strongly positively correlated with AUPRC bias amplification favoring high-prevalence groups (Spearman’s ρ = 0.905, p = 0.002), despite limited disparities.Source: §3.2 · p. 8

Practical guidance

• These recommendations guide metric selection for model development while still supporting reporting both metrics, broader metric sets, and ROC or PR curves when appropriate.The guidance concerns which metric to emphasize for model selection, hyperparameter tuning, or critical reporting in a given scenario.Source: §4 · p. 8

Context-independent model evaluation

• Use AUROC for context-independent evaluation because it uniformly values corrections across the entire score space when error costs and deployment thresholds are unknown.This supports impartial model evaluation without preferring mistakes at particular score thresholds.Source: §4 · p. 8

Elevated false negative costs

• Use AUROC when false negatives are especially costly, because low-threshold corrections that improve recall matter more than the high-score corrections prioritized by AUPRC.This applies to settings such as early screening for critical illnesses, where missed positive cases are the principal concern.Source: §4 · p. 8

Ethical resource distribution among diverse populations

• Use AUROC for equitable resource distribution across diverse populations because AUPRC can favor high-prevalence subgroups and induce bias.AUROC avoids selectively prioritizing improvements in high-score regions when all subpopulations should be treated equally.Source: §4 · p. 8

Claim dissemination and impact

• 424 publications made Claim 1, including high-profile medical and fairness-critical studies that selected AUPRC because of class imbalance.The review combined manual and automated searches of over 1.5M arXiv papers and found the claim affecting applications where recall or fairness matters.Source: §5 · p. 9

Attribution failures

• 167 of 424 papers cited no source, while 135 of the 257 citing sources cited no paper that actually made Claim 1.The unsupported or misattributed claim appeared across venues including NeurIPS, ICML, and ICLR; was incorrectly cited 144 times as support.Source: §5 · p. 9
• The frequently cited argues that precision-recall curves may be more informative than ROC curves under imbalance, not that AUPRC should replace AUROC for model comparison.Curves communicate performance across thresholds, whereas their areas are single-point summaries primarily used for deployment-agnostic comparison.Source: §5 · p. 9

Over-generalization

• Literature arguments often over-generalize valid cases for preferring AUPRC, conflate misleading metric values with poor model comparison, and promote AUPRC optimization even where false negatives are costly [409].The review distinguishes deployment metrics associated with precision-recall curves from the broader claim that AUPRC is preferable in every imbalanced setting.Source: §5 · p. 9

Theoretical Extensions

• The theoretical results could be generalized to account for subgroup task difficulty, uncalibrated models, and more than two subpopulations.These extensions would enable more nuanced comparisons beyond pairwise subgroup analyses.Source: §6 · p. 10

Empirical Extensions

• Future experiments should include more fairness datasets and develop finer-grained methods for probing how metric choice affects disparity measures.Source: §6 · p. 10

Alternative Metrics

• Analyses could examine additional metrics, including precision-recall-gain area, net benefit area [307], and single-threshold deployment-centric measures.Source: §6 · p. 10

Calibration

• Theorem 3’s perfect-calibration requirement is a major limitation, motivating analyses for imperfectly calibrated models and studies of post-hoc calibration safety after AUPRC- or AUROC-based optimization.Source: §6 · p. 10

• AUPRC is not generally superior to AUROC: AUROC minimizes FPR unbiasedly over positive scores, whereas AUPRC emphasizes higher-score regions.This distinction follows from the study’s theoretical and empirical analyses.Source: §7 · p. 10
• AUPRC can be explicitly discriminatory by favoring model performance improvements in higher-prevalence subgroups.The finding is supported across synthetic and real-world fairness datasets.Source: §7 · p. 10
• Metric selection should be balanced and context-aware to advance technically sound, equitable, and just machine-learning models.Source: §7 · p. 10

A Broader Impact and Ethical Considerations

• AUPRC may favor models serving subpopulations with more frequent positive labels, potentially amplifying algorithmic disparities and motivating rigorous metric validation in high-stakes domains.The paper highlights healthcare, finance, and criminal justice as contexts where biased evaluation can have profound societal repercussions.Source: A Broader Impact and Ethical… · p. 37
• LLM-assisted literature analysis is presented as a novel way to scrutinize long-standing ML assumptions and encourage more comprehensive, empirically rigorous, and ethically aware investigations.This approach is described as potentially setting a precedent for broader scientific examination in the field.Source: A Broader Impact and Ethical… · p. 37
• Metric selection can skew model performance toward certain groups, raising fairness concerns when algorithmic decisions affect individuals and communities.The paper frames this as an ethical dimension of evaluating ML models rather than only a technical measurement issue.Source: A Broader Impact and Ethical… · p. 37
• The COMPAS recidivism dataset is used because it is common in fairness research, but the paper does not advocate deploying automated recidivism predictions.The authors explicitly recognize the dataset’s societal issues.Source: A Broader Impact and Ethical… · p. 37

C Notation

• The formal setup draws i.i.d. paired feature-label samples (x, y) from X × Y, with binary labels y ∈ {0, 1} and a model f_θ producing continuous scores in (0, 1).The notation also allows score distributions conditioned on positive and negative labels, s+ and s−.Source: C Notation · p. 37
• The notation defines NP and NN as the counts of positive and negative examples and introduces threshold-based quantities for evaluating model scores.These definitions support the subsequent treatment of AUROC and AUPRC.Source: C Notation · p. 38

D Proof of Theorem 1

• The proof of Theorem 1 rewrites AUROC as 1 − E_s+[FPR] and AUPRC as E_s+[Prec], then expands precision using Bayes’ rule.The derivation uses threshold-conditioned true-positive rates and score distributions, with synthetic validation provided in public code; the AUPRC formulation differs from earlier formulations such as those in optimization literature [409].Source: D Proof of Theorem 1 · pp. 38–39

Proof of Theorem 2

• Correcting any atomic mistake decreases false positives by one at exactly the lower score threshold.Thresholds below or above the adjacent scores are unchanged; only the lower score threshold is affected.Source: E Proof of Theorem 2 · p. 40
• AUROC improves by a constant amount for every corrected atomic mistake, while AUPRC favors higher-score mistakes and disfavors lower-score mistakes.AUROC weights false-positive rates equally across positive samples, whereas AUPRC weights false positives inversely by model firing rate.Source: E Proof of Theorem 2 · p. 40
• Separating tied opposite-label scores has no effect on AUROC or AUPRC under the strict-inequality decision rule.Such ties induce no false positive, and perturbing the negative score leaves both the false-positive rate and firing rate unchanged.Source: E Proof of Theorem 2 · p. 40
• Theorem 2 is also supported by synthetic empirical validation available in the authors’ public code.The passage identifies the validation resource but reports no additional quantitative results.Source: E Proof of Theorem 2 · p. 40

Preliminary lemmas

• The proof first establishes calibration-based lemmas relating positive and negative score densities and bounding the firing rate by positive prevalence.Lemma 1 derives p+(t) = [t/(1−t)] [Py(y=0)/Py(y=1)]p−(t), while Lemma 2 gives FR(f, τ) ≤ Py(y=1).Source: F Proof of Theorem 3 · pp. 40–41

Theorem 3 setup

• Theorem 3 assumes two subgroups and perfect calibration for subgroup 0, then identifies the maximal-score mistake as producing the largest possible AUPRC improvement when corrected.The proof considers a threshold τ above which subgroup 1 has mistake probability at least δ ∈ (0, 1], while subgroup parameters remain fixed as subgroup 2 prevalence varies.Source: F Proof of Theorem 3 · p. 41

Theorem 3 proof

• As subgroup 2 prevalence tends to zero, the probability of observing any subgroup 2 mistake above τ tends to zero, whereas subgroup 1 mistakes remain at least δ, establishing the theorem’s claim.This excludes both cross-group mistakes and purely subgroup 2 mistakes in the limit.Source: F Proof of Theorem 3 · pp. 41–42

G.1 Sampling a random model with a given AUROC

• The sampling procedure generates score-label sets whose expected AUROC equals a user-specified target by sampling positive scores and placing negative scores across probability-weighted intervals.Negative scores are sampled within intervals between ascending positive scores using binomially derived probabilities; the process is repeated until the desired number of negatives is obtained.Source: §G.1 · p. 43

G.2 Calibration includes prevalence matching

• Generated scores may be rescaled to match prevalence-based expected probabilities or calibrated without changing AUROC, with calibration being the stronger condition.A calibrated model’s expected probability equals label prevalence, so prevalence matching alone is strictly weaker than calibration.Source: §G.1 · p. 43, §G.2 · p. 43

G.1 Sampling a random model with a given AUROC

• Applying the sampler independently within subpopulations guarantees target AUROC only within each subpopulation, because the overall value also depends on an unspecified xAUC term.In the reported experiments, this limitation did not meaningfully affect results, and joint AUROCs were sufficiently close to the target to avoid a more complex method.Source: §G.1 · p. 43

G.2 Calibration includes prevalence matching

• The appendix defines calibration as the condition that, for predicted probability q, the conditional probability of a positive label equals q.This calibration formulation underlies the accompanying prevalence-matching discussion.Source: §G.2 · p. 43

G.3 Details on optimization procedures

• The optimization appendix evaluates random-noise search, sequential atomic-mistake correction, and sequential nearby-score permutation as alternative procedures.M1 repeats random perturbation 100 times with δ ∈ [0, 0.1], while M2 sequentially fixes atomic mistakes for 50 steps.Source: §G.3 · pp. 43–44
• Random fixed-magnitude perturbations are biased toward improving the lower-prevalence group because its scores occupy a more compressed probability range.This effect creates more score permutations for the low-prevalence group independently of whether AUROC or AUPRC is optimized.Source: §G.3 · p. 44
• Figures 4–6 compare AUROC- and AUPRC-based optimization on per-group metrics across mistake-fixing, nearby-score permutation, and random-noise procedures.The synthetic comparisons use group prevalences of 0.05 and 0.01; Figure 5 additionally starts from AUROC 0.65 rather than 0.85.Source: Figure 4 · p. 45, Figure 5 · p. 46, Figure 6 · p. 47

H.1 Dataset Details

• Sex and race serve as protected attributes across all four datasets.Source: §H.1 · p. 48
• The study uses adult, compas, lsac, and mimic for binary prediction tasks involving income, recidivism, bar passage, and ICU mortality.The datasets restrict race to specified binary groups where needed, including African-American/Caucasian, White/Black, and White/Black patients.Source: §H.1 · p. 48
• Groups are balanced by subsampling the majority group, then datasets are split into 50% training, 25% validation, and 25% test sets stratified by group.Dataset statistics are reported in Table 1.Source: §H.1 · p. 48, Table 1 · p. 48

H.2 Hyperparameter Grid

• The hyperparameter grid varies tree depth, learning rate, estimator count, minimum child weight, protected-attribute inclusion, and higher-prevalence-group weighting.The specified ranges include max depth 1–9, estimators 50–1000, and group weights from 1 to 50.Source: §H.2 · p. 48

H.3 Additional Results

• Additional raw AUROC results are available in the experiment repository, with summarized views provided in Figures 7 and 8.Source: §H.3 · p. 48
• Figure 7 relates signed AUROC gaps to validation-set overall AUPRC and overall AUROC, reporting 95% confidence intervals from 20 random data splits.Parenthetical values indicate prevalence ratios between groups, and datasets are ordered by those ratios.Source: Figure 7 · p. 49
• Figure 8 reports a Spearman correlation of 0.905 (p = 0.002) between prevalence ratios and differences in AUROC-gap correlations with AUROC versus AUPRC.Each point represents a dataset–attribute combination.Source: Figure 8 · p. 49

I.1 Paper Acquisition

• The review began with approximately 1.5 million arXiv texts and manually scraped conference proceedings, including 9,680 NeurIPS papers from 1987–2019.The arXiv dataset was approximately 93.8 GB in JSONL format.Source: §I.1 · p. 49

I.2 Keyword-Driven Filtering Process

• Keyword scripts reduced the arXiv pool from 16,022 texts mentioning either metric to 8,244 mentioning both, while 78 of 9,680 NeurIPS texts mentioned both.The screening used separate AUROC and AUPRC keyword lists and a dual-mention criterion.Source: §I.2 · pp. 49–50

I.3 AI-Assisted Screening and Refinement; I.4 Manual Review

• GPT-3.5 identified 2,728 arXiv papers with explicit comparative claims, and GPT-4.0 Turbo refined these to 201 relevant arXiv papers and 2 relevant NeurIPS papers.The selected papers and GPT-identified claims were then compiled for collaborative manual review.Source: §I.3 · p. 50, §I.4 · p. 50

I.4.1 Final Papers

• Manual review identified 424 papers making or referencing some version of the claim that “AUPRC is better than AUROC in cases of class imbalance.”The review recorded claim quotations and whether they included citations.Source: §I.4.1 · p. 50, §I.4 · p. 50

1. Claims; 2. Limitations; 3. Theory Assumptions and Proofs

• The checklist states that the paper’s abstract reflects its contributions and scope, discusses limitations and future work in Section 6, and provides full theorem assumptions and proofs in the appendix.These checklist responses also emphasize matching claims to results, stating assumptions, and discussing scope and robustness.Source: §1 · p. 53, §2 · p. 53, §3 · p. 54

4. Experimental Result Reproducibility; 5. Open access to data and code; 6. Experimental Setting/Details

• The study reports complete experimental descriptions, fully released code, synthetic or publicly available datasets, and sufficient training details where applicable.Because the work studies machine-learning metrics rather than presenting a modeling result, the training-detail item is marked not applicable to the main contribution.Source: §4 · p. 54, §5 · p. 55, §6 · p. 55

7. Experiment Statistical Significance; 8. Experiments Compute Resources

• The paper reports appropriate statistical significance tests, and its synthetic experiments can be replicated in a Colab notebook while real-data experiments are described with released code.The checklist separately records the compute-resource justification and statistical-significance response.Source: §7 · p. 55, §8 · p. 56

9. Code Of Ethics

• The authors report compliance with the NeurIPS Code of Ethics, discuss positive and negative societal impacts related to metric misunderstanding and fairness, and state that no new data or models are released.They also report using only appropriately cited public datasets and conducting no crowdsourcing or human-subjects research, making asset documentation and IRB items not applicable.Source: §9 · pp. 56–58