Local Discrimination of Binary Classifiers via Localized AUC Functional

Results

• In the NHANES hold-out subset, conventional AUC was .78 [.69; .77] in the low-performance subgroup and .88 [.82; .93] in the high-performance subgroup; external UK Biobank AUCs were .76 [.74; .78] and .71 [.70; .72].The reported SHAP drivers of AUC variation were education, age, sex, current smoking, and prevalent hypertension.Source: Results · p. 2

1 Introduction

• LOCATOR uses LAUC to identify suboptimal-discrimination subgroups without subsetting, while the paper develops methods to locate, quantify, and test performance heterogeneity.A single cohort-level AUC cannot show whether discrimination varies across covariate-space regions; predefined subgrouping can lose power and miss multidimensional patterns.Source: §1 · pp. 3–4

2.1 Definition of Localized AUC Functional

• LAUC estimates a reference model’s AUC for an index participant by weighting participants according to similarity, with the paper using a Gower-distance-based kernel.The localized score is intended to reflect discrimination among participants phenotypically similar to the index participant.Source: §1 · p. 4, §2.1 · p. 5

2.3 Estimator Construction and Asymptotic Consistency

• The paper constructs an estimator of localized AUC (LAUC), establishes its asymptotic distribution, and provides a closed-form standard-error estimator; it also gives sufficient conditions under which LOCATOR identifies high- and low-performance covariate-space regions.Source: §1 · p. 4

2.4 Asymptotic Normality

• The paper establishes an asymptotic distribution for LAUC and gives an asymptotically valid confidence interval; its plug-in standard-error approach uses a logit transformation.The logit transformation approximately removes mean–variance dependence in the AUC outcome.Source: §1 · p. 4, §2.5 · p. 7

Conclusions

• Lower performance in a low-competence region may reflect validation-cohort statistical characteristics as well as model heterogeneity.The authors propose testing procedures to support inference and remediation in that region.Source: §1 · p. 4

2.2.1 Relationship of LAUC(xi) to global AUC under uniform similarity

• When similarity weights are uniform, LAUC is algebraically equivalent to conventional AUC.This connects the local functional to the global metric as a special case.Source: §2.2.1 · p. 6

2.2.3 Monotonicity Guarantees

• Under stated assumptions, expected LAUC is ordered across model-competence regions, supporting its use to distinguish regional discrimination.The paper identifies sufficient conditions for this ordering.Source: §2.2.3 · pp. 6–7

3.1 Mathematical Formulation

• For each index participant, LOCATOR computes a weighted AUC from the reference model’s predictions and outcomes, with weights reflecting phenotypic similarity.The input data must include the evaluated model’s predictors and variables needed to distinguish performance regions.Source: §3.1 · p. 9

3.2 LOCATOR algorithm and implementation details

• LOCATOR splits data into training and test sets, uses Gower distances to form local neighborhoods, and trains XGBoost to predict logit-transformed LAUC.Test-set predictions are used to identify high- and low-performance subgroups.Source: §3.2 · p. 10

3.3 Interpretation of High and Low Performance Regions with SHAP analysis

• SHAP values rank features contributing to predicted local performance, while gain-based feature importance is used as a sensitivity check for potential overfitting.The evaluation compares conventional AUCs between median-split test subgroups and examines AUC across cumulative prediction percentiles.Source: §3.3 · pp. 10–11

formance (Positive Control)

• In the positive control, predicted logit(LAUC(xi)) rankings reversed at x3 = 0.5, matching the designed change in which reference model was more precise.gM ranked higher when x3 < 0.5, while gN ranked higher when x3 ≥ 0.5.Source: formance (Positive Control) · pp. 12–13, Figure 1 · p. 13

4.2 Simulation Scenario 2: negative control experiment

• In the negative control, the noise-trained subgroup model produced overlapping ROC confidence bands, as expected when it could not distinguish performance groups.The XGBoost model used to identify the groups was trained on pure noise.Source: Figure 2 · p. 14, §4.2 · p. 14

4.3 Simulation Scenario 3: Low Performance in the Overlap Region

• AUC was 0.69 (95% CI 0.66–0.71) in the low-performance subgroup versus 0.88 (95% CI 0.86–0.90) in the high-performance subgroup, with separable confidence bands.The low- and high-performance groups were identified in the test set using the median of trained XGBoost predictions.Source: §4.3 · pp. 14–15, Figure 3 · p. 15
• The low-performance subgroup fell between two high-performance regions, where overlapping event and non-event distributions made separation difficult.The simulation used x1 ∼ N(A · y, 1), with distribution centers around x1 = 0 and x1 = 1.190232.Source: §4.3 · p. 15, Figure 4 · p. 16

Case-Mix

• The two-axis framework distinguishes model heterogeneity from cohort case-mix or risk-range variance, because a localized AUC drop alone does not establish structural model failure.It is intended to inform model remediation versus cohort selection.Source: Case-Mix · p. 16

5.1 Local Underperformance Scenarios

• Four underperformance scenarios are classified by covariate distribution, predicted-risk range, and true model specification, including gaps that arise even when the model is correctly specified.The scenarios distinguish cohort characteristics from model defects.Source: §5.1 · p. 16

5.1.1 Scenario 1.1: Performance Degradation in Overlap Regions

• When event and non-event risk distributions overlap in a low-performance region, additional information such as biomarkers or imaging may be needed to discriminate outcomes better.This scenario is diagnosed by wide risk range in the low-performance region paired with low model heterogeneity.Source: §5.1.1 · p. 17

5.1.2 Scenario 1.2: Homogeneous Covariate Clusters (Validation Cohort Artefact)

• A dense cluster with nearly identical covariates and risks can create a cohort-selection artifact; refitting on the same covariates will not improve discrimination.The proposed response is greater covariate diversity in the validation cohort or novel biomarkers that reintroduce risk variance.Source: §5.1.2 · p. 17

5.1.3 Scenario 1.3: Covariate Diversity with Flat Risk Range (Biomarker Exhaustion)

• With covariate diversity but flat predicted risk, omitted predictors call for model augmentation and refitting, whereas latent biomarker clustering points to cohort selection.The latter occurs when a high-leverage latent biomarker is predicted by observed covariates but is nearly constant in the validation cohort.Source: §5.1.3 · pp. 17–18

Test for Flat Predicted Risk

• The flat-risk test compares predicted-risk variance in the low-performance subgroup S with its complement Sc against threshold δ; reject when the upper bound RU is below δ.The alternative is a smaller variance in S, making this a lower-tail test.Source: Test for Flat Predicted Risk · p. 18

5.1.4 Scenario 2: Genuine Local Model Misspecification (Structural Failure)

• Structural misspecification is indicated when the low-performance subgroup has ample risk variance but observed AUC falls well below expected AUC; remedies include refitting or subgroup-specific interactions.Simulations report severe discrimination collapse, for example ΔAUC ≥ +0.35.Source: §5.1.4 · pp. 18–19

• The National Health and Nutrition Examination Survey (NHANES) is named.Source: §6 · p. 20

6.1 NHANES 1999–2010 Cohort Setup and Preprocessing

• NHANES 1999–2010 included 17,748 participants with linked mortality follow-up, and data were split into training and test subsets for LOCATOR development and evaluation.The described preprocessing included exclusions, imputation, and training an XGBoost model to predict logit(LAUC(xi)).Source: §6.1 · p. 20

6.2 Identification of PREVENT Competence Regions in NHANES

• In the NHANES test set, conventional AUC was 0.88 in the high-performance subgroup and 0.78 in the low-performance subgroup.The subgroups were formed by splitting at the median predicted logit(LAUC(xi)).Source: §6.2 · p. 21
• The ROC curves show separation between high- and low-performance subgroups for XGBoost models using maximum, 70th-percentile, and median delta.The figure describes the subgroups as those below and above the median predicted logit(LAUC(xi)), respectively.Source: Figure 5 · p. 22

6.3 External Validation of PREVENT Competence Regions in UK Biobank

• In UK Biobank, the high-performance subgroup had AUC 0.761 (95% CI 0.742–0.781) versus 0.708 (95% CI 0.697–0.719) in the low-performance subgroup.The regions learned in NHANES were applied to UK Biobank; the reported confidence intervals did not overlap and were below the .05 statistical significance threshold.Source: §6.3 · p. 23, Figure 7 · p. 23

SHAP analysis

• SHAP analysis identified age, sex, antihypertensive treatment, current smoking, and statin use among the most important performance drivers.The analysis also associated low performance with low poverty income ratio and histories of type 2 diabetes and cancer.Source: SHAP analysis · p. 23, Figure 8 · p. 24

7 Discussion

• In NHANES, PREVENT AUC was 0.88 in high-LOCATOR regions versus 0.73 in low-competence regions, and the regions were validated in UK Biobank.The discussion also reports SHAP factors associated with lower PREVENT performance.Source: §7 · p. 25
• LOCATOR uses localized LAUC to identify covariate-space performance regions without subsetting, while its diagnostic matrix and F-test distinguish model misspecification from validation-cohort case-mix restrictions.The discussion describes plug-in variance estimators and logit-stabilized confidence intervals based on non-parametric two-sample U-statistic asymptotic theory.Source: §7 · p. 25

7.2 Strengths and Limitations

• NHANES lacks non-fatal cardiovascular outcomes and is cross-sectional; the analysis is U.S.-restricted, and SHAP associations are not causal.The fixed similarity-weighting exponent may also not be optimal for every dataset.Source: §7.2 · pp. 25–26

7.3 Extensions and Future Directions

• Future work could explore adaptive weighting, other risk models, time-to-event outcomes, diverse cohorts, and prospective studies of clinical utility.The authors also suggest comparing XGBoost subgroup division with SHAP-based division.Source: §7.3 · p. 26

7.4 Conclusions

• LOCATOR and localized LAUC are proposed for identifying high- and low-performance regions of binary classifiers, with broader application beyond clinical risk prediction under development.Source: §7.4 · p. 26

8.1 Statement on AI use

• DeepSeek and Gemini assisted with Python code writing, while Claude Desktop assisted with R code writing.The authors state that AI-assisted code was reviewed, tested, and adapted by the authors.Source: §8.1 · p. 27
• Gemini Pro and Claude Pro assisted with LaTeX debugging, selected proof checks, and academic writing; the authors attribute the LAUC definition, LOCATOR algorithm, and Propositions 1–3 and proofs to themselves.Source: §8.1 · p. 27
• Gemini 3.6 Flash generated an initial draft of proofs for Propositions 4–6, which the authors subsequently reworked.The authors also used it to scan text for errors and take responsibility for the content.Source: §8.1 · p. 27

9 Proofs from Section 2

• Under constant weights, the localized AUC equals the traditional AUC.Source: §9 · p. 28

9.1 Assumptions

• The ordering argument assumes strict first-order stochastic dominance between regions and that the groups are separated with high probability.Source: §9.1 · p. 31

9.2 Useful Lemma

• The useful lemma writes conditional concordance as a convex combination of intra- and cross-region concordances, reducing the ordering result to inequalities among those terms.The stated reduction includes C2 ≤ C1,2 and C2,1 ≤ C1.Source: §9.2 · p. 33

1. Lower Bound: C1,2 ≥C2.

• Cases in S1 stochastically dominate cases in S2 if and only if F+1(t) ≤ F+2(t) for all t.Source: §1 · p. 33
• The stated ordering condition is F+1(t) ≤ F+2(t) for all t, with cases in S1 stochastically dominating cases in S2.Source: §1 · p. 33
• The derivation uses an integral involving F−2(t) and F+1(t), expressed as a boundary term.Source: §1 · p. 34

2. Upper Bound: C1,2 ≤C1.

• The upper-bound proof also invokes an ordering in which controls in S1 have stochastically lower scores than controls in S2.This condition appears alongside the upper-bound derivation.Source: §2 · p. 34
• C2 ≤ C1,2 ≤ C1 is the stated upper-bound ordering.The main theorem concerns ordering of population regional expectations across model-competence regions.Source: §2 · p. 34, §9.3 · p. 34

9.4 Estimator Construction and Asymptotic Consistency

• The LAUC estimator is unbiased and consistent under Definitions 1 and 2, with asymptotic normality and an asymptotically valid confidence interval also stated.Source: §9.4 · pp. 37–39

9.5 Test of homogeneous predicted risk

• The homogeneous-risk test compares logit-transformed predicted-risk variance in the low-performance subgroup S with its complement, using a lower-tail one-sided F-test.The null and alternative are framed using a non-flatness threshold ratio δ.Source: §9.5 · pp. 40–41

Assumptions

• The F-test requires S to be fixed independently of the test data; discovering and evaluating S on the same sample invalidates its nominal F-distribution.The passage also notes independent observations, with block-resampling adjustments needed for clustered or longitudinal settings.Source: Assumptions · p. 41

9.6 Supplementary Figures and Tables

• NHANES preprocessing reduced 62,160 respondents to a final analytical cohort of 17,748 mortality-linkage-eligible participants.UMAP showed no obvious clusters in the baseline variables displayed for the training data.Source: Figure 9 · p. 42, §9.6 · p. 42, Figure 10 · p. 43

9.7 Inclusion and Exclusion Criteria into External Validation Cohort.

• UK Biobank participants were selected using PREVENT-compatible age and exclusion criteria, and variables were harmonized to NHANES coding and measurement conventions.Source: §9.7 · p. 43

9.8 Cardiovascular Endpoint Definition

• 0.744 and 0.822 were the mean standard AUCs in the low- and high-performance NHANES subgroups, respectively, across 100 seeds.The cardiovascular endpoint used for validation was CVD death, matching the NHANES training outcome; three subgroup ROC curves also showed increasing performance from low to high.Source: §9.8 · p. 43, Figure 11 · p. 44, Figure 13 · p. 46