PaperlayerView exact revision on arXiv

Source-linked AI summary

When uncertainty quantification fails: structural vulnerabilities and corrections for selective prediction under clinical class imbalance

Walter Endrizzi, Flavio Ragni, Raffaele Marchesi, Stefano Bovo, Monica Moroni, Giuseppe Jurman, Venet Osmani

arXiv:2610.37561v1cs.LGcs.AI

TL;DR

Selective prediction defers ambiguous clinical cases, but severe class imbalance can make uncertainty scores reflect class membership rather than clinical ambiguity. The paper combines threshold-aware scoring, class-conditioned rejection, and class-stratified evaluation; its findings identify global minority penalties and describe ways to mitigate and expose them.

  • Problem

    In imbalanced clinical prognosis, uncertainty scores can reflect minority-class sparsity and class membership rather than patients' clinical ambiguity.

    Source: §1 · p. 2, §2.2.3 · p. 8

  • Method

    The framework pairs threshold-aware Hτ scoring with class-conditioned rejection and evaluates trajectory-specific risk-coverage using ccAUGRC.

    Source: §1 · p. 4, §2.3.3 · p. 11

  • Results

    Under severe imbalance, global entropy and margin structurally penalize minority patients, while Hτ with CCR mitigates minority exclusion and ccAUGRC makes subgroup failures observable.

    Source: §4 · pp. 24–25, §5 · p. 27

  • Takeaways & Limitations

    Threshold-aligned uncertainty and class-conditioned evaluation support safer selective prediction without systematically excluding minority patients under the evaluated conditions.

    Source: §4 · p. 25, §5 · p. 27

    The framework is formulated for binary prognostic tasks, and multiclass extension requires rethinking entropy rescaling and CCR partitioning.

    Source: §4 · p. 26

Abstract

from arXiv · show

Background. Selective prediction systems improve the safety of clinical artificial intelligence by deferring ambiguous cases to human experts while automating high-confidence decisions. However, this safety mechanism relies entirely on the premise that uncertainty scores accurately reflect clinical ambiguity. In prognostic tasks characterized by severe class imbalance, where the clinical outcome of interest is a minority, this assumption breaks down. Standard uncertainty measures cease to capture clinically aligned ambiguity; instead, they become a proxy for class membership, systematically penalizing minority patients regardless of their individual predictive clarity. Results. Across 56 synthetic conditions and three neurodegenerative cohorts spanning severe class imbalance to near-balanced conditions (multiple sclerosis, Parkinson's Disease, Alzheimer's Disease), our framework consistently maintained bounded, balanced risk across trajectories. In contrast, global metrics either collapsed minority Sensitivity or protected the minority at the expense of majority patients. Applying CCR with standard entropy fails to protect the minority because it breaks within-class monotonicity when tau=0.5. Combining CCR with threshold-aware metrics restores strict monotonicity, correctly identifies ambiguous patients, and safely mitigates minority exclusion. Conclusions. Under clinical class imbalance, standard uncertainty quantification measures dataset sparsity rather than clinical ambiguity, creating a systemic illusion of safety. This bias is a correctable structural flaw. Clinical pipelines deploying selective prediction under imbalance should adopt threshold-aligned uncertainty measures and class-stratified evaluation. This reduces class-specific risk imbalance, supporting safer selective prediction under the evaluated conditions without silently excluding the most vulnerable patients.

1 Introduction

The introduction argues that under clinical class imbalance, standard uncertainty scores can confuse class membership with clinical ambiguity, and global evaluation can hide minority exclusion. It presents threshold-aware scoring, class-conditioned rejection, and stratified evaluation as corrections to these structural vulnerabilities.

1 Introduction

  • Under imbalance, standard uncertainty scores can proxy class membership and dataset sparsity rather than distinguish triage-relevant ambiguity.The introduction links minority patients' sparse feature representations to structurally elevated uncertainty estimates.Source: §1 · pp. 2–3
  • Entropy's fixed 0.50 peak misaligns with shifted thresholds, margins can over-reject confident majority cases, and minority disagreement can remain higher at equal boundary distance.The introduction describes these as threshold-entropy mismatch, asymmetric spatial penalties, and structural uncertainty divergence.Source: §1 · p. 3
  • Global risk-coverage metrics can improve by excluding harder-to-predict minority patients, masking that exclusion as better overall safety.The described masking arises because these metrics aggregate risk across the full population.Source: §1 · pp. 3–4
  • The paper proposes Hτ for threshold-aligned uncertainty ranking, CCR for class-conditioned rejection, and ccAUGRC for trajectory-specific risk-coverage evaluation.The framework is evaluated on synthetic and real-world neurodegenerative prognostic tasks.Source: §1 · p. 4

1.1 Related Work

  • Related work leaves structural correction of threshold-entropy mismatch and imbalance-specific risk decomposition underexplored, despite work on imbalance handling and selective evaluation.The related-work review distinguishes class-conditional conformal prediction, which bounds error with variable-sized prediction sets, from fixed-bandwidth selective classification.Source: §1.1 · pp. 4–5

2 Methods

The methods define selective prediction around a prevalence-aligned decision threshold and develop and compare corrections to uncertainty scoring, rejection, and evaluation. They assess these approaches using class-stratified risk-coverage trajectories and summary metrics across synthetic conditions and three neurodegenerative cohorts within a shared modeling pipeline.

2.2 Structural vulnerabilities of standard uncertainty quantification

  • Under imbalance, entropy can misrank clinical ambiguity, global margin cutoffs can distort class retention, and minority uncertainty floors can drive systematic deferral.Entropy peaks at p = 0.5 rather than the clinical boundary; asymmetric margin ranges can eliminate or underrepresent the stable class, while compressed minority probabilities elevate uncertainty regardless of individual clarity.Source: §2.2.1 · p. 7, §2.2.2 · p. 7, §2.2.3 · p. 8

2.3.1 Threshold-aware ambiguity score

  • The threshold-aware score Hτ maps the clinical boundary to its entropy maximum and preserves within-class ranking by distance from τ.Hτ reduces to Shannon entropy when τ = 0.5; global use alone does not remove the minority uncertainty floor.Source: §2.3.1 · p. 9

2.3.2 Class-Conditioned Rejection

  • CCR ranks uncertainty separately within predicted classes, preserving their proportions and requiring a τ-centered metric to rank progressors correctly when τ < 0.5.Within the predicted-progressor queue, standard entropy peaks at p = 0.5 rather than at boundary false positives; class-specific ranking also bypasses the minority uncertainty floor.Source: §2.3.2 · pp. 9–10

2.3.3 Class-conditioned Area Under the Generalized Risk-Coverage Curve (ccAUGRC)

  • ccAUGRC evaluates risk and coverage within each class, exposing minority exclusion that can make global risk appear artificially low.It substitutes class-specific denominators and coverage for global values and integrates the risk-coverage curve through rmax = 0.80.Source: §2.3.3 · p. 11

2.4.1 Real-World clinical cohorts

  • The real-world evaluation uses three cohorts spanning severe to near-balanced prevalence, with each clinical decision threshold set to its observed class prior.The cohorts are multiple sclerosis (π = 0.13), Parkinson’s disease (π = 0.39), and Alzheimer’s disease (π = 0.54).Source: §2.4.1 · pp. 11–12

2.4.2 Synthetic data

  • Synthetic experiments vary class prior, separability, and sample size across 56 Gaussian conditions to characterize standard uncertainty failures under controlled settings.The grid uses seven priors, four Cohen’s d separability values, and N ∈ {300, 1000}; real cohorts provide non-Gaussian validation.Source: §2.4.2 · pp. 12–13

2.5.1 Asynchronous ensemble pipeline

  • A shared ensemble pipeline uses stratified bootstrap splits, independently optimized classifiers, held-out Platt calibration, and patient-level aggregation of calibrated predictions.Thirty iterations use 80/20 splits; the ensemble mean supplies predicted probability and its standard deviation serves as a proxy for epistemic uncertainty.Source: §2.5.1 · pp. 13–14

2.5.2 Baseline and Control Methods

  • The comparison includes entropy and margin baselines, global Hτ, CCR variants, and a random-order control; Margin + CCR is omitted because it exactly matches Hτ + CCR.HTotal + CCR tests the need for threshold-aware scoring, while Random + CCR separates class-conditioned rejection from the uncertainty signal.Source: §2.5.2 · pp. 14–15
  • The comparison set included Hτ + CCR, global Margin and HTotal baselines, Hτ and HTotal + CCR ablations, and a Random + CCR control.Source: Table 1 · p. 15

2.5.3 Evaluation protocol and statistical analysis

  • Methods were evaluated across rejection rates from 0 to 0.80 in 0.05 steps, tracking class-stratified Sensitivity, Specificity, and retained-cohort AUPRC.Source: §2.5.3 · p. 15
  • Clinical summaries included Sensitivity stability over 0%–50% rejection, class-conditioned AUGRC for each trajectory, and Global AUGRC.Sensitivity stability was estimated as a linear-regression slope; ccAUGRC integrated each trajectory over the full rejection range.Source: §2.5.3 · p. 15
  • Inference used 1,000 patient-level bootstrap resamples of aggregated predictions, bootstrap sign tests, and cohort- and metric-specific Holm-Bonferroni correction at family-wise α = 0.05.Reported point estimates used 95% confidence intervals; significance required intervals to exclude zero and comparisons to satisfy the adjusted threshold.Source: §2.5.3 · pp. 15–16
  • Synthetic experiments mapped the π and d grid using Sensitivity at 30% rejection and its stability slope; full curves for 56 conditions were in the Supplementary Material.Source: §2.5.3 · p. 16

3 Results

Across synthetic conditions and clinical cohorts, class-conditioned triage with threshold-aware uncertainty bounded risk across trajectories, while global uncertainty metrics could exclude minority patients or reduce majority retention. The results also show that uncertainty divergence and these vulnerabilities diminish near class balance.

3.1.1 Phase diagram of standard UQ failure

  • At π = 0.10 and d = 2.0, Hτ + CCR recovered up to +0.99 Sensitivity, while at π = 0.50 its difference from baseline was zero.Standard HTotal collapsed under severe imbalance and overlap; at π = 0.10 and d = 1.0, baseline minority Sensitivity was 0.11.Source: §3.1.1 · p. 16, Figure 3 · p. 17

3.1.2 Sensitivity stability scaling law

  • Hτ alone reduced the exclusion slope at π = 0.10 from −2.09 to −0.26, but Hτ + CCR maintained a positive slope between +0.13 and +0.24 across tested priors.The full dual architecture addressed the remaining spatial penalty that left minority patients leaking under Hτ alone.Source: §3.1.2 · p. 17
  • At π = 0.10, standard UQ achieved Specificity of approximately 0.94 by displacing rejections onto the minority class; Hτ + CCR reallocated deferrals to recover minority Sensitivity.The resulting trade-off was a modest Specificity reduction for substantial minority Sensitivity recovery.Source: §3.1.2 · pp. 17–18, Figure 4 · p. 18

3.2.1 Empirical confirmation of uncertainty divergence

  • At equivalent relative boundary distances, minority ensemble disagreement was higher in MS and PD (p < 0.01), but the asymmetry was not significant in near-balanced AD.The synthetic analysis likewise found the gap narrowed as minority prevalence increased and dissipated at parity.Source: §3.2.1 · p. 19, Figure 5 · p. 20

3.2.2 Rejection curve analysis

  • In MS, HTotal drove minority Sensitivity to zero by 35% rejection, while global Margin reduced majority Specificity to nearly 0.20 at high rejection.Hτ + CCR simultaneously improved target-class Sensitivity and stable-class Specificity; all frameworks converged to nearly identical trajectories in near-balanced AD.Source: §3.2.2 · pp. 19, 20, 22, Figure 6 · p. 21
  • At 30% rejection in MS, the framework deferred false negatives at 58.1% versus 27.9% for true negatives and false positives at 33.5% versus 19.7% for true positives.The targeted filtering trends held across evaluated cohorts and were associated with increased AUPRC as uncertain predictions were deferred.Source: §3.2.2 · pp. 20–21

3.2.3 Class-conditioned safety evaluation

  • Under severe and moderate imbalance, Hτ + CCR was the only evaluated method to bound risk for both clinical trajectories while minimizing class disparity.The comparison was evaluated with class-conditioned AUGRC; the table reports lower values as better safety/lower risk.Source: §3.2.3 · p. 22, Table 2 · p. 23

4 Discussion

Under severe imbalance, global uncertainty scores can reflect class membership rather than clinically aligned ambiguity, while threshold-aware uncertainty paired with class-conditioned rejection addresses this structural failure. The discussion also qualifies where the approach applies and how its operational trade-offs differ from conformal methods.

4 Discussion

  • Standard uncertainty measures can penalize minority patients through threshold-entropy mismatch, asymmetric spatial penalties, and structural uncertainty divergence.These mechanisms conflate minority-class sparsity with decision-level ambiguity.Source: §4 · p. 24
  • When class prior and separability jointly fall below π ≈0.20 and d ≈1.0, Hτ realigns the uncertainty signal and CCR ranks patients within predicted trajectories.Both components are required to recover performance in this phase-diagram region.Source: §4 · p. 24
  • Class preservation alone substantially mitigates minority exclusion, while Hτ provides an additional benefit by ordering ambiguous cases within protected queues.Random + CCR isolates the contribution of class-conditioned rejection.Source: §4 · p. 24
  • At π = 0.54 in the near-balanced Alzheimer’s cohort, the proposed framework converges to baseline behavior without measurable penalty, while HTotal retains a substitution artifact.In this setting, HTotal preferentially protects the larger progressor class at stable patients’ expense.Source: §4 · p. 24
  • CCR trades formal distribution-free coverage bounds for predictable review quotas distributed proportionally across predicted trajectories.Class-conditional conformal prediction can yield variable rejection volumes and uninformative minority prediction sets under severe imbalance.Source: §4 · p. 26
  • The framework is formulated for binary prognostic tasks, assumes a known fixed threshold, and uses bootstrap intervals that omit model-training variance.When τ shifts substantially from π, CCR queue sizes still reflect the empirical dataset distribution.Source: §4 · p. 26

5 Conclusion

Under severe clinical class imbalance, standard global uncertainty metrics can mistake dataset sparsity for clinical ambiguity and contribute to disproportionate deferral of minority patients. The paper finds that threshold-aware uncertainty combined with class-conditioned rejection mitigates this failure, supporting subgroup-specific risk evaluation in clinical triage.

5 Conclusion

  • Under severe imbalance, standard uncertainty measures may reflect dataset sparsity rather than clinically aligned ambiguity, while global metrics can conceal disproportionate deferral of minority patients.Source: §5 · p. 27
  • Combining Hτ with CCR mitigates minority exclusion by aligning uncertainty with the decision threshold and distributing rejection proportionally across predicted trajectories.The stated proportionality allows finite-sample rounding errors.Source: §5 · p. 27
  • Clinical pipelines should measure and control risk within patient trajectories using class-stratified metrics rather than infer subgroup safety from global risk-coverage evaluation.Source: §5 · p. 27

6 Declarations

The declarations report study funding, no competing interests, data-access restrictions, and code availability.

6.1 Funding

  • The study was partially funded through the National Plan for Complementary Investments to the NRRP, project D34H.Funding is attributed to the Italian Ministry of University and Research.Source: §6.1 · p. 27

Conflict of interest

Data availability

  • ADNI data were used under license and are not publicly available, while the remaining analyzed datasets are private but available from the corresponding author on reasonable request.The private datasets’ baseline characteristics and validation protocols were previously described in and.Source: Data availability · p. 28

Code availability

A.1 Data Source and Ethics Statement

The appendix identifies ADNI as the source for the Alzheimer’s disease data and describes the cohort used for a 36-month conversion prediction task. The analytic cohort excludes cases without ascertainable 36-month outcomes.

A.1 Data Source and Ethics Statement

  • ADNI data were used to study progression of mild cognitive impairment and early Alzheimer’s disease using imaging, biomarkers, and clinical and neuropsychological assessments.Source: §A.1 · p. 30

A.2 Cohort Selection Criteria

  • The 36-month conversion task included baseline MCI subjects, with 170 progressors and 144 stable subjects, yielding 314 patients after excluding those without definitive follow-up outcomes.Source: §A.2 · p. 30

A.3 Feature Engineering and Preprocessing

The study predicts 36-month outcomes from baseline demographic and clinical or cognitive features, using training-fold preprocessing to prevent data leakage. Its experiments use a shared ensemble pipeline, with classifier hyperparameters searched independently by grid search cross-validation.

A.3 Feature Engineering and Preprocessing

  • Baseline demographic and clinical or cognitive assessments were used to predict 36-month outcomes.The assessments include MMSE, CDR-SB, and ADAS-Cog.Source: §A.3 · p. 30
  • Features with more than 50% missingness were excluded; retained values were imputed using training-fold statistics and continuous features were z-score standardized.Continuous values used the training-distribution median, while categorical values used the mode.Source: §A.3 · pp. 30–31

B.1 Hardware and software environment

  • The asynchronous pipeline was run identically across clinical and synthetic cohorts in a containerized SLURM environment.Each task used 16 CPU cores, 60 GB RAM, and a 72-hour wall-clock limit.Source: §B.1 · p. 31

B.2 Hyperparameter search space

  • Each base classifier was tuned to maximize AUPRC, sampling up to 20 configurations per classifier during each of 30 bootstrap iterations.The search spaces are listed in Table B2.Source: §B.2 · p. 32, Table B2 · p. 32

C.1 Phase diagram of standard UQ failure (n=300 scenario).

The n=300 synthetic phase map shows that standard global uncertainty can exclude minority patients under severe imbalance and overlapping distributions, while the proposed architecture restores coverage. Figure C2 reports sensitivity stability and sensitivity and specificity at a fixed rejection rate.

C.1 Phase diagram of standard UQ failure (n=300 scenario).

  • Under severe imbalance (π ≤0.20) and overlapping distributions (d ≤1.0), standard global HTotal yields minority Sensitivity ≈0, while Hτ + CCR rescues coverage and converges to baseline behavior at π = 0.50.Figure C1 maps minority Sensitivity at a fixed 30% global rejection rate across class prior and separability.Source: Figure C1 · p. 33, §C.1 · p. 33

C.2 Sensitivity stability scaling law (n=300 scenario).

  • Figure C2 reports Sensitivity stability over 0%–50% rejection and Sensitivity and Specificity at a fixed 30% rejection rate for n = 300.The text identifies these as the measures reported, without giving their values.Source: §C.2 · p. 33

C.3 Rejection curves on synthetic data.

Across synthetic rejection trajectories, Hτ + CCR maintains bounded minority protection, unlike global HTotal, which rapidly loses minority coverage under severe imbalance and overlap. At a fixed rejection rate, the proposed framework restores minority retention and redistributes deferrals toward a class-proportional profile.

C.4 Rejection curves on synthetic data.

  • At 30% rejection, standard UQ collapses minority Sensitivity under severe imbalance, while Hτ + CCR recovers it and yields a balanced, class-proportional rejection profile.Figure C2 also reports Sensitivity stability across 0%–50% rejection; standard HTotal shows nonlinear sensitivity scaling as imbalance increases.Source: Figure C2 · p. 34
  • Under severe imbalance (π ≤0.20) and overlap (d ≤1.0), HTotal rapidly collapses minority Sensitivity, while Hτ + CCR maintains stable, bounded protection through 80% rejection and approaches baseline behavior as π approaches 0.50.Figures C3 and C4 show trajectories for N = 1000 and N = 300, respectively; global Hτ alone delays depletion but fails at higher rejection rates.Source: §C.3 · pp. 33–34

C.4 Controlled evaluation of structural uncertainty divergence

Controlled synthetic datasets isolate how class prevalence relates to uncertainty-estimation asymmetry, separating that analysis from varying clinical feature spaces and biological complexity.

C.4 Controlled evaluation of structural uncertainty divergence

  • The controlled synthetic analysis isolates class-prevalence effects on uncertainty estimation from clinical feature-space and biological-complexity variation.Source: §C.4 · p. 35
  • At fixed separability d = 1.0, the minority prior was varied across π ∈ {0.15, 0.30, 0.50} at dataset sizes N = 300 and N = 1000.The N = 300 and N = 1000 analyses are reported in Figures C5 and C6, respectively.Source: §C.4 · p. 35

Appendix D Results on real-world data

Table D3 recalculates class-conditioned and global generalized risk-coverage metrics over a shared integration domain. It also reports class disparity and adjusted significance information.

Appendix D Results on real-world data

  • Table D3 recalculates ccAUGRC and Global AUGRC over a standardized integration domain shared across methods.Lower values indicate lower risk.Source: Table D3 · p. 35
  • The table adds class disparity and reports point estimates with 95% confidence intervals from 1,000 post-prediction bootstrap iterations.Source: Table D3 · p. 35
  • Arrows show whether the baseline has higher or lower risk, and red asterisks mark significance retained after Holm-Bonferroni correction.Source: Appendix D · p. 35
Loading 2610.37561v1…