Source-linked AI summary
Uncertainty Identifies Difficult Samples Across Methods: A Multi-Task Study on a Heterogeneous Skin Lesion Dataset
Leon Koole, Jiapan Guo, Matias Valdenegro-Toro
TL;DR
Uncertainty quantification methods are rarely compared across families on heterogeneous skin-lesion data, leaving agreement about difficult cases unclear. This study compares five methods across two diagnostic tasks and finds that difficulty rankings are largely method-agnostic, while Deep Ensembles leads calibration.
Problem
Evidence on how different uncertainty-quantification families identify difficult lesions in pooled, heterogeneous data remains limited.
Method
The study compares five uncertainty-quantification methods on a shared backbone with binary and five-class diagnostic heads across accuracy, calibration, decomposition, and risk-coverage.
Results
Per-sample entropy rankings correlate 0.54 to 0.91 across methods, while Deep Ensembles achieves the lowest ECE on both heads.
Takeaways & Limitations
Uncertainty can support selective referral by ranking difficult cases, while method choice matters more for calibration and uncertainty decomposition than difficulty ranking.
Takeaways & Limitations
The background-swap evidence is suggestive rather than conclusive because it uses only 16 images and changes lighting and contrast alongside the background.
Abstract
from arXiv · showhide
Skin lesion classifiers can be confidently wrong on the cases that matter most, so knowing when a prediction should not be trusted is clinically as useful as the prediction. We study uncertainty quantification on a dataset pooled from many ISIC sources, with a shared backbone and two jointly learned heads: a binary malignant versus non-malignant head and a five-class diagnostic head. Five UQ methods (MC Dropout, DropConnect, Flipout, Deep Ensembles, DUQ) are compared on accuracy, calibration, uncertainty decomposition, and risk-coverage. Difficulty is largely method-agnostic: even methods with narrow entropy distributions rank the same samples as hard (per-sample entropy correlations of $0.54$ to $0.91$). The choice of method matters more for calibration and uncertainty decomposition, where Deep Ensembles is the clear winner, than for finding difficult cases. The ranking is also good enough that deferring the most uncertain cases removes a disproportionate share of errors, supporting uncertainty-based selective referral, evaluated here in-distribution only.
1 Introduction
Uncertainty quantification can support clinician referral for difficult skin-lesion cases, but existing dermatology studies often use limited methods and curated benchmarks. This study evaluates five methods on a heterogeneous ISIC dataset and finds that methods largely agree on difficult samples, while uncertainty fails on confidently misleading cases and is intended for triage rather than autonomous diagnosis.
- Motivation: Melanoma can closely resemble benign lesions, making reliable malignant-versus-benign classification clinically important while histopathology remains invasive and reserved for suspicious cases.Localized melanoma has a five-year survival above 99%, versus roughly one-third after spread.
- Motivation: Uncertainty quantification enables classifiers to abstain on least-confident cases and defer them to clinicians.The introduction describes Bayesian approximations, deep ensembles, and single-pass DUQ as distinct UQ families.
- Research gap: Prior dermatology work commonly evaluates only one to three UQ methods from one family on a single curated benchmark.Existing studies nevertheless show that uncertainty can distinguish reliable from unreliable predictions and flag difficult or out-of-distribution cases.
- Study design: The study compares five UQ methods and a softmax baseline on a heterogeneous ISIC dataset using a shared backbone and two prediction heads.The pooled test set combines dermoscopic, clinical, and total-body-photography images at roughly a 70/30 benign/malignant split.
- Main findings: 0.54 to 0.91 rank correlations show that methods largely agree on which lesions are difficult, suggesting difficulty is primarily a property of samples.The agreement supports selective referral, but entropy dips on systematically misleading cases where every model is confidently wrong.
- Scope and limitations: The work supports uncertainty-based triage rather than autonomous diagnosis and notes a fairness gap on darker skin.The stated limits include this fairness concern, while the evaluation is framed as referral support.
2 Evaluating Uncertainty as Difficulty Estimation
The study evaluates uncertainty-based difficulty estimation on a deliberately heterogeneous, imbalanced ISIC dataset using a shared multi-task backbone and five UQ methods. It assesses predictive quality, calibration, uncertainty, and model-independent difficulty through consensus and leave-one-model-out tiers.
- Evaluation setting: 92,092 images from 18 data sources form a deliberately heterogeneous dataset spanning clinical and dermoscopic imaging protocols, devices, and patient populations.About 90% of images are dermoscopic, with the remainder comprising clinical close-ups and TBP tiles.
- Dataset construction: The pooled data preserve a near 70/30 benign-to-malignant ratio across splits, while 3,289 indeterminate cases are excluded from training for separate difficult-sample evaluation.The remaining 88,803 benign and malignant images are used for train, validation, and test splits.
- Multi-task architecture: A shared ImageNet-pretrained EfficientNet-B3 backbone jointly trains binary H1 and five-class H2 heads under weighted cross-entropy.H1 separates malignant from non-malignant lesions, while H2 predicts consolidated diagnostic categories.
- UQ methods: Five UQ methods—MC Dropout, DropConnect, Flipout, Deep Ensembles, and DUQ—are compared with a plain deterministic softmax baseline.The methods span Bayesian approximations, ensembles, and single deterministic approaches.
- Difficulty and metrics: Difficulty tiers count how many UQ methods misclassify each sample, with leave-one-model-out recomputation reducing circularity when evaluating an individual method.Predictive quality uses accuracy; calibration uses ECE and confidence AUROC; total predictive uncertainty is Shannon entropy of the mean predictive distribution.
3 Results
Deep Ensembles performs best overall for calibration and uncertainty decomposition, while methods largely agree on which samples are difficult. Uncertainty supports selective referral, but some systematically misleading cases are confidently wrong and dataset artifacts create safety concerns.
- Model Comparison: Deep Ensembles leads or ties on accuracy and has the lowest ECE on both heads: 0.013 on H1 and 0.019 on H2.DropConnect is accurate but overconfident, with H2 ECE 0.078; Flipout trails the baseline by about four points on H1 and more than ten on H2.
- Model Comparison: Only Deep Ensembles and MC Dropout yield meaningful epistemic entropy components, whereas DropConnect assigns ≈3% and Flipout < 0.1% to model uncertainty.DUQ is single-pass and has no epistemic decomposition; its entropy instead comes from normalized RBF-kernel outputs.
- Heterogeneity from Pooling Sources: Diagnosis confirmation type is the strongest independent entropy predictor for every method, while data source has a moderate controlled effect of partial η2 ≈0.01–0.03.The Royal Prince Alfred Hospital subset has the highest mean entropy and lowest accuracy for every method; Deep Ensembles retains 0.72 accuracy versus 0.63 for the next-best method.
- Difficult Samples: Entropy rises with LOMO difficulty for every method, but tier 4 reverses this pattern with an entropy dip and overconfidence peak, indicating confidently wrong cases.Non-malignant samples with very low entropy almost all carry clinician-applied measurement markers, an unquantified observation associated with false-negative risk.
- Difficult Samples: Per-sample entropy correlations range from 0.54 for DropConnect–Flipout to 0.91 for DUQ–Deep Ensembles on H1, with a similar band on H2.Every method assigns higher mean entropy to the 3,289-case indeterminate set than to the test set, independently corroborating the difficulty result.
- Selective Referral: Deferring the most uncertain predictions should remove a disproportionate share of errors, and Deep Ensembles achieves the lowest H1 AURC at 0.012.Which method ranks best depends on the head, but uncertainty ranks errors well enough to support selective referral in-distribution.
4 Discussion and Conclusion
Across methods, uncertainty largely identifies which samples are difficult, and its ranking is sufficient to support selective referral in-distribution. However, uncertainty can miss confidently wrong cases, while fairness and architecture limitations constrain generalization.
- Difficulty is largely method-agnostic: entropy rankings agree closely across methods, and cheaper single-pass DUQ recovers almost the same ordering.Methods with narrow entropy distributions flag the same images as methods with wide distributions.
- Selective referral removes a disproportionate share of model errors, producing low AURC values for Deep Ensembles, DropConnect, and MC Dropout on H1.This evidence is limited to the in-distribution setting because no dedicated out-of-distribution set was tested.
- At the highest difficulty tiers, entropy dips while overconfidence peaks, so high-entropy deferral can miss cases where models fail most consistently.This is described as a limitation of uncertainty-based triage rather than a tuning problem.
- Fairness claims are constrained by the predominantly light-skinned test population and sparse malignant samples for darker Fitzpatrick types.The test set has effectively zero malignant samples for type VI and only three to six for types IV and V.
- Deep Ensembles performs most consistently for calibration and uncertainty decomposition, while all findings require confirmation beyond the single EfficientNet-B3 backbone.DUQ is nearly as effective for difficulty ranking alone, but generalization to other architectures remains unconfirmed.
Detailed Per-Model Multivariate Regression Results
The section reports full Type II ANOVA multivariate regressions of predictive entropy across all five UQ methods. Across models, confirmation type is the strongest entropy predictor.
- Confirmation type is the strongest predictor of entropy across all models.The regressions use Type II ANOVA tests, with η2 as the effect size.
- Full regression results are provided for Flipout, MC-Dropout, MC-DropConnect, DUQ, and Deep Ensembles in Tables 2–4.Tables report Type II sums of squares and significance markers for p < 0.05, p < 0.01, and p < 0.001.
- Entropy is modeled from diagnosis, confirmation type, approximate age, attribution, image type, sex, and anatomical site.The model includes N = 3617 observations.
Entropy Growth in Indeterminate Test Set
The indeterminate test set produces significantly higher predictive entropy across all UQ methods, indicating that it is difficult to recognize this unlabeled class. Table 5 summarizes these entropy differences using relative increases.
- All UQ methods show a significant entropy increase on the indeterminate test set.This supports the expectation that the indeterminate set reflects difficulty in recognizing an unlabeled class.
- The analysis compares mean predictive entropy between test and indeterminate sets.Results are presented in Table 5 for all UQ methods.
- A larger relative entropy increase indicates that a method responds more strongly to sample difficulty.Table 5 reports this relative increase as ∆%.
Supplementary Material
Supplementary analyses probe dataset structure and shortcut sensitivity. On 16 Royal Prince Alfred images, removing the white background generally reduced predictive entropy, though the result is suggestive rather than conclusive.
- Shortcut sensitivity: 16 Royal Prince Alfred images were edited by replacing the white background with black, then additionally removing surgical markers, and entropy was measured for both heads.The experiment tests whether the uniform white background acts as a shortcut.
- Shortcut sensitivity: H1 entropy fell for all five methods after background removal, while H2 entropy fell for four of five methods.Deep Ensembles was the exception, with H2 entropy rising slightly in both edited variants.
- Limitations: n = 16 makes the background-editing result suggestive rather than conclusive, and marker removal deepened most entropy drops without changing their direction.Accuracy deltas were not reported because each flipped case changes accuracy by about 6.3 points.
- Diagnostic grouping: 22 fine-grained secondary diagnosis categories were collapsed into five classes for the H2 task: Benign, Melanoma, Malignant Non-Epidermal, and Other, with the fifth category defined in the grouping scheme.The supplied grouping descriptions enumerate the included diagnostic categories and proliferations.