Source-linked AI summary
Beyond Boundary Noise: Aggregated Aleatoric Uncertainty Fails to Capture Presence Ambiguity in 3D Lung Nodule Segmentation
Simon Baur, Arne Schernich, Ekin Böke, Wojciech Samek, Jackie Ma
TL;DR
Clinically meaningful case-level disagreement about whether a pathology is present may not be captured by standard entropy-based uncertainty in medical image segmentation. This paper evaluates that question in 3D lung nodule segmentation and finds that entropy maps capture boundary noise but not presence ambiguity, whereas a supervised ambiguity head performs consistently better across architectures and cohorts.
Problem
Whether entropy-based uncertainty captures clinically meaningful case-level disagreement about pathology presence, rather than only pixel-wise segmentation variation, remains poorly understood.
Method
The study evaluates entropy-based uncertainty across four lung nodule segmentation architectures and trains a supervised ambiguity head on frozen bottleneck features.
Results
Across architectures and both LIDC and LNDb cohorts, entropy captures boundary noise but fails on presence ambiguity, while the ambiguity head remains strongest across metrics.
Takeaways & Limitations
Entropy-based aleatoric uncertainty should not be relied on as a proxy for clinical ambiguity in safety-critical lung nodule segmentation.
Takeaways & Limitations
Case-level ambiguity labels with multiple annotators remain rare in public datasets, limiting broader evaluation and motivating studies across other structures and modalities.
Abstract
from arXiv · showhide
Uncertainty estimation is critical for the safe clinical deployment of deep learning in medical image segmentation, with aleatoric uncertainty theoretically designed to capture irreducible data ambiguity. However, whether entropy-based measures reflect clinically meaningful ambiguity, i.e. case-level disagreement about whether a pathology is present at all, remains poorly understood. Contrary to most prior work, which focused on pixel-wise boundary disagreement, we systematically evaluate how well aleatoric uncertainty captures presence ambiguity. Our evaluation spans 3D lung nodule segmentation across four architectures with Monte Carlo dropout and deep ensembles, on LIDC-IDRI and an external validation cohort (LNDb). We find that entropy-based uncertainty maps align with boundary noise and minor drawing variation but carry insufficient discriminative signal for presence ambiguity. In contrast, a lightweight supervised ambiguity head trained on frozen segmentation features substantially outperforms all entropy-aggregation-based baselines across architectures, metrics, and both cohorts, and matches or exceeds methods that explicitly model ambiguity under disagreement supervision (Probabilistic U-Net, Annotator-Confusion 3D-UNet). A qualitative feature-space analysis shows that presence ambiguity is already encoded in the frozen encoder features of pixel-wise-trained networks, only to be discarded by the segmentation output and its entropy aggregation. Our findings expose a fundamental mismatch between the theoretical promise of aleatoric uncertainty and its practical behavior, and suggest that practitioners should not rely on entropy-based uncertainty as a proxy for clinical ambiguity in safety-critical applications.
1 Introduction
The paper argues that safe clinical deployment requires reliable uncertainty estimates, then shows that entropy-based aleatoric uncertainty captures boundary-level annotator noise but not clinically relevant presence ambiguity. It introduces a supervised ambiguity head that uses frozen segmentation features and explains that pixel-wise-trained networks already encode presence ambiguity internally.
- Motivation: Safe clinical deployment requires accurate predictions and reliable uncertainty estimates because erroneous or overconfident medical decisions can affect patient outcomes.
- Key finding: Entropy-based aleatoric uncertainty highlights object boundaries but fails to capture case-level disagreement about whether a lung nodule is present.This failure holds whether entropy is aggregated over the ground-truth or predicted mask.
- Proposed approach: A lightweight supervised ambiguity head trained on frozen segmentation features recovers annotator disagreement better than entropy aggregation from MC-dropout and ensemble predictions.It matches or exceeds Probabilistic 3D-UNet and Annotator-Confusion 3D-UNet across architectures on LIDC and LNDb data.
- Interpretation: UMAP analysis shows that networks trained with pixel-wise supervision already encode presence ambiguity in their internal representations.The segmentation output and entropy aggregation fail to retain this clinically relevant signal.
2 Related Work
Related work frames predictive uncertainty as entropy-based and commonly decomposes it into aleatoric and epistemic components. Other approaches model aleatoric uncertainty through globally consistent segmentation hypotheses sampled from conditional latent spaces, rather than independent pixel-wise fluctuations.
- Aleatoric Uncertainty From Spatial Inter-Annotator Disagreement: An ambiguous LIDC-IDRI sample with annotator disagreement about structure presence produces virtually no aleatoric uncertainty signal in MC-Dropout visualization.The figure contrasts non-ambiguous and ambiguous patches using annotator overlays, uncertainty maps, and mean predicted masks.
- Bayesian Uncertainty Estimation and Disentanglement: Predictive uncertainty is quantified using Shannon entropy and commonly decomposed into aleatoric and epistemic components.Aleatoric uncertainty represents irreducible input-dependent ambiguity, whereas epistemic uncertainty reflects reducible lack of knowledge.
- Bayesian Uncertainty Estimation and Disentanglement: In practice, aleatoric uncertainty is approximated from expected entropy across Monte Carlo dropout samples or deep ensembles.The separation of aleatoric and epistemic uncertainty has been studied in general benchmarks and medical image classification.
- Aleatoric Uncertainty From Spatial Inter-Annotator Disagreement: Probabilistic segmentation models use conditional latent spaces to sample globally consistent segmentation hypotheses that capture spatial annotation ambiguity.This approach differs from independent pixel-wise fluctuations by representing coherent segmentation variants.
3 Method and Experimental Setup
The study evaluates case-level lung-nodule presence ambiguity on LIDC-IDRI and external LNDb data, distinguishing annotator disagreement about presence from pixel-wise boundary variation. It compares aggregated voxel-wise uncertainty with a post-hoc ambiguity head trained on frozen segmentation bottleneck features.
- Datasets: Experiments use LIDC-IDRI with a 70/15/15 train/validation/test split and LNDb as an external validation cohort.LIDC annotations were independently produced and then revised after peer review, while LNDb provides external evaluation.
- Experimental comparison: Presence-ambiguity detection compares pixel-wise AU aggregation from Ensemble and MC-Dropout with the post-hoc Ambiguity Head across four architectures, LIDC, and LNDb.The comparison evaluates whether aggregated uncertainty aligns with the case-level ambiguity label.
- Ambiguity definition: Cases are labeled ambiguous when annotators disagree on nodule presence, covering partial-disagreement configurations distinct from boundary disagreement.Non-ambiguous cases have unanimous presence or absence, whereas ambiguous cases satisfy 0 < A(x) < 1; configurations include 1-vs-3, 2-vs-2, and 3-vs-1 on LIDC and 1-vs-2 and 2-vs-1 on LNDb.
- Uncertainty aggregation: Voxel-wise uncertainty maps are aggregated into scalar scores over annotator-union, ground-truth, or predicted-mask regions and evaluated against the case-level ambiguity label.The annotator union includes every voxel labeled foreground by at least one annotator and is non-empty for every evaluated case.
- Ambiguity head: The post-hoc ambiguity head applies attention pooling and a 3-layer ReLU MLP with sigmoid output to frozen bottleneck features from a pretrained segmentation backbone.The head is trained with binary cross-entropy and produces the ambiguity estimate Uamb(x).
4 Results
Across both cohorts and architectures, entropy aggregation remains near-random for detecting presence ambiguity, while the ambiguity head is strongly discriminative and better ranks disagreement severity. Feature-space analysis indicates that presence ambiguity is already encoded in frozen encoder representations but discarded by the standard segmentation output.
- Binary ambiguity detection: Entropy aggregation stays close to random across disagreement levels, whereas the ambiguity head remains strongly discriminative across both cohorts.This comparison averages over all four architectures and includes both MC-dropout and ensemble entropy aggregation.
- Binary ambiguity detection: At most 0.48 AUROC is achieved by ground-truth-mask entropy aggregation, versus at most 0.58 AUROC with predicted-mask aggregation.Predicted-mask aggregation consistently improves entropy-based baselines but remains far below the ambiguity head.
- Binary ambiguity detection: Annotator-Confusion 3D-UNet improves from 0.60 to 0.73 AUROC, while Probabilistic 3D-UNet improves from 0.51 to 0.64 with predicted-mask aggregation.These supervised disagreement-modelling baselines benefit more from predicted-mask aggregation than entropy-based methods.
- Disagreement severity: The ambiguity head increases monotonically across disagreement tiers and attains the strongest Spearman correlation across both cohorts and all backbones.Ground-truth-mask entropy aggregation is near-zero and flat, while pixel-wise aggregation and supervised baselines track severity weakly or inconsistently.
- Feature-space analysis: Consensus and ambiguous cases form clearly separable clusters across architectures, showing that presence ambiguity is already encoded in frozen encoder representations.The ambiguity head therefore recovers a latent signal that the standard segmentation pipeline fails to transport to its output.
5 Conclusion
Across four segmentation architectures and two datasets, entropy-based aleatoric uncertainty was ill-suited to detecting case-level presence ambiguity in lung nodule segmentation. The study presents the ambiguity head as a simple proof of concept and calls for richer training methods and broader evaluation despite limited public datasets with multiple annotator labels.
- 5 Conclusion: Entropy-based aleatoric uncertainty is ill-suited for detecting case-level presence ambiguity in lung nodule segmentation across four architectures and two datasets.One dataset was an external validation cohort.
- 5 Conclusion: The ambiguity head is only a simple proof of concept, motivating richer ways to incorporate presence ambiguity during training.Future work should extend the analysis to other structures and modalities.
- 5 Conclusion: Case-level ambiguity with multiple annotator labels remains rare in public datasets, a limitation the community should address.This scarcity constrains broader study of presence ambiguity.
Appendix
The appendix details the ambiguity-detection evaluation protocol and test-set composition, then provides supplementary metrics for ground-truth and prediction-mask uncertainty aggregation. It also includes comparisons across disagreement levels, architectures, cohorts, and supervised competitors.
- Evaluation protocol: Ambiguity detection uses AUROC, AUPRC, and F1 on held-out test cases, with the head trained post hoc and its optimal F1 threshold selected on validation data.LIDC test sets contain 88 (1vs3), 47 (2vs2), 47 (3vs1), and 80 non-ambiguous cases; LNDb uses a class-balanced down-sampled subset.
- Supplementary results: Tables 1 and 2 provide detailed metrics for disagreement-level detection and comparisons with supervised competitors.These tables supply the detailed results underlying the reported figure metrics.
- Prediction-mask aggregation: Tables 3 and 4 report prediction-mask aggregation results across architectures, disagreement thresholds, LIDC and LNDb, and ordinal correlations with disagreement severity.Prediction-mask aggregation is presented as an alternative to ground-truth-mask aggregation.