PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Trustworthy AI for Glioma Diagnosis: A Task-Aware Evaluation of Uncertainty Quantification

Gonzalo Esteban Mosquera Rojas, Sebastian R. van der Voort, Carolin M. Pirkl, Sandeep Kaushik, Marion Smits, Stefan Klein

arXiv:2609.39429v1cs.CVcs.AI

TL;DR

Multi-task AI for glioma diagnosis needs uncertainty estimates that can communicate prediction reliability, but uncertainty across interacting segmentation and classification tasks remains insufficiently characterized. This study evaluates several uncertainty properties and methods in a joint MRI-based diagnostic framework. Uncertainty supported error detection, while decomposition and a composite trust score did not consistently improve on predictive uncertainty.

  • Problem

    Uncertainty behavior across interacting segmentation and classification tasks, including its relation to prediction reliability, remains insufficiently characterized in glioma diagnosis.

    Source: §1 · p. 2, §2.4 · p. 4

  • Method

    The study evaluates MCD uncertainty in a multi-task MRI framework for tumor segmentation and prediction of IDH status, 1p/19q co-deletion, and grade, and compares DE and MCDE.

    Source: §1 · p. 2

  • Results

    Across tasks, uncertainty supported error detection and selective-prediction analyses, but decomposition did not consistently improve error detection over predictive uncertainty alone.

    Source: §5 · p. 17, §6 · p. 20

  • Takeaways & Limitations

    The task-aware evaluation provides practical guidance for assessing uncertainty in deep-learning diagnostic tools for glioma.

    Source: §5 · p. 17, §6 · p. 20

    Convergence analysis measured average uncertainty by sample count and dropout rate, not case-wise ranking stability, repeatability, or downstream-metric variability across repeated MC realizations.

    Source: §5 · p. 20

Abstract

from arXiv · show

Uncertainty Quantification (UQ) is a key requirement for trustworthy AI in high-stakes medical image analysis. In this work, we evaluate UQ in a multi-task Deep Learning framework for MRI-based glioma diagnosis that performs tumor segmentation and predicts IDH mutation status, 1p/19q co-deletion status, and tumor grade. Monte Carlo Dropout (MCD) is used for a detailed task-aware analysis of predictive, aleatoric, and epistemic uncertainty. We assess MC sample convergence, calibration, error detection, selective prediction, associations with segmentation performance, and the effect of voxel-wise uncertainty aggregation on case-level reliability. We also compare MCD with Deep Ensembles (DE) and Monte Carlo Deep Ensembles (MCDE), examine interactions between segmentation quality and classification, and evaluate a composite trust score integrating segmentation and classification uncertainty. Across tasks, uncertainty estimates supported meaningful error detection, while calibration depended on the dropout rate, with moderate rates yielding the most reliable probabilities. Uncertainty decomposition provided task-dependent interpretability but did not consistently improve error detection over predictive uncertainty alone. DE and MCDE showed comparable operational utility, with no method consistently dominating across tasks and metrics. The composite trust score did not consistently outperform classification uncertainty for selective prediction. Overall, our results provide a task-aware evaluation strategy and practical guidance for the development of trustworthy AI for glioma diagnosis.

1. Introduction

Glioma diagnosis increasingly relies on molecular markers, while MRI-based multi-task models can jointly segment tumors and predict subtype features. This work evaluates whether uncertainty estimates make such systems more reliable and useful for clinical decision support.

1. Introduction

  • Molecular markers such as IDH mutation and 1p/19q co-deletion, alongside tumor grade, inform glioma classification and clinical decision-making.MRI-based segmentation and molecular prediction offer a way to characterize tumors beyond simple diameter measures.Source: §1 · p. 2
  • Multi-task models jointly segment tumors and classify genetic features, but black-box predictions without reliable confidence scores limit clinical translation.Shared representations may improve efficiency and generalization relative to isolated single-task approaches.Source: §1 · p. 2
  • Uncertainty quantification can support safer integration by communicating prediction reliability and identifying cases for human review.The paper frames UQ as a way to improve transparency in decision-support systems.Source: §1 · p. 2
  • Multi-task UQ remains insufficiently characterized because segmentation and classification outputs may respond differently to shared representations and uncertainty propagation.The authors also ask whether segmentation quality influences subtype-prediction reliability and whether decomposing uncertainty has practical value.Source: §1 · p. 2
  • The study evaluates MCD in a multi-task glioma pipeline and compares it with DE and MCDE across convergence, calibration, operational utility, aggregation, and cross-task interactions.The pipeline segments tumors and predicts IDH status, 1p/19q co-deletion status, and tumor grade.Source: §1 · pp. 2–3

2. Related Works

Prior work establishes MRI radiogenomics and multi-task learning as promising approaches, while clinical translation and the practical value of uncertainty decomposition remain open questions. The related work highlights a specific gap: uncertainty interactions between segmentation and molecular prediction in glioma have not been systematically examined.

2.1 MRI-based radiogenomics for glioma characterization

  • MRI radiogenomics uses imaging features to characterize glioma molecular subtypes and other outcomes, but reproducibility, generalizability, distribution shifts, uncertainty, and interpretability constrain clinical translation.Deep learning models have been reported to outperform classical radiomics approaches in molecular subtype prediction.Source: §2.1 · p. 3

2.2 Multi-task Deep Learning

  • Multi-task medical imaging combines related objectives, and prior glioma models jointly segmented tumors and predicted IDH status, 1p/19q co-deletion, and grade without systematic UQ analysis.The cited work of van der Voort et al. (2023) left interactions between segmentation quality, tumor-feature prediction reliability, and task-specific uncertainty unanalyzed.Source: §2.2 · pp. 3–4

2.3 Uncertainty Quantification and Decomposition

  • UQ methods model aleatoric and epistemic uncertainty in different ways, but decomposition's practical benefit depends on task, dataset, and evaluation context.Kahl et al. (2024) and Mukhoti et al. (2023) motivate validating operational value rather than assuming it.Source: §2.3 · p. 4

2.4 Uncertainty Quantification in Multi-Task Settings

  • Few studies examine uncertainty across interacting tasks, and glioma segmentation-to-subtype uncertainty links and cross-task trust aggregation remain uninvestigated systematically.Prior multi-task UQ work focused mainly on segmentation or cascaded pipelines rather than tightly integrated tasks.Source: §2.4 · p. 4

3. Methods

The methods combine tumor segmentation and glioma-feature classification to evaluate uncertainty across tasks using MCD, calibration and error-detection analyses, and ensemble comparisons. They also examine relationships between segmentation and classification performance and uncertainty, including whether segmentation uncertainty can identify classification errors or improve selective prediction when combined with classification uncertainty.

3.1 Multi-task Deep Learning Glioma Subtyping and Uncertainty Quantification Framework

  • The multi-task model uses four structural MRI sequences to jointly segment tumors and predict IDH status, 1p/19q co-deletion, and tumor grade.An encoder-decoder extracts multi-resolution features for fully connected classification branches; preprocessing includes registration, bias-field correction, and skull stripping.Source: §3.1 · p. 4, Figure 1 · p. 5

3.2 Data

  • Training and test sets combine in-house and public glioma datasets, while masked loss accommodates missing molecular or grade labels and the feature distributions are imbalanced.Training data include BraTS, Brain tumor Progression, CPTAC-GBM, EGD, IvyGAP, and REMBRANDT; test data include TCGA-GBM and TCGA-LGG.Source: §3.2 · p. 5

3.3.1 Bayesian Modeling; 3.3.2 Uncertainty Decomposition and MCD

  • Bayesian prediction averages over parameter uncertainty; MCD approximates it with dropout-enabled stochastic passes, decomposing predictive uncertainty into aleatoric and epistemic components.Aleatoric uncertainty is the average entropy across passes, while epistemic uncertainty is the mutual-information difference between predictive and aleatoric uncertainty (Gal and Ghahramani, 2016).Source: §3.3.1 · p. 6, §3.3.2 · p. 6

3.4.1 MC Sample Convergence and Decomposition of Uncertainty Estimates

  • MCD convergence and decomposition were evaluated across dropout rates δ = 0.2–0.6 in 0.05 increments and 10–100 MC samples in steps of 10.A sufficient sample count was defined by limited further change in average uncertainty; higher dropout can reduce model capacity, while more samples increase inference time.Source: §3.4.1 · p. 7

3.4.2 Calibration and Quality Analysis

  • Classification evaluation used ECE and NLL for probability calibration and AUC for performance, while segmentation uncertainty was correlated with Dice using Pearson and Spearman measures.ECE and NLL assess softmax probability calibration, not calibration of uncertainty estimates; case-level segmentation uncertainty was aggregated over the brain, predicted tumor, dilated tumor, or distance-weighted region.Source: §3.4.2 · pp. 7–8

3.4.3 Operational Utility of Uncertainty Estimates

  • Operational utility was assessed through uncertainty-based error ranking and selective prediction, with U-AUC, Average Precision, Lift, and risk–coverage area metrics.Segmentation error definitions were tested at Dice thresholds τ = 0.60, 0.70, and 0.80; lower AURC indicates lower residual risk among retained low-uncertainty cases.Source: §3.4.3 · pp. 8–9

3.4.4 Comparison of MCD with other UQ methods

  • After selecting the MCD configuration, the study compared it with five-model Deep Ensembles and Monte Carlo Deep Ensembles using the same operational-utility metrics.Deep Ensembles average deterministic member predictions; MCDE additionally enables dropout at inference and averages predictions across members and MC samples.Source: §3.4.4 · p. 9

3.4.5 Interaction Between Tumor Segmentation and Classification

  • Cross-task analyses compared segmentation Dice between correct and incorrect classifications using Mann–Whitney U tests and correlated segmentation with classification uncertainty components.Pearson correlations were computed separately for predictive, aleatoric, and epistemic uncertainty.Source: §3.4.5 · pp. 9–10
  • Segmentation uncertainty was assessed as a predictor of classification errors using U-AUC, AP, Lift, and AURC.The analysis tested whether uncertainty in the shared segmentation backbone propagated to the reliability of tumor-feature predictions.Source: §3.4.5 · p. 10
  • The composite trust score integrated normalized predictive uncertainty from classification and segmentation, with task-specific weights based on each source’s Lift for identifying classification errors.The weighting favored the source with greater error-identification performance relative to baseline prevalence; higher trust scores were designed to indicate more reliable predictions.Source: §3.4.5 · p. 10
  • Selective prediction analysis tested whether combining the uncertainty sources improved classification-error identification over classification uncertainty alone.The trust score was also designed to support comparison with confidence-based referral strategies.Source: §3.4.5 · p. 10

4. Results

Uncertainty estimates showed task-dependent calibration and error-detection behavior, while ensemble methods provided utility comparable to MCD without a consistent winner. Segmentation uncertainty related to segmentation and classification reliability, but adding it to a trust score did not consistently improve selective prediction.

4. Results

  • Uncertainty estimates generally plateaued after about 20–30 MC samples; aleatoric uncertainty dominated classification, while epistemic uncertainty dominated segmentation.The study selected T = 30 for subsequent analyses.Source: §4.1 · p. 10, Figure 2 · p. 11

4.2 Calibration and Quality Analysis

  • At dropout 0.25, MCD offered a favorable trade-off between predictive performance and calibration; calibration metrics and their optima varied by task.IDH and tumor-grade ECE and NLL were lowest at 0.25, while 1p/19q ECE and NLL were minimized at 0.2; AUC generally declined as dropout increased.Source: §4.2 · pp. 10–11, Figure 3 · p. 12
  • Predicted-tumor aggregation gave the strongest, most stable negative uncertainty–DSC association; mean segmentation DSC peaked at dropout 0.25.Brain-mask aggregation had weak correlations, while dilated and boundary-weighted aggregation associations weakened as dropout increased.Source: §4.2 · p. 11, Figure 4 · p. 13

4.3 Operational Utility of Uncertainty Estimates

  • Predictive uncertainty discriminated high- from low-quality predictions across tasks, and all uncertainty components detected errors better than baseline, without consistent gains from decomposition.For segmentation, the three components had U-AUC values of 0.95–0.96 and AP values of 0.81–0.83; their AURC values were all 0.13.Source: §4.2 · p. 12, Figure 5 · p. 13, §4.3 · p. 14

4.4 Comparison of MCD with other UQ methods

  • Across tasks and metrics, DE and MCDE provided operational utility comparable to MCD, with no method consistently dominating.MCDE tended to have favorable AURC, whereas DE or MCD more often achieved higher AP or Lift; several confidence intervals overlapped.Source: §4.4 · p. 14, Table 5 · p. 15
  • Higher segmentation DSC accompanied correct IDH and tumor-grade predictions significantly, while segmentation uncertainty was positively associated with predictive and aleatoric classification uncertainty.For 1p/19q, the DSC difference by classification correctness was not statistically significant; epistemic cross-task associations were weaker and less consistent.Source: §4.5 · pp. 14–16, Figure 6 · p. 15
  • The composite trust score identified classification errors, but did not consistently improve on predictive classification uncertainty across tasks.It improved point estimates for 1p/19q, but not for IDH or tumor grade; confidence intervals overlapped across comparisons.Source: §4.5 · pp. 16–17

5. Discussion

Uncertainty estimates provided useful but task-dependent reliability information, while their calibration and operational value depended on dropout, uncertainty component, task, and aggregation choices. The discussion also identifies limits in the evaluated imaging scope and convergence analysis.

Convergence

  • Average uncertainty generally plateaued after approximately 20–30 MC samples, supporting a computationally practical sample count for subsequent analyses.Increasing dropout generally raised uncertainty magnitudes, but excessive dropout reduced predictive performance and uncertainty quality.Source: §5 · p. 17

Uncertainty decomposition

  • Aleatoric uncertainty dominated classification, while epistemic uncertainty was more prominent in segmentation; decomposition did not consistently improve error detection over predictive uncertainty.Predictive uncertainty was operationally comparable to its components across tasks, despite providing less separation of uncertainty sources.Source: §5 · p. 17

Calibration and dropout

  • Moderate dropout levels achieved the best calibration and predictive likelihood, whereas excessive dropout degraded both uncertainty quality and predictive performance.Increasing dropout generally raised uncertainty magnitudes; calibration and predictive performance capture related but distinct properties of probability outputs.Source: §5 · pp. 17–18

Segmentation uncertainty

  • Predicted-tumor aggregation had the strongest and most stable negative association with segmentation accuracy; dilated and boundary-weighted approaches added no clearer advantage.Full-brain aggregation showed weak DSC associations, while localized alternatives introduced additional hyperparameters.Source: §5 · p. 18

Operational utility

  • Across tasks, uncertainty estimates detected errors better than baseline rates, and DE and MCDE were comparable to MCD without a consistently dominant method.Uncertainty also reduced residual risk among retained high-confidence cases; segmentation uncertainty remained informative across evaluated DSC thresholds, although AP and Lift varied with error prevalence.Source: §5 · pp. 18–19

Cross-task interactions

  • Segmentation quality was significantly associated with IDH and grade correctness, but not significantly with 1p/19q correctness; predictive and aleatoric uncertainty showed more shared cross-task patterns than epistemic uncertainty.The association between segmentation quality and classification correctness was task-specific, rather than universal.Source: §5 · p. 19

Classification reliability

  • Segmentation uncertainty had Lift above 1 for all three classification tasks, but classification-specific predictive uncertainty achieved higher Lift and lower AURC across them.A composite trust score also detected low-trust cases, yet did not consistently improve on classification uncertainty; its point estimates improved for 1p/19q but not IDH or grade.Source: §5 · p. 19

Limitations and future work

  • The evaluation used structural MRI only, and convergence analysis did not assess case-wise ranking stability, repeatability across independently trained models or seeds, or downstream metric variability across repeated MC runs.The proposed framework can also be extended to UQ methods beyond MCD, DE, and MCDE.Source: §5 · pp. 19–20

6. Conclusion

Uncertainty estimates supported error detection and selective-prediction analyses, but their utility varied by task and metric. Uncertainty decomposition offered task-specific insight, while segmentation-focused aggregation and classification predictive uncertainty were useful for their respective tasks.

6. Conclusion

  • Across tasks, uncertainty estimates identified prediction errors and supported selective-prediction analyses, but utility varied by task, uncertainty component, and operational metric.MCD averages converged with a practical number of MC samples, while calibration and uncertainty quality depended on dropout configuration.Source: §6 · p. 20
  • Aleatoric uncertainty was more prominent in classification, whereas epistemic uncertainty was more prominent in tumor segmentation, without consistent operational advantage over predictive uncertainty alone.Uncertainty decomposition offered insight into task-specific uncertainty sources.Source: §6 · p. 20
  • For segmentation, uncertainty identified low-quality cases more informatively when aggregated within tumor-focused regions rather than across the full brain.Source: §6 · p. 20
  • Classification predictive uncertainty remained the most direct and informative indicator of classification error; the composite trust score did not consistently improve over it.Source: §6 · p. 20
  • MCD, DE, and MCDE showed operational value that was not restricted to a single uncertainty-quantification method.Source: §6 · p. 20
  • The task-aware evaluation framework provides practical guidance for assessing uncertainty in deep-learning diagnostic tools for glioma.Source: §6 · p. 20

Ethical Standards

The authors state that research conduct and writing followed applicable ethical standards and regulations.

Ethical Standards

Conflicts of Interest

The disclosure lists institutional ties, GE HealthCare funding and employment, and one author's declaration of no competing interests.

Conflicts of Interest

Appendix A. Implementation details

The appendix describes a standardized MRI preprocessing pipeline intended to reduce inter-subject variability, improve alignment, and support numerical stability. It covers registration, bias correction, skull stripping and cropping, and brain-mask-based intensity normalization.

A.1 Pre-processing

  • The preprocessing steps were designed to reduce inter-subject variability, improve anatomical alignment, and ensure numerical stability during optimization.Source: §A.1 · p. 24
  • All scans were spatially registered to MNI152 atlas space, using modality-specific templates to align anatomical contrasts.The atlas has 1×1×1 mm3 resolution and dimensions of 197×233×189 voxels (Fonov et al., 2009, 2011).Source: §A.1 · p. 24
  • Bias field correction was applied to mitigate low-frequency intensity inhomogeneities, including scanner-related variation in the multi-center dataset.Source: §A.1 · p. 24
  • HD-BET skull stripping removed non-brain tissue, and the brain mask supported cropping scans to 145x182x152 voxels.The authors state that cropping reduced computational burden and background redundancy.Source: §A.1 · p. 24
  • Voxel intensities were z-score standardized using the mean and standard deviation calculated within each scan's brain mask.Source: §A.1 · p. 24

A.2 Architecture and training procedure

The multi-task network processes full 3D volumes and is trained with a weighted loss that accounts for task-specific label availability and class imbalance.

A.2 Architecture and training procedure

  • The van der Voort et al. (2023) architecture processes full 3D volumes and was trained for up to 100 epochs with ADAM.Its weighted objective combines grade and segmentation losses with task-specific terms.Source: §A.2 · p. 24
  • A masked, class-weighted categorical cross-entropy loss uses only samples with available labels and accounts for class imbalance.Class weights are computed from each task’s labeled training samples and class counts.Source: §A.2 · p. 24
  • For tumor segmentation, the training objective uses DSC loss to optimize overlap between predicted and ground-truth masks.A small constant is added for numerical stability.Source: §A.2 · pp. 24–25

Appendix B. Data

The dataset overview reports train and test case counts and task-label availability, including missing annotations and class imbalance.

Appendix B. Data

  • Table 10 summarizes train and test cases and label distributions for IDH, 1p/19q, and tumor grade, with missing annotations marked N/A.The reported missing labels and class imbalance motivated task-specific masking during training.Source: Appendix B · p. 25

Appendix C. Segmentation error threshold sensitivity analysis

Segmentation uncertainty remained informative across the tested definitions of segmentation error, despite changes in error prevalence.

Appendix C. Segmentation error threshold sensitivity analysis

  • The threshold analysis tested robustness rather than optimizing the primary DSC ≤ 0.70 definition of segmentation error.The thresholds represented severe failures, the primary operating point, and stricter low-quality segmentation.Source: Appendix C · p. 25
  • Across all three thresholds, AP exceeded the corresponding error-rate baseline and Lift remained greater than 1; aleatoric and epistemic uncertainty showed similar patterns.The tested thresholds were τDSC = 0.60, 0.70, and 0.80.Source: Appendix C · p. 25
  • Predictive-uncertainty U-AUC was 0.96 at τDSC = 0.60 and 0.93 at τDSC = 0.80, while error rates rose from 0.10 to 0.31.The table reports threshold-specific error-detection metrics for predictive, aleatoric, and epistemic uncertainty with bootstrap confidence intervals.Source: Table 9 · p. 25, Appendix C · p. 25

Appendix D. Correlation patterns between segmentation uncertainty estimates and tumor segmentation DSC

Tumor-localized predictive uncertainty was negatively associated with segmentation DSC, whereas brain-level aggregation showed no meaningful association.

Appendix D. Correlation patterns between segmentation uncertainty estimates and tumor segmentation DSC

  • Tumor-localized uncertainty showed strong negative associations with DSC: predicted-tumor correlations were Pearson ρ = −0.832 and Spearman ρs = −0.772; dilated-tumor correlations were −0.791 and −0.798.The figure presents case-wise associations for brain mask, predicted tumor, dilated tumor, and boundary-weighted aggregation.Source: Figure 8 · p. 25, Appendix D · p. 26
  • Boundary-weighted aggregation also correlated negatively with DSC, with Pearson ρ = −0.630 and Spearman ρs = −0.717.Source: Figure 8 · p. 25, Appendix D · p. 26
  • Brain-mask aggregation showed no meaningful association with DSC: Pearson ρ = 0.015 and Spearman ρs = −0.032.The rank-based analysis supported the same overall interpretation as Pearson correlation.Source: Figure 8 · p. 25, Appendix D · p. 26

Appendix E. Comparison of UQ Methods

Aleatoric uncertainty showed trends broadly consistent with predictive uncertainty, while epistemic uncertainty varied more across tasks and methods. Epistemic uncertainty was informative for segmentation but generally weaker for classification error detection.

Appendix E. Comparison of UQ Methods

  • For tumor segmentation, uncertainty was aggregated within the predicted tumor region, matching the strategy used in the main method comparison.The component-wise analyses complement the main predictive-uncertainty comparison.Source: Appendix E · p. 26, Table 11 · p. 27
  • Epistemic uncertainty remained informative for tumor segmentation across MCD, DE, and MCDE, with MCD strongest on U-AUC, AP, and Lift, but it was generally weaker for classification error detection.Aleatoric uncertainty showed trends broadly consistent with predictive uncertainty, with useful error detection across classification tasks and high performance for segmentation.Source: Appendix E · p. 26
  • The component-wise comparison assessed aleatoric and epistemic uncertainty using U-AUC, AP, Lift, and AURC, with 95% confidence intervals from 1000× bootstrap resampling.Classification errors were method-specific; segmentation errors were cases with DSC ≤0.70.Source: Table 11 · p. 27
Loading 2609.39429v1…