Evaluation of Uncertainty-Aware Multi-Software Ensembles for Hippocampal Segmentation

1. Introduction

• Manual segmentation is time-consuming and variable, and multiple protocols complicate the definition of ground truth.These issues limit reliable volumetric measurement despite its potential as a disease biomarker.Source: §1 · p. 2
• Automated hippocampal segmentation has limited clinical adoption amid concerns about silent failure, distribution shift, calibration, and computational cost.Ensemble uncertainty estimation requires running multiple models or pipelines, so inference cost scales with ensemble size.Source: §1 · p. 2
• High computational demands increase carbon footprint and limit access for resource-constrained hospitals and research institutions.The study frames uncertainty reliability and computational efficiency as joint concerns for broader clinical adoption.Source: §1 · p. 2
• Ensemble uncertainty is explored as a quality-control signal for motion-corrupted scans and as an additional feature for Alzheimer’s disease classification.The classification hypothesis concerns disagreement between methods designed for atrophy and methods trained only on healthy individuals.Source: §1 · pp. 2–3

1.1 Contributions

• The study benchmarks five heterogeneous hippocampal segmentation methods for accuracy, calibration, and carbon footprint across two dementia cohorts and one epilepsy cohort.It identifies a resource-efficient triad, finds simple probability averaging the best integration strategy, and evaluates uncertainty for quality control and disease classification.Source: §1.1 · p. 3

2.1 Segmentation tools

• Five segmentation techniques were selected to span atlas-based and deep-learning methods and variation in training data.Table 1 summarizes their methodologies and primary clinical training cohorts or atlases; the ASHS atlas used was the Penn Memory Center 3T atlas for T1-weighted MRI.Source: §2.1 · p. 3, Table 1 · p. 4

2.2 Data

• Segmentation performance was evaluated on three independent datasets with clinician-defined manual segmentations confirmed by a consultant neuroradiologist.The segmentation definition followed an amended harmonised protocol that also included the fimbria; the datasets comprised ADNI and local dementia and epilepsy cohorts.Source: §2.2 · p. 3
• Motion-artefact analysis used MR-ART scans rated for clinical image quality, while Alzheimer’s classification used a separate, larger ADNI cohort without manual ground-truth segmentations.MR-ART included motion-free and two induced-motion conditions; the classification cohort included 50 cognitively normal, 50 mild cognitive impairment, and 50 Alzheimer’s disease participants.Source: §2.2 · pp. 3–4

2.3 Processing

• Public ADNI and MR-ART inference ran on an HPC cluster, while local hospital data ran on a MacBook Pro because of data-sharing limitations.Carbon benchmarking used a dedicated cluster node; reported emissions covered inference CPU, GPU, and RAM energy, not training or other lifecycle emissions.Source: §2.3 · p. 4
• FreeSurfer v6.0.0 was selected to fit local-machine memory constraints, and epilepsy hippocampi were processed separately by side.The left/right separation was intended to avoid averaging out possible unilateral atrophy; no additional scan preprocessing was performed.Source: §2.3 · pp. 4–5

2.4 Uncertainty quantification

• Voxel-level sample variance, mutual information, and predictive entropy were computed across the winning triad’s probabilistic outputs.The methods use each tool’s voxel prediction and the mean prediction across tools; calibration was evaluated with expected calibration error using 15 equal-width bins and per-hippocampus averaging.Source: §2.4 · p. 5
• Adaptive expected calibration error used 15 equal-mass bins, reducing sensitivity to sparsely populated bins dominated by near-certain background voxels.Class-specific AECE was also calculated separately for foreground and background, and tool rankings were checked across bin counts from 10 to 50.Source: §2.4 · p. 5

2.5 Ensemble performance & calibration

• The ADNI advantage of the selected triad over other sub-ensembles may be inflated because selection and evaluation used the same data.The two local datasets were not used for triad selection.Source: §2.5 · p. 5
• Triads and the full ensemble were compared using DSC, ECE, and estimated CO2 emissions, with majority voting producing their final masks.ADNI was used both to select the best-performing triad and report its performance; Local Dementia and Local Epilepsy provide held-out cross-domain assessment.Source: §2.5 · p. 5
• Triad ECE treated the proportion of member votes for a voxel as discrete confidence levels.For the winning triad, code changes enabled saving probabilistic outputs; amended scripts are publicly available on GitHub.Source: §2.5 · p. 5

2.5.1 Ensemble integration strategy

• The three integration strategies were simple probability averaging, confidence-weighted predictions, and winner-takes-all selection of the most confident model.Friedman tests with Holm-Bonferroni adjustment assessed performance differences; resource use was estimated by duration and CO2 expenditure.Source: §2.5.1 · p. 5

2.6 Uncertainty-based quality control

• The quality-control analysis asks whether ensemble uncertainty can identify unreliable segmentation outputs and flag low-image-quality cases.For silent-failure detection, uncertainty is aggregated over the union of predicted segmentations with a one-voxel margin.Source: §2.6 · p. 5, §2.6.1 · p. 5

2.6.1 Silent failure detection

• Uncertainty maps show sample variance, mutual information, and predictive entropy for high-Dice and low-Dice examples, with the gold-standard boundary marked.The maps are min-max normalized.Source: Figure 1 · p. 6
• Silent-failure thresholds were optimized per dataset and metric on validation data, then applied to held-out test predictions.Failures were defined as DSC < 0.82 or HD95 > 2.5 mm; the optimal threshold maximized F1-score after a stratified 30% validation/70% test split.Source: §2.6.1 · pp. 5–6

2.6.2 Motion artefact detection

• Motion-artifact analysis relates uncertainty to MR-ART image-quality scores using within-subject permutation tests and paired score comparisons.Scores were assigned by expert neuroradiologists; Spearman correlation tested monotonic association, and Wilcoxon signed-rank tests compared paired categories with Holm-Bonferroni adjustment across six tests.Source: §2.6.2 · p. 6

2.7 Uncertainty for Alzheimer’s disease classification

• The ADNI baseline cohort included 150 balanced participants: 50 cognitively normal, 50 with mild cognitive impairment, and 50 with Alzheimer’s disease.Age and sex were included as covariates.Source: §2.7 · p. 6
• Models compared hippocampal-volume features alone with individual or combined uncertainty features across CN vs AD, CN vs MCI, and MCI vs AD classification.Uncertainty metrics were averaged across the hippocampal ROI for each hemisphere and then across hemispheres; evaluation used three classifiers and repeated stratified cross-validation.Source: §2.7 · pp. 6–7

3.1.1 Benchmarking cost and performance

• The InnerEye, FastSurfer, and HippoSeg triad matched the five-software ensemble’s accuracy while reducing emissions from 4.65 kg to 0.85 kg CO2eq for 30 ADNI patients.The triad retained a comparably well-calibrated uncertainty measure.Source: §3.1.1 · p. 7

3.1.2 Ensemble integration strategy

• Simple probability averaging yielded the highest Dice performance across all datasets (p < 0.05, Holm-Bonferroni corrected).For HD95, the mean ensemble did not significantly differ from other integration strategies on epilepsy; on Local Dementia it did not differ from InnerEye but outperformed the other methods.Source: §3.1.2 · p. 7

3.1.3 Uncertainty calibration

• The triad’s averaged probabilistic output had the best calibration, with ECE 0.05 across all three datasets.HippoSeg was next, with slightly higher average ECE; InnerEye had ECE ≈0.08 and FastSurfer ECE ≈0.16.Source: §3.1.3 · p. 7

3.2.1 Silent failure detection

• For Dice < 0.82 outliers, entropy achieved 91.7% recall and 44.2% precision; for HD95 > 2.5 mm, it achieved 88.9% recall and 40.6% precision.For Dice outliers, variance and mutual information each had 83% recall and approximately 35% precision; for HD95 outliers, both had 55.6% recall, with precision of 28.9% and 30.6%, respectively.Source: §3.2.1 · p. 7

3.2.2 Motion artifact detection

• Image quality score and uncertainty had a significant monotonic relationship across all three metrics (p < 0.01; Spearman ρ = 0.30–0.36).Score 3 exceeded score 2 in uncertainty for all metrics, but score 1 versus score 2 was not significant; entropy had the strongest correlation (ρ = 0.36).Source: §3.2.2 · p. 7

3.3 Uncertainty for Alzheimer’s disease classification

• For CN vs AD, Full_mean logistic regression achieved AUC 0.87 ± 0.03, a +0.06 improvement over the volume-only baseline (AUC 0.81 ± 0.04; p < 0.001).Individual uncertainty features did not improve over volume alone; uncertainty features also did not improve CN vs MCI or MCI vs AD classification.Source: §3.3 · p. 7

4.1 Ensemble performance & computational cost

• The InnerEye–FastSurfer–HippoSeg triad matched the full ensemble’s performance while reducing computational cost by approximately 82% in the ADNI benchmark.The comparison is a within-study inference benchmark on fixed hardware and an energy mix, not a full lifecycle carbon assessment.Source: §4 · p. 7, Figure 2 · p. 8, §4.1 · p. 8

4.2.1 Silent failure detection

• The entropy-based detector achieved 91.7% recall and 44.2% precision, flagging up to 35% of cases for manual review; failures were defined as DSC < 0.82 or HD95 > 2.5 mm.Source: §4.2.1 · p. 9

4.2.2 Motion artifact detection

• Uncertainty increased with motion artefact severity (Spearman ρ = 0.30–0.36), but the difference between good- and medium-quality scans was not significant.Uncertainty was significantly higher for clinically unusable scans than medium-quality scans; the authors suggest future evaluation against task-quality endpoints rather than scan-quality labels.Source: Figure 5 · p. 10, §4.2.2 · p. 10

4.3 Uncertainty for Alzheimer’s disease classification

• Combining volume with all uncertainty metrics improved CN vs. AD classification by +0.06 AUC (p < 0.001), and Full_mean reclassified five additional subjects versus volume alone.The discussion describes the classification gain as modest and notes that the study reduced voxel-wise uncertainty maps to scalar features, discarding spatial patterns.Source: §4.3 · p. 10, Figure 6 · p. 11, Figure 7 · p. 11

4.4 Uncertainty decomposition

• The study does not assign its entropy, variance, or mutual-information measures cleanly to aleatoric or epistemic uncertainty.Image-quality changes may increase aleatoric uncertainty and also create distribution shifts that affect ensemble disagreement differently across members.Source: §4.4 · p. 12

4.5 Limitations

• Small sample sizes, including n=8 for the local epilepsy dataset, limit generalisability, and partly overlapping ADNI data for triad selection and evaluation may introduce selection bias.Other stated boundaries include not evaluating specialized probabilistic segmentation models or alternative post-hoc methods, and aggregating voxel-wise uncertainty into a scalar for classification.Source: §4.5 · p. 12

4.6 Conclusion

• The conclusion frames uncertainty as promising for motion-artefact detection and modest Alzheimer’s classification gains, while proposing richer uncertainty features and longitudinal data as future directions.The stated longitudinal aim is to investigate uncertainty-supported detection of hippocampal atrophy.Source: §4.6 · p. 12

• The work states that it followed appropriate ethical standards and applicable laws and regulations for research involving human or animal subjects.Source: Ethical Standards · p. 13

• Figure S1 and Figure S3 provide heatmaps for CN vs. MCI and MCI vs. AD classification, respectively.Source: Figure S1 · p. 17, Figure S3 · p. 18
• For CN vs. MCI classification, baseline AUC was 0.691 and BAcc was 0.650; the full model had AUC 0.660 and BAcc 0.638.Source: Figure S2 · p. 17
• For MCI vs. AD classification, baseline AUC was 0.698 and BAcc was 0.646; the full model had AUC 0.700 and BAcc 0.677.Source: Figure S4 · p. 18
• Figure S5 shows good, medium, and bad MR-ART image-quality examples, with the latter two acquired during head motion.Source: Figure S5 · p. 19
• Supplementary calibration metrics are reported by dataset and tool, including equal-width ECE, adaptive equal-mass ECE, and foreground/background ECE.Source: Table S1 · p. 19