PaperlayerView exact revision on arXiv

Source-linked AI summary

Bayesian QuickNAT: Model Uncertainty in Deep Whole-Brain Segmentation for Structure-wise Quality Control

Abhijit Guha Roy, Sailesh Conjeti, Nassir Navab, Christian Wachinger

arXiv:1811.09800v1cs.CV

TL;DR

Whole-brain segmentation lacks an inherent quality estimate, while manual QC is time-consuming and global. Bayesian QuickNAT uses test-time dropout sampling to estimate voxel- and structure-wise uncertainty. Its structure-wise metrics correlate with Dice, and uncertainty-weighted regression estimates are more similar to those from manual annotation.

  • Problem

    Whole-brain segmentation methods do not inherently estimate quality, and manual QC is time-consuming, variable, binary, and global to a scan.

    Source: §1 · p. 1

  • Method

    Bayesian QuickNAT keeps dropout active at test time to sample segmentations and uses four metrics to quantify structure-wise uncertainty.

    Source: §1 · p. 2, §3.3 · p. 3

  • Results

    Structure-wise uncertainty measures are highly correlated with Dice, with intersection over union across Monte Carlo samples serving as a proxy when manual annotations are unavailable.

    Source: §5.7 · p. 11

  • Takeaways & Limitations

    Weighting observations by segmentation quality in regression yields estimates more similar to those from manual annotation.

    Source: §5.7 · p. 11

    The MALC training set contained no scans with Alzheimer’s disease or children, limiting its represented training data for those populations.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

We introduce Bayesian QuickNAT for the automated quality control of whole-brain segmentation on MRI T1 scans. Next to the Bayesian fully convolutional neural network, we also present inherent measures of segmentation uncertainty that allow for quality control per brain structure. For estimating model uncertainty, we follow a Bayesian approach, wherein, Monte Carlo (MC) samples from the posterior distribution are generated by keeping the dropout layers active at test time. Entropy over the MC samples provides a voxel-wise model uncertainty map, whereas expectation over the MC predictions provides the final segmentation. Next to voxel-wise uncertainty, we introduce four metrics to quantify structure-wise uncertainty in segmentation for quality control. We report experiments on four out-of-sample datasets comprising of diverse age range, pathology and imaging artifacts. The proposed structure-wise uncertainty metrics are highly correlated with the Dice score estimated with manual annotation and therefore present an inherent measure of segmentation quality. In particular, the intersection over union over all the MC samples is a suitable proxy for the Dice score. In addition to quality control at scan-level, we propose to incorporate the structure-wise uncertainty as a measure of confidence to do reliable group analysis on large data repositories. We envisage that the introduced uncertainty metrics would help assess the fidelity of automated deep learning based segmentation methods for large-scale population studies, as they enable automated quality control and group analyses in processing large data repositories.

1. Introduction

Existing whole-brain segmentation methods do not estimate segmentation quality, leaving manual QC time-consuming and global. Bayesian QuickNAT adds model-derived uncertainty, including structure-wise measures intended to support automated QC and group analysis.

1. Introduction

  • Whole-brain segmentation methods do not estimate quality, while manual QC is time-consuming, variable across raters, binary, and global to the scan.Cohort-level biomarker summaries have also been used to identify outliers when manual QC is costly.Source: §1 · p. 1
  • Bayesian segmentation can provide a posterior distribution, but sampling-based approaches have been computationally intensive and used for substructures rather than whole-brain segmentation.The cited approaches used Markov Chain Monte Carlo sampling or the Laplace approximation.Source: §1 · p. 1
  • Monte Carlo dropout approximates posterior sampling, and multiple samples can be generated with CNN inference in acceptable time.Dropout at test time approximates sampling from a Bernoulli distribution over network weights.Source: §1 · p. 2
  • Bayesian QuickNAT estimates segmentation uncertainty within the segmentation model, with four structure-wise metrics intended to predict Dice accuracy and support quality control and group analysis.The authors state that the approach does not increase segmentation accuracy, but adds a confidence estimate.Source: §1 · p. 2
  • Structure-wise quality estimates provide continuous, scan-independent QC and represent within-scan variation more specifically than a global pass/fail assessment.The authors present these properties as advantages over manual QC, particularly for large neuroimaging repositories.Source: §1 · p. 2
  • The proposed uncertainty measures target scans with low expected accuracy, potentially because of image quality or differences from training data; the authors do not claim improved segmentation accuracy.The motivation notes that learning-based performance depends closely on the scans used during training.Source: §1 · p. 2

2. Prior Art

Earlier approaches estimated segmentation quality through agreement measures or separate predictors. The paper distinguishes its approach as model-derived uncertainty, and presents structure-wise uncertainty for whole-brain segmentation and downstream group analysis as a contribution.

2. Prior Art

  • Earlier work estimated segmentation performance using STAPLE, a separate Dice regressor, or reverse classification accuracy (Valindria et al., 2017).RCA was also extended to quality control in large-scale cardiac MRI scans.Source: §2 · p. 2
  • Unlike separate quality predictors, this approach derives a quality measure from uncertainty in the segmentation model and requires no independently trained evaluation classifier.The authors note that a separate classifier could itself make prediction errors.Source: §2 · pp. 2–3
  • The authors describe this as the first work, to their knowledge, to provide structure-wise uncertainty for whole-brain segmentation and apply it to group analysis.The group-analysis application is framed as supporting reliable estimation.Source: §2 · p. 3

3. Method

Bayesian QuickNAT extends whole-brain segmentation with dropout-based uncertainty estimates at voxel and structure levels, then uses structure-wise uncertainty to weight group analyses.

3.1. Background on Bayesian Inference

  • Variational inference approximates the intractable posterior over network weights, and Monte Carlo samples from the approximation yield different segmentations whose mean is the final prediction.Dropout at a layer approximates sampling from its variational weight distribution.Source: §3.1 · p. 3

3.2. QuickNAT architecture

  • QuickNAT combines three 2D fully convolutional models across coronal, axial, and sagittal views, then aggregates their segmentations.Each model uses encoder-decoder blocks with dense and skip connections.Source: §3.2 · p. 3

3.3. Bayesian QuickNAT

  • Keeping dropout active at test time generates multiple probability maps and segmentations; averaging probability maps produces the final segmentation.The method inserts dropout after each encoder and decoder block and feed-forwards the scan N times with different random masks.Source: §3.3 · pp. 3–4

3.4.1. Voxel-wise Uncertainty

  • Voxel-wise model uncertainty is entropy across Monte Carlo probability maps, summed across structures for a global voxel-wise uncertainty map.Low entropy corresponds to predictions that remain similar across different dropout patterns.Source: §3.4.1 · p. 4

3.4.2. Structure-wise Uncertainty

  • Four structure-wise measures capture volume variation, pairwise Dice overlap, all-sample IoU, and mean voxel-wise uncertainty within each structure.The measures except mean voxel-wise uncertainty use label maps, which are much smaller than probability maps and can be computed faster.Source: §3.4.2 · p. 4

3.5. Segmentation Uncertainty in Group Analyses

  • Structure-wise uncertainty weights a linear regression so scans with more reliable segmentations receive greater emphasis.Using constant weights gives standard regression.Source: §3.5 · pp. 4–5

4. Experimental Setup

The experiments use one trained model and evaluate it on four datasets, including data with age, pathology, scanner, and image-quality variation.

4.1. Architecture and Training Procedure

  • Inference uses dropout rate 0.2 and 15 Monte Carlo samples, taking under two minutes for sampling.Other dropout rates reduced segmentation performance compared with no test-time dropout.Source: §4.1 · p. 5
  • The model is pretrained on 581 IXI volumes with FreeSurfer labels, then fine-tuned on 15 manually annotated MALC volumes and tested on the remaining 15.The experiments segment 33 brain structures.Source: §4.1 · p. 5

4.2. Test Datasets

  • Testing covers MALC-15 and three datasets not seen during training: ADNI-29, CANDI-13, and IBSR-18.ADNI-29 varies in diagnosis and scanner field strength; CANDI-13 includes children aged 5–15 with psychiatric disorders, and IBSR-18 includes low-resolution, poor-contrast scans.Source: §4.2 · pp. 5–6

5. Experimental Results and Discussion

On CANDI-13, increasing Monte Carlo sample counts preserved Dice performance, while uncertainty maps were stable by the 15-to-18-sample transition; the experiments use 15 samples.

5.1. Number of MC Samples

  • The CANDI-13 experiment tests sample-count effects on unseen data, using counts from 3 to 18 and assessing both Dice performance and uncertainty-map stability.The stability criterion is whether entropy estimates change substantially as more samples are added.Source: §5.1 · p. 6
  • Mean global Dice remained nearly constant from 3 to 18 Monte Carlo samples and close to the no-dropout performance.The authors attribute this to aggregation across three principal axes, which they say compensates for a slight performance decrease from test-time dropout.Source: §5.1 · p. 6, Table 1 · p. 6
  • The 15→18-sample transition produced a small mean absolute change in voxel-wise entropy, indicating stable uncertainty maps.The comparison measures changes between maps estimated with consecutive sample counts.Source: §5.1 · p. 6, Table 2 · p. 6
  • The authors use 15 samples as a compromise between accurate segmentation, reliable uncertainty estimates, and computational demands.More samples increase scan time and require loading all intermediate probability maps into RAM.Source: §5.1 · p. 6

5.2. Uncertainty based quality control across different datasets

Across seen and unseen datasets, structure-wise uncertainty tracks segmentation accuracy, with IoU showing the strongest correlations. IoU also approximates Dice closely enough to serve as an unsupervised proxy, though similarity varies by structure.

5.2. Uncertainty based quality control across different datasets

  • IoU has the highest correlation with Dice across all four datasets, while Dice is highest on MALC at 88% and falls 5–7% on ADNI, CANDI, and IBSR.The datasets differ in whether they were seen during training and in age range, pathology, and scan resolution.Source: §5.2 · pp. 6–7

5.3. IoU as a proxy for the Dice score

  • IoUs achieve 2–7% MAE and 80–88% classification accuracy against Dice, with similarity metrics comparable to or better than a dedicated Dice-prediction model.The IoUs are computed from MC samples without supervision.Source: §5.3 · p. 7
  • IoU values closely resemble Dice scores, making IoU a suitable proxy without manual annotations.For most structures, IoU is close to Dice; the inferior lateral ventricles show the largest reported difference, about 15%, potentially because of their small size.Source: §5.3 · p. 7

5.4. Sensitivity of Uncertainty to scan Quality

The Rician-noise experiment tests whether uncertainty responds to degraded scan quality. As noise increases, both Dice and mean IoU decrease, with IoU falling faster.

5.4. Sensitivity of Uncertainty to scan Quality

  • Table 5 reports mean Dice, mean IoU, MAE, and bad/medium/good classification accuracy across Rician noise levels.The reported outcomes show Dice and mean IoU decreasing as noise increases.Source: Table 5 · p. 8, §5.4 · p. 8
  • As Rician noise increases, mean Dice and mean IoU decrease, and mean IoU falls faster, indicating greater sensitivity to noise than segmentation accuracy.The experiment uses noise levels of 3, 5, 7, and 9 dB on the MALC test dataset; real scans did not produce Dice scores below 0.5, while 9 dB resembles an artificially induced failure case.Source: §5.4 · pp. 7–8, Table 5 · p. 8

5.5. Qualitative Analysis

The qualitative examples pair MRI scans and segmentations with voxel-wise uncertainty and structure-wise IoU heat maps. The selected cases are each dataset’s worst-Dice scan, illustrating how uncertainty varies across datasets and structures.

5.5. Qualitative Analysis

  • The worst MALC example retained high segmentation quality, reflected by thin voxel-wise uncertainty lines and a red IoU heat map.MALC was also used for training.Source: §5.5 · p. 8
  • The worst ADNI-29 case had Dice 0.78, while the worst IBSR-18 case had Dice 0.79; their heat maps show elevated uncertainty in some subcortical structures.The ADNI case involved severe Alzheimer’s pathology absent from training; the IBSR scan showed poor contrast and ringing artifacts.Source: §5.5 · p. 8
  • The worst CANDI-13 case had Dice 0.73 and the lowest-confidence heat map, with high voxel-wise uncertainty and low-reliability blue shades in some subcortical structures.The scan belonged to a 5-year-old subject and had strong motion artifacts and poor contrast; this age range and scan quality were absent from training.Source: §5.5 · pp. 8–9
  • Figure 4 pairs MRI, segmentation, voxel-wise uncertainty, and IoU heat maps for the worst-Dice scan from each of four datasets.The heat-map scale runs from 0.4 to 1, with darker red indicating higher segmentation reliability.Source: §5.5 · p. 8, Figure 4 · p. 9

5.6. Uncertainty for Group Analysis

The paper incorporates structure-wise uncertainty into regression analyses as a confidence weight. In ADNI-29, weighting moved estimates closer to manual-annotation coefficients; in ABIDE-I, weighted analyses found autism associations matching a large ENIGMA study.

5.6.1. Group analysis on ADNI-29

  • For all 12 ADNI-29 structures, uncertainty weighting moved diagnosis-coefficient estimates closer to manual-annotation values; dMC-based weighting improved estimates for five structures.The comparison included manual annotations, FreeSurfer, Bayesian QuickNAT, and uncertainty-weighted Bayesian QuickNAT.Source: §5.6.1 · p. 9, Table 6 · p. 10
  • Figure 5 presents per-structure Dice scores (red) and IoUs (blue) on the CANDI-13 dataset as boxplots, with center lines marking medians and boxes spanning the 25th to 75th percentiles.Only left-hemisphere structures are shown; whiskers reach the most extreme non-outlier values, and outliers are marked with red crosses.Source: Figure 5 · p. 10

5.6.2. ABIDE-I

  • In ABIDE-I, uncertainty-weighted regression found significant autism associations matching a large ENIGMA study, while unweighted regression was significant only for amygdala.The analysis included 1,112 scans from 20 sites and used site as a covariate.Source: §5.6.2 · p. 10

5.6. Uncertainty for Group Analysis

  • The approach assigns continuous weights to all scans, avoiding outlier detection and manual inspection during group analysis.The authors frame structure-wise uncertainty as a confidence measure for the volume estimates used in regression.Source: §5.6 · p. 9, §5.6.2 · p. 10

5.7. General Discussion

Structure-wise uncertainty tracks segmentation quality and can support quality control when manual Dice annotations are unavailable. Weighting group analyses by uncertainty produced regression estimates closer to those from manual annotations, while uncertainty also rises for scans unlike the training data.

5.7. General Discussion

  • Uncertainty increases for scans that differ from training data in demographics, pathology, or image quality, and may flag scans for reacquisition or guide which scan types to add to training.Source: §5.7 · pp. 10–11
  • Structure-wise uncertainty measures are highly correlated with Dice, and the intersection-over-union of Monte Carlo samples is a proxy for Dice when manual annotations are unavailable.The metric is reported to have the same range as Dice and to be highly correlated with it in unseen datasets.Source: §5.7 · p. 11
  • Weighting regression samples by segmentation quality yielded estimates more similar to manual-annotation estimates, a potential advantage for large repositories where manual quality control is time-consuming.Source: §5.7 · p. 11

6. Conclusion

Bayesian QuickNAT estimates structure-wise uncertainty for whole-brain MRI segmentation using test-time dropout samples and four uncertainty metrics. The metrics correlated with segmentation accuracy across diverse unseen datasets, and uncertainty can also be incorporated into group analysis as confidence information.

6. Conclusion

  • The proposed structure-wise metrics correlated highly with segmentation accuracy across unseen datasets, and intersection-over-union of Monte Carlo samples closely approximated Dice.Validation included scans from children and scans with pathologies, low resolution, and low contrast.Source: §6 · p. 11
  • Bayesian QuickNAT is an F-CNN for whole-brain segmentation that uses test-time dropout to generate Monte Carlo samples and four metrics to quantify structure-wise uncertainty.Source: §6 · p. 11
  • The authors propose using uncertainty as confidence in group analysis, reporting reliable effect sizes and describing a fail-safe signal for manual intervention when the system is unsure.Source: §6 · p. 11
  • Experiments were limited to neuroimaging applications, although the authors state that the basic idea can be extended to other segmentation applications.Source: §6 · p. 11

Appendix

The appendix lists the brain-segmentation classes, including left- and right-sided structures and additional midline or other labels.

Appendix

  • The listed left-sided classes include white matter, cortex, ventricles, cerebellar tissues, thalamus, caudate, putamen, pallidum, hippocampus, amygdala, accumbens, and ventral DC.Source: Appendix · p. 12
  • The listed right-sided classes include white matter, cortex, ventricles, cerebellar tissues, thalamus, caudate, putamen, pallidum, hippocampus, amygdala, and accumbens.Source: Appendix · p. 12
  • Other listed labels are the 3rd and 4th ventricles, brain stem, CSF, Left ventral, and Optic Chiasma.Source: Appendix · p. 12
Loading 1811.09800v1…