Disentangling Human Error from the Ground Truth in Segmentation of Medical Images

1 Introduction

• Medical image segmentation faces high inter-reader variability and scarce curated labels because annotation is costly.The passages cite variability in MS lesion location, size, shape, and anatomy, and report 74–85% inter-reader variability for glioblastoma segmentation.Source: §1 · pp. 1–2
• Earlier label-fusion methods assume equal expert reliability or lack a mechanism to learn across training images.The cited discussion also notes that their relatively simple functions may not capture complex annotator behavior.Source: §1 · p. 2
• The method jointly estimates annotator reliability and true segmentation labels from noisy annotations using two coupled CNNs.One CNN estimates true segmentation probabilities; the other models annotator behavior through per-image, pixel-wise confusion matrices.Source: §1 · p. 2
• Its parameters are optimized across images, allowing it to use correlations between similar samples even when each image has only one annotation.The introduction contrasts this with methods that estimate annotator parameters separately for each target image.Source: §1 · p. 2
• Across MNIST and three medical imaging datasets, the method outperforms label-fusion approaches and relevant baselines, especially with few labels and high disagreement.The evaluated datasets are MSLSC, BraTS, and LIDC-IDRI; MNIST uses simulated annotators.Source: §1 · p. 2

2 Related Work

• STAPLE-family methods model annotator reliability, while STEP incorporates local morphological similarity between atlases and target images.These label-fusion approaches are fitted per subject and lack a mechanism to integrate information across training images.Source: §2 · p. 3
• Probabilistic U-net and PHiSeg treat variable expert annotations as plausible true segmentations, whereas this work treats them as noisy estimates of one unknown true map.The paper argues this single-truth assumption may suit physical boundaries while allowing ambiguity in human interpretations.Source: §2 · p. 3
• Joint annotator-label learning had shown promise in classification, but received little attention for structured prediction with high-dimensional outputs.The paper proposes the first simultaneous approach to this segmentation problem, drawing inspiration from the two-stage STAPLE framework.Source: §2 · p. 3

3.1 Problem Set-up

• Images have noisy annotations from identified annotators, but no ground-truth labels; at inference, the segmentation network predicts without fusing test-image annotations.Each image is assumed to have at least one annotator.Source: §3.1 · p. 3

3.2 Probabilistic Model and Proposed Architecture

• Two coupled CNNs estimate the true segmentation probabilities and each annotator’s image-dependent, pixelwise confusion matrices, under independence assumptions for annotators and pixels given the image.The segmentation network’s output is used to segment test images.Source: §3.2 · pp. 4–5

3.3 Learning Spatial Confusion Matrices and True Segmentation

• Cross-entropy fits annotator-specific predictions to noisy labels, while minimizing the estimated confusion-matrix trace encourages unreliable annotators and separates noise from the true-label distribution.The combined loss is optimized by stochastic gradient descent to learn both networks.Source: §3.3 · p. 5

3.4 Justification for the Trace Norm

• When predictions closely fit noisy-label distributions and the average confusion matrix’s correct-class diagonal entry is largest in its row, trace minimization recovers that class’s annotator-specific CM column.Under the single-ground-truth assumption, the remaining CM values are uniformly 1/L, so recovering the correct-class column suffices.Source: §3.4 · p. 6

4 Experiments

• Evaluation covers synthetic binary labels for MNIST and MS, synthetic multi-class labels for BraTS, and real expert annotations for LIDC-IDRI.The experiments report CM root-MSE, segmentation Dice, and generalized energy distance, and compare with label-fusion baselines, Global CM, and Probabilistic U-Net.Source: §4 · pp. 6–7

4.1 MNIST and MS lesion segmentation datasets

• On MNIST and MS, the model achieves higher Dice than STAPLE, especially with one label per image, and improves confusion-matrix estimation by 6.3%.It is also consistently more accurate than the Global CM model.Source: §4.1 · p. 7, Table 1 · p. 7, Table 2 · p. 7

4.2 BraTS Dataset and LIDC-IDRI Dataset

• The LIDC-IDRI examples include a case where the model predicts an abnormality missed completely by annotator 4.The paper also reports visualizing predicted segmentations alongside annotator contours and consensus.Source: Figure 7 · p. 8, §4.2 · p. 9
• BraTS evaluation uses four simulated annotator patterns, while LIDC-IDRI uses four radiologists’ annotations and a STAPLE-aggregated gold standard.For LIDC-IDRI, labels are unavailable at test time; lesion-centered 2D crops are used in the experiment.Source: §4.2 · pp. 8–9
• On BraTS and LIDC-IDRI, the model achieves higher Dice than STAPLE and Spatial STAPLE for dense and single-label scenarios; BraTS confusion-matrix estimation improves over STAPLE by 14.4%.The reported figures also show successful prediction of lesion segmentation and annotator variations.Source: §4.2 · p. 9
• The model outperforms Probabilistic U-Net on generalized energy distance across four test datasets, indicating better capture of inter-annotator variation.Table 3 reports GED using DICE as its distance metric.Source: Table 3 · p. 7, §4.2 · p. 9

5 Discussion and Conclusion

• Experiments show considerable improvement over averaging, majority vote, STAPLE, and its extensions in segmentation accuracy and confusion-matrix estimation.The method jointly estimates spatial characteristics of annotator errors and the ground-truth label distribution.Source: §5 · p. 9
• The method assumes one unknown true segmentation map, although the authors suggest real variation may combine intrinsic ambiguity with human-specific errors.They propose annotator metadata or a curated reference set as possible ways to help separate these sources of variation.Source: §5 · p. 9
• Future applications include selecting samples and annotators for labeling, and using estimated error patterns to help inexperienced annotators improve.The authors describe these as future work or potential applications.Source: §5 · p. 9

Boarder Impact Statement

• Medical image segmentation assigns biologically meaningful labels to pixels or voxels and supports applications including surgical planning and disease-progression studies.High inter- and intra-reader variability is common in expert segmentation labels.Source: Boarder Impact Statement · p. 10
• Labeling an MS patient’s three average-visit scans takes 1.5 to 2 hours, and expert voxel-by-voxel labels have approximately 10% volume variability.The passage connects this burden and inconsistency to potential time and economic costs.Source: Boarder Impact Statement · p. 10
• Estimated annotator information could support education and data-efficient label acquisition, while the current work has been demonstrated only on medical images.The passage also names diagnosis, prognosis, surgical planning, and disease studies as potential areas of impact.Source: Boarder Impact Statement · p. 10

• The fragment names medical-image segmentation.Source: Supplementary Material: Disentangling… · p. 14

A.1 Annotation Simulation Details

• Synthetic annotations are generated from assumed ground truth on MNIST, MS lesion, and BraTS datasets to evaluate the approach when ground truth is known.Source: §A.1 · p. 14
• Five simulated annotators represent faithful, over-segmenting, under-segmenting, fracture-prone over-segmenting, and blank-labeling behaviors.The simulations use morphological transformations such as thinning, thickening, and fractures.Source: §A.1 · p. 14
• For multi-class simulations, a target class receives four synthetic label patterns, and noise-level robustness is tested on MNIST and MS lesion data.Source: §A.1 · p. 14

A.2 Additional Qualitative Results on MNIST and MS Dataset

• The model achieved higher Dice similarity coefficients than STAPLE and Spatial STAPLE on MNIST and MS lesion examples.The segmentation figures compare ground truth and simulated annotator labels with supervised-model predictions.Source: §A.2 · p. 14, Figure 7 · p. 14, Figure 9 · p. 15
• The model outperformed STAPLE and Spatial STAPLE in confusion-matrix estimation, with and without trace norm.The estimated-label and confusion-matrix visualizations show patterns of annotator mistakes, including in the single-label-per-image setting.Source: §A.2 · p. 14, Figure 8 · p. 15, Figure 10 · p. 15
• The figures contrast ground-truth and annotator segmentations with predictions from supervised models on MNIST and MS lesion images.The companion visualizations show estimated true labels and confusion matrices; white, green, red, and black encode true positive, false negative, false positive, and true negative, respectively.Source: Figure 7 · p. 14, Figure 9 · p. 15, Figure 8 · p. 15

A.3 Quantitative and Extra Qualitative Results on BraTS and LIDC-IDRI

• LIDC-IDRI uses Spatial STAPLE-aggregated labels as ground truth and compares estimated with measured inter-reader consensus; BraTS experiments use high-grade cases and cropped 2D images.The additional figures show BraTS and LIDC-IDRI segmentations and annotator predictions.Source: §A.3 · p. 16, Figure 11 · p. 17, Figure 12 · p. 17
• 14.4% on BraTS is the reported margin by which the trace-regularized model outperforms STAPLE in confusion-matrix estimation; Dice is higher on both datasets and label settings.The segmentation and confusion-matrix comparisons are reported for BraTS and LIDC-IDRI with dense and single labels.Source: §A.3 · p. 16, Table 4 · p. 16, Table 5 · p. 17
• The model improves over baselines in all three LIDC-IDRI consensus subgroups, including low-consensus cases where experts disagree substantially.Estimated annotator consensus correlates strongly with real inter-reader variation, though variation appears under-estimated for easy, high-consensus samples.Source: Figure 14 · p. 18, Figure 15 · p. 19

A.4 Low-rank Approximation

• The rank-1 approximation halves confusion-matrix parameters and annotator-prediction FLOPs while reasonably retaining segmentation and confusion-matrix performance.On four-class BraTS, overall GPU-use reduction is marginal; the authors expect greater gains with more classes.Source: §A.4 · p. 20, Table 6 · p. 20
• The approximation represents each annotator’s spatial confusion matrix as a product of two smaller rectangular matrices, reducing complexity growth from quadratic to linear in class count.Figure 16 compares parameter counts and FLOPs for the default and low-rank implementations.Source: §A.4 · p. 20, Figure 16 · p. 20

B Implementation details

• A four-stage 2D U-net shares parameters between segmentation and annotator networks except for their decoder output layers.The segmentation head outputs class channels; the annotator head outputs L×L channels per location, or 2×L×l with low-rank approximation.Source: B Implementation details · p. 21

B.1 Pytorch implementation of loss function

• The loss maps predicted true-label probabilities through each annotator’s confusion matrix, then sums noisy-label cross-entropy and weighted trace regularization.The implementation applies softmax to segmentation logits and computes pixelwise cross-entropy against each annotator’s labels.Source: §B.1 · p. 21

C Proof of Theorem 1

• With perfect modeling of noisy labels and diagonal-dominant true and estimated confusion matrices, trace is minimized when the matrices match; equal trace recovers the true label and true-class column.The argument is introduced for one annotator and then extended to multiple annotators.Source: C Proof of Theorem 1 · p. 22
• For a one-hot true class k, the proof establishes that the estimated confusion matrix’s kth diagonal entry is at least the true matrix’s entry.The argument uses diagonal dominance and the estimated label distribution at that pixel.Source: C Proof of Theorem 1 · p. 22
• The proof distinguishes its sample-specific result from earlier equivalent results stated in expectation over the data population.The authors characterize their result as a slightly weaker version of those prior results.Source: C Proof of Theorem 1 · p. 22