Exploiting Epistemic Uncertainty of Anatomy Segmentation for Anomaly Detection in Retinal OCT

Philipp Seeböck; José Ignacio Orlando; Thomas Schlegl; Sebastian M. Waldstein; Hrvoje Bogunović; Sophie Klimscha; Georg Langs; Ursula Schmidt-Erfurth

Source-linked AI summary

Exploiting Epistemic Uncertainty of Anatomy Segmentation for Anomaly Detection in Retinal OCT

Philipp Seeböck, José Ignacio Orlando, Thomas Schlegl, Sebastian M. Waldstein, Hrvoje Bogunović, Sophie Klimscha, Georg Langs, Ursula Schmidt-Erfurth

arXiv:1905.12806v1May 29, 2019eess.IVcs.CVcs.LGstat.ML

TL;DR

Supervised pathology segmentation requires predefined regions, extensive annotations, and representative disease cohorts, whereas anomaly detection can learn normal anatomy from healthy data. This paper uses Bayesian anatomical segmentation and epistemic uncertainty to detect retinal OCT anomalies, outperforming alternative approaches across several conditions and achieving a Dice index of 0.789.

Problem
Supervised pathology segmentation requires predefined pathology definitions, large-scale annotations, and representative patient cohorts, motivating anomaly detection from healthy data for biomarker discovery.
Method
A weakly supervised Bayesian U-Net learns healthy retinal layers from automatically generated labels, then uses Monte Carlo epistemic uncertainty and majority-ray-casting post-processing to segment anomalies.
Results
The method outperformed alternative approaches across several disease conditions, with Dice index improving from 0.768 to 0.789 after layer-flattening.
Takeaways & Limitations
Using healthy anatomical structure and uncertainty enables anomaly detection without target-class annotations and supports biomarker discovery across diseases.
Takeaways & Limitations
Majority-ray-casting can oversegment when anomalies are arranged in configurations that violate its surrounding-uncertainty assumption and can produce foveal false positives.

Abstract

from arXiv · show

Diagnosis and treatment guidance are aided by detecting relevant biomarkers in medical images. Although supervised deep learning can perform accurate segmentation of pathological areas, it is limited by requiring a-priori definitions of these regions, large-scale annotations, and a representative patient cohort in the training set. In contrast, anomaly detection is not limited to specific definitions of pathologies and allows for training on healthy samples without annotation. Anomalous regions can then serve as candidates for biomarker discovery. Knowledge about normal anatomical structure brings implicit information for detecting anomalies. We propose to take advantage of this property using bayesian deep learning, based on the assumption that epistemic uncertainties will correlate with anatomical deviations from a normal training set. A Bayesian U-Net is trained on a well-defined healthy environment using weak labels of healthy anatomy produced by existing methods. At test time, we capture epistemic uncertainty estimates of our model using Monte Carlo dropout. A novel post-processing technique is then applied to exploit these estimates and transfer their layered appearance to smooth blob-shaped segmentations of the anomalies. We experimentally validated this approach in retinal optical coherence tomography (OCT) images, using weak labels of retinal layers. Our method achieved a Dice index of 0.789 in an independent anomaly test set of age-related macular degeneration (AMD) cases. The resulting segmentations allowed very high accuracy for separating healthy and diseased cases with late wet AMD, dry geographic atrophy (GA), diabetic macular edema (DME) and retinal vein occlusion (RVO). Finally, we qualitatively observed that our approach can also detect other deviations in normal scans such as cut edge artifacts.

I. INTRODUCTION

The paper motivates anomaly detection as a way to identify retinal deviations without predefined pathology categories or diseased training cohorts, then proposes epistemic uncertainty from healthy-anatomy segmentation for this purpose.

Motivation: Anomaly detection learns normal appearance from healthy data and detects deviations without requiring representative pathological cohorts or predefined disease categories.This can support biomarker discovery and screening by focusing clinical review on segmented anomalous regions.
Motivation: Epistemic uncertainty is more suitable than aleatoric uncertainty for detecting changes from normal samples because it increases for appearances different from training data.Epistemic uncertainty reflects model uncertainty that can decrease with additional training data.
Contribution: The proposed approach learns healthy retinal anatomy with weak labels, uses Bayesian segmentation uncertainty to detect anomalies, and avoids manual labels during training.Weak labels are automatically generated by a surrogate segmentation method, enabling larger training sets and more appearance variability.
Application: OCT provides high-resolution retinal volumes whose B-scans support clinical decisions about diagnosis and treatment.The paper situates anomaly detection in a clinically important imaging modality.
Application: In retinal OCT, the method targets anatomical and pathological deviations, including layer-related abnormalities, without using anomaly labels for training.Normal retinas are defined as lacking pathological changes beyond permitted age-related alterations.

B. Related Work

Prior work includes handcrafted, supervised, generative, and deep anomaly-detection approaches for retinal imaging, while this paper introduces epistemic uncertainty from healthy anatomy segmentation as a new anomaly-segmentation strategy.

Existing approaches: Supervised biomarker methods target predefined pathological markers and require representative examples of their appearances, whereas anomaly detection models deviations from healthy data.The latter avoids selecting features around a fixed clinical target before anomaly discovery.
Existing approaches: Earlier OCT anomaly methods used shape models, Gaussian mixture models, or deep unsupervised models to identify anomalous scans or regions.These approaches span handcrafted shape modeling, statistical appearance modeling, and learned anomaly detection.
Novelty: Earlier Bayesian segmentation studies used aleatoric uncertainty for training, epistemic uncertainty for refinement or feedback, or dropout to average predictions rather than directly segment anomalies.The paper distinguishes its proposed use of epistemic uncertainty for anomaly detection from these prior applications.
Novelty: The paper presents epistemic uncertainty from a Bayesian U-Net trained on healthy anatomy as a novel basis for detecting and segmenting retinal OCT anomalies.It combines weak labels with majority-ray-casting to produce compact anomaly masks and reports improved performance over prior OCT anomaly detection.

II. METHODS

The method trains a Bayesian U-Net on normal retinal anatomy using automatically generated weak labels, then structures uncertainty-based outputs into anomaly segmentations.

Healthy-anatomy training: A Bayesian U-Net is trained on normal cases to segment retinal layers using weak labels automatically generated by a graph-based segmentation method.The overview describes healthy B-scans and automatically produced layer labels as the training inputs.
Uncertainty estimation: At test time, Monte Carlo dropout produces pixel-level epistemic uncertainty estimates for unseen B-scans.The model applies sampled predictions to quantify uncertainty while segmenting new images.
Post-processing: Majority-ray-casting transforms uncertainty maps into compact anomaly segmentations whose shape is more consistent with blob-like abnormalities than raw thresholded maps.The post-processing step is introduced to bridge the layered appearance of retinal anatomy and compact anomaly shapes.
Pipeline: The method description organizes the pipeline around weakly supervised healthy-population segmentation, epistemic-uncertainty anomaly detection, and a retinal-OCT-specific application.These components are presented as consecutive method stages.
Network architecture: The U-Net uses an encoder-decoder architecture with skip connections to reconstruct segmentation outputs from multiscale feature maps.The described implementation has five depth levels and applies dropout after convolutional blocks.

B. Exploiting Epistemic Uncertainty for Anomaly Detection

The anomaly detector estimates epistemic uncertainty from repeated dropout-based segmentation predictions and aggregates class-specific variability into a pixel-wise uncertainty map.

Rationale: Epistemic uncertainty is estimated because it increases when unseen image appearances differ substantially from the training data.The method uses this property to identify anomalies in new scans.
Bayesian formulation: Bayesian training approximates the posterior distribution over network weights with a variational distribution, optimized through dropout-based learning.The approximation makes posterior-based uncertainty tractable during model training.
Uncertainty estimation: For an unseen image, the model generates n predictions under sampled weights, then computes pixel-wise variance for each class.The variance measures prediction disagreement across stochastic forward passes.
Uncertainty aggregation: The final uncertainty map averages the class-specific variances at each pixel across the K segmentation classes.This produces one scalar uncertainty value per pixel for subsequent anomaly processing.

C. Application of anomaly detection in retinal OCT scans

The method trains a Bayesian U-Net on healthy OCT anatomy using weak retinal-layer labels, then converts test-time epistemic uncertainty into smooth anomaly segmentations. Majority-ray-casting iteratively fills uncertainty-derived regions using horizontal and vertical ray evidence before morphological cleanup.

Application setup: Healthy OCT B-scans and automatically generated retinal-layer labels train the Bayesian multiclass U-Net, whose test-time uncertainty—not layer accuracy—is used for anomaly segmentation.The labels contain 11 classes: background and 10 retinal layers.
Uncertainty preprocessing: Thresholded uncertainty maps produce an initial binary anomaly map, with small connected components removed before shape refinement.The binary map is not sufficiently smooth or compact for the intended anomaly representation.
Majority-ray-casting: Majority-ray-casting iteratively casts cardinal-direction rays from background pixels, voting for pixels enclosed by detected regions to create shape-consistent anomaly segmentations.The method assumes the retina is approximately horizontally oriented in the B-scan.
Post-processing: The ray-casting process can repeat with iteration-specific vote thresholds, followed by morphological closing and opening to remove artifacts.The resulting binary map from one iteration becomes the input to the next.

III. EXPERIMENTAL SETUP

The evaluation studies anomaly segmentation, component contributions, lesion-wise detection, and volume-wise classification in six macula-centered Spectralis OCT datasets. Healthy volumes train and validate the model, while independent diseased datasets provide anomaly tests.

Evaluation objectives: The experiments evaluate anomaly-region accuracy, component ablations, lesion-wise detection, and volume-wise classification based on anomalous pixels per B-scan.These four objectives define the experimental evaluation.
Data: The normal dataset contains 226 healthy volumes, split by patient into 198 training and 28 validation cases, while a separate 33-volume normal-evaluation set is reserved for evaluation.Volumes with pathological changes beyond age-related alterations were excluded from the healthy data.
Data: Late wet AMD contributes 31 OCT volumes, including 5 validation and 26 test cases with active neovascular AMD and manual annotations.The supplied setup identifies this cohort as an anomaly dataset.

A. Segmentation accuracy

Segmentation accuracy is assessed with precision, recall, and Dice, while ablations isolate the effects of label granularity, ray-casting, morphology, and retinal flattening. The test set was not used for design decisions.

Evaluation metrics: Precision, recall, and Dice evaluate binary segmentation, whereas ROC analysis is infeasible because the method outputs binary anomaly labels rather than pixel-level likelihoods.The metrics are standard for binary segmentation tasks.
Ablation protocol: The full method is called WeakAnD, and its test-set ablations illustrate how changing individual components affects results.Design decisions were based on validation-set performance rather than the test set.
Ablations: A binary retina/background model tests whether replacing 11-class layer labels with binary segmentation changes anomaly-detection performance.This ablation is named WeakAnD(binary).
Ablations: Other ablations remove majority-ray-casting, replace it with thresholding or convex hulls, remove morphological operations, or flatten the retina before processing.Flattening tests whether horizontal retinal orientation better satisfies the ray-casting assumption.

B. Lesion-wise Detection

Lesion-wise evaluation treats each connected anomaly within a B-scan as one lesion and relates lesion detection to segmentation overlap. Recall and precision curves are computed across all Dice thresholds to assess stability.

Lesion definition: Each connected anomaly within a B-scan is treated as one lesion, and its overlap with manual annotation is measured using an individual Dice index.This evaluates detection performance on a lesion-wise basis.
Lesion metrics: For a reference Dice threshold d, lesions exceeding d count as true positives for computing lesion-detection Recall and Precision.False negatives and false positives are also counted at each threshold.
Curve interpretation: Evaluating every d from 0 to 1 yields LD-Re and LD-Pr curves that assess Dice-value stability but cannot select an operating point.The curves span Dice values rather than lesion probabilities.

C. Volume-wise Disease Detection

The study tested whether average anomalous area per B-scan could distinguish healthy from diseased volumes across four disease comparisons.

The volume-wise experiments were conducted alongside quantitative evaluations on the late wet AMD validation set.
Four comparisons evaluated healthy volumes against late wet AMD, GA, RVO, and DME cases using the average anomalous area per B-scan.
The anomalous-area feature was used directly for discrimination without additional training.

IV. RESULTS

The proposed method produced strong anomaly segmentations and showed that epistemic uncertainty tracked annotated anomaly extent in late wet AMD OCT volumes. Ablations and qualitative analyses further examined the contributions of fine-grained anatomy segmentation and post-processing.

0.789 Dice was achieved after layer flattening, improving significantly from 0.768 on the late wet AMD test set.The paired Wilcoxon signed-rank test gave p = 0.00007.
0.91 Pearson correlation linked total segmentation uncertainty with manually annotated anomaly area across late wet AMD OCT volumes.
Segmentation Accuracy: All ablations reduced performance, with binary segmentation particularly harming metrics and highlighting the value of fine-grained layer segmentation.
Lesion-wise Detection: The proposed method was evaluated for lesion-wise detection using precision and recall curves alongside baseline methods.
Volume-wise Disease Detection: Volume-wise experiments used average anomalous pixels per B-scan to distinguish diseased and healthy volumes, with results shown for both baselines and the proposed method.

V. DISCUSSION

The method links epistemic uncertainty from healthy retinal anatomy segmentation to anomalies, producing useful anomaly segmentations and strong disease separation, while remaining constrained by anatomy coverage and post-processing assumptions.

Uncertainty analysis: Epistemic uncertainties were more powerful than softmax entropy for reflecting abnormal changes relative to the training set.The comparison used quantitative metrics and lesion-detection experiments.
Quantitative and lesion-level results: The proposed approach achieved significantly better quantitative results than the OCT anomaly-detection baseline.The discussion notes that manual anomaly boundaries are difficult and subjective, with the worst observed Dice value reported as 0.72.
Volume-wise classification: The predicted anomalous-area size almost perfectly discriminated normal from diseased subjects across AMD, DME, RVO, and GA experiments.Further evaluation is needed in early diseased cases and screening settings.
Anatomical supervision: Using whole-retina binary labels reduced anomaly-detection Dice by 14.7% because the resulting uncertainty maps were less detailed and produced inconsistent shapes and gap artifacts.Layer-wise retinal labels better incorporated anatomical knowledge into the model.
Post-processing and ablation: Dice index decreased by 28.4% when majority-ray-casting was replaced with uncertainty thresholding, which also introduced retinal-layer border artifacts.The threshold t=0.03 was selected on the validation set.
Scope and limitations: The model detected anomalies only between the top of the NFL and the RPE because jointly labeled background regions were not represented in training.Adding labels for layers such as the choroid might extend uncertainty estimates to other deviations.
Scope and limitations: Majority-ray-casting can oversegment when anomalies are arranged in ways that violate its assumption that uncertainty-surrounded areas are anomalous.False positives were also observed in the foveal depression.

VI. CONCLUSION

The paper presents a weakly supervised anomaly detector that uses epistemic uncertainty from a Bayesian multiclass U-Net trained on normal retinal anatomy. Evaluations show anomaly detection across several conditions, outperforming alternative approaches and supporting biomarker discovery.

VI. CONCLUSION: The method uses epistemic uncertainty from a Bayesian multiclass U-Net trained on normal subjects, without annotations of the target anomalies.Healthy retinal-layer labels are generated automatically by a traditional segmentation method.
VI. CONCLUSION: The approach needs only healthy samples and is not limited to a specific disease or pathology because it detects deviations from normal anatomy.This broad scope follows from exploiting anatomical information learned from healthy scans.
VI. CONCLUSION: The method was evaluated on 33 normal and 115 diseased OCT volumes, comprising 1617 and 5635 B-scans, respectively.
VI. CONCLUSION: The uncertainty-driven method detected anomalies under several conditions and outperformed alternative approaches.The authors identify this result as promising for biomarker discovery involving atypical visual variability.