Source-linked AI summary
DR$\vert$GRADUATE: uncertainty-aware deep learning-based diabetic retinopathy grading in eye fundus images
Teresa Araújo, Guilherme Aresta, Luís Mendonça, Susana Penas, Carolina Maia, Ângela Carneiro, Ana Maria Mendonça, Aurélio Campilho
TL;DR
DR grading is complex, and deep-learning systems’ black-box behavior can make their decisions difficult for specialists to understand and accept. DR|GRADUATE provides a grade with an explanation and uncertainty estimate; it achieved state-of-the-art performance across several DR-labeled datasets, with lower κ tending to accompany higher uncertainty.
Problem
DR grading requires integrating different abnormalities, specialists may disagree on complex cases, and deep-learning systems’ black-box behavior can make their decisions difficult to understand and accept.
Source: §1 · pp. 2–3Method
DR|GRADUATE predicts a DR grade with an uncertainty estimate and an explanation map indicating lesion presence across image patches.
Source: §1 · p. 3, §2.1 · p. 4Results
DR|GRADUATE achieved state-of-the-art performance across several DR-labeled datasets, and lower κ tended to accompany higher uncertainty.
Source: §6 · p. 15Takeaways & Limitations
Uncertainty can help specialists rank predictions for second evaluation, while explanation maps offer a tool for mitigating deep learning’s black-box behavior.
Source: §6 · p. 15R4 classification was the least accomplished task; the model failed to detect NVs and other R4 lesions and often associated photocoagulation marks with R4.
Source: §5.1 · p. 10, §5.3 · p. 14, §6 · p. 15
Abstract
from arXiv · showhide
Diabetic retinopathy (DR) grading is crucial in determining the adequate treatment and follow up of patients, but the screening process can be tiresome and prone to errors. Deep learning approaches have shown promising performance as computer-aided diagnosis(CAD) systems, but their black-box behaviour hinders the clinical application. We propose DR$\vert$GRADUATE, a novel deep learning-based DR grading CAD system that supports its decision by providing a medically interpretable explanation and an estimation of how uncertain that prediction is, allowing the ophthalmologist to measure how much that decision should be trusted. We designed DR$\vert$GRADUATE taking into account the ordinal nature of the DR grading problem. A novel Gaussian-sampling approach built upon a Multiple Instance Learning framework allow DR$\vert$GRADUATE to infer an image grade associated with an explanation map and a prediction uncertainty while being trained only with image-wise labels. DR$\vert$GRADUATE was trained on the Kaggle training set and evaluated across multiple datasets. In DR grading, a quadratic-weighted Cohen's kappa (QWK) between 0.71 and 0.84 was achieved in five different datasets. We show that high QWK values occur for images with low prediction uncertainty, thus indicating that this uncertainty is a valid measure of the predictions' quality. Further, bad quality images are generally associated with higher uncertainties, showing that images not suitable for diagnosis indeed lead to less trustworthy predictions. Additionally, tests on unfamiliar medical image data types suggest that DR$\vert$GRADUATE allows outlier detection. The attention maps generally highlight regions of interest for diagnosis. These results show the great potential of DR$\vert$GRADUATE as a second-opinion system in DR severity grading.
1. Introduction
DR screening matters because disease can progress without symptoms, yet grading requires integrating multiple abnormalities and remains vulnerable to errors. DR|GRADUATE is proposed to pair severity grades with interpretable explanations and uncertainty estimates.
1. Introduction
- Screening is essential because many patients with DR lack visual symptoms, while image quality and lesion variety contribute to diagnostic errors.Specialists grade fundus images by visually assessing abnormalities in retinal structures and tissue.Source: §1 · p. 2
- DR severity is staged from non-proliferative to proliferative disease, with ordered grades reflecting progression risk and differing lesion patterns.Mild NPDR is characterized by microaneurysms; proliferative DR involves abnormal new vessels.Source: §1 · p. 2, Table 1 · p. 2
- Deep-learning grading is more complex than lesion detection because it must identify and integrate multiple abnormalities, and specialists can disagree on complex cases.The black-box behavior of deep-learning systems can make their decisions difficult for specialists to understand and accept.Source: §1 · pp. 2–3
1.1. State-of-the-art
- Krause et al. (2018) used high-resolution Inception-v4 with descending grade thresholds, while de la Torre et al. (2018) treated grading as ordinal with a quadratic-weighted-kappa loss.De la Torre et al. (2018) reported improved performance over cross-entropy training.Source: §1.1 · p. 3
- Explanation research focused mainly on binary detection and referral, including saliency comparisons, MIL patch maps, and iterative inpainting approaches.Quellec et al. (2017), Costa et al. (2019), and González-Gonzalo et al. (2018) describe these respective approaches.Source: §1.1 · p. 3
- Leibig et al. (2017) found dropout-based Bayesian uncertainty more informative than softmax output for DR detection and reported benefits from uncertainty-aware referral.The method estimates uncertainty from disagreement across repeated predictions with dropout active during inference.Source: §1.1 · p. 3
1.2. Contributions
- DR|GRADUATE combines a severity grade with an uncertainty estimate and interpretable attention map, and is described as extensively validated on internationally recognized DR levels.The uncertainty is intended to help identify cases for specialist inspection; the map highlights regions relevant to the classification.Source: §1.2 · pp. 3–4
2. DR|GRADUATE for DR grading
DR|GRADUATE jointly predicts an ordinal DR grade, an explanation map, and explicit uncertainty using image-wise grade labels. Its design combines Multiple Instance Learning, Gaussian class probabilities, grade-wise maps, and iterative batch balancing.
2. DR|GRADUATE for DR grading
- The model produces a DR grade, explanation, and uncertainty from image-wise grade labels, treating grades as ordinal and lesion types as grade-specific.The framework uses the standard Multiple Instance Learning assumption that the highest-scoring lesion patch determines the image grade.Source: §2 · p. 4
2.1. Predicting a grade with uncertainty
- A 1×1×1 convolution forms lesion map L, whose maximum gives the preliminary grade; Gaussian class probabilities centered there use learned variance to estimate uncertainty.The variance is inferred from the backbone features, and the model is trained end-to-end.Source: §2.1 · p. 4
2.2. Explainability of the DR grade
- For each grade, lesion presence is converted into an input-sized explanation map and Gaussian processing weights pixels according to the network’s receptive field.The study does not aim to predict pixel-level lesion segmentations.Source: §2.2 · p. 5
2.3. Dealing with class imbalance
- Iterative batch balancing addresses grade imbalance while avoiding excessive rebalancing, which could increase healthy-image misclassifications.The batch distribution begins balanced and progressively shifts toward a more realistic, though still less skewed, distribution.Source: §2.3 · p. 5
3. Experiments
The experiments assess DR grading, uncertainty, and explainability across datasets. They also examine uncertainty in relation to image quality and unfamiliar medical image types.
3. Experiments
- DR grading performance is evaluated quantitatively across several labeled fundus datasets, with class-wise performance also analyzed.The protocol includes both intra- and inter-dataset experiments.Source: §3 · p. 6
- The experiments assess whether lower prediction uncertainty is associated with better grading performance and whether poor-quality images have higher uncertainty.Source: §3 · p. 6
- Uncertainty on colonoscopy, skin, cataract-surgery, and breast-biopsy images is analyzed to inspect potential outlier detection.Source: §3 · p. 6
- Attention maps are evaluated through visual inspection and quantitative comparison with ophthalmologist lesion annotations.Source: §3 · p. 6
3.1. Datasets
The grading evaluation uses the Kaggle training set and several public and private datasets. Separate image-quality datasets support uncertainty experiments, while SCREEN-DR also supplies lesion annotations.
3.1.1. DR grading datasets
- DR|GRADUATE was trained on Kaggle DR detection images (approximately 35,000 in the training set), labeled from absence through proliferative DR, and evaluated on several public and private datasets.Messidor-2 ground truth was an adjudicated consensus of three specialists; SCREEN-DR included a selected subset of 966 images graded in five levels.Source: §3.1.1 · pp. 6–7
- DMR uses modified Davis grades; the international grades are mapped as NDR=R0, SDR=R1∪R2, PPDR=R3, and PDR=R4.Source: §3.1.1 · p. 7, Table 2 · p. 7
- SCREEN-DR provides pixel-wise lesion annotations for 348 images, with grade maps grouping lesions by severity; EX annotations are excluded from those maps.R1 maps include MAs, R2 HEMs, R3 HEMs, CWSs and IRMAs, and R4 NVs, PHEMs and PFIB.Source: §3.1.1 · p. 7, Figure 6 · p. 7
3.1.2. Quality assessment datasets
- Uncertainty-related quality experiments use DR1, DRIMDB, and HRF, which include quality labels or paired good- and poor-quality images.HRF pairs images from the same eye and its poor-quality images have decreased sharpness; contrast and illumination are not assessed.Source: §3.1.2 · p. 7
3.2. Evaluation
The evaluation measures ordinal grading agreement, tests uncertainty against prediction performance and image quality, assesses explanation-map overlap, and uses nonparametric tests with effect sizes. It also checks whether replacing the multiple-instance layer changes grading performance.
3.2.1. DR grading
- The quadratic-weighted kappa evaluates ordinal agreement, while normalized confusion matrices support grade-wise assessment because κ alone can be dominated by common classes.The confusion-matrix rows are normalized by their totals; κ ranges from −1 to 1.Source: §3.2.1 · pp. 7–8
- Replacing the multiple-instance layer with a fully connected layer produced no significant performance difference, though the non-MIL model tended to detect R0 better and confuse disease cases more.No further experiments with the non-MIL model were performed.Source: §3.2.1 · p. 8
3.2.2. Uncertainty estimation
- Uncertainty quality is tested by calculating κ on subsets below increasing uncertainty thresholds, beginning at 0.15 to retain at least 48 SCREEN-DR images, and by comparing grade-wise uncertainty distributions.The evaluation also averages uncertainty within each predicted-grade versus ground-truth-grade confusion-matrix position and tests blur and contrast changes.Source: §3.2.2 · p. 8
3.2.3. Explainability
- Explanation maps are evaluated by lesion overlap and image-level measures; allowing overlap with the same or lower grades reflects the pipeline’s max-pooling assumption.The measures include object overlap, maximum-activation overlap, class-level overlap, and ground-truth lesion coverage.Source: §3.2.3 · p. 8
3.2.4. Statistical tests
- Dataset uncertainty differences are assessed with the nonparametric Kruskal–Wallis H test and Cohen’s d, reporting effect size because large samples can make small differences statistically significant.Cohen’s d uses the difference between group means relative to pooled standard deviation; the paper reports effect size alongside significance given dataset sizes commonly exceeding 1,000 samples.Source: §3.2.4 · pp. 8–9
3.3. Parameter setting
The network uses a 640×640×3 input and a 20×20 output feature map, with settings selected for attention-map coverage, loss regularization, and class balancing.
3.3. Parameter setting
- The 640×640×3 input yields a 20×20 feature map, with receptive field 156 and stride 32; attention-map Gaussian σ′ is set to RF/6.The loss weight α is 0.7, and class balancing uses r = 0.99 and wf = (0.5, 2, 2, 3, 3) with f = 300.Source: §3.3 · p. 9
3.4. Training details
The model is trained on cropped and resized fundus images using the Kaggle DR detection training set and specified optimization settings.
3.4. Training details
- Images are cropped around the field of view and resized to 640×640 pixels; training runs for 240 epochs with batch size 30, Adam, and learning rate 3×10^-4.Training uses the Kaggle DR detection training set.Source: §3.4 · p. 9
4. Results
DR|GRADUATE achieves κ similar to state-of-the-art methods and shows consistent grading performance across multiple test datasets. Its uncertainty is higher for poor-quality images, and its explanation maps are assessed against specialist lesion annotations.
4.1. DR grading
- DR|GRADUATE achieves κ similar to state-of-the-art methods, with grade-wise confusion matrices showing a diagonal tendency across test datasets.R4 is the most misclassified grade; over-diagnosis ranges from 24% on Kaggle to 55% on SCREEN-DR, while under-diagnosis ranges from 45% to 76%. The confusion matrices show row-normalized percentages.Source: §4.1 · p. 9, Table 3 · p. 10, Figure 7 · p. 10
4.2. Uncertainty estimation
- For all three quality-labeled datasets, uncertainty is statistically higher on bad-quality images (p-value < 0.05); blur sensitivity is assessed across five datasets, while contrast adjustment shows no suggested relation to uncertainty.For DRIMDB the effect size is huge, and for the other datasets it is medium/large.Source: §4.2 · p. 9, §4.2.1 · p. 9
- Across unfamiliar image types, neighboring datasets have statistically different uncertainty (p-value < 0.05) with large to very large effect sizes, while differences among eye-fundus datasets are small to very small.The BACH-to-SCREEN-DR difference is statistically significant with a medium effect size.Source: §4.2 · p. 9
4.3. Explainability
- Attention-map examples include correctly explained predictions and incorrect predictions or explanations; higher thresholds produce smaller, more specific regions but lower sensitivity.Threshold 0.3 is described as a compromise that retains many lesions without producing overly coarse regions; Table 4 assesses maps against specialist-annotated lesions.Source: §4.3 · pp. 9–10, Table 4 · p. 10
5. Discussion
DR|GRADUATE generalizes across datasets, but retains grade-specific error modes. Its uncertainty estimates track prediction reliability and image quality, while its maps highlight plausible diagnostic regions without capturing every lesion.
5.1. DR grading
- Performance does not degrade beyond Kaggle, but R3/R4 confusion persists and errors also reflect annotation criteria, artifacts, subtle lesion detection, and clustered-hemorrhage counting.The MIL assumption can fail when R2/R3 grading hinges on hemorrhage counts rather than a novel lesion.Source: §5.1 · p. 10
5.1.1. Comparison with the state-of-the-art
- At 2% of de la Torre et al. (2017)’s parameter count, DR|GRADUATE achieves similar performance but trails Krause et al. (2018)’s 0.84 κ on EyePACS-2.Krause et al.’s model used over 1.6 million images with adjudicated labels and had 72% normalized over-grading; DR|GRADUATE showed lower over-estimation.Source: §5.1.1 · p. 11
5.2. Uncertainty estimation
- Higher κ accompanies lower uncertainty, which increases with distance from the correct grade; reviewing uncertain cases may reduce workload, with 25% removed reaching 0.8 κ on Kaggle.Low-uncertainty misclassifications can still arise from lookalike lesions or acquisition errors.Source: §5.2 · p. 11
5.2.1. Low quality images
- Bad-quality images received significantly higher uncertainty across datasets; uncertainty also rose with increasing blur, while contrast changes showed no correlation, plausibly reflecting training augmentation.The HRF and DR1 bad-quality subsets tested blur only, not low contrast or poor illumination.Source: §5.2.1 · pp. 12–13
5.2.2. Unfamiliar data types
- Unfamiliar non-fundus medical images had higher average uncertainty than fundus images, supporting their potential detection as outliers; local structural resemblance affected relative uncertainty.The authors suggest locally similar structures in microscopy images may produce lower uncertainty than globally similar colonoscopy images.Source: §5.2.2 · p. 13
5.3. Explainability
- Maps highlight plausible diagnosis-relevant regions, including small MAs and varied R3 signs, but miss some R4 lesions and may mark photocoagulation as R4.Because specialists may omit abnormalities from annotations, lesion-wise detection is not reliable enough to characterize an image; explanations also exposed camera-lens dust behind misclassifications.Source: §5.3 · pp. 13–15
6. Conclusions
DR|GRADUATE provides DR grades with an uncertainty estimate and explanation map, and achieves state-of-the-art performance across several DR-labeled datasets. Higher uncertainty tends to accompany lower κ; R4 classification was least successful, with Kaggle labeling identified as a likely factor.
6. Conclusions
- R4 classification was the least accomplished task; the authors suggest Kaggle labels may include laser-treated images as PDR despite lacking typical PDR signs.The paper reports state-of-the-art performance in several DR-labeled datasets overall.Source: §6 · p. 15
- Lower κ tended to occur in higher-uncertainty cases, suggesting uncertainty can help rank predictions and identify images for specialist re-evaluation.The authors frame uncertainty as a way to judge how much to trust predictions when test-time performance is unknown.Source: §6 · p. 15
- Compared with state-of-the-art systems, DR|GRADUATE adds uncertainty estimates and maps of relevant regions without compromising DR-grading performance.The authors expect the combination to ease clinical application.Source: §6 · p. 15
- Figure 15 shows examples predicted as R4, with contours marking attention maps at a threshold of 0.3.The authors report R4 as the least accomplished grade.Source: Figure 15 · p. 15, §6 · p. 15