Source-linked AI summary
Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy
Jonathan Krause, Varun Gulshan, Ehsan Rahimy, Peter Karth, Kasumi Widner, Greg S. Corrado, Lily Peng, Dale R. Webster
TL;DR
Grader variability and limited examination of disagreement-resolution methods complicate reference-standard construction for diabetic retinopathy algorithms. This study compares grading and reference-standard approaches, finding that adjudicating a small subset of grades improved performance, reaching approximately ophthalmologist-level results.
Problem
Complex diabetic retinopathy grading produces grader variability, while the differences among disagreement-resolution methods have not been examined extensively.
Method
The study compares grading and reference-standard methods, using multiple label sources and a subspecialist adjudication protocol to develop deep learning models.
Results
Adjudicating 0.22% of training image grades significantly improved model performance, which was approximately on par with individual ophthalmologists and retinal specialists.
Takeaways & Limitations
Consensus grades from multiple retina specialists provide a more rigorous reference standard and improved algorithmic accuracy for retinal disease detection and screening.
Takeaways & Limitations
The study used hard exudates as a proxy for diabetic macular edema, which generally requires stereoscopic images or OCT for proper diagnosis and characterization.
Abstract
from arXiv · showhide
Diabetic retinopathy (DR) and diabetic macular edema are common complications of diabetes which can lead to vision loss. The grading of DR is a fairly complex process that requires the detection of fine features such as microaneurysms, intraretinal hemorrhages, and intraretinal microvascular abnormalities. Because of this, there can be a fair amount of grader variability. There are different methods of obtaining the reference standard and resolving disagreements between graders, and while it is usually accepted that adjudication until full consensus will yield the best reference standard, the difference between various methods of resolving disagreements has not been examined extensively. In this study, we examine the variability in different methods of grading, definitions of reference standards, and their effects on building deep learning models for the detection of diabetic eye disease. We find that a small set of adjudicated DR grades allows substantial improvements in algorithm performance. The resulting algorithm's performance was on par with that of individual U.S. board-certified ophthalmologists and retinal specialists.
1 Introduction
Diabetic retinopathy grading requires identifying fine retinal features, creating substantial variability among graders. This study examines how grading disagreement-resolution methods and reference standards affect deep learning for diabetic eye disease.
- DR grading requires identifying and quantifying microaneurysms, intraretinal hemorrhages, intraretinal microvascular abnormalities, and neovascularization.
- Intergrader kappa scores for DR grading range from 0.40 to 0.65.
- Reference standards can use majority decisions, senior-grader arbitration, or consensus discussion among multiple graders.
- The relative performance of these disagreement-resolution methods has not been examined extensively.
- The study examines grading variability, reference-standard definitions, and their effects on deep learning models for diabetic eye disease detection.
2 Methods
The study combines large-scale fundus-image training with multiple grading sources and a small adjudicated tuning set. It trains an ensemble convolutional neural network to predict DR, DME, and image gradability, while tuning model choices against adjudicated consensus grades.
- The development data combine EyePACS clinic images, three Indian eye hospitals, and Messidor-2, with EyePACS, ophthalmologist, and retinal-specialist adjudication grades.
- Unadjudicated images formed the train set, while adjudicated consensus images formed the tune set for selecting hyperparameters and modeling choices.
- The clinical validation set used nonoverlapping EyePACS images graded by three retina specialists through face-to-face adjudication and by three distinct ophthalmologists.
- A convolutional neural network predicts a 5-point DR grade, referable DME, and image gradability from fundus images.
- The model uses distributed stochastic gradient descent, tuning-set evaluation, and an ensemble of 10 models to improve performance and robustness.
- Algorithmic improvements include a larger training set, separate prediction targets for grading sources, 779 x 779-pixel inputs, Inception-v4, and 5-class DR prediction.
3 Results
Adjudication changed some retinal-specialist grades and exposed variability among individual graders. Models evaluated against adjudicated consensus achieved strong DR performance, approximately matching individual ophthalmologists and retinal specialists across grading tasks.
- 3.1 Grading and Adjudication: 1.5% of images differed by 2 DR severity steps between retinal-specialist majority grades and adjudicated consensus, with weighted kappa of 0.91.
- 3.1 Grading and Adjudication: For moderate or worse DR, individual specialists had 74.4% to 82.1% sensitivity and at least 99% specificity, while nonadjudicated majority sensitivity was 88.1%.
- 3.1 Grading and Adjudication: Ophthalmologists had quadratic-weighted kappa values of 0.80 - 0.84 individually, rising to 0.87 for their majority decision.
- 3.1 Grading and Adjudication: Missed microaneurysms, artifacts, and MA-versus-hemorrhage disagreements were the most common causes of ophthalmologist grading errors, at 36%, 20%, and 16%.
- 3.2 Model Results: 0.942 to 0.986 AUC improvement occurred for moderate or worse DR when adjudicated consensus, rather than majority decision, was used as the reference standard.
- 3.2 Model Results: 0.84 quadratic-weighted kappa was achieved for 5-class DR prediction, on par with individual ophthalmologists and retinal specialists.
- 3.2 Model Results: For binary DR tasks, algorithm performance was approximately on par with individual ophthalmologists and retinal specialists; mild-or-worse DR yielded sensitivity 0.970, specificity 0.917, and AUC 0.986.
4 Discussion
The discussion emphasizes that adjudicated consensus grading can improve clinically useful diabetic eye disease algorithms, even when only a small subset of training grades is adjudicated. It also identifies grading-process insights, operational trade-offs, and limits on generalizability.
- Reference standards: Consensus adjudication by multiple retina specialists provides a more rigorous reference standard by correcting artifacts, missed microaneurysms, and ambiguous lesion classifications.Disagreements included distinguishing microaneurysms from intraretinal hemorrhages and separating moderate from severe disease.
- Algorithm versus manual grading: 20.36% overgrading versus 1.2% undergrading characterized the algorithm, whereas ophthalmologist grading showed 7.3% overgrading versus 11.4% undergrading.The algorithm’s imbalance reflected operation at a very high-sensitivity screening set point.
- Algorithm versus manual grading: Combining algorithmic initial screening with manual review of positive images would likely preserve high sensitivity and specificity while limiting unnecessary referrals.This proposed workflow is intended for screening referable diabetic retinopathy.
- Limitations: The finding that resolutions above 450 x 450 pixels did not significantly improve performance may not generalize beyond single nonstereoscopic 45° color fundus photographs and the ICDR scale.Different grading scales, image types, or additional imaging such as OCT and fluorescein angiography could change the result.
- Limitations: Using hard exudates as a proxy for diabetic macular edema limited the study because proper diagnosis and characterization generally require stereoscopic images or OCT.The authors identify inclusion of OCT in future ground-truth determination as valuable.
- Algorithm development: Adjudicating 0.22% of training image grades in a tune set significantly improved model performance, without adjudicating the entire training corpus.The resulting model performed approximately on par with individual ophthalmologists and retinal specialists.
A Model Hyperparameters
The models used Inception-based architectures, Adam optimization, image augmentation, and specified training configurations. The full model increased input resolution to 779 × 779 and used distinct hyperparameters.
- The Gulshan et al. implementation used Inception-v3 with 299 × 299 input images, learning rate 0.001, batch size 32, and weight decay 4 · 10−5.
- The implementation used Adam with β1 = 0.9, β2 = 0.999, and ϵ = 0.1, rather than the RMSProp optimizer used by Gulshan et al.
- The full model used Inception-v4, 779 × 779 input images, learning rate 0.0014339, batch size 24, and weight decay 1.15 · 10−5.
- The implementation applied reflections and randomized brightness, saturation, hue, and contrast changes as data augmentation.
- Each model in the 10-way ensemble was trained for 250,000 steps, with evaluations based on a running average of parameters.
Tables and Figures
The tables and figures describe dataset standards, grader agreement, disagreement causes, algorithm comparisons, and model evaluation against adjudicated references. Together, they organize how grading protocols and reference standards relate to clinical and algorithmic performance.
- Table 1: Table 1 contrasts majority and adjudicated reference standards across training, tuning, and validation datasets, including different DME definitions.
- Table 2: Table 2 compares retinal-specialist majority grades with adjudicated consensus using a diabetic-retinopathy confusion matrix.
- Table 3: Table 3 reports each retinal specialist’s sensitivity, specificity, and quadratic-weighted kappa against the adjudicated reference standard for moderate or worse DR.
- Table 4: Table 4 compares ophthalmologist-majority grades with retinal-specialist adjudicated consensus for DR and DME.
- Table 5: Table 5 reports ophthalmologists’ sensitivity, specificity, and quadratic-weighted kappa against adjudicated grades for moderate or worse DR and referable DME.
- Table 6: Table 6 categorizes disagreements between retinal-specialist adjudication and ophthalmologist-majority grades, with positive values indicating higher adjudicated grades.
- Table 7: Table 7 compares final AUCs for moderate or worse DR across the original model, a high-resolution variant, and the full model.
- Table 8: Table 8 compares algorithm grades with retinal-specialist adjudicated grades for DR and DME, reporting sensitivity and specificity for moderate or worse DR and referable DME.