Source-linked AI summary
Replication study: Development and validation of deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs
Mike Voets, Kajsa Møllersen, Lars Ailo Bongo
TL;DR
The paper asks whether a published deep learning method for detecting diabetic retinopathy can be replicated when source code and some methodological details are unavailable. It re-implements the method using public EyePACS and Messidor-2 data, but obtains lower AUCs than the original study. The authors conclude that replication of deep learning medical-image methods remains challenging and warrants further studies.
Problem
Published deep learning medical-image methods require replication, but source code, data, and methodological details are not always publicly available.
Method
The authors re-implemented the original diabetic-retinopathy detection method using public Kaggle EyePACS and Messidor-2 data while selecting unspecified normalization and validation settings experimentally.
Results
AUCs were 0.94 on the Kaggle EyePACS test set and 0.80 on Messidor-2, compared with 0.99 on both corresponding original-study test sets.
Takeaways & Limitations
The study illustrates challenges in replicating deep learning methods and motivates additional replication studies, especially for medical image analysis.
Takeaways & Limitations
Interpretation is limited because the replication used different data and grading, while normalization and validation hyper-parameters remained unspecified in the original study.
Abstract
from arXiv · showhide
Replication studies are essential for validation of new methods, and are crucial to maintain the high standards of scientific publications, and to use the results in practice. We have attempted to replicate the main method in 'Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs' published in JAMA 2016; 316(22). We re-implemented the method since the source code is not available, and we used publicly available data sets. The original study used non-public fundus images from EyePACS and three hospitals in India for training. We used a different EyePACS data set from Kaggle. The original study used the benchmark data set Messidor-2 to evaluate the algorithm's performance. We used the same data set. In the original study, ophthalmologists re-graded all images for diabetic retinopathy, macular edema, and image gradability. There was one diabetic retinopathy grade per image for our data sets, and we assessed image gradability ourselves. Hyper-parameter settings were not described in the original study. But some of these were later published. We were not able to replicate the original study. Our algorithm's area under the receiver operating curve (AUC) of 0.94 on the Kaggle EyePACS test set and 0.80 on Messidor-2 did not come close to the reported AUC of 0.99 in the original study. This may be caused by the use of a single grade per image, different data, or different not described hyper-parameter settings. This study shows the challenges of replicating deep learning, and the need for more replication studies to validate deep learning methods, especially for medical image analysis. Our source code and instructions are available at: https://github.com/mikevoets/jama16-retina-replication
1 Introduction
This study assesses whether a published deep learning method for detecting referable diabetic retinopathy can be replicated from its reported methodology. Using public data and a re-implementation, the authors found substantially lower performance than originally reported and identify missing methodological detail as a central challenge.
- Replicability is essential for validating medical technologies and maintaining scientific standards, but concerns about non-replicability span biomedical, computational, and machine-learning research.
- The study targets a well-known JAMA deep learning algorithm for detecting referable diabetic retinopathy because its source code was unpublished and no prior replication was known.
- The original algorithm achieved an AUC of 0.99 on both the EyePACS-1 and Messidor-2 test sets.
- The replication re-implemented the method with a public EyePACS data set, the public Messidor-2 set, and self-determined validation hyper-parameters and normalization.
- AUCs of 0.94 on the replication EyePACS test set and 0.80 on Messidor-2 did not reproduce the original results.
- The authors attribute possible deviations to different data and grading, unspecified hyper-parameters, possible original-study or methodological errors, or replication mistakes, while not invalidating the original findings.
2 Methods
The replication used public EyePACS and Messidor-2 images, adapted the original grading and data procedures, and re-implemented its deep learning architecture and validation approach. Missing normalization and validation details required the authors to select settings experimentally, while limited grading information prevented some original analyses.
- 2.1 Data sets: The replication used 57,146 EyePACS images for training and validation, 8,790 for testing, and all 1,748 Messidor-2 images for testing.
- 2.2 Grading: Unlike the original study, the replication lacked complete macular-edema grades and therefore did not train the algorithm to detect macular edema.
- 2.2 Grading: Image gradability was assessed for all Kaggle EyePACS and Messidor-Original images by one non-ophthalmologist author using a simple grading tool and the original study’s instructions.
- 2.2 Grading: Kaggle EyePACS images had one clinician-provided diabetic-retinopathy grade per image, whereas the original images were graded by ophthalmologists for quality, retinopathy, and macular edema.
- 2.3 Algorithm: The replica used the original InceptionV3 architecture, fundus preprocessing to 299 × 299 pixels, batch normalization, ImageNet initialization, and an 80/20 training-validation split.
- 2.3 Algorithm: Because normalization and validation hyper-parameters were unspecified, the authors experimented with image standardization, [0, 1] normalization, and other settings.
- 2.4 Algorithm validation: Validation performance was measured using AUC from thresholded network predictions, with validation AUC serving as the early-stopping criterion.
- 2.4 Algorithm validation: The replication omitted original supplementary experiments comparing training-set performance across image and grade subsets because it focused on the main rDR results and had only one grade per image.
3 Results
The replica algorithm performed best with images normalized to [–1, 1], but its test-set AUCs remained substantially below the original study’s reported performance.
- The replica algorithm was evaluated on two independent test sets.
- Table 1 compares replication performance across normalization methods with the original study’s results shown in parentheses.
4 Discussion
The replication differed substantially from the original study, with likely deviations involving data quality, grading, normalization, and unspecified hyper-parameters or architecture details.
- 4 Discussion: Performance differences likely reflected different public data, hyper-parameters, and grading procedures, including expert re-grading in the original study.
- 4 Discussion: Several original-study method details remained unspecified, including validation hyper-parameters, normalization, handling of ungradable images, and additional algorithms.
- 4 Discussion: Image normalization was essential, with [–1, 1] normalization performing best, possibly because InceptionV3 was pretrained on similarly normalized ImageNet images.
- 4.1 Hyper-parameters: The main hyper-parameter challenge was avoiding convergence on a local maximum of the validation AUC function.The authors also considered whether resizing fundus images to 299 pixels hindered lesion detection.
- 4 Discussion: The original architecture may have used image gradability as an input parameter.
- 4.2 Kaggle images: Kaggle included grades for all images, and 19.9% were ungradable, potentially allowing the algorithm to learn features associated with ungradable images.Excluding non-gradable images did not significantly improve performance.
5 Conclusion
The re-implementation did not match the original study’s performance, with deviations attributed mainly to hyper-parameters and data quality or grading. The replication study’s code and running instructions are publicly available.
- The re-implemented method did not achieve the original study’s reported performance using publicly available data.The authors identify hyper-parameters, data quality, and grading as the main sources of deviation.
- Image normalization to the [–1, 1] range performed best among tested methods but still diverged from the original algorithm’s results.The authors infer that the original study most likely used this normalization range.
- The authors regard unspecified hyper-parameter settings as a missing level of methodological detail, although they assume their impact was minor.
- Higher-quality original data and differences in the number of grades per image may explain part of the performance gap.The authors consider the smaller replication data set a less likely major source because performance reportedly levels off around 40 000 images.
- The replication study’s source code and execution instructions are publicly available on GitHub.