Source-linked AI summary
Evaluation of a deep learning system for the joint automated detection of diabetic retinopathy and age-related macular degeneration
Cristina González-Gonzalo, Verónica Sánchez-Gutiérrez, Paula Hernández-Martínez, Inés Contreras, Yara T. Lechanteur, Artin Domanian, Bram van Ginneken, Clara I. Sánchez
TL;DR
Existing deep-learning retinal systems analyze diabetic retinopathy and age-related macular degeneration separately, although the diseases can co-exist. This study validates RetCAD for joint detection and finds its performance comparable to human experts, with higher sensitivity but lower specificity for both diseases.
Problem
Existing deep-learning systems analyze diabetic retinopathy and age-related macular degeneration independently, despite their possible co-existence.
Source: Introduction · pp. 3–4Method
The study validates commercially available RetCAD v.1.3.0 for joint detection of referable diabetic retinopathy and age-related macular degeneration and compares its performance with human observers.
Source: Introduction · p. 4Results
RetCAD performed comparably to human experts, with lower specificity but higher sensitivity for both diabetic retinopathy and age-related macular degeneration.
Source: Discussion · p. 10Takeaways & Limitations
The findings support the potential of automated joint detection for screening for diabetic retinopathy and age-related macular degeneration.
Source: Introduction · p. 4, Discussion · p. 10Joint detection was evaluated on one dataset of 600 images from 288 patients, and the authors note that future studies would benefit from more and larger validation datasets.
Source: Limitations and future work · p. 12
Abstract
from arXiv · showhide
Purpose: To validate the performance of a commercially-available, CE-certified deep learning (DL) system, RetCAD v.1.3.0 (Thirona, Nijmegen, The Netherlands), for the joint automatic detection of diabetic retinopathy (DR) and age-related macular degeneration (AMD) in color fundus (CF) images on a dataset with mixed presence of eye diseases. Methods: Evaluation of joint detection of referable DR and AMD was performed on a DR-AMD dataset with 600 images acquired during routine clinical practice, containing referable and non-referable cases of both diseases. Each image was graded for DR and AMD by an experienced ophthalmologist to establish the reference standard (RS), and by four independent observers for comparison with human performance. Validation was furtherly assessed on Messidor (1200 images) for individual identification of referable DR, and the Age-Related Eye Disease Study (AREDS) dataset (133821 images) for referable AMD, against the corresponding RS. Results: Regarding joint validation on the DR-AMD dataset, the system achieved an area under the ROC curve (AUC) of 95.1% for detection of referable DR (SE=90.1%, SP=90.6%). For referable AMD, the AUC was 94.9% (SE=91.8%, SP=87.5%). Average human performance for DR was SE=61.5% and SP=97.8%; for AMD, SE=76.5% and SP=96.1%. Regarding detection of referable DR in Messidor, AUC was 97.5% (SE=92.0%, SP=92.1%); for referable AMD in AREDS, AUC was 92.7% (SE=85.8%, SP=86.0%). Conclusions: The validated system performs comparably to human experts at simultaneous detection of DR and AMD. This shows that DL systems can facilitate access to joint screening of eye diseases and become a quick and reliable support for ophthalmological experts.
Introduction
The study evaluates RetCAD v.1.3.0 for joint detection of referable diabetic retinopathy (DR) and age-related macular degeneration (AMD), alongside individual-disease validation and human comparisons.
Introduction
- The study targets scalable joint screening because manual, disease-specific protocols burden health systems and patients, while DR and AMD can co-exist and early detection can avert blindness.The introduction identifies DR as a leading cause of preventable blindness and AMD as the most common cause of blindness in developed countries.Source: Introduction · pp. 3–4
Evaluation data
- Joint validation used 600 DR-AMD images with referable and non-referable cases; individual-disease validation used 1,200 Messidor images for DR and 133,821 AREDS images for AMD.An experienced ophthalmologist established the joint-dataset reference standard, and four independent observers enabled human-performance comparisons.Source: Evaluation data · pp. 4–6
Automated grading approach
- RetCAD v.1.3.0 uses separate ensembles of convolutional neural networks for DR and AMD, producing averaged 0–100 likelihood-related scores after image preprocessing and quality assessment.None of the validation images were used to train the system.Source: Automated grading approach · pp. 7–8
Evaluation design
- Evaluation compared referable with non-referable cases using sensitivity, specificity, ROC curves, and AUC; bootstrap resampling generated 95% confidence intervals.On the joint dataset, disease cases were evaluated both against controls plus cases of the other disease and against controls alone.Source: Evaluation design · pp. 8–9
- AUC was 95.1% (SE=90.1%, SP=90.6%) for referable DR detection and 94.9% (SE=91.8%, SP=87.5%) for referable AMD detection on the joint dataset.Average observer SE and SP were 61.5% and 97.8% for DR, and 76.5% and 96.1% for AMD.Source: Evaluation design · p. 9
- AUC was 97.5% (SE=92.0%, SP=92.1%) for referable DR detection in Messidor and 92.7% (SE=85.8%, SP=86.0%) for referable AMD detection in AREDS.The Messidor specificity confidence interval appears in the following passage.Source: Evaluation design · p. 10
Discussion
RetCAD v.1.3.0 distinguished referable DR from AMD and showed higher sensitivity but lower specificity than human observers in joint detection. The authors note limitations in staging and validation diversity and propose broader validation and detection of additional eye diseases.
Discussion
- False positives reflected cross-disease confusion: 17.4% of false-positive DR detections were referable AMD, while 24.2% of false-positive AMD detections were referable DR.The system could differentiate between the two diseases, and its performance was not significantly altered when individual disease detection was assessed on the same dataset.Source: Discussion · p. 10
- RetCAD v.1.3.0 showed higher sensitivity but lower specificity than human observers for joint DR and AMD detection, supporting its potential use in screening or grading support.The authors emphasize that screening systems should miss fewer referable cases.Source: Discussion · pp. 10–11, Limitations and future work · p. 13
- Observers disagreed more on AMD, while DR stage 2 cases were often graded as stage 1, complicating referable/non-referable classification despite relatively high interobserver scores.The authors suggest AMD may need screening protocols like those already used for DR.Source: Discussion · p. 11
- Ting et al. [45] reported lower AUCs for referable DR and AMD but used larger datasets and also evaluated glaucoma; fewer observers and separate disease validation sets left joint-screening effects unclear.The authors note that their observers were ophthalmologists or trainees accustomed to clinical prevalence and grading tasks, which differ from real screening settings.Source: Discussion · p. 11, Limitations and future work · p. 12
- In Messidor, 97.6% of false negatives were DR stage 2 and 92.7% of false positives were stage 1, although detection errors remained low.The authors report that Messidor's original set was used to enable comparison because no publicly available image-based reference standard existed for Messidor-2; they also validated on Messidor-2 using a patient-based reference standard [34,40,61].Source: Discussion · p. 11
- In AREDS, 86.0% of false positives were AMD stage 2 and 67.1% of false negatives were stage 3; the authors note the images were digitized analog photographs.Burlina et al. [42] used the whole AREDS dataset for training, which might explain their better performance.Source: Discussion · p. 12
Limitations and future work
- The output score has no clear disease-staging cutoff, and the joint-validation set contained 600 images from 288 patients, with possible repeated visits and multiple images per visit.The authors call for larger, more diverse datasets with graded severity levels across DR, AMD, and additional eye diseases.Source: Limitations and future work · pp. 12–13
- The authors conclude that joint DR and AMD detection could support screening and reduce subjectivity; adding automated detection of co-existing diseases such as glaucoma and cataracts could increase usability.They note that color fundus imaging facilitates screening, including in developing countries, while optical coherence tomography could provide additional diagnostic information.Source: Limitations and future work · pp. 12–13
Supplementary Online Content
Supplementary analyses assessed individual referable DR and AMD detection in the enriched DR-AMD set and referable DR detection in Messidor-2. RetCAD achieved AUCs of 95.6% for DR, 95.2% for AMD, and 98.0% for Messidor-2 DR detection.
DR-AMD dataset extraction
- The DR-AMD validation set comprised 600 randomly selected images: 150 referable AMD, 150 referable DR, and 300 controls.The source set contained 8,871 images; after quality exclusions, 7,086 were preliminarily graded before sampling.Source: Supplementary Online Content · p. 24
Individual detection in the DR-AMD dataset
- 95.6% AUC for referable DR and 95.2% AUC for referable AMD were reported in the DR-AMD dataset.At the optimal operating point, DR sensitivity and specificity were 91.7% and 90.9%; AMD sensitivity and specificity were 88.6% and 92.1%.Source: Figure S2 · p. 25, Supplementary Online Content · p. 25
Validation of individual detection of referable DR in Messidor-2
- Messidor-2 combined 690 extension images with 1,058 original Messidor images, totaling 1,748 images from 874 examinations; specialist consensus gradings classified 684 subjects as referable and 190 as non-referable.Referability was assigned at participant level based on the higher-stage eye, with both eyes at level 0 or 1 defining non-referability.Source: Supplementary Online Content · p. 30
- 98.0% AUC, 92.6% sensitivity, and 93.4% specificity were achieved for participant-based referable DR detection in Messidor-2.The participant score was the maximum of the two eye-image scores.Source: Supplementary Online Content · p. 30