Source-linked AI summary
DeepSeeNet: A deep learning model for automated classification of patient-based age-related macular degeneration severity from color fundus photographs
Yifan Peng, Shazia Dharssi, Qingyu Chen, Tiarnan D. Keenan, Elvira Agrón, Wai T. Wong, Emily Y. Chew, Zhiyong Lu
TL;DR
Manual AREDS severity grading is time-consuming, and prior automated systems did not assign patient-based scores using both eyes. DeepSeeNet combines bilateral eye-level AMD risk-factor detection into an AREDS score and achieved higher overall accuracy than retinal specialists, while the study notes limitations from case imbalance and reliance on color fundus photographs.
Problem
The study addresses the lack of automated patient-based AREDS severity scoring from bilateral color fundus photographs despite the time required for expert manual grading.
Method
DeepSeeNet detects AMD risk factors in each eye and combines the bilateral findings to assign an AREDS Simplified Severity Scale score.
Results
67% accuracy versus 60% was achieved by Fine-tuned DeepSeeNet compared with retinal specialists on patient-based classification.
Takeaways & Limitations
The study concludes that DeepSeeNet can support automated assignment of patient-based AMD severity categories and may enhance clinical decision-making.
Takeaways & Limitations
The current model is limited by imbalance among cases in the dataset and the sole use of color fundus photographs.
Abstract
from arXiv · showhide
In assessing the severity of age-related macular degeneration (AMD), the Age-Related Eye Disease Study (AREDS) Simplified Severity Scale predicts the risk of progression to late AMD. However, its manual use requires the time-consuming participation of expert practitioners. Although several automated deep learning systems have been developed for classifying color fundus photographs (CFP) of individual eyes by AREDS severity score, none to date has used a patient-based scoring system that uses images from both eyes to assign a severity score. DeepSeeNet, a deep learning model, was developed to classify patients automatically by the AREDS Simplified Severity Scale (score 0-5) using bilateral CFP. DeepSeeNet was trained on 58,402 and tested on 900 images from the longitudinal follow-up of 4549 participants from AREDS. Gold standard labels were obtained using reading center grades. DeepSeeNet simulates the human grading process by first detecting individual AMD risk factors (drusen size, pigmentary abnormalities) for each eye and then calculating a patient-based AMD severity score using the AREDS Simplified Severity Scale. DeepSeeNet performed better on patient-based classification (accuracy = 0.671; kappa = 0.558) than retinal specialists (accuracy = 0.599; kappa = 0.467) with high AUC in the detection of large drusen (0.94), pigmentary abnormalities (0.93), and late AMD (0.97). DeepSeeNet demonstrated high accuracy with increased transparency in the automated assignment of individual patients to AMD risk categories based on the AREDS Simplified Severity Scale. These results highlight the potential of deep learning to assist and enhance clinical decision-making in patients with AMD, such as early AMD detection and risk prediction for developing late AMD. DeepSeeNet is publicly available on https://github.com/ncbi-nlp/DeepSeeNet.
Methods and Materials
The study used longitudinal bilateral color fundus photographs from AREDS, Reading Center grades as reference labels, and a model that combines eye-level findings into patient-based severity scores.
- Reference grading: Reading Center grades served as gold-standard labels for model training and performance assessment.Photographs were graded independently by senior and junior graders, with discrepancies adjudicated by a senior investigator.
- Severity scoring: The AREDS Simplified Severity Scale assigns scores 0–4 from bilateral drusen and pigment status, and score 5 when late AMD occurs in either eye.Late AMD was defined as neovascular AMD or central geographic atrophy.
- Datasets: 58,402 training images and 900 testing images were drawn from 4,549 AREDS participants.The testing set used bilateral baseline images from 450 participants, while the training set included images from 4,099 participants across multiple visits.
- Model design: DeepSeeNet detects large drusen, pigmentary abnormalities, and late AMD for each eye before computing a bilateral patient-based score from 0 to 5.Its constituent subnetworks are Drusen-Net, Pigment-Net, and Late AMD-Net.
Results
Fine-tuned DeepSeeNet outperformed retinal specialists on overall patient-based AREDS severity classification, while performance varied across severity categories and risk-factor tasks.
- Overall classification: 0.671 accuracy and 0.558 kappa were achieved by fine-tuned DeepSeeNet on patient-based severity classification.Fine-tuning was the best of three evaluated training strategies.
- Overall classification: 67% accuracy versus 60% was achieved by Fine-tuned DeepSeeNet compared with human retinal specialists.Agreement with the ground truth was moderate by kappa.
- Risk-factor prediction: D-Net and P-Net outperformed retinal specialists for assessing large drusen and pigmentary abnormalities, respectively.LA-Net had similar accuracy to specialists for late AMD, although its kappa was lower.
- Class-specific performance: DeepSeeNet correctly classified AREDS scores 0–4 more often, whereas retinal specialists correctly classified late AMD more often.The comparison was based on confusion matrices against Reading Center grades.
- Internal representations: t-SNE separated pigmentary-abnormality and late-AMD presence from absence more clearly than small/none, medium, and large drusen classes.Many misclustered drusen points were associated with medium drusen, which were difficult to identify.
- Transparency: Saliency maps highlighted image regions corresponding to drusen and pigmentary changes, supporting interpretation of model predictions.The maps identify pixels contributing most to classification but do not fully explain the final decision.
Discussion
DeepSeeNet achieved higher overall patient-based AMD severity classification accuracy than retinal specialists, while showing weaknesses for late AMD and image-quality-sensitive cases. The model’s bilateral, clinically interpretable design and public release support further validation and refinement.
- Performance: 67% accuracy versus 60% for retinal specialists, with moderate agreement with ground truth.Fine-tuned DeepSeeNet achieved accuracy = 0.671 and kappa = 0.558; retinal specialists achieved accuracy = 0.599 and kappa = 0.467.
- Performance: Severity Scale scores 0 to 4 were classified correctly more often by Fine-tuned DeepSeeNet, whereas retinal specialists performed better for late AMD.The difference may reflect the relatively low number of late AMD images available for model training.
- Error analysis: Most incorrect predictions for actual Severity Scale scores 0 to 4 differed from the true score by only one scale step.Among incorrect late-AMD classifications, 50% had noncentral GA in at least one eye, which was not defined as late AMD in this study.
- Error analysis: Image artifacts and brightness affected errors, including false late-AMD and geographic-atrophy classifications.Among participants incorrectly classified as having late AMD, 25.0% had digital artifacts obscuring the fovea; pale retinas or artifacts increased misclassification as GA.
- Limitations: The dataset was limited by case imbalance, particularly the relatively low proportion of late AMD photographs.The authors suggest that further training with more late AMD images may address this limitation.
- Transparency and future work: Further validation across diverse datasets and countries is needed before clinical integration can be considered.Future work includes testing other color-fundus datasets, adding demographic, medical, genetic, and multimodal imaging information, and evaluating AREDS2 data.
- Transparency and future work: DeepSeeNet used sub-networks with explicit purposes and saliency maps to improve transparency, while combining risk factors from both eyes into a patient score.The model, data partition, and code were made publicly available to support reproducibility and further methodological development.
Glossary
The glossary defines core deep-learning terms used in the study, including training data, model parameters, optimization, and image-classification architectures.
- Model architectures: A convolutional neural network is a deep-learning architecture used largely for image classification, while Inception-v3 is a CNN with an inception architecture.The glossary also describes fully connected layers as connecting every output neuron to all activations in the preceding layer.
- Evaluation and visualization: A test set is a mutually exclusive group of images not used for training and reserved for evaluating model performance.An epoch is one complete pass through the training set, and over-fitting occurs when a model fits training data too closely to generalize reliably.
- Training and optimization: A training set contains images used to train a model, whose predictions are compared with ground-truth labels to optimize its parameters.Back-propagation is used to calculate gradients for updating network weights.
- Training and optimization: Transfer learning initializes a new model with weights learned from a large dataset, enabling more efficient training on another dataset.ImageNet is commonly used for pre-training because it contains more than 14 million labeled natural images.
- Model architectures: Deep learning learns image features directly from training data rather than requiring human experts to prespecify them.The glossary contrasts this with traditional machine-learning approaches.
- Evaluation and visualization: t-SNE maps high-dimensional retinal-image features into two dimensions so similar datapoints appear near one another and dissimilar datapoints farther apart.The method is used to visualize and explore complex datasets.
Training strategies
The study compared three DeepSeeNet training strategies and found that fine-tuning an ImageNet-pretrained Inception-v3 model on AREDS images performed best.
- Training strategies: Three strategies were evaluated: a fixed-feature-extractor MLP, a fully trained model initialized randomly, and a fine-tuned model initialized from ImageNet.All three models were assessed on the testing dataset.
- MLP: MLP training updated only the final two layers while using the pre-trained Inception-v3 network as a fixed feature extractor.The model consisted of two densely connected layers with 256 and 128 units.
- Fine-tuned DeepSeeNet: Fine-tuned DeepSeeNet used an ImageNet-pretrained Inception-v3 model and retrained all layers with AREDS images.Unlike the MLP, all layers were permitted to change during AREDS training.
- Fully-trained DeepSeeNet: Fully-trained DeepSeeNet used randomly initialized Inception-v3 weights and trained all layers directly on AREDS images without ImageNet pre-training.This strategy was contrasted with the fine-tuned model's pre-initialized weights.
- Results: Accuracy=0.671 and kappa=0.558 were achieved by Fine-tuned DeepSeeNet, the best-performing of the three models.The study therefore focused subsequent discussion on Fine-tuned DeepSeeNet.
- Results: Fine-tuned DeepSeeNet performed more accurately than both Fully-trained DeepSeeNet and the MLP model.The authors suggest that pre-trained weights were beneficial, although training from scratch was possible with the AREDS dataset.