Source-linked AI summary
REFUGE Challenge: A Unified Framework for Evaluating Automated Methods for Glaucoma Assessment from Fundus Photographs
José Ignacio Orlando, Huazhu Fu, João Barbossa Breda, Karel van Keer, Deepti R. Bathula, Andrés Diaz-Pinto, Ruogu Fang, Pheng-Ann Heng, Jeyoung Kim, JoonHo Lee, Joonseok Lee, Xiaoxiao Li, Peng Liu, Shuai Lu, Balamurali Murugesan, Valery Naranjo, Sai Samarth R. Phaye, Sharath M. Shankaranarayana, Apoorva Sikka, Jaemin Son, Anton van den Hengel, Shujun Wang, Junyan Wu, Zifeng Wu, Guanghui Xu, Yongli Xu, Pengshuai Yin, Fei Li, Xiulan Zhang, Yanwu Xu, Xiulan Zhang, Hrvoje Bogunović
TL;DR
Glaucoma is frequently undiagnosed because its progression is asymptomatic. REFUGE analyzes an open challenge and its unified evaluation framework, finding that top-ranked automated approaches outperformed two glaucoma specialists in classification.
Problem
Glaucoma can progress asymptomatically and remain undiagnosed, ultimately resulting in blindness.
Method
REFUGE analyzes outcomes and methodological contributions from an open glaucoma challenge using a unified evaluation framework.
Results
The two top-ranked teams achieved better glaucoma-classification results than two glaucoma specialists.
Takeaways & Limitations
REFUGE’s unified evaluation framework helped identify common practices from participating methods and results.
Takeaways & Limitations
REFUGE’s data set lacks diverse ethnicities, and its 1200 images remain a stated limitation.
Abstract
from arXiv · showhide
Glaucoma is one of the leading causes of irreversible but preventable blindness in working age populations. Color fundus photography (CFP) is the most cost-effective imaging modality to screen for retinal disorders. However, its application to glaucoma has been limited to the computation of a few related biomarkers such as the vertical cup-to-disc ratio. Deep learning approaches, although widely applied for medical image analysis, have not been extensively used for glaucoma assessment due to the limited size of the available data sets. Furthermore, the lack of a standardize benchmark strategy makes difficult to compare existing methods in a uniform way. In order to overcome these issues we set up the Retinal Fundus Glaucoma Challenge, REFUGE (\url{https://refuge.grand-challenge.org}), held in conjunction with MICCAI 2018. The challenge consisted of two primary tasks, namely optic disc/cup segmentation and glaucoma classification. As part of REFUGE, we have publicly released a data set of 1200 fundus images with ground truth segmentations and clinical glaucoma labels, currently the largest existing one. We have also built an evaluation framework to ease and ensure fairness in the comparison of different models, encouraging the development of novel techniques in the field. 12 teams qualified and participated in the online challenge. This paper summarizes their methods and analyzes their corresponding results. In particular, we observed that two of the top-ranked teams outperformed two human experts in the glaucoma classification task. Furthermore, the segmentation results were in general consistent with the ground truth annotations, with complementary outcomes that can be further exploited by ensembling the results.
1. Introduction
Glaucoma is a largely undiagnosed cause of irreversible vision loss, while CFP-based automated assessment remains limited by subtle signs, costly complementary testing, scarce labeled data, and inconsistent evaluation. REFUGE addresses these gaps through a public dataset and unified challenge framework for glaucoma classification and optic disc/cup segmentation.
- Clinical motivation: Glaucoma progresses asymptomatically and can cause irreversible vision loss before patients recognize the disease.Early detection is therefore essential to prevent vision loss.
- Clinical motivation: At least half of patients with glaucoma remain undiagnosed, creating a major detection challenge.Generalized screening is hindered by false positives that current healthcare infrastructures cannot readily absorb.
- Imaging and assessment gap: CFP is economical and non-invasive, but early glaucoma signs are difficult to recognize and clinical assessment commonly requires complementary IOP, perimetry, and OCT studies.This combination is not cost-effective for large-scale population screening.
- Benchmarking gap: Existing automated glaucoma methods could not be properly compared because no unified evaluation framework or large public labeled dataset was available.Limited labeled datasets also hampered the deployment of deep learning for glaucoma detection.
- REFUGE contribution: REFUGE released 1200 CFPs with reference annotations for glaucoma identification, optic disc/cup segmentation, and fovea localization.The database was approximately two times larger than the largest previously available dataset.
- REFUGE contribution: REFUGE established a standardized, fair protocol for comparing algorithms on glaucoma classification and optic disc/cup segmentation.The challenge analyzed participating methods, compared results with two glaucoma experts, and kept the data and evaluation open for further development.
2. Automated glaucoma assessment: state-of-the-art and current evaluation protocols
Automated glaucoma assessment has progressed from handcrafted features to deep learning for classification and optic disc/cup segmentation, but methods remain constrained by limited data and inconsistent evaluation protocols.
- Glaucoma classification: Handcrafted features limited classification accuracy because they could not comprehensively characterize disease-appearance variability.
- Glaucoma classification: Transfer learning outperformed networks trained from scratch for glaucoma detection, although fine-tuning relied on more than 14,000 images.
- Glaucoma classification: Restricting analysis to the optic nerve head improved performance over full-size images but hampered learning of alternative features from other regions.
- Optic disc/cup segmentation: Optic disc/cup segmentation commonly uses modified U-Net architectures, while shallow networks prevent overfitting but limit learning of rare, specific features.
- Evaluation protocols: Evaluation protocols differed substantially in datasets, metrics, and train/test partitions, making direct comparison of existing models difficult.
- Evaluation protocols: REFUGE provides a comparison dataset with clinical labels and segmentation information within the surveyed database landscape.
3. The REFUGE challenge
REFUGE introduced a publicly released, clinically labeled fundus-photo dataset and a standardized challenge framework for optic disc/cup segmentation and glaucoma classification. Its partitions and annotations were designed to support reliable evaluation and generalization across acquisition settings.
- Dataset and annotations: REFUGE provides 1200 retinal color fundus photographs with clinically derived glaucomatous or non-glaucomatous labels and optic disc, optic cup, and fovea annotations.Labels were based on follow-up clinical records and multiple diagnostic examinations rather than a single photograph.
- Dataset and annotations: Glaucomatous cases required optic nerve head damage and reproducible visual-field defects under predefined criteria, including vCDR > 0.7 or retinal nerve fiber layer thinning.The visual-field criterion required significant sensitivity reduction at specified contiguous locations and p-value thresholds.
- Dataset composition: The non-glaucomatous group included healthy subjects and patients with diabetic retinopathy, myopia, and megalopapilae, conditions that can resemble glaucoma anatomically.Myopic and megalopapillae cases were included because their atypical optic nerve head appearance can lead to misclassification.
- Dataset and annotations: Seven glaucoma specialists independently delineated optic disc and cup boundaries, with majority voting and senior quality control producing reference segmentations.Specialists annotated without access to patient information or disease prevalence; only a few cases required correction.
- Challenge setup: The dataset was split into training, offline-test, and online-test subsets, each stratified to contain 10% glaucomatous and 90% non-glaucomatous cases.Training images came from the Zeiss camera, whereas test sets included lower-resolution Canon images to encourage cross-device generalization.
- Challenge setup: REFUGE evaluated optic disc/cup segmentation, glaucoma classification, and optionally fovea detection using fixed partitions and uniform task-specific criteria.Participants could use other public datasets, and the offline set supported up to five evaluations per participant.
4. Results
The REFUGE test-set results showed strong and complementary performance across classification and segmentation methods. Top classification methods exceeded the ground-truth vCDR baseline in some comparisons, while different teams led optic-disc and optic-cup segmentation.
- Glaucoma classification: VRT achieved significantly better glaucoma-classification performance than the ground-truth vCDR baseline (p = 0.006).SDSAIRC and CUHKMED also had higher AUC values than ground-truth vCDR, but without statistically significant differences.
- Glaucoma classification: A three-team likelihood ensemble improved AUC only marginally and did not differ significantly from the best individual team (p = 0.576).The ensemble averaged normalized likelihoods from the three top-ranked methods.
- Segmentation: CUHKMED obtained the highest average optic-disc DSC, whereas Masker obtained the highest average optic-cup DSC.VRT and BUCT ranked second and third for optic-disc DSC, while Masker’s cup result was highest among the reported approaches.
- Segmentation: Masker significantly outperformed every other method in vCDR mean absolute error (p < 0.014), while CUHKMED ranked second and BUCT third.For DSC, Masker differed significantly from every team except CUHKMED (p = 0.387).
- Ensembling: A complementary-method majority vote improved difficult optic nerve head segmentations, with significant gains only when the methods were complementary.The individual-method behavior was otherwise stable across most cases.
5. Discussion
REFUGE's unified evaluation framework revealed complementary strengths across glaucoma classification and optic disc/cup segmentation methods, while exposing important limits in data diversity, scale, annotation fidelity, and evaluation design.
- Challenge findings: Attention mechanisms and domain-specific heuristics helped top classification methods focus on the optic nerve head without requiring manual annotations of glaucoma-related abnormalities.
- Challenge findings: Ensembling improved segmentation performance, particularly when top-ranked models disagreed, indicating that their predictions remained complementary.
- Challenge findings: REFUGE's unified evaluation framework enabled technical comparisons across automated optic disc/cup segmentation and glaucoma classification methods.
- Challenge strengths: 1200 publicly available fundus images with clinical labels and expert reference annotations made REFUGE the largest public glaucoma image source at the time.
- Limitations: Evaluation design and annotations also constrain interpretation: offline-test submissions were limited, possible overfitting remained, and CFP-based masks may diverge from true anatomy.
- Limitations: REFUGE results have limited generalizability because the dataset lacks diverse ethnicities, contains high-quality images, and remains small for broadly general deep-learning solutions.
6. Conclusions
REFUGE summarized twelve teams’ automated approaches for glaucoma classification and optic disc/cup segmentation, identifying strong classification results, domain-shift-aware segmentation practices, and benefits from ensembling.
- Twelve teams’ approaches for glaucoma classification and optic disc/cup segmentation were analyzed from the on-site MICCAI 2018 competition.
- The strongest glaucoma classifiers combined deep learning with glaucoma-specific biomarkers, including vertical cup-to-disc ratio and retinal nerve fiber layer defects.
- Two top-ranked teams achieved better glaucoma-classification results than two glaucoma specialists, supporting automated identification of glaucoma suspects from fundus images.
- The best segmentation solutions addressed training–test domain shift and image variability, while ambiguous optic-disc or cup edges remained challenging.
- For both challenge tasks, integrating multiple models improved individual performance and helped identify common practices for future baselines.
- REFUGE data and its evaluation framework were made publicly accessible through challenge websites to encourage future glaucoma-assessment development.
AIML
AIML used ensembles of ResNet-based fully convolutional networks for optic disc/cup segmentation and combined classification models, averaging outputs for final predictions.
- AIML used a two-stage optic-disc/cup segmentation approach combining multiple dilated fully convolutional ResNet networks.
- A ResNet-50 FCN first coarsely segmented the optic nerve head, after which the region was cropped to approximately one quarter of the original resolution.
- Final segmentation predictions averaged multi-view outputs from all networks across augmented image versions.
- The final classification result was obtained by averaging the outputs of the combined architectures.
BUCT
BUCT separately segmented optic disc and cup regions with U-Nets, then used cropped grayscale fundus images and an Xception classifier for glaucoma prediction.
- BUCT trained separate U-Nets for optic-disc and optic-cup segmentation using cropped image regions and augmented REFUGE training data.
- For optic-disc segmentation, a 817 × 817-pixel square was cropped and resized to 256 × 256 pixels before applying a reduced-filter U-Net.
- False positives were removed by retaining the largest connected component, and an ellipse was fitted to the optic-disc segmentation.
- For cup segmentation, the optic-disc bounding rectangle was extended by 100 pixels per side and resized to 128 × 128 pixels before U-Net prediction.
- For glaucoma classification, cropped color fundus photographs were converted to grayscale, augmented, and used to train an Xception network from scratch.
CUHKMED
CUHKMED addressed segmentation domain shift with output-space adversarial learning and morphology-aware loss, then derived glaucoma probabilities from normalized vertical cup-to-disc ratios.
- CUHKMED used a patch-based Output Space Adversarial Learning framework to reduce segmentation degradation under unsupervised target-domain shift.
- A U-Net extracted optic-disc regions of interest before DeepLabv3+ with a MobileNetV2 backbone performed segmentation.
- A morphology-aware segmentation loss encouraged smooth predictions based on optic-disc and cup shape.
- Adversarial learning encouraged target-domain segmentation predictions to resemble source-domain predictions.
- The final segmentation prediction was formed by ensembling five models.
- For glaucoma classification, fitted optic-disc and cup ellipses produced vertical cup-to-disc ratios that were normalized to probabilities between 0 and 1.
Cvblab
The method uses separate segmentation and classification pipelines, with preprocessing, augmentation, and ensemble averaging across models or outputs.
- OD/OC segmentation: A two-stage modified U-Net process segments the optic disc first, then uses its mask to crop and segment the optic cup.
- OD/OC segmentation: CLAHE preprocessing and 256 × 256 resizing are applied before network inference.
- OD/OC segmentation: The segmentation models are trained using DRIONS-DB, DRISHTI-GS, RIM-ONE v3, and REFUGE data.
- Glaucoma classification: An ensemble of VGG19, GoogLeNet, ResNet-50, and Xception networks is fine-tuned to identify glaucomatous images.
- Training: Data augmentation and SMOTE balancing address training variation and bias toward the more common Normal class.
- Glaucoma classification: The final predictions are combined by averaging the models’ glaucoma likelihoods.
Mammoth
The method combines Mask-RCNN and Dense U-Net segmentation with ensemble-based glaucoma classification using optic-nerve-head crops and semi-supervised learning.
- OD/OC segmentation: Mask-RCNN and Dense U-Net are ensembled for optic-disc and optic-cup segmentation.
- OD/OC segmentation: Dense U-Net uses the optic-disc probability mask as an additional attention channel for optic-cup segmentation.
- OD/OC segmentation: Both segmentation networks are trained with a linear combination of cross-entropy and Dice losses, and their probability outputs are averaged.
- Glaucoma classification: The classification pipeline crops a 1.5-times-disc-radius optic-nerve-head patch, resizes it to 448 × 448, and applies CLAHE and mean color normalization.
- Glaucoma classification: ResNet-18 and semi-supervised CatGAN networks classify glaucoma, with CatGAN-generated fake images enlarging the training set.
- Glaucoma classification: Eight ResNet-18 models arise from 4-fold cross-validation with weighted and unweighted cross-entropy losses, and predictions are averaged at inference.
Masker
The section presents multiple teams’ approaches that combine optic-nerve-head localization, specialized segmentation architectures, preprocessing, and ensemble or hybrid glaucoma-risk prediction.
- Masker: Mask-RCNN, U-Net, and M-Net models trained on 14 bagged partitions produce final segmentation by accepting regions predicted by 80% of networks.
- Masker: One glaucoma-risk formulation combines vertical cup-to-disc ratio with a binary ensemble vote, using 14 classification networks.
- Masker: CoarseNet localizes the optic-nerve head and FineNet segments the optic disc and cup using dense blocks and modified pooling.
- Masker: F-Net optic-disc and optic-cup features are reduced to 64 dimensions each, concatenated, and classified by a four-layer neural network.
- Masker: A 10-fold cross-validation ensemble produces the final glaucoma likelihood by taking the maximum likelihood across seven selected models.
- Masker: DeepLabv3+ segmentation uses cropped optic-nerve-head regions, cross-entropy training, pixel quantization, and an ASPP-based multiscale assumption.
- Masker: Another pipeline averages a network score with logistic-regression output using the estimated vertical cup-to-disc ratio, while histogram matching compensates for camera-vendor variance.
- Masker: A two-stage optic-disc/optic-cup separation and ellipse-fitting post-processing are used to improve segmentation masks.