PaperlayerView exact revision on arXiv

Source-linked AI summary

GAMMA Challenge:Glaucoma grAding from Multi-Modality imAges

Junde Wu, Huihui Fang, Fei Li, Huazhu Fu, Fengbin Lin, Jiongcheng Li, Lexing Huang, Qinji Yu, Sifan Song, Xinxing Xu, Yanyu Xu, Wensai Wang, Lingxiao Wang, Shuai Lu, Huiqi Li, Shihua Huang, Zhichao Lu, Chubin Ou, Xifei Wei, Bingyuan Liu, Riadh Kobbi, Xiaoying Tang, Li Lin, Qiang Zhou, Qiang Hu, Hrvoje Bogunovic, José Ignacio Orlando, Xiulan Zhang, Yanwu Xu

arXiv:2202.06511v4cs.CV

TL;DR

Glaucoma grading algorithms rarely combine fundus photography and OCT, despite clinical use of both modalities. GAMMA evaluates paired-image grading methods and reports an advantage for combined data over either modality alone. The challenge’s results suggest combined automated screening may be appropriate for community screening, although dataset size and population diversity constrain generalization.

  • Problem

    Few computer-aided glaucoma methods use both fundus and OCT, despite clinicians often using both screenings for more reliable diagnosis.

    Source: §1 · pp. 3–4

  • Method

    GAMMA evaluates multi-modal glaucoma grading and analyzes challenge methods, including dual-modality architectures and ordinal ensemble strategies.

    Source: §4.2.1 · p. 21, §4.2.2 · p. 22, §5.2 · p. 24

  • Results

    About 10% improvement against single-modality methods is reported for simple fundus-OCT fusion, with another about 10% improvement from advanced model design.

    Source: §5.3 · p. 28

  • Takeaways & Limitations

    Combined automated fundus-OCT detection seems appropriate for large-scale community screening, with top-three teams achieving sensitivities above 0.9.

    Source: §5.3 · p. 28

    The 300-sample dataset is not large enough for capable deep-learning models, and its Chinese population limits diversity, so algorithms may need retraining for other populations.

    Source: §5.2 · pp. 26–27

Abstract

from arXiv · show

Color fundus photography and Optical Coherence Tomography (OCT) are the two most cost-effective tools for glaucoma screening. Both two modalities of images have prominent biomarkers to indicate glaucoma suspected. Clinically, it is often recommended to take both of the screenings for a more accurate and reliable diagnosis. However, although numerous algorithms are proposed based on fundus images or OCT volumes in computer-aided diagnosis, there are still few methods leveraging both of the modalities for the glaucoma assessment. Inspired by the success of Retinal Fundus Glaucoma Challenge (REFUGE) we held previously, we set up the Glaucoma grAding from Multi-Modality imAges (GAMMA) Challenge to encourage the development of fundus \& OCT-based glaucoma grading. The primary task of the challenge is to grade glaucoma from both the 2D fundus images and 3D OCT scanning volumes. As part of GAMMA, we have publicly released a glaucoma annotated dataset with both 2D fundus color photography and 3D OCT volumes, which is the first multi-modality dataset for glaucoma grading. In addition, an evaluation framework is also established to evaluate the performance of the submitted methods. During the challenge, 1272 results were submitted, and finally, top-10 teams were selected to the final stage. We analysis their results and summarize their methods in the paper. Since all these teams submitted their source code in the challenge, a detailed ablation study is also conducted to verify the effectiveness of the particular modules proposed. We find many of the proposed techniques are practical for the clinical diagnosis of glaucoma. As the first in-depth study of fundus \& OCT multi-modality glaucoma grading, we believe the GAMMA Challenge will be an essential starting point for future research.

1. Introduction

Glaucoma can silently progress to irreversible vision loss, making early detection important. GAMMA addresses the limited availability of methods and public data for grading glaucoma from paired fundus and OCT images by releasing a dataset and evaluating submitted methods.

1. Introduction

  • Glaucoma may progress without warning signs until advanced stages, and early diagnosis can slow or prevent vision loss.The paper describes glaucoma as a leading cause of irreversible blindness and notes the importance of timely detection and treatment.Source: §1 · p. 3
  • Fundus photography shows the optic disc, cup, and vessels, while OCT measures retinal nerve fiber layer thickness; clinicians often use both because neither alone can exclude early glaucoma.A cited report estimates that 46.3% of glaucoma cases would be missed using fundus images or OCT alone.Source: §1 · p. 4
  • Most computer-aided glaucoma algorithms use one modality, while few combine fundus and OCT because public paired datasets are unavailable and the modalities differ in characteristics and dimensionality.Source: §1 · p. 4
  • GAMMA asks teams to classify fundus-image and OCT-volume pairs as normal, early-glaucoma, or progressive-glaucoma.Progressive-glaucoma combines intermediate and advanced stages.Source: §1 · pp. 3, 5
  • GAMMA released the first publicly available paired fundus-photography and OCT-volume dataset for deep-learning glaucoma grading and evaluated submitted methods.The challenge also provides optic disc and cup masks and fovea locations for auxiliary segmentation and localization tasks.Source: §1 · p. 5
  • The paper analyzes top teams’ methods and source code and conducts ablations to identify effective techniques for glaucoma screening.The authors expect the challenge and its analyses to inform future algorithm design.Source: §1 · pp. 5–6

2. The GAMMA challenge

GAMMA ran a preliminary stage followed by a final stage, using a shared training set and restricting teams to that dataset for development. Seventy teams submitted 1,272 valid preliminary results, and ten teams advanced to the final stage.

2. The GAMMA challenge

  • Teams could train on the released set and submit up to five results daily during the 30-day preliminary stage, but could not use private datasets.Final-stage teams were ranked on a test set and could no longer modify their models.Source: §2 · p. 6
  • 70 teams submitted 1,272 valid results during the preliminary stage, and ten were selected for the final stage based on performance and willingness to participate in the workshop.Source: §2 · p. 6

2.1. GAMMA Dataset

The GAMMA dataset contains 300 paired fundus and OCT samples from 276 Chinese patients, with glaucoma grades and auxiliary annotations. The paper describes how grading labels and fovea and optic disc/cup references were assigned.

2.1. GAMMA Dataset

  • The dataset contains 300 fundus-OCT pairs from 276 Chinese patients, divided into 100-pair training, preliminary, and final sets.The samples were collected at Sun Yat-sen Ophthalmic Center; intermediate and advanced glaucoma were grouped as progressive-glaucoma for the main task.Source: §2.1 · pp. 7–8

2.1.1. Glaucoma Grading

  • Glaucoma grades were determined from same-day visual-field mean deviation values: early above -6 dB, intermediate from -6 to -12 dB, and advanced worse than -12 dB.The visual-field reports were described as reliable using fixation-loss and false-positive and false-negative thresholds.Source: §2.1.1 · p. 8

2.1.2. Fovea Localization

  • Four ophthalmologists independently marked each fundus-image fovea, and a senior ophthalmologist selected markers to average into the reference coordinate.Source: §2.1.2 · pp. 8–9

2.1.3. Optic Disc & Cup Segmentation

  • Four ophthalmologists manually annotated optic cup and disc regions, after which a senior ophthalmologist selected intersections of annotations as the reference mask.The dataset also provides glaucoma grades, fovea coordinates, and cup and disc masks.Source: §2.1 · p. 8, §2.1.3 · p. 9

2.2. Challenge Evaluation

The challenge evaluates glaucoma grading with quadratically weighted Cohen’s kappa, fovea localization with reciprocal normalized coordinate error, and optic disc/cup segmentation using Dice and vCDR error.

2.2.1. Glaucoma Grading

  • Glaucoma grading uses quadratically weighted Cohen’s kappa for the ordered normal, early-glaucoma, and progressive-glaucoma classes.The score uses observed accuracy and the chance probability of predicting the correct categories.Source: §2.2.1 · p. 9

2.2.2. Fovea Localization

  • Fovea localization is scored using the reciprocal of average Euclidean distance between estimated and reference coordinates normalized by image size.A missing fovea is represented by coordinate (0, 0), and the score denominator adds 0.1 to keep the score within 10 points.Source: §2.2.2 · pp. 9–10

2.2.3. Optic Disc & Cup Segmentation

  • Optic disc and cup segmentation is evaluated with Dice coefficients and mean absolute error in vertical cup-to-disc ratio.The vCDR is the ratio of maximum vertical cup and disc diameters; the final task score weights vCDR most heavily and disc segmentation more than cup segmentation.Source: §2.2.3 · p. 10

2.3. Baseline method

The baseline uses separate fundus and OCT encoders with feature fusion for glaucoma grading. The authors report that multimodal input and two additional techniques improve grading performance.

2.3. Baseline method

  • The baseline concatenates features from separate ResNet34-based fundus and OCT encoders, then classifies them with a fully connected layer.The fundus encoder takes 3 input channels and the OCT encoder 256; training uses cross-entropy loss.Source: §2.3 · p. 11
  • The dual-branch baseline outperformed either single-modality branch by a large margin with less variance.The comparison supports the value of combining fundus and OCT inputs, even with simple feature fusion.Source: Table 1 · p. 11, §2.3 · p. 12
  • Optic-disc-region cropping improved mean kappa by 7.2% and standard deviation by 0.2% compared with the baseline.The crop focuses the network on the optic disc and cup and is obtained using a pretrained optic-disc segmentation network.Source: §2.3 · p. 12
  • Ordinal regression splits the three ordered glaucoma grades into two binary classifications, and independently improved models by an average 4.5%.The two binary cross-entropy losses double-penalize severe errors; early glaucoma predicted as normal is treated as less severe than progressive glaucoma predicted as normal.Source: §2.3 · pp. 12–13

3. Methods of participating teams

The ten top-performing teams used varied preprocessing, architectures, and ensemble strategies for fundus-and-OCT glaucoma grading. Most retained dual-branch networks, while some exploited auxiliary tasks, multi-scale inputs, or the ordinal structure of the grades.

3.1. Data Preprocessing

  • Most teams used the default random crop, flip, and rotation augmentation; others added multi-scale resizing, optic-disc cropping or enhancement, or converted OCT volumes to retinal-thickness heatmaps.DIAGNOS-ETS used multi-scale test-time augmentation; MedIPBIT used dataset masks for training crops and predicted masks at inference.Source: §3.1 · pp. 13, 15

3.2. Architecture

  • Most teams used dual branches to encode fundus and OCT separately before feature concatenation, while FATRI-AI concatenated modalities at input and HZL used a multi-task UNet.HZL jointly predicted glaucoma grade, optic-disc-and-cup segmentation, and fovea location; other variants included 3D OCT networks and disc-aware fundus ensembles.Source: §3.2 · pp. 15–16
  • Most teams used cross-entropy supervision; some aligned modality features or averaged independently supervised branch predictions, with no observed ablation differences between these strategies.DIAGNOS-ETS used KL divergence for feature alignment; EyeStar and WZMedTech averaged branch predictions.Source: §3.2 · p. 16

3.3. Ensemble strategy

  • Several teams averaged predictions from their best models across validation folds, while SmartDSP and WZMedTech used staged decisions that exploit the ordinal class structure.SmartDSP selected models strongest on each grade and applied class-specific thresholds; WZMedTech used two models to distinguish normal from glaucoma, then early from progressive cases.Source: §3.3 · pp. 16–17
  • DIAGNOS-ETS averaged multi-scale predictions with temperature scaling, while FATRI-AI used high-confidence first-model predictions as pseudo labels for a second model.FATRI-AI used first-model instances with confidence over 0.7.Source: §3.3 · p. 17

4. Results

The challenge results show variation in test-set generalization and clinically meaningful differences across glaucoma classes. The methodological analyses favor a 3D OCT branch with a DENet fundus branch, ordinal ensembling, and optic-disc region use, with the benefit of disc cropping smaller in stronger models.

4.1. Challenge Results

  • Glaucoma-as-normal errors were generally 4% to 8%, and teams differed in early- versus progressive-glaucoma accuracy; the authors consider higher progressive-glaucoma accuracy preferable when overall performance is similar.The reported glaucoma-as-normal error was lower than the misdiagnosed rate reported for junior ophthalmologists.Source: §4.1 · p. 18
  • SmartDSP, Voxelcloud, EyeStar, HZL, and IBME retained or improved their rankings on the test set, while lower-ranked teams generally showed worse generalization.MedIPBIT, IBME, WZMedTech, DIAGNOS-ETS, and MedICAL had a dramatic performance decrease on the final test set.Source: §4.1 · p. 18, Table 3 · p. 19

4.2. Methodological Findings

  • The authors conclude that a 3D Net and DENet dual-branch architecture with ordinal ensembling performs best, and focusing on the optic-disc/cup region helps glaucoma grading.Source: §4.2 · pp. 19–20

4.2.1. Ablation study on architectures in GAMMA

  • 9.15% higher mean kappa for 3D-DEN than basic DualRes; it also exceeded Res-3D by 5.74% and Res-DEN by 2.73%, with the lowest standard deviation.Res-DEN improved mean kappa by 6.42% and reduced standard deviation by 0.53% versus DualRes; Res-3D improved mean kappa by 3.41% and reduced kappa standard deviation by 0.46%.Source: Table 5 · p. 21, §4.2.1 · p. 21

4.2.2. Ablation study on ensemble strategies in GAMMA

  • 4.52% (2-ordinal) and 7.53% (3-ordinal) mean kappa improvements over five-fold averaging, with standard-deviation reductions of 0.09% and 0.23%, respectively.The authors attribute the gains to dividing the three-class task into binary subtasks and selecting each subtask's best-performing models; SmartDSP also defaults uncertain cases to early glaucoma.Source: §4.2.2 · p. 22, Table 6 · p. 23

4.2.3. Effects of auxiliary tasks

  • 6.79% (70.2% to 77.0%) improvement from disc-region cropping in models without ordinal regression, versus 4.40% (76.8% to 81.2%) in models with ordinal regression.The authors suggest optic-disc cropping aligns fundus and OCT branch features, while stronger models may extract the disc region without that prior.Source: §4.2.3 · pp. 22–23

5. Discussion

GAMMA’s multimodal grading results were stronger than single-modality results, while the discussion highlights the challenge’s dataset, evaluation framework, and methodological analysis. It also notes constraints in fusion, dataset scope, and interpretability, and identifies further clinical validation and added diagnostic inputs as future directions.

5.1. Multimodal Fusion strategies in GAMMA

  • Straightforward dual-branch concatenation predominated because 2D fundus images and 3D OCT volumes differ in dimensions, scale, and spatial correspondence.The discussion says pixel-level fusion is inapplicable to this 2D–3D task, while feature-level methods are difficult to transfer directly.Source: §5.1 · pp. 23–24

5.2. Challenge strengths and limitations

  • GAMMA provided the largest publicly available fundus–OCT paired dataset to date, with clinical grading labels, optic-disc/cup masks, and fovea positions.The authors describe these annotations as helpful for calibrating grading methods and note that the only similar dataset they cite had 50 image pairs.Source: §5.2 · pp. 24–25
  • An open, uniform online evaluation interface enables participants to test methods and compare them on a common leaderboard.Submitted source code also enabled the authors to reproduce top teams’ methods and conduct ablations to identify module effectiveness.Source: §5.2 · pp. 25–26
  • Because preliminary-set evaluations were limited and final methods were tested once, many teams divided the released training data offline; the authors suggest separate training, validation, online-evaluation, and test sets.The final test result was evaluated once, which the authors say helps keep their analysis unbiased by repeated test evaluation.Source: §5.2 · pp. 25–26
  • The dataset remains too small for capable deep-learning models and reflects a Chinese population, so methods may need retraining before use in other populations.The authors call for large-scale, multi-ethnicity data collection to support generalization.Source: §5.2 · pp. 26–27
  • GAMMA methods were black-box neural networks and few were interpretable; the authors note that highlighting image regions alone misses the diagnostic disc-to-cup ratio.They identify explainability as important for clinical adoption and propose considering it as a challenge metric.Source: §5.2 · p. 27

5.3. Clinical implications of the results and future work

  • Simple fundus–OCT fusion improved performance by about 10% over single-modality methods, with another 10% gain from advanced model design.Top-three teams’ sensitivities were 0.959, 0.918, and 0.959, exceeding the reported junior ophthalmologist range of 0.694 to 0.862.Source: §5.3 · p. 28
  • Top-three methods achieved sensitivities above 0.9, supporting potential for large-scale community screening, although the results are limited to a specific image population.Future work will explore adding visual-field and intraocular-pressure measurements to the model.Source: §5.3 · pp. 28–29

6. Conclusion

The paper presents GAMMA as a challenge for automated glaucoma grading from fundus images and OCT volumes. It introduces the dataset, challenge process, evaluation framework, and top-ranked algorithms as a starting point for further research.

6. Conclusion

  • GAMMA introduced an automated glaucoma-grading challenge using both fundus images and OCT volumes, alongside its dataset, evaluation framework, and analyses of top-ranked algorithms.The authors describe it as the first in-depth study of this multimodal grading task.Source: §6 · p. 29
  • The GAMMA data and evaluation framework are publicly accessible through the challenge website.The passage also notes that top-10 teams’ code and technical reports are available.Source: §6 · p. 29
  • Future participants are invited to use the dataset and submit results on the website to benchmark their methods..Source: §6 · p. 29

Appendix

The appendix introduces methods for auxiliary tasks.

Appendix

  • The appendix introduces methods proposed for auxiliary tasks.Source: Appendix · p. 41

Fovea Localization

The fovea-localization section describes how challenge teams approached prediction and summarizes their ranking and methods. Approaches included coordinate regression, segmentation, heatmap prediction, detection, and coarse-to-fine localization.

Fovea Localization

  • The challenge ranked fovea-localization methods by fovea localization score and also reported Euclidean distance.The section points to Table 7 for rankings and Table 9 for method summaries; its baseline used a fundus image to predict a 2D fovea-center coordinate.Source: Fovea Localization · p. 41, Table 7 · p. 41
  • Fovea-localization teams used coordinate regression, binary segmentation, heatmap prediction, or object detection to locate the fovea.Regression predicted coordinates; segmentation used a centered circle, heatmaps used a Gaussian target, and detection used a centered square.Source: Fovea Localization · pp. 41–42
  • Almost half of the teams used coarse-to-fine localization, cropping a region of interest from coarse predictions for later refinement.EyeStar’s Two-Stage Self-Adaptive localization Architecture fused multi-scale regions and coarse features using ROI Align, concatenation, self-attention, and a Fuse layer.Source: Fovea Localization · p. 42

OD/OC Segmentation

OD/OC segmentation methods were ranked by a score incorporating Dice values and vertical cup-to-disc ratio; most teams used a coarse-to-fine strategy, while VoxelCloud added blood-vessel information.

OD/OC Segmentation

Related papers

Loading 2202.06511v4…