PaperlayerView exact revision on arXiv

Source-linked AI summary

Applications of Deep Learning in Fundus Images: A Review

Tao Li, Wang Bo, Chunyu Hu, Hong Kang, Hanruo Liu, Kai Wang, Huazhu Fu

arXiv:2101.09864v1eess.IVcs.CVcs.LG

TL;DR

Early fundus-image screening is clinically important, and deep learning has expanded across segmentation, diagnosis, and synthesis. This review synthesizes 143 application papers within a structured hierarchy and presents datasets, task analyses, and shared limitations. It concludes that deep learning can support biomarker analysis and ophthalmic disease screening, while data quality, sharing, interpretability, and other constraints remain unresolved.

  • Problem

    Early screening of eye diseases using fundus images is clinically important because patients may be unaware of worsening asymptomatic conditions.

  • Method

    The review synthesizes deep learning applications in fundus images using a carefully designed hierarchy across lesion and biomarker segmentation, disease diagnosis, and image synthesis.

  • Results

    Deep learning can detect and segment biomarkers and directly predict ophthalmic disease, providing suggestions and assistance for physicians in screening.

  • Takeaways & Limitations

    The review identifies common limitations and discusses possible solutions involving weak supervision, federated learning, and interpretability methods.

  • Takeaways & Limitations

    Deep learning applications remain limited by unresolved performance constraints, data-sharing challenges, and inherent lack of interpretability.

Abstract

from arXiv · show

The use of fundus images for the early screening of eye diseases is of great clinical importance. Due to its powerful performance, deep learning is becoming more and more popular in related applications, such as lesion segmentation, biomarkers segmentation, disease diagnosis and image synthesis. Therefore, it is very necessary to summarize the recent developments in deep learning for fundus images with a review paper. In this review, we introduce 143 application papers with a carefully designed hierarchy. Moreover, 33 publicly available datasets are presented. Summaries and analyses are provided for each task. Finally, limitations common to all tasks are revealed and possible solutions are given. We will also release and regularly update the state-of-the-art results and newly-released datasets at https://github.com/nkicsl/Fundus Review to adapt to the rapid development of this field.

1. Introduction

Fundus images provide a non-invasive, cost-effective basis for large-scale screening, while deep learning supports classification, segmentation, and synthesis applications. This review organizes 143 papers across diseases and tasks using a multi-layer hierarchy, with task analyses and limitations.

  • Fundus images: Fundus images can be acquired non-invasively and cost-effectively, making them suitable for large-scale screening.
  • Fundus images: Fundus images capture biomarkers and lesions relevant to diagnosing glaucoma, diabetic retinopathy, AMD, cataract, ROP, and DME.Examples include the optic disc, optic cup, macula, fovea, blood vessels, microaneurysms, hemorrhages, and exudates.
  • Deep learning applications: Deep learning applications in fundus imaging are broadly divided into classification, segmentation, and synthesis tasks.Classification includes disease diagnosis and grading; commonly used architectures include VGG-Net, Inception, ResNet, FCN, U-Net, MaskRCNN, DeeplabV3+, and GANs.
  • Review scope: The review collected 143 papers published from January 2016 to August 2020 across major academic databases and venues.
  • Review contributions: Compared with two similar reviews covering 34 and 62 papers, this review includes more recent papers, diseases, challenging tasks, and a multi-layer hierarchy.
  • Review contributions: The review provides summaries and analyses for each task and discusses common limitations with possible solutions.

2. Lesion detection/segmentation

Lesion detection and segmentation use diverse datasets and deep learning methods, but remain constrained by limited labels, class imbalance, lesion size, and visual variation. The review surveys datasets, experimental results, selective sampling, annotation refinement, and unresolved research needs.

  • Datasets: Retinal lesion detection and segmentation datasets overlap with those used for diabetic retinopathy diagnosis and include DIARETDB0, DIARETDB1, RC-RGB-MA, E-ophtha MA, Messidor, and Messidor-2.The datasets differ in image counts, lesion labels, normal-image proportions, acquisition settings, and annotation formats.
  • Hemorrhage detection: Patch-based methods can generate tens of thousands of patches from dozens of images, helping address limited datasets and computational costs.
  • Hemorrhage detection: Lesion detection and segmentation face imbalance because only a few patches contain lesions, while many background patches contribute little to the task.
  • Hemorrhage detection: Selective sampling dynamically increases the likelihood that informative samples are used in later CNN training iterations.
  • Hemorrhage detection: BBR-Net refines coarse bounding-box annotations before RetinaNet hemorrhage detection on a private dataset.
  • Limitations: Current hemorrhage applications still require further study of imbalance, greater research attention, and more pixel-level annotated datasets.

2.2. Microaneurysms

Microaneurysm segmentation is challenging because lesions have low contrast, resemble other lesions, and vary with imaging conditions. Reviewed approaches use multiscale processing, clinical-report guidance, and discriminative losses, but efficiency remains limited.

  • Microaneurysms are early clinical signs of diabetic retinopathy, but their detection is hindered by low contrast, similar-colored lesions, and variable image conditions.
  • Two-stage multiscale networks: A two-stage architecture uses multiscale FCNs to generate regions of interest, then refines patches with a modified ResNet-50 and triplet loss.Selective sampling reduces computational cost and addresses data imbalance.
  • Clinical report guided CNNs: Clinical-report guidance combines a weak image-to-text model with MS-CNN to produce rough and then high-quality microaneurysm segmentation.
  • Discussion: Multiscale and report-guided methods improve lesion-focused processing, but two-stage and cascaded architectures require many parameters and lack efficiency.

2.3. Exudates

Exudate detection and drusen segmentation address clinically relevant retinal lesions despite low contrast, variable appearance, and confusing structures. Reviewed methods modify preprocessing, losses, or network design, while the evidence base remains limited for drusen and innovative architectures for exudates.

  • 2.3. Exudates: Exudate detection is important for diabetic-retinopathy diagnosis, but low contrast, varied sizes, and similarity to other lesions complicate segmentation.
  • CNN with circular Hough conversion: A three-layer CNN classifies exudated versus exudate-free images after optic-disc removal using adaptive histogram equalization, Canny edges, and circular Hough conversion.
  • Modification to loss function: Top-k and bin losses address loss imbalance introduced by class-balanced cross-entropy, but misclassification remains unresolved.
  • Discussion: Exudate work has mainly tested baseline models, leaving newer encoder-decoder architectures unexplored.
  • 2.4. Drusen: Drusen segmentation faces color similarity, uneven brightness, biomarker interference, irregular shapes, and blurred boundaries.
  • Deep random walk: A deep random walk method fuses semantic and low-level features before affinity learning for pixel-level drusen segmentation.
  • Discussion: Only one effective drusen-segmentation approach is identified, motivating exploration of other architectures and links to AMD diagnosis.

2.5. Multiple lesions

Multiple-lesion segmentation is pursued because identifying individual lesions is more practical than grouping them as red or bright lesions. Reviewed approaches include CNNs, dual decoders, multiscale networks, and a deep membrane system, while sensitivity and pixel-level annotation remain priorities.

  • 2.5. Multiple lesions: Multiple-lesion segmentation is more practical than detecting one lesion type or grouping all lesions as red or bright lesions.Challenges include imbalance, contrast, illumination, and inter-class similarity.
  • Early multi-lesion methods: A 10-layer CNN first demonstrated simultaneous pixel-level segmentation of exudates, haemorrhages, and microaneurysms.
  • Early multi-lesion methods: A patching-based CNN classified five lesion-related classes and achieved good performance with limited training samples after ophthalmologist verification and relabeling.
  • Multiscale networks: L-Seg uses a modified VGG-16-based multiscale architecture to segment microaneurysms, soft exudates, hard exudates, and haemorrhages simultaneously.
  • Dual-decoders: Dual decoders extend U-Net for simultaneous red- and bright-lesion segmentation, with each decoder specialized for one lesion category.
  • Multiscale networks: GlobalNet and LocalNet combine contextual and detailed patch features for multiscale segmentation of four lesion types.
  • Discussion: The review identifies sensitivity improvement, individual-lesion segmentation, and pixel-level lesion annotations as continuing priorities.

3.1. Vessel segmentation

Retinal vessel segmentation has progressed from patch-based pixel classification toward fully convolutional encoder-decoder systems incorporating multiscale features, refinement, attention, and specialized vessel handling. Remaining concerns include thin-vessel accuracy, limited datasets, class imbalance, and sampling-resolution trade-offs.

  • 3.1. Vessel segmentation: Vessel segmentation supports diagnosis of diabetic retinopathy and glaucoma but remains challenging because contrast and intensity vary across datasets.
  • Limitations: Common datasets contain relatively few images, and vessel-shape imbalance is especially difficult; additional collection, annotation, synthesis, and selective sampling are suggested.
  • Early approaches: Early methods classified center pixels from image patches using CNNs with fully connected layers, while structured prediction extended outputs to multiple pixels.
  • Fully convolutional networks: Fully convolutional networks provide end-to-end vessel segmentation and can incorporate multiscale side outputs, CRF modeling, and additional wavelet-derived input channels.
  • Encoder-decoder architectures: U-Net remains the most popular fundus-image segmentation framework, motivating improvements in vessel-specific architecture and operations.
  • Architectural improvements: Architectural strategies distinguish thick from thin vessels, use coarse-to-fine refinement, and combine multiscale supervision paths.
  • Architectural improvements: Dual-encoder U-Net variants fuse spatial and contextual features through separate encoders and a feature-fusion module.
  • Limitations: Thin and edge vessel segmentation still needs higher sensitivity and accuracy while maintaining specificity and AUC.

3.2. OD /OC Segmentation

OD/OC segmentation supports glaucoma-related measurement, but optic-cup boundaries and class imbalance make the task difficult. Reviewed approaches include multiscale, atrous, multitask, domain-adaptive, regression, proposal-based, and VAE-based methods, with domain shift and generation performance remaining open issues.

  • Motivation: Cup-to-disc ratio uses vertical cup and disc diameters, making accurate optic-cup and optic-disc segmentation important for glaucoma diagnosis.Optic-cup segmentation is harder because its boundaries are subtle and the cup region is imbalanced.
  • Existing approaches: Reviewed OD/OC methods include FCN, Fine-Net, M-Net, parameter-shared branching, weak-ROI segmentation, depth-guided networks, pixel-wise regression, and ellipse proposal networks.These methods use strategies such as edge-weighted losses, atrous convolutions, multiscale inputs, polar transformations, depth guidance, distance maps, and elliptical anchors.
  • Existing approaches: Pixel-wise regression reformulates OD and fovea detection as distance prediction, using a bi-distance map and a U-Net-like network for globally consistent outputs.The map encodes each pixel’s distance to its nearest optic-disc or fovea landmark.
  • Existing approaches: Domain-adaptive methods extract source- and target-domain regions and learn spatial or shape information to address inaccurate target-domain boundaries and noisy entropy maps.The reviewed frameworks include pOSAL and the unsupervised BEAL approach.
  • Discussion: Future work emphasizes optic-disc quantification, domain adaptation, and the continuing REFUGE2 challenge because domain shift and poor generation performance remain unresolved.The review also identifies optic-disc quantification as a promising direction.

3.3. Fovea segmentation

Fovea segmentation is clinically relevant for defining lesion risk, but few studies address it because publicly available datasets are scarce. The review identifies only one notable work and calls for more architectures and research.

  • Motivation: Fovea segmentation can help define the risk of particular retinal lesions, yet publicly available datasets are scarce.The review therefore describes few works focused on this task.
  • Existing approach: Sedai et al. proposed a two-stage framework that first localizes the fovea region coarsely and then performs fine segmentation on the resulting region of interest.The coarse network fuses upsampled feature maps from multiple levels, while the fine network processes the localized region.
  • Discussion: Only one remarkable work had been introduced for fovea segmentation, motivating further research and exploration of architectures beyond the existing two-stage design.The review explicitly calls for better performance and more architectural alternatives.

3.4. A/V classification

Artery/vein classification is important for disease-related vascular assessment and is increasingly performed directly as a multiclass segmentation problem. However, it remains more challenging than vessel segmentation and can encounter mixed artery-vein segments.

  • Motivation: Artery/vein subdivision supports early diagnosis because a low arteriolar-to-venular width ratio can predict diabetes and cardiovascular diseases.Dedicated datasets add artery/vein labels to vessel imagery, including DRIVE-AV, LES-AV, and INSIPRE-AVR.
  • Methods: Galdran et al. formulated artery/vein classification as four-class segmentation of background, artery, vein, and uncertain pixels.Their U-Net-like model classified arteries and veins directly without first segmenting the vessel tree.
  • Discussion: The field generally favors direct artery/vein classification without prior vessel segmentation, although this task is more challenging than vessel segmentation.Existing work also reports artery and vein labels appearing within a single vessel segment, which is uncommon in reality.

4. Disease diagnosis/grading

Deep learning disease diagnosis and grading in fundus images is dominated by CNN classification, often using ensembles and increasingly incorporating lesion localization or segmentation. The review highlights clinical deployment potential but also a trade-off between grading performance and lesion localization.

  • Background: Diabetic retinopathy is graded under ICDRS into five severity classes, from no apparent DR through proliferative DR.DR is described as a vascular disease and a leading cause of preventable blindness worldwide.
  • Clinical-style papers: Clinical-style studies typically use author-collected datasets and emphasize clinical meaning rather than network-architecture improvements.These works assess deep learning in actual clinical diagnosis settings.
  • Clinical-style papers: CNN ensembles are commonly used for DR classification, including ten-network Inception ensembles and feature-fusion systems for two-class and four-class grading.The reviewed systems vary in backbone, dataset size, input resolution, and downstream feature fusion.
  • Approaches combining lesion detection: Lesion heatmaps, segmentation, and attention mechanisms are combined with DR grading to provide information about image regions contributing to diagnostic decisions.Examples include two-stage local-global networks, encoder-decoder models, and attention networks trained with image-level supervision.
  • Benchmark works: Maximum accuracy reached 0.8284 in DR grading on DDR, while lesion segmentation and detection were particularly poor across the evaluated models.On Messidor-2, deeper and larger networks improved classification but worsened localization, demonstrating a trade-off between the two objectives.
  • Other approaches: An offline smartphone system can provide immediate referable-DR screening, offering a low-cost approach for areas with limited medical resources.The described system delivers diagnosis results through a downloadable application.
  • Discussion: Current systems often generate coarse heatmaps without lesion labels, while joint grading and lesion segmentation face a trade-off because classification features may lack segmentation’s required spatial information.The review identifies balancing these objectives as important for the multi-task problem.

4.2.4. Generating evidence maps

Evidence-map approaches aim to make fundus-image disease diagnosis more interpretable by localizing clinically relevant regions, especially for glaucoma. Related diagnosis work also covers AMD, ROP, cataracts, and multiple diseases using CNN-based and multi-task strategies.

  • Glaucoma evidence maps: Glaucoma evidence-map methods combine diagnosis with localization, optic-disc segmentation, or attention prediction to identify clinically relevant regions.WSMTL performs evidence identification, optic-disc segmentation, and glaucoma diagnosis simultaneously; AG-CNN predicts attention maps and localizes pathological areas.
  • Glaucoma evidence maps: EAMNet links low-level localization with high-level semantics through multi-layer average pooling and produces evidence activation maps.The maps are obtained by weighted summation of feature maps.
  • Limitations: Glaucoma evidence guidance remains limited because accurate optic-disc/optic-cup segmentation and factors such as age, race, and family history receive insufficient attention.The review cautions that glaucoma diagnosis should not rely only on cup-to-disc-ratio estimation.
  • Related disease diagnosis: AMD diagnosis includes CNN, hybrid, ensemble, and lesion-guided approaches, including severity grading and five-year progression-risk estimation.DeepSeeNet separately detects drusen, pigment abnormalities, and late AMD to grade disease severity.
  • Related disease diagnosis: Deep CNNs for ROP plus-disease diagnosis outperformed six of eight invited ROP experts in a 5,511-image multicenter cohort.The pipeline used U-Net preprocessing followed by an Inception-v1 diagnostic network.
  • Related disease diagnosis: Multiple-disease diagnosis is considered promising because clinical practice needs more comprehensive systems, with further experiments encouraged on datasets such as ODIR.The review discusses systems covering simultaneous and multi-class disease diagnosis.

5. Image synthesis

Deep learning image synthesis addresses limited medical-image datasets by generating or reconstructing fundus images, with GANs dominating the reviewed approaches. Applications include anatomical, vascular, disease-conditioned, smartphone, AMD, and cross-modality synthesis.

  • Motivation and overview: Image synthesis can enlarge fundus-image training resources, support image understanding, improve model performance, and alleviate overfitting.The review motivates synthesis partly because high-quality medical-image annotations are costly and datasets are often small.
  • Fundus-image synthesis applications: Anatomical synthesis generates high-quality optic-nerve-head images from optic-disc, optic-cup, and vessel segmentation masks using a multi-U-Net generator.The generated images reproduce vessel bending at optic-cup edges, and a discriminator evaluates synthetic and original images with masks.
  • Fundus-image synthesis applications: Vessel-synthesis methods learn mappings from vessel structures or maps to synthetic retinal images, including approaches designed for small datasets.Tub-GAN and Tub-sGAN use tubular annotations, while another method combines an adversarial autoencoder with a GAN.
  • Fundus-image synthesis applications: DR-GAN generates high-resolution fundus images conditioned on arbitrary DR grades and lesion information using vessel and lesion masks.Fine-grained local-detail modeling and multiscale discriminators are used to improve realism.
  • Fundus-image synthesis applications: For AMD, ophthalmologists obtained similar diagnoses on real and synthetic images, and synthetic-only training achieved similar performance to real-only training.The GAN was trained on 133,821 AREDS fundus images.
  • Fundus-image synthesis applications: Cross-modality reconstruction can translate retinography into angiography, offering a self-supervised route to learn features and obtain images from non-invasive modalities.Hervella et al. used U-Net for this reconstruction task.
  • Motivation and overview: GANs dominate fundus-image synthesis, while CycleGAN appears among newer architectural variants.The review characterizes synthesis as a relatively new but rapidly developing direction.

6. Other applications

Other applications extend fundus-image deep learning to rare pathologies, systemic disease prediction, image registration, enhancement, and quality assessment. These directions broaden clinical use while remaining constrained by limited data and the need for robust, interpretable processing.

  • Rare pathologies: Rare-pathology studies include lesion-aware segmentation for pathological myopia and attention-generating diagnosis of refractive error.The pathological-myopia method segments atrophy and detachment, while the refractive-error model identifies the foveal region as important for prediction.
  • Rare pathologies: Rare-pathology diagnosis is considered important despite receiving less attention, but its main current challenge is lack of data.The review expects experience from more prevalent diseases to extend to these diagnoses.
  • Systemic diseases: Fundus images have been used to predict systemic characteristics and diseases, including cardiovascular risk factors, ischemic stroke, cardiac vessels, and biological age.Approaches include attention heatmaps, vessel-tree feature isolation, cross-domain vessel synthesis, and CNN-based age estimation.
  • Systemic diseases: Systemic-disease diagnosis from fundus images is presented as promising for remote, non-invasive, offline, and convenient prediction.The review connects this direction with multi-disease and smartphone-based offline diagnosis systems.
  • Image processing: Image registration, enhancement, and quality assessment support downstream fundus-image processing and selection.Reviewed examples include non-rigid registration, weakly supervised enhancement of blurred images, and ternary retinal-image quality labels.
  • Image processing: The review identifies domain invariance and visualization as common goals across fundus-image applications, alongside continued exploration of architectures and methods.These goals are described as useful for clinicians and relevant across tasks.

7. Conclusions and Discussions

Deep learning shows strong performance across fundus-image diagnostic tasks, but practical limitations in data, generalization, computation, privacy, class imbalance, and interpretability remain. The review identifies strategies including weak supervision, image synthesis, federated learning, domain adaptation, and heatmaps to address these challenges.

  • Conclusions: Deep learning can detect and segment biomarkers and assist or directly support screening for glaucoma, DR, AMD, and other ophthalmic diseases.The review reports comparable or better performance than human experts in some practical DR diagnosis settings.
  • Lack of high-quality labeled data: Fundus datasets are limited because expert annotation is difficult, motivating weakly supervised learning and image synthesis as ways to reduce dependence on high-quality labels.Weak supervision addresses incomplete, inexact, or inaccurate supervision, while synthesis can generate realistic labeled images.
  • Lack of high-quality labeled data: Private datasets, privacy requirements, institutional competition, and regulations constrain data sharing, making federated learning a potential solution to the resulting “data islands” problem.The review distinguishes horizontal, vertical, and federated transfer learning according to overlap in users and features.
  • Domain shift: Dataset differences in cameras, resolution, illumination, and settings create domain shifts that can reduce model generalization across datasets.The review identifies domain adaptation as a strategy used to improve optic-disc segmentation on target domains.
  • High computational consumption: Deep learning requires substantial computation and training time, which can prevent deployment on portable ophthalmic devices.The review attributes this cost to models having much larger parameter sizes than traditional machine-learning methods.
  • Lack of interpretability: Clinical adoption is constrained by deep learning’s lack of interpretability, while heatmaps can show image regions used for predictions and support diagnostic assistance.The review discusses CAMs, evidence maps, Grad-CAM-based approaches, uncertainty estimation, and relevance-like scores as interpretability strategies.
Loading 2101.09864v1…