PaperlayerView exact revision on arXiv

Source-linked AI summary

DR-GAN: Conditional Generative Adversarial Network for Fine-Grained Lesion Synthesis on Diabetic Retinopathy Images

Yi Zhou, Boyang Wang, Xiaodong He, Shanshan Cui, Ling Shao

arXiv:1912.04670v3eess.IVcs.CV

TL;DR

Severe diabetic-retinopathy grades are scarce in training data, despite the need for diverse, balanced data for grading models. DR-GAN generates fundus images conditioned on lesion and grading information; EyePACS experiments report consistent grading improvements when synthesized data augment training.

  • Problem

    DR datasets are imbalanced, with levels 3 and 4 comprising only 2.35% and 2.16% of EyePACS images, respectively.

    Source: §I · p. 1

  • Method

    DR-GAN uses a conditional high-resolution generator to synthesize retina images with controllable lesion and grading information.

    Source: §I · p. 2

  • Results

    Synthesized-data augmentation consistently improved results across all five compared approaches on real EyePACS test images.

    Source: D. Data Augmentation by Synthesis for… · p. 9

  • Takeaways & Limitations

    Synthesized data can augment training, particularly for severe DR images, to improve grading-model performance.

    Source: §V · p. 10

    Generated images can inherit low illumination from EyePACS because it lacks manipulation labels, and the structural and lesion masks are inferred rather than ground truth.

    Source: 2) Limitations · p. 6

Abstract

from arXiv · show

Diabetic retinopathy (DR) is a complication of diabetes that severely affects eyes. It can be graded into five levels of severity according to international protocol. However, optimizing a grading model to have strong generalizability requires a large amount of balanced training data, which is difficult to collect particularly for the high severity levels. Typical data augmentation methods, including random flipping and rotation, cannot generate data with high diversity. In this paper, we propose a diabetic retinopathy generative adversarial network (DR-GAN) to synthesize high-resolution fundus images which can be manipulated with arbitrary grading and lesion information. Thus, large-scale generated data can be used for more meaningful augmentation to train a DR grading and lesion segmentation model. The proposed retina generator is conditioned on the structural and lesion masks, as well as adaptive grading vectors sampled from the latent grading space, which can be adopted to control the synthesized grading severity. Moreover, a multi-scale spatial and channel attention module is devised to improve the generation ability to synthesize details. Multi-scale discriminators are designed to operate from large to small receptive fields, and joint adversarial losses are adopted to optimize the whole network in an end-to-end manner. With extensive experiments evaluated on the EyePACS dataset connected to Kaggle, as well as the FGADR dataset, we validate the effectiveness of our method, which can both synthesize highly realistic (1280 x 1280) controllable fundus images and contribute to the DR grading task.

I. INTRODUCTION

DR grading models need diverse, balanced data, but high-severity cases are scarce and conventional augmentation offers limited diversity. DR-GAN synthesizes fundus images conditioned on structural, lesion, and grading information, with experiments reporting promise for image synthesis and downstream training.

I. INTRODUCTION

  • EyePACS contains only 2.35% level 3 and 2.16% level 4 images, compared with 73.67% normal images, while common augmentation leaves limited diversity.The authors motivate synthesis as a way to provide more varied images across DR grades for training.Source: §I · p. 1
  • DR-GAN conditions a high-resolution retina generator on vessel, optic-disk, and lesion masks and uses multi-scale attention and discriminators for image synthesis.The training objective includes adversarial, feature-matching, and grading-classification losses.Source: §I · pp. 1–2
  • Adaptive grading vectors sampled from learned latent grading spaces manipulate synthesis blocks to control the grading information in generated images.The vectors are learned using real DR images with grading annotations.Source: §I · p. 2
  • Experiments on EyePACS and FGADR report promising fidelity and diversity, and find that synthesized images can augment training for DR grading and lesion segmentation.The paper also reports comparison experiments and ablations.Source: §I · p. 2
  • The extended model adds laser marks and proliferate membranes, an optic-disk mask, and FGADR training, alongside a new spatial-and-channel attention module.Weakly supervised activation maps supply masks for laser marks and proliferate membranes because their annotations are unavailable.Source: §I · p. 2

II. RELATED WORK

Prior retinal synthesis approaches leave gaps in controllable grading, clear lesion detail, or global grade representation. DR-GAN uses mask conditioning and progressive generation to produce high-resolution fundus images.

II. RELATED WORK

  • Earlier retinal synthesis methods lack controllable grading, clear DR lesions, or global grade representation, while one descriptor-based method still required ophthalmologist review for gradability.DR-GAN is presented as enabling manipulation of both grading and lesion information.Source: §II · p. 2

A. High-Resolution DR Image Generation

  • The generator uses a two-stage encoder-decoder design to produce 1280 × 1280 images, first generating 640 × 640 features and then adding local detail.The higher-resolution stage combines its features with those from the first stage and is fine-tuned after that stage is pretrained.Source: A. High-Resolution DR Image Generation · pp. 2–3
  • Vessel, optic-disk, and lesion masks condition the generator; predicted masks extend training to EyePACS images, while FGADR annotations improve lesion-mask training.The EyePACS masks are model-predicted rather than ground truth, and the authors describe their noise as tolerable.Source: A. High-Resolution DR Image Generation · p. 3

B. Multi-Scale Spatial and Channel Attention Module

The spatial and channel attention module is added to address an encoder-decoder generator’s difficulty preserving tiny lesions and fine vessels. It processes features and resized masks to model spatial context and channel dependencies.

B. Multi-Scale Spatial and Channel Attention Module

  • The SCA module follows adaptive grading manipulation and combines encoder and synthesis features with resized structural and lesion masks before attention.The resulting features are passed through a convolution before attention and upsampling for the next synthesis block.Source: B. Multi-Scale Spatial and Channel… · p. 3
  • The attention design aims to model richer spatial context and strengthen semantics such as input masks, improving dependencies between channel maps for small-detail synthesis.Source: B. Multi-Scale Spatial and Channel… · p. 3
  • Spatial attention forms an attention map over spatial positions from reshaped feature branches, while channel attention forms an attention map across channels.The module combines the spatially and channel-attended features with a weighted element-wise sum and additional convolutions.Source: B. Multi-Scale Spatial and Channel… · pp. 3–4

C. Multi-Scale Discriminators with Multi-Task Optimization

DR-GAN uses image scales with different discriminator receptive fields to assess holistic structure and local detail. A multi-task objective combines adversarial training with feature, grading, and perceptual losses.

C. Multi-Scale Discriminators with Multi-Task Optimization

  • Three discriminators process 1280 × 1280, 640 × 640, and 320 × 320 images, with smaller-scale inputs emphasizing holistic structure and larger-scale inputs emphasizing local details.The largest-scale discriminator particularly targets small lesion regions.Source: C. Multi-Scale Discriminators with… · p. 4
  • The training objective includes adversarial and feature-matching losses, plus auxiliary DR grading classification with focal loss for imbalanced data.A VGG-19-based perceptual loss is added for the largest-scale input.Source: C. Multi-Scale Discriminators with… · p. 4
  • The generator receives fused vessel and lesion masks as its condition, while the discriminator receives channel-wise concatenated conditions and real or synthesized images.The loss weights λ1, λ2, and λ3 are set to 10, 10, and 1, respectively.Source: C. Multi-Scale Discriminators with… · p. 4

D. Adaptive Grading Manipulation

DR-GAN learns latent grading spaces from image features and samples grading vectors to control severity during synthesis. Those vectors modulate generator features through AdaIN.

D. Adaptive Grading Manipulation

  • Latent grading vectors are sampled from grade-specific normal distributions learned from feature means and variances.A ResNet-50 grading model extracts the features used to form five spaces, one for each DR level.Source: D. Adaptive Grading Manipulation · pp. 4–5
  • During training, the input pair’s ground-truth grade determines which latent grading space supplies its vector.Source: D. Adaptive Grading Manipulation · p. 5
  • A four-layer nonlinear mapping network encodes affine transformations, and AdaIN embeds the grading vectors into synthesis-block features.Features are concatenated with random noise before a 1 × 1 convolution fuses them; grading vectors supply the scale and bias vectors.Source: D. Adaptive Grading Manipulation · p. 5

E. Testing Phase and Implementation Details

At test time, DR-GAN predicts a grade from the input masks to select a latent grading distribution. The grading spaces are pre-trained with ResNet-50 on EyePACS.

E. Testing Phase and Implementation Details

  • At testing, the grading function predicts a label from the vessel and lesion masks to select a latent grading distribution.After training, the model can imitate multiple lesion masks for one vessel mask to generate images with controllable grading labels.Source: E. Testing Phase and Implementation… · p. 5

1) Pre-training the Grading Space:

  • Five grading-level latent spaces are pre-trained with ResNet-50 using EyePACS training and testing data, with 30 training epochs.The pre-training uses ADAM with a 0.001 learning rate, 0.5 momentum, and batch size 64.Source: 1) Pre-training the Grading Space · p. 5

2) End-to-End Training:

The end-to-end training uses staged generator training, while mask supervision combines annotated datasets with inferred EyePACS masks. FGADR annotations also train lesion segmentation models.

2) End-to-End Training:

  • 85 hours of training produce a model that synthesizes one image in 0.21 seconds.The generator uses seven residual blocks; Gm trains for 10 epochs before Gl is added for 10-epoch fine-tuning.Source: 2) End-to-End Training · p. 5

A. Datasets and Pre-processing

  • EyePACS images have grading labels but no pixel-wise masks, so masks predicted by segmentation models are used as weak ground truth for DR-GAN training.The authors use EyePACS for its larger scale and diversity, while avoiding direct DR-GAN training on FGADR’s smaller pixel-annotated set because they expect overfitting.Source: A. Datasets and Pre-processing · p. 5
  • FGADR’s 1,842-image Seg-set supplies pixel-level annotations for four lesions and image-level labels for laser marks and proliferate membranes.The reported PR-curve AUCs on FGADR are 0.405 for microaneurysms, 0.678 for hemorrhages, 0.798 for hard exudates, and 0.692 for soft exudates.Source: A. Datasets and Pre-processing · p. 5
  • Weakly supervised classifiers generate activation maps for laser marks and proliferate membranes because these global features are not suitable for pixel-wise annotation.The classifiers achieve 94.97% and 92.21% accuracy, respectively, before activation maps are extracted for EyePACS images.Source: A. Datasets and Pre-processing · pp. 5–6

B. Qualitative Evaluation of Image Synthesis

The qualitative evaluation examines image fidelity and the effects of grading and lesion conditions. DR-GAN generates varied, grade-controllable images, but several training-data and mask constraints produce occasional failures.

1) Performance Demonstration:

  • 10,000 images per grading level are synthesized by varying lesion quantity and position within fixed vessel and optic-disk masks.The authors report promising fidelity and grading manipulation, with lesions including microaneurysms, hemorrhages, hard and soft exudates, laser marks, and proliferate membranes.Source: 1) Performance Demonstration · p. 6

2) Limitations:

  • Occasional failures include low illumination, blurred images from unclear inferred masks, unrealistic fovea regions, and missing lesions for some grade-1 inputs.The authors also report that the model cannot manipulate IRMA or neovascularization because too few training examples prevented adequate segmentation masks.Source: 2) Limitations · p. 6

C. Quantitive Evaluation of Image Synthesis

Human review assessed synthesized-image fidelity, real/fake discrimination, and grading accuracy. DR-GAN scored above Tub-sGAN on fidelity and grading accuracy, while real images received the highest fidelity score.

C. Quantitive Evaluation of Image Synthesis

  • Fidelity was assessed through expert review and Frechet inception distance and sliced wasserstein distance scores.Three ophthalmologists independently evaluated images; reported ratings were averaged.Source: C. Quantitive Evaluation of Image… · pp. 6, 8, 1) Evaluation by Ophthalmologists · p. 8
  • Fig. 5 shows images at five grading levels for an input vessel and optic disk, plus lesion examples identified by colored boxes.The boxes indicate hemorrhages, hard exudates, soft exudates, microaneurysms, laser marks, and proliferate membranes.Source: Figure 5 · p. 7
  • Fig. 6 presents four failure cases alongside local-detail patches comparing DR-GAN without and with SCA.The failure cases include low illumination, blurred retinal structure, a foveal defect, and a grade-1 sample without lesions.Source: Figure 6 · p. 8

1) Evaluation by Ophthalmologists:

2) Evaluation with Frechet Inception Distance:

DR-GAN’s FID improved slightly with SCA, and its SCA version achieved the best reported SWD score. The reported comparisons also identify lesion masks and adaptive grading manipulation as relevant to synthesis fidelity.

2) Evaluation with Frechet Inception Distance:

  • 4.53 FID was achieved by DR-GAN without SCA, and adding SCA produced a slight further decrease.The authors also report that removing lesion masks largely decreased the fidelity of synthesized lesion patterns.Source: 2) Evaluation with Frechet Inception… · p. 9

3) Evaluation with Sliced Wasserstein Distance:

  • 6.17 SWD was DR-GAN with SCA’s best score; SWD decreased by 2.5 versus DR-GAN without AGM.Lower SWD is better; the authors attribute the decrease to adaptive instance normalization by grading vectors contributing to synthesis blocks.Source: 3) Evaluation with Sliced Wasserstein… · p. 9

D. Data Augmentation by Synthesis for DR Grading

DR-GAN-generated data is evaluated as an augmentation for EyePACS grading and for transfer to FGADR. It improves average grading metrics on both datasets, while MixUp and CutMix provide little or no benefit in the reported comparison.

D. Data Augmentation by Synthesis for DR Grading

  • The grading evaluation compares real-only training, real-data training and testing, and training on combined real and synthesized data.The combined-data setting is evaluated on the real EyePACS test set; Fig. 7 presents training-label distributions before and after augmentation.Source: D. Data Augmentation by Synthesis for… · p. 9, Figure 7 · p. 9
  • On EyePACS, synthesized-data augmentation raises average accuracy by 1.75% and kappa by 1.87% across five compared approaches.The evaluation table also covers FGADR Grade-set testing without fine-tuning, where average accuracy rises by 1.08% and kappa by 1.15%.Source: D. Data Augmentation by Synthesis for… · p. 9, Table III · p. 10
  • On FGADR Grade-set testing without fine-tuning, synthesized-data augmentation raises average accuracy by 1.08% and kappa by 1.15%.FGADR Grade-set is used only for testing to evaluate generalization.Source: D. Data Augmentation by Synthesis for… · p. 9, Table III · p. 10
  • With real-data-trained grading models tested on generated images, AFN achieves 90.46% accuracy and 89.15% kappa, the best reported performance in that setting.The generated set contains 50,000 images, with 10,000 for each grading level.Source: D. Data Augmentation by Synthesis for… · p. 9
  • On EyePACS with ResNet-50, MixUp decreases average accuracy by 0.45%, while CutMix increases it by only 0.21%.The authors attribute the limited benefit to unrealistic images and incorrect lesion counts for the corresponding grades.Source: D. Data Augmentation by Synthesis for… · p. 9
  • For per-class true positive rates on EyePACS, generated-data augmentation improves all grades, particularly grades 3 and 4, which have less real data.The evaluation uses a ResNet-50 backbone.Source: D. Data Augmentation by Synthesis for… · pp. 9–10, Table IV · p. 10

E. Ablation Studies

The ablations test lesion masks, adaptive grading manipulation, loss terms, and spatial-channel attention, and assess the quality required of the pretrained grading space. The conclusion identifies pixel-level lesion-mask annotation as a future need.

E. Ablation Studies

  • Removing lesion masks significantly harms grading performance because synthesized lesion patterns are poor, preventing generated-data augmentation from benefiting the grading model.The module evaluation is reported in Table V.Source: E. Ablation Studies · p. 10, Table V · p. 10
  • Removing adaptive grading manipulation lowers kappa by 1.68% relative to the design with AGM.The authors report that AGM improves the fidelity and diversity of synthesized lesion appearances within corresponding grading levels.Source: E. Ablation Studies · p. 10
  • Removing perceptual loss or discriminator classification loss reduces grading performance; without classification loss, accuracy falls by 1.17% and kappa by 1.83%.The comparisons appear in Table V.Source: E. Ablation Studies · p. 10, Table V · p. 10
  • Adding multi-scale spatial and channel attention improves synthesized-image fidelity and slightly increases grading-model performance.The authors report these changes for the DR-GAN extension with SCA.Source: E. Ablation Studies · p. 10
  • An effective latent grading space requires a pretrained grading accuracy of around 80%; at 75.40%, AGM negatively affects synthesis.At 79.26%, AGM performs similarly to the model without AGM.Source: E. Ablation Studies · p. 10

V. CONCLUSION

  • The conclusion proposes adding more real, annotated pixel-level lesion masks to improve DR-GAN training.It presents the model as a high-resolution generator conditioned on grading and lesion information, with synthesized data intended to augment grading, particularly for severe DR.Source: §V · p. 10
Loading 1912.04670v3…