PaperlayerView exact revision on arXiv

Source-linked AI summary

GANs for Medical Image Synthesis: An Empirical Study

Youssef Skandarani, Pierre-Marc Jodoin, Alain Lalande

arXiv:2105.05318v2eess.IVcs.CVcs.LG

TL;DR

The paper asks whether GANs that generate realistic images can also provide useful medical data, where annotated datasets are limited. It compares multiple GANs across three medical datasets using FID and downstream segmentation, finding that visual realism does not reliably reproduce dataset richness.

  • Problem

    The study addresses limited evidence about whether realistic-looking GAN outputs are useful medical data for downstream tasks, especially with small annotated datasets.

  • Method

    The authors compare multiple GAN architectures across three medical datasets, selecting models through hyperparameter search and evaluating outputs with FID and U-Net segmentation.

  • Results

    GAN effectiveness varies substantially: StyleGAN and SPADE GAN achieve the best FID results, but FID does not consistently predict segmentation performance.

  • Takeaways & Limitations

    Realistic-looking medical images are not always reliable synthetic data, and task-based evaluations can be more informative than traditional GAN metrics.

  • Takeaways & Limitations

    Very small or highly unbalanced datasets constrain the benefit of GAN-based enlargement because models may overfit or worsen downstream performance.

Abstract

from arXiv · show

Generative Adversarial Networks (GANs) have become increasingly powerful, generating mind-blowing photorealistic images that mimic the content of datasets they were trained to replicate. One recurrent theme in medical imaging is whether GANs can also be effective at generating workable medical data as they are for generating realistic RGB images. In this paper, we perform a multi-GAN and multi-application study to gauge the benefits of GANs in medical imaging. We tested various GAN architectures from basic DCGAN to more sophisticated style-based GANs on three medical imaging modalities and organs namely : cardiac cine-MRI, liver CT and RGB retina images. GANs were trained on well-known and widely utilized datasets from which their FID score were computed to measure the visual acuity of their generated images. We further tested their usefulness by measuring the segmentation accuracy of a U-Net trained on these generated images. Results reveal that GANs are far from being equal as some are ill-suited for medical imaging applications while others are much better off. The top-performing GANs are capable of generating realistic-looking medical images by FID standards that can fool trained experts in a visual Turing test and comply to some metrics. However, segmentation results suggests that no GAN is capable of reproducing the full richness of a medical datasets.

1. Introduction

This section motivates evaluating whether GANs can generate useful medical data, not merely realistic-looking images, amid limited annotated datasets. The study therefore examines GAN-generated data across multiple medical imaging applications and datasets.

  • GANs have achieved highly realistic image synthesis, motivating evaluation of whether they perform similarly on medical data.
  • Medical image analysis includes segmentation, which outlines image regions belonging to clinically relevant classes such as cardiac structures.
  • Medical imaging datasets are often small because collecting sensitive data and producing reliable annotations require substantial cost and specialized knowledge.
  • Prior work reported realistic synthetic medical images, including examples that experts could not reliably distinguish from real images.
  • Synthetic-data studies have produced mixed downstream segmentation results, with synthetic-only training falling short of real-data performance in reported examples.
  • This paper evaluates the richness and benefit of GAN-generated data across three datasets spanning different organs and imaging modalities.

2. Generative Adversarial Networks

This section introduces GANs as adversarial generator–discriminator systems, reviews common training challenges and model variants, and describes FID-based evaluation. It also motivates comparing architectures suited to medical image synthesis.

  • GANs train a generator and discriminator in a minimax game, with the generator producing samples intended to be classified as real.
  • The study selects widely adopted or quality-improving GANs that can fit on a single 12Gb GPU.
  • GAN training can suffer from convergence problems, vanishing gradients, and mode collapse, which can reduce learning quality or generated-data diversity.
  • The compared models include DCGAN, LSGAN, WGAN, HingeGAN, StyleGAN, and SPADE GAN, representing different architectural or loss-function choices.
  • FID measures the distance between real and generated image distributions in a learned Inception-feature space and correlates with human visual perception.
  • FID has important limitations, including high bias and inability to detect training-set memorization.

3. Material and Methods

The methods aim to make GAN comparisons fair across architectures and medical datasets by searching broad hyperparameter spaces. Runs are selected using FID as the reference metric.

  • The study covers different GANs, image modalities, and a wide range of hyperparameters to assess their effects on medical-image synthesis.
  • Approximately 500 GPU-days were used for a broad hyperparameter search, with the best runs retained according to FID.
  • The search used sensible initial configurations for each GAN–dataset pair because exhaustive sweeps grow exponentially with the number of optimized hyperparameters.
  • Table 1 lists the hyperparameters optimized in the study.

3.2. GANs setup

The setup standardizes several GAN implementations while preserving the original network architectures of StyleGAN and SPADE GAN. The resulting architectures are summarized in Figure 1.

  • DCGAN, LSGAN, WGAN, and HingeGAN share a fully convolutional generator–discriminator architecture, differing primarily in their loss functions.
  • StyleGAN and SPADE GAN use publicly available implementations without network-architecture changes.
  • Figure 1 schematically summarizes the architecture of each GAN evaluated.

3.3. GAN Training Tricks

The study uses label smoothing, feature matching, and differentiable augmentation as training techniques intended to stabilize GAN optimization and improve convergence.

  • Label smoothing replaces the discriminator’s true classification labels with a smooth value α.
  • Feature matching adds a generator objective that minimizes the distance between discriminator activations for real and generated data.
  • Differentiable augmentation applies various augmentations to fake and real samples before they reach the discriminator.The passage states that this yields more stable training and better convergence.

3.4. GAN evaluation in medical imaging

The evaluation separates visual fidelity from medical usefulness by assessing both generated-image appearance and downstream segmentation performance. This protocol tests whether synthetic data preserves the richness and diversity needed for subsequent medical-imaging tasks.

  • FID measures image fidelity, but visual acuity alone is insufficient to assess the true performance of GANs in medical imaging.
  • A separately trained segmentation network is evaluated on GAN-generated datasets and compared with results from the original real datasets.
  • The common evaluation protocol reflects GANs’ intended use for enlarging datasets and supplying additional training data to downstream tasks.

3.5. Datasets

The study covers cardiac cine-MRI, liver CT, and retinal fundus imaging across datasets that differ in modality, organ, and size. These datasets provide varied anatomical and data-scale conditions for evaluating GANs.

  • The three datasets represent cardiac cine-MR images, liver CT, and retina imaging across different organs and imaging modalities.
  • Dataset size ranges from large SLIVER07 to moderate ACDC and small IDRID.
  • ACDC contains 150 cardiac cine-MRI exams, split into 100 training and 50 testing exams.
  • SLIVER07 contains 40 contrast-enhanced liver CT volumes, split into 20 training, 10 validation, and 10 testing volumes.
  • IDRiD contains 516 retinal fundus images, including 81 with segmentation masks, of which 54 training images were used.

3.6. Dataset generation

The selected GANs learn the datasets’ image distributions and generate synthetic medical datasets for downstream segmentation evaluation. Generation uses joint image-mask modeling for most GANs and produces 10,000 raw samples per model.

  • The GANs are trained to synthesize new medical data and are assessed by sampling images for training a segmentation network.
  • Most GANs model the joint distribution of images and masks by concatenating them along the channel axis, while SPADE is conditioned on a segmentation mask.
  • Each trained GAN generates a dataset of 10,000 images by randomly sampling its input latent space.
  • The generated datasets receive no further processing so the study can evaluate the quality of the GANs’ raw image outputs.

4. Experiments and Results

The experiments compare GAN architectures across medical datasets using FID, generated-image inspection, and downstream U-Net segmentation. StyleGAN and SPADE GAN perform best visually and on segmentation, but FID does not fully capture application utility or dataset richness.

  • 4.1. Hyperparameter search and overall results: HingeGAN had the lowest overall FID variance among the simpler GANs, while DCGAN and LSGAN were more sensitive to hyperparameter choices.SPADE and StyleGAN were more stable but required approximately 10 and 30 days of training, respectively.
  • 4.1. Hyperparameter search and overall results: Larger reference datasets produced better FID values, with IDRID generally above 150, ACDC roughly between 100 and 150, and Sliver07 mostly below 100.FID also varied with the number of discriminator convolutional filters.
  • 4.1. Hyperparameter search and overall results: StyleGAN and SPADE GAN were the two best models by FID across datasets, including SPADE GAN’s IDRID FID of 1.09.Figure 5 provides examples of generated images for each GAN on ACDC, Sliver07, and IDRID.
  • 4.2. Segmentation evaluation: StyleGAN and SPADE GAN achieved the highest Dice scores on all datasets, with StyleGAN reaching 87% Dice on ACDC, 2% below training on original data.The downstream evaluation generated 10,000 images per dataset and trained a U-Net on synthetic data.
  • 4.2. Segmentation evaluation: Simpler GANs performed systematically poorly on every dataset despite intensive hyperparameter searches, suggesting they may be ill-suited for medical imaging applications.The generated datasets also achieved performance similar to traditional augmentation techniques when used as augmentation data.
  • 4.2. Segmentation evaluation: FID did not correlate reliably with Dice: StyleGAN’s Sliver07 FID of 29.06 was lower than SPADE GAN’s 47.62, although SPADE GAN achieved better segmentation.U-Net latent-space projections distinguished StyleGAN from DCGAN more clearly than InceptionNet projections, exposing a limitation of ImageNet-based FID for medical images.

5. Discussion

The discussion highlights training volatility, dataset-dependent performance, and limits in evaluating whether synthetic medical images retain task-relevant information. GANs can produce visually convincing images, but downstream utility remains constrained by data scale, imbalance, dimensionality, and computational demands.

  • Training Volatility: Hyperparameter adjustments can substantially affect GAN training, with DCGAN and LSGAN showing the highest variability while WGAN and HingeGAN are less sensitive.
  • Computational and Modeling Costs: State-of-the-art GANs can produce high-quality images but may require long training times, overfit, or reproduce training examples through “Memory GAN.”
  • FID and Image Quality: Lower FID does not consistently translate into better segmentation performance, making task-based evaluation important for medical image synthesis.
  • Data Scale: Very small or highly imbalanced datasets can cause overfitting or amplify training-data biases, reducing the benefit of GAN-based augmentation.The discussion identifies IDRID as exceedingly small and Sliver07 as having only 5% foreground pixels.
  • Medical Data Structure: Because much medical data is acquired volumetrically, typical 2D GANs may fail to capture the full extent of medical information.

6. Conclusion

The conclusion evaluates GANs as medical image synthesis tools in response to limited access to large annotated datasets. It finds that their effectiveness is not consistently reliable despite near-indistinguishable images, and supports further development of models tailored to medical data.

  • Conclusion: Limited access to huge annotated datasets motivates evaluating GANs as medical imaging data sources.
  • Conclusion: GAN effectiveness as a source of medical imaging data is not always reliable, even when generated images are nearly indistinguishable from real data.
  • Conclusion: The study points toward further research on GANs that account for medical-data subtleties to improve generative models.
Loading 2105.05318v2…