PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Adversarial Network in Medical Imaging: A Review

Xin Yi, Ekta Walia, Paul Babyn

arXiv:1809.07294v4cs.CVcs.LG

TL;DR

Medical imaging lacks a unified account of how GANs are being applied across reconstruction, synthesis, segmentation, classification, and detection. This review surveys the literature through January 1, 2019, finding rapid adoption concentrated in image synthesis and in reconstruction or segmentation, while noting that clinical adoption remains limited.

  • Problem

    The review addresses the need for a comprehensive account of GAN applications and remaining challenges in medical imaging.

  • Method

    The authors searched major medical-imaging databases and venues, added cross-referenced works, and organized studies by task and imaging modality.

  • Results

    About 46% of reviewed papers studied image synthesis, while another 37% focused on reconstruction and segmentation.

  • Takeaways & Limitations

    GANs show promise for cross-modality synthesis, shape and texture regulation, data scarcity, and reuse of labels across modalities.

  • Takeaways & Limitations

    GAN adoption in medical imaging remains in its infancy, with no breakthrough GAN-based application yet adopted clinically.

Abstract

from arXiv · show

Generative adversarial networks have gained a lot of attention in the computer vision community due to their capability of data generation without explicitly modelling the probability density function. The adversarial loss brought by the discriminator provides a clever way of incorporating unlabeled samples into training and imposing higher order consistency. This has proven to be useful in many cases, such as domain adaptation, data augmentation, and image-to-image translation. These properties have attracted researchers in the medical imaging community, and we have seen rapid adoption in many traditional and novel applications, such as image reconstruction, segmentation, detection, classification, and cross-modality synthesis. Based on our observations, this trend will continue and we therefore conducted a review of recent advances in medical imaging using the adversarial training scheme with the hope of benefiting researchers interested in this technique.

1. Introduction

This review introduces GANs and surveys their growing use in medical imaging. It uses a literature search through January 1, 2019 to organize applications and identify remaining challenges.

  • Motivation: GANs train a generator and discriminator simultaneously, supporting image generation and helping counteract domain shift.The generator produces images while the discriminator distinguishes generated from real images.
  • Scope: The review focuses on GAN applications in medical imaging and identifies challenges relevant to extending their use.
  • Review method: The literature search covered PubMed, arXiv, MICCAI, SPIE Medical Imaging, ISBI, and MIDL, with cross-referenced works added.Preliminary-only arXiv papers were excluded, and the search cutoff was January 1, 2019.
  • Organization: The review categorizes GAN studies by reconstruction, synthesis, segmentation, classification, detection, registration, and other canonical tasks.

2. Background

The background explains vanilla GAN training, its optimization challenges, and major conditional and architectural variants. These variants adapt generation to auxiliary information, image translation, and high-resolution synthesis.

  • Vanilla GAN: A vanilla GAN maps random noise z through generator G to generated samples, while discriminator D distinguishes them from real samples.Successful training aims for the generated distribution pg(x) to approximate the real distribution pr(x).
  • Vanilla GAN: The discriminator learns to classify real and generated images, while gradients from it train the generator to produce samples that can fool D.
  • Vanilla GAN: With optimal discriminator updates, generator training is equivalent to minimizing the Jensen–Shannon divergence between pr(x) and pg(x).
  • Optimization challenges: Imbalanced adversarial training can leave the discriminator too strong, causing near-zero gradients and limited guidance for the generator, especially at high resolution.
  • Optimization challenges: Mode collapse occurs when pg(x) concentrates on a few modes of pr(x), producing a limited set of samples rather than diverse images.
  • Conditional and translation models: Conditional GANs use auxiliary information c, such as labels or images, to guide generation through xg = G(z, c).Image-conditioned frameworks include paired pix2pix and unpaired CycleGAN or UNIT translation.
  • Architectural variants: Progressive and convolutional architectures improve generation of high-resolution images by building outputs or representations incrementally.DCGAN uses fully convolutional downsampling and upsampling layers, while LAPGAN and SGAN use cascaded generation.

3. Applications in Medical Imaging

GANs are used in medical imaging both to generate images and to provide learned priors for discrimination or regularization. These roles address data scarcity, privacy, and abnormal-image analysis.

  • Generative use: The generative use of GANs explores training-data structure and produces new images, making GANs promising for data scarcity and patient privacy.
  • Discriminative use: The discriminative use treats the discriminator as a learned prior for normal images, allowing it to regularize or detect abnormal images.

3.1. Reconstruction

GAN-based reconstruction methods enhance medical images from noisy or incomplete inputs, using task-specific losses and, when available, additional data fidelity constraints. The review also highlights unresolved evaluation and hallucination risks.

  • 3.1. Reconstruction: GAN reconstruction has been applied to low-dose CT, MR, PET, and endomicroscopy imaging tasks.Applications include denoising, reconstruction, and super-resolution across several modalities.
  • 3.1. Reconstruction: Pix2pix frameworks support low-dose CT denoising, MR reconstruction, and PET denoising, while perceptual losses improve similarity to reference images.Focus maps and frequency-domain fidelity can further emphasize important regions or raw K-space consistency.
  • 3.1. Reconstruction: Adversarial loss generally produces more visually appealing reconstructions than pixel-wise loss alone but can hallucinate unseen structures under domain mismatch.Pixel-wise reconstruction loss can reduce hallucination when paired samples are available, but healthy-only training remains vulnerable when reconstructing pathological images.
  • 3.1. Reconstruction: Evaluation lacks reliable comparisons of reconstruction losses, with limited large-scale human-observer analysis and datasets not tailored to downstream medical-image analysis.The review calls for new reference-standard datasets to compare GAN-based reconstruction methods.
  • 3.1. Reconstruction: Figure 5 illustrates reconstruction examples including low-dose CT denoising and vessel-map-to-fundus image generation.The examples show preserved liver contrast regions and retinal vessel structures matching the conditioning map.

3.2. Medical Image Synthesis

Medical image synthesis uses GANs to address privacy, limited positive cases, and augmentation gaps while generating unconditional, cross-modality, and conditioned images. Reviewed studies report benefits for classification and anatomical image translation, with shape constraints used to reduce distortions.

  • 3.2. Medical Image Synthesis: GAN-based synthesis can address diagnostic-image privacy and insufficient positive cases, while conventional augmentation misses protocol, sequence, and pathology variation.GANs are presented as a more generic augmentation solution for medical images.
  • 3.2.1. Unconditional Synthesis: Unconditional synthesis generates images from random noise; DCGAN, WGAN, and PGGAN are commonly used, with PGGAN supporting higher resolutions.Studies often train separate generators for individual classes to support downstream tasks.
  • 3.2.1. Unconditional Synthesis: Combining generated samples with real training data improved both sensitivity and specificity for lesion classification.The reviewed evidence also reports comparable perceived quality to real MR images but discrepancies in anatomical accuracy.
  • 3.2.2. Cross modality synthesis: Cross-modality synthesis can reduce acquisition time and cost while generating training samples whose appearance is constrained by anatomy.Pix2pix is used with co-registered modalities, whereas CycleGAN handles cases where registration is challenging.
  • 3.2.2. Cross modality synthesis: Shape-consistency losses from segmentation networks provide implicit anatomical constraints during cross-modality translation and address geometric distortions.Each segmentor produces semantic labels for the corresponding modality.
  • 3.2. Medical Image Synthesis: Conditioning on segmentation maps, text, locations, or synthetic images enables synthesis in uncommon conditions and can produce coarse-to-fine images with clear tumour boundaries.These conditioning strategies support controlled medical-image generation.

3.3. Segmentation

GANs are used in segmentation to enforce higher-order spatial consistency beyond pixel-wise losses. Reviewed approaches include adaptive similarity measures, multi-scale constraints, and semi-supervised use of unannotated images.

  • 3.3. Segmentation: Pixel-wise segmentation losses do not guarantee spatial consistency, motivating adversarial methods that impose higher-order structure on final segmentation maps.Traditional CRF and graph-cut refinements incorporate spatial correlation but have limitations described in the review.
  • 3.3. Segmentation: Adversarial loss acts as an adaptively learned similarity measure by comparing segmented outputs and annotations in a learned low-dimensional manifold.The review contrasts this with perceptual loss computed by a pretrained classifier on natural images.
  • 3.3. Segmentation: Multi-scale L1 features in the discriminator enforce spatial constraints at different depths and achieved state-of-the-art performance in the BRATS 13 and 15 challenges.Other reviewed work uses unannotated images to compute segmentation maps that confuse the discriminator.
  • 3.3. Segmentation: Adversarial training can also enforce network invariance to small perturbations, reducing overfitting on small datasets.This use differs from enforcing higher-order consistency on the final segmentation maps.

3.4. Classification

GANs support medical-image classification by extracting features, acting as classifiers, augmenting training data, and incorporating unlabeled samples to address domain variation.

  • GANs support classification either by using generator or discriminator features, or by treating the discriminator as a classifier.
  • Adversarial loss can reduce domain overfitting by supplying unlabeled test-domain images to the discriminator.
  • GAN-based augmentation can make classification comparable with a traditional supervised CNN using an order of magnitude less labeled data.
  • A two-stage pipeline separately augments images with GANs and classifies them using a traditional classification network.The stages are trained disjointly, without communication between them.
  • Using a separate DCGAN for each lesion class produced better lesion-classification performance than a unified ACGAN, although the reason remains unexplored.The separate-model design requires N models for N classes and is less memory- and computation-efficient.
  • GAN-generated augmentation may be effective in the medium-data regime but not in high- or low-data regimes.

3.5. Detection

GAN discriminators can detect abnormalities by learning the distribution or manifold of normal medical images; distribution matching that removes lesions can be exploited for this purpose.

  • The discriminator can detect abnormalities by learning the probability distribution of normal pathology and flagging images outside that distribution.
  • A learned manifold of normal anatomical variability supports anomaly scoring based on how well a test image’s latent code fits that manifold.
  • CycleGAN distribution matching may remove lesions when the target domain contains pathology-free medical images.
  • When source and target domains share an imaging modality but differ in normal versus abnormal tissue, lesion removal can instead support abnormality detection.

3.6. Registration

GAN-based registration uses adversarial discrimination to regularize transformations or generate aligned images, supporting multimodal, deformable, and joint registration-segmentation tasks.

  • In cGAN registration, the generator produces transformation parameters, deformation fields, or transformed images, while the discriminator distinguishes aligned from unaligned image pairs.
  • A discriminator regularizing the generator’s displacement field was more effective than other regularizers for MR-to-TRUS registration.
  • CycleGAN has been applied to multimodal retinal and MR registration, with the generator producing both transformed images and deformation fields.
  • Joint segmentation and registration with CycleGAN performed better than separate approaches for lung X-ray images.
  • CycleGAN-based MR-to-CT registration achieved at best similar performance to traditional multimodal deformable registration methods.

3.7. Other works

Beyond canonical imaging tasks, GANs have been used for patient-specific motion modeling, disease-relevant region highlighting, video recolorization, and radiotherapy treatment planning.

  • cGAN can model patient-specific motion distributions from a single preoperative image.
  • GAN-based methods can highlight regions most accountable for a disease.
  • GANs have been used to recolor endoscopic video data.
  • pix2pix has been applied to treatment planning in radiotherapy.

4. Discussion

GAN applications in medical imaging expanded rapidly, especially in image synthesis, reconstruction, and segmentation, while evaluation, data fidelity, and clinical adoption remain unresolved challenges.

  • Studies applying GANs rose significantly in 2017–2018, with image synthesis comprising about 46% of reviewed papers.Cross-modality image synthesis was identified as the most important application, and MR was the most common modality.
  • Another 37% of studies focused on reconstruction and segmentation, where adversarial training regulates generated shape and texture.In liver segmentation from non-contrast CT, adversarial loss significantly improved performance over graph cut and CRF.
  • Classification accounted for 8% of studies, primarily using GANs to address domain shift or generate small aligned objects such as nodules, lesions, and cells.The review reports diminishing returns for GAN-based generation in classification as large labeled datasets become available.
  • Evidence for detection, registration, and other applications was too limited to support conclusions.
  • Quantitative evaluation often relied on MAE, PSNR, or SSIM, although these reference metrics may not correspond to perceived visual quality.The review notes that direct pixel-wise optimization can produce blurry images despite better metric values.
  • Unpaired translation lacks a data-fidelity term, so preservation of small abnormality regions is not guaranteed and generated images should not be used directly for doctors’ interpretation.
Loading 1809.07294v4…