Source-linked AI summary
GANs for Medical Image Analysis
Salome Kazeminia, Christoph Baur, Arjan Kuijper, Bram van Ginneken, Nassir Navab, Shadi Albarqouni, Anirban Mukhopadhyay
TL;DR
Medical image analysis faces scarce labeled data, class imbalance, unlabeled-data and domain-shift challenges. This review synthesizes GAN architectures and medical applications through 2018, covering 77 relevant papers, and discusses their opportunities, shortcomings, and future directions. It highlights GAN use across seven application categories and modalities while noting unresolved evaluation, trust, and training-stability limitations.
Problem
Medical deep learning is constrained by scarce costly labels, class imbalance, available unlabeled data, and domain shift.
Method
The paper reviews GAN-based architectures and medical image-processing applications published through 2018, organizing 77 relevant papers into seven categories.
Results
The review finds GAN methods applied across synthesis, segmentation, reconstruction, detection, de-noising, registration, and classification, including cross-modality synthesis such as MR-to-CT mapping.
Takeaways & Limitations
GANs offer reviewed approaches for enlarging datasets, using unlabeled data, synthesizing across modalities, and supporting multimodal medical-image information.
Takeaways & Limitations
The review identifies inadequate objective denoising metrics, limited clinician trust in generated data, and unstable GAN training as important limitations.
Abstract
from arXiv · showhide
Generative Adversarial Networks (GANs) and their extensions have carved open many exciting ways to tackle well known and challenging medical image analysis problems such as medical image de-noising, reconstruction, segmentation, data simulation, detection or classification. Furthermore, their ability to synthesize images at unprecedented levels of realism also gives hope that the chronic scarcity of labeled data in the medical field can be resolved with the help of these generative models. In this review paper, a broad overview of recent literature on GANs for medical applications is given, the shortcomings and opportunities of the proposed methods are thoroughly discussed and potential future work is elaborated. We review the most relevant papers published until the submission date. For quick access, important details such as the underlying method, datasets and performance are tabulated. An interactive visualization which categorizes all papers to keep the review alive, is available at http://livingreview.in.tum.de/GANs_for_Medical_Applications.
1. Introduction
This review surveys GAN-based architectures for medical image processing, covering their applications, methods, datasets, performance, shortcomings, and future directions. It includes 77 relevant papers spanning seven application categories and multiple imaging modalities.
- The review summarizes GAN-based architectures for medical image processing published through the end of 2018.
- The reviewed methods are categorized into synthesis, segmentation, reconstruction, detection, de-noising, registration, and classification.
- GAN methods were applied across MRI, CT, OCT, chest X-Ray, dermoscopy, ultrasound, PET, and microscopy.
- Tables summarize methods, metrics, datasets, image modalities, architectures, and performance, while a live tree supports paper categorization.
- The paper discusses advantages and shortcomings of reviewed methods and identifies directions for future work.
- 77 papers were selected as relevant, covering a broad spectrum of applications and GAN variants.
2. Opportunities for Medical Image Analysis
GANs are presented as a response to limited labeled medical data, class imbalance, unlabeled-data availability, and domain shift. Their applications include controllable synthesis, cross-modality translation, and learned image similarity.
- Medical deep learning depends heavily on large-scale labeled data, which is costly and scarce, while medical datasets often exhibit class imbalance.
- GANs can generate realistic-looking images to enlarge training datasets and support semi-supervised or unsupervised learning.
- Conditional GANs allow generation to be seeded with class labels, image attributes, or images themselves.
- Adversarial networks can use labeled and arbitrary amounts of unlabeled data, while domain shift can impair generalization beyond training distributions.
- Adversarial networks can function as learned similarity metrics that incorporate spatial context beyond pixel-level ℓ1 or ℓ2 distances.
3. Basic GAN models employed in medical applications
The paper introduces GAN architectures and extensions used in medical applications, from unconditional and conditional generation to cycle-consistent translation and modified adversarial losses. These variants address controllability, image quality, and training stability.
- A GAN uses a generator and discriminator in a two-player minimax game, with the generator mapping latent noise to synthetic data.
- The discriminator distinguishes real from generated data, while the generator learns to make generated samples appear real.
- The vanilla GAN minimizes the Jensen-Shannon divergence between real and synthetic data distributions.
- Vanilla GAN training is sensitive to hyperparameters and can suffer from vanishing or exploding gradients, mode collapse, and mode hopping.
- DCGAN replaces basic network components with deep convolutional architectures, batch normalization, and Leaky-ReLU to improve stability and resolution.
- Conditional GANs provide generation control by jointly supplying noise and prior information to the generator and discriminator.
- Conditioning improves detailed target-image features and training stability.
- Pix2Pix performs paired high-resolution image-to-image translation using a U-Net generator, skip connections, and an ℓ1 loss.
4. Applications in Medical Image Processing
The review organizes GAN-based medical image processing methods into seven application categories and compares their architectures, losses, modalities, datasets, performance, and peer-review status.
- The application review covers synthesis, segmentation, reconstruction, detection, de-noising, registration, and classification.
- Each application subsection summarizes architectures, loss functions, imaging modalities, dataset properties, evaluated performance, and peer-review status.
4.1. Synthesis
GAN-based medical image synthesis spans unconditional and conditional generation across diverse imaging tasks and modalities. Reviewed studies report promising realism and downstream utility, while also highlighting limitations in data requirements, artifact formation, and evaluation validity.
- Unconditional and Conditional Synthesis: The review distinguishes unconditional GANs, which generate images from random noise, from conditional GANs, which use prior knowledge for supervised generation.Conditional inputs can include class labels, image attributes, or images themselves.
- Applications: GAN-based synthesis has been applied to medical images including CT, MR, PET, retinal, microscopy, skin-lesion, lung-nodule, and ultrasound images.Applications include cross-modality translation, staining, lesion synthesis, and simulation.
- Downstream Utility: Synthetic images supported downstream tasks such as segmentation, classification, and tumor detection in several reviewed studies.Synthetic data improved a segmentation model by 16% in one cardiac MR study, while blended cGAN and FCN PET images achieved very high tumor detection performance.
- Reported Realism: Several studies found synthetic images difficult for human observers or experts to distinguish from real images, including MR slices, lung nodules, skin lesions, and intravascular ultrasound.The IVUS study generated samples at 64 × 64 and 256 × 256 pixels that could not reliably be distinguished from real images in a visual Turing test.
- Unpaired Translation: Unpaired CycleGAN methods reduced reliance on paired, co-registered training data for stain transfer and cross-domain medical image synthesis.Paired datasets can be difficult to obtain, and imperfect co-registration can introduce artifacts.
- Limitations and Evaluation: The review cautions that synthesis validity often remains uncertain because artifacts can make generated images distinguishable from real images and expert feedback may still be required.The authors call for more elaboration and evaluation of method validity.
4.2. Segmentation
GAN-based segmentation methods address challenges across brain, chest, eye, abdominal, spine, and microscopic medical images. Reviewed approaches combine adversarial training with specialized architectures or losses, but evaluation and detail preservation remain difficult.
- Segmentation is essential for detection, classification, and shape analysis, while manual annotation is tedious and time-consuming.
- Adversarial strategies support supervised, semi-supervised, unsupervised, and domain-adaptive segmentation across varied anatomical and imaging settings.Examples include brain domain adaptation, synthetic CT generation from MRI, and cycleGAN or LSGAN approaches.
- Human-level heart/lung segmentation was achieved with SCAN on chest X-rays by addressing instability with generator pre-training using pixel-wise loss.
- GAN-based methods combine adversarial training with pixel- or voxel-wise losses, while U-Net and ResNet are common generator architectures.CycleGAN and reconstruction losses are also used to incorporate non-ROI features.
- Reviewed methods train with 10 to 1000+ input samples, but inconsistent reporting and reliance on debatable Dice-only evaluation hinder clear judgment.
- Microscopic-image segmentation showed significant enhancement with GANs, although overall comparisons between adversarial and non-adversarial segmentors were often close.
4.3. Reconstruction
GAN-based reconstruction targets fast MR acquisition and reconstruction without sacrificing detail, using modified architectures and added losses. The reviewed methods emphasize perceptual, frequency, pixel-wise, and regularization objectives, with ResNet commonly used in generators.
- Fast MR reconstruction is desirable because rapid acquisition and reconstruction directly reduce motion artifacts.
- GAN-based MR reconstruction primarily modifies established architectures and combines them with task-specific loss functions.
- DAGAN-based strategies: DAGAN-based methods add perceptual losses to adversarial and pixel-wise losses, with later variants targeting frequency preservation and training refinement.
- 3D super-resolution: Three-dimensional approaches use 3D convolutions or 3DSRGAN objectives to improve volumetric reconstruction, reduce blurring, control interpolation, and limit over-fitting.
- Other reconstruction strategies: Other reconstruction methods add pixel-wise, perceptual, saliency, Voronoi-vectorization, or regularization losses to adversarial objectives.
- ResNet is the most popular generator architecture in the reviewed reconstruction methods, all of which include pixel-wise and adversarial losses.
4.4. Detection
GANs address medical-image anomaly detection by augmenting scarce annotated datasets or modeling normal-image distributions. Reviewed studies report effective anomaly detection, but heterogeneous datasets and metrics limit fair comparison.
- GANs tackle annotation demands in supervised anomaly detection by generating synthetic samples or modeling distributions for outlier detection.
- Unsupervised anomaly detection: AnoGAN reconstructs the most similar normal image to a query, allowing anomalies to be identified from discrepancies between the two images.
- Unsupervised anomaly detection: AnoGAN detected retinal OCT anomalies, while adaptations targeted brain MR images and VA-GAN extracted disease-related change maps for Alzheimer’s detection.
- Unsupervised anomaly detection: VA-GAN also detected low-contrast Alzheimer’s lesions, beyond the high-contrast lesions detected by the referenced CNNs.
- Detection methods show good anomaly-detection performance while reducing training-data requirements, but varying datasets and metrics impede fair comparison.
4.5. De-noising
GAN-based denoising seeks to preserve texture and sharpness while improving low-contrast medical images, addressing the blurring common in deep-learning reconstruction. The review reports compelling visual results but identifies a lack of adequate objective evaluation metrics.
- Radiological imaging trades image contrast against radiation exposure, while reduced radiation can produce lower contrast and lower signal-to-noise ratios.
- GAN-based denoising methods learn tissue texture, remove CT metal artifacts, improve training stability, and address blurring through perceptual or sharpness-aware objectives.
- Visual results from reviewed denoising methods are compelling, but commonly used PSNR, MSE, SSIM, SD, and mean metrics are not sensitive enough to texture details.
- Region-of-interest segmentation is recommended for measuring preserved medical information, but it is expensive and motivates development of a meaningful new metric.
- Faster and more accurate denoising frameworks remain an open direction despite reported benefits from domain-feature learning and texture-aware losses.
4.6. Registration
GANs have been applied to medical image registration and deformation correction, using learned representations to model image discrepancies. Although they improve registration performance, the review notes that real medical settings may still require higher performance.
- Motivation: GANs emerged as candidates for medical image registration because they can learn image-transformation mappings in a single forward pass.This addresses the parameter dependency and heavy optimization load of traditional registration methods.
- Methods: Fan et al. proposed an unsupervised GAN to register structural patterns defined in patches across different brain images.
- Methods: Yan et al. and Hu et al. proposed GAN-based methods for registration and deformation correction in 3D prostate MRI and intra-procedural TRUS.In Yan et al., the discriminator also served as a certainty evaluator during testing.
- Advantages: GANs provide local and global feature information for registration by learning features at multiple scales to model distribution discrepancies.
- Limitations: GANs significantly enhance registration performance, but the required performance may still not be reached in some real medical settings.
4.7. Classification
GAN-based classification methods address limited medical training data by augmenting datasets and extracting class-specific features. Reviewed results indicate that classification tasks benefit from GAN-generated samples.
- Motivation: GANs can potentially mitigate classification data limitations by augmenting training data and extracting domain-specific features for each class.CNN classifiers require large training datasets, while medical data are difficult to access.
- Methods: SCGAN uses two generators to produce useful and non-informative cardiac ultrasound samples, while its discriminator distinguishes both authenticity and sample type.
- Methods: Ren et al. used a GAN to extract class-specific features for classifying prostate histopathology patches into two Gleason grades.They added a classification loss to prevent patches from the same image receiving different Gleason-grade labels.
- Results: Classification tasks benefit from GAN-generated samples according to the reviewed performance results.
5. Discussion
The review highlights GANs' broad use across medical imaging tasks and their advantages in realistic synthesis, feature learning, and data augmentation. It also identifies major barriers involving trust, training stability, evaluation, interpretability, and practical deployment.
- Scope: GAN-based medical imaging research spans synthesis, segmentation, reconstruction, classification, denoising, registration, and detection applications.The reviewed architectures include examples for synthesis, segmentation, reconstruction, denoising, and classification.
- Advantages: GANs mimic data distributions, produce realistic images, and learn rich similarity metrics that benefit discriminative deep-learning frameworks.
- Advantages: GANs can enlarge training datasets, support semi- and unsupervised settings, enable cross-modality synthesis, and help detect unseen abnormalities.The review gives MR-to-CT synthesis as an example of cross-modality translation.
- Advantages: Adversarial feature extraction has been leveraged in segmentation, registration, denoising, and classification applications.
- Drawbacks: Generated medical data raise trust concerns because GAN mechanisms are poorly understood and synthesized intensities may lack clinically meaningful mappings such as CT Hounsfield units.
- Drawbacks: GAN training can be unstable, causing mode collapse or mode-hopping, while unclear medical-image modes make unstable or unrealistic outputs difficult to identify.The review also notes a lack of proper evaluation tools and comparable medical benchmarks.
- Drawbacks: Medical GAN evaluation is difficult because pixel-wise metrics may not capture GAN performance and often require ground-truth images unavailable for unsupervised methods.
- Drawbacks: GANs remain difficult to interpret, limiting their reliability and practical application in medical environments.