Source-linked AI summary
Deep Learning Approaches for Data Augmentation in Medical Imaging: A Review
Aghiles Kebaili, Jérôme Lapuyade-Lahorgue, Su Ruan
TL;DR
Limited medical imaging training data and the shortcomings of traditional augmentation motivate the use of deep generative models. This review synthesizes evidence on VAEs, GANs, and diffusion models across medical imaging applications, concluding that they can generate realistic samples while retaining model-specific limitations and open research challenges.
Problem
Medical imaging deep learning is constrained by limited training data, while traditional augmentation often produces limited and unconvincing results.
Method
The review surveys VAEs, GANs, and diffusion models for medical image augmentation across classification, segmentation, and cross-modal translation.
Results
Deep generative models offer advantages over traditional augmentation by generating realistic new images that capture the underlying training-data distribution.
Takeaways & Limitations
These models have potential for improving deep learning algorithms in medical image analysis across multiple downstream tasks.
Takeaways & Limitations
Generative models face a trade-off among sample quality, sampling speed, and diversity, while diffusion models additionally have long sampling times.
Abstract
from arXiv · showhide
Deep learning has become a popular tool for medical image analysis, but the limited availability of training data remains a major challenge, particularly in the medical field where data acquisition can be costly and subject to privacy regulations. Data augmentation techniques offer a solution by artificially increasing the number of training samples, but these techniques often produce limited and unconvincing results. To address this issue, a growing number of studies have proposed the use of deep generative models to generate more realistic and diverse data that conform to the true distribution of the data. In this review, we focus on three types of deep generative models for medical image augmentation: variational autoencoders, generative adversarial networks, and diffusion models. We provide an overview of the current state of the art in each of these models and discuss their potential for use in different downstream tasks in medical imaging, including classification, segmentation, and cross-modal translation. We also evaluate the strengths and limitations of each model and suggest directions for future research in this field. Our goal is to provide a comprehensive review about the use of deep generative models for medical image augmentation and to highlight the potential of these models for improving the performance of deep learning algorithms in medical image analysis.
1. Introduction
Medical imaging deep learning often operates with limited training data, making augmentation important, while traditional methods may produce limited and unconvincing results. This review examines deep generative models as a way to generate more realistic data and surveys their applications across medical imaging tasks.
- Motivation: Limited training samples make deep learning applications challenging in medical imaging, where modalities are high-dimensional and some diseases are rare.The passage identifies MRI, CT, and PET as examples of medical imaging modalities.
- Motivation: Data augmentation artificially synthesizes new samples to increase training-set size when rich datasets are unavailable.It is also used when modalities are missing in multimodal image segmentation.
- Motivation: Traditional augmentation schemes can produce limited and unconvincing results, motivating methods that reproduce the real data distribution more closely.The stated goal is to make newly sampled data difficult to distinguish from real data.
- Review scope: The review focuses on VAEs, GANs, and diffusion models for medical image augmentation and examines their use across classification, segmentation, and translation tasks.The review selected 72 publications after database searches and manual filtering for relevance.
- Review scope: The literature spans multiple imaging modalities and downstream applications, with CT and MRI prominent and segmentation and classification among the most common tasks.Cross-modal translation counts both source and target modalities in the publication distribution.
2. Background
Deep generative models learn mappings from low-dimensional latent variables or noise to high-dimensional medical data, enabling synthetic samples that resemble the underlying distribution. The background contrasts GANs, VAEs, and diffusion models through their training mechanisms, sampling behavior, and trade-offs among quality, speed, and diversity.
- General framework: Deep generative models learn a mapping from a low-dimensional latent vector z to a high-dimensional data point x resembling the real data distribution.The latent vector is sampled from a prior distribution, and the learned mapping generates new data points.
- Generative Adversarial Networks: GANs train a generator and discriminator adversarially so generated data become difficult to distinguish from real data.The generator maps a random latent vector to data, while the discriminator classifies inputs as real or generated.
- Variational Autoencoders: VAEs use variational inference and an encoder-decoder structure to reconstruct data while learning a compact latent representation.Their evidence lower bound separates reconstruction and regularization terms, and the decoder generates samples from latent vectors.
- Diffusion Models: Diffusion models gradually add Gaussian noise through a Markov chain and learn the reverse denoising process to reconstruct data from noise.The reverse process maps x_T sampled from N(0, I) back to the initial data point x_0 through multiple steps.
- Model trade-offs: VAEs, GANs, and diffusion models trade off fast sampling, sample quality, and coverage of data modes.GANs can generate realistic images quickly but may suffer mode collapse; VAEs offer coverage and smooth latent representations but often produce blurry samples.
- Model trade-offs: Diffusion models can represent complex high-dimensional distributions and produce representative samples, but their reverse process is computationally intensive and slow.Sampling may require several hundreds to thousands of steps, motivating variants designed to improve speed while preserving quality and diversity.
3. Deep Generative Models for Medical Image Augmentation
This section reviews VAEs, GANs, and diffusion models for synthetic medical-image augmentation across classification, segmentation, and cross-modal translation. Reported studies show task-specific gains, while the review also highlights differences in adoption and architecture design.
- Generative Adversarial Networks: 62% versus 54%: WGAN deceived physicians more successfully than DCGAN for most evaluated MRI sequences.The exception was FLAIR images; combining PGGAN with traditional augmentation slightly improved classifier performance over PGGAN alone.
- Generative Adversarial Networks: 85.7% sensitivity and 92.4% specificity: adding ACGAN-generated liver lesions improved classification over conventional augmentation alone.The conventional-augmentation baseline achieved 78.6% sensitivity and 88.4% specificity.
- Generative Adversarial Networks: GAN-based augmentation spans classification, segmentation, and cross-modal translation, with conditional synthesis especially useful for segmentation and translation.Two-stage pipelines can generate synthetic images conditioned on segmentation masks.
- Variational Autoencoders: 3.72% and 3.17%: conditional WGAN and CVAE augmentation, respectively, improved classification accuracy on 3D functional MRI data.The generated samples were described as diverse and high quality.
- Variational Autoencoders: VAEs appear in fewer medical-image augmentation studies than GANs, with nearly half of the VAE studies using hybrid adversarial architectures.Common approaches include conditional VAEs and hybrids such as IntroVAE, PAVAE, and ALVAE.
- Diffusion Models: Diffusion-model studies in the review are recent, with the earliest dating to 2022, and they received more attention in 2022 than GANs and VAEs.Examples include latent diffusion for high-resolution 3D MR synthesis, brainSPADE for labeled brain MRI, and conditional diffusion for MRI-to-CT translation.
4. Key Findings and Implications
The review finds that VAEs, GANs, and diffusion models offer complementary approaches to medical image augmentation, but each faces trade-offs involving realism, diversity, stability, complexity, or sampling speed. Diffusion models show particularly strong recent potential, while domain coverage and evaluation practices remain important considerations.
- The review centers on VAEs, GANs, and diffusion models, whose strengths and limitations reflect the trade-off between quality, sampling speed, and mode coverage.The authors describe this as the generative learning trilemma.
- Basic augmentation operations can produce irrelevant or deformed medical images because anatomical structures and tumor shapes are complex.The review identifies this limitation as a motivation for more capable generative approaches.
- GANs: GANs have been widely applied across modalities and tasks, with reported improvements of 3% in fMRI classification accuracy and 9.1% in Dice score for segmentation and translation.DCGAN and WGAN were commonly used for classification, while Pix2Pix, CycleGAN, and SPADE were promising for conditional generation.
- VAEs: VAEs appear in fewer augmentation studies, but hybrid and improved variants have shown potential for classification, segmentation, translation, and higher-quality image generation.A Riemannian-manifold VAE reportedly improved accuracy by 8% using synthetic images, while hybrid models can be complex and challenging to train.
- Diffusion models: Diffusion models have attracted increasing attention since 2022 and can synthesize realistic images with good mode coverage and stable training, although sampling is slow.DDPM and LDM are prevalent variants; a text-conditioned LDM study reported a 5.8% accuracy improvement for lesion identification.
- Applications and implications: The reviewed studies span MRI, CT, ultrasound, dermoscopy, and otoscopy, with classification most common and conditional synthesis used for controlled generation and segmentation masks.The review also emphasizes incorporating anatomical and physiological knowledge to improve realism and utility.
5. Conclusions
Deep generative models can improve medical image augmentation by producing realistic images that capture the training-data distribution, but each model family has important limitations. The review highlights VAEs, GANs, and diffusion models while identifying directions for further research.
- Deep generative models can generate realistic new images that capture the underlying distribution of the training dataset.
- VAEs: VAEs learn meaningful and disentangled data representations that support interpretability and latent space manipulation.
- VAEs: VAEs may produce fuzzy images lacking important details, motivating improved variants such as vector quantized VAEs.
- The review evaluates VAEs, GANs, and diffusion models for medical image augmentation and discusses their strengths, limitations, and future research directions.