Source-linked AI summary
Generation of 3D Brain MRI Using Auto-Encoding Generative Adversarial Networks
Gihyun Kwon, Chihye Han, Dae-shik Kim
TL;DR
Limited and costly medical imaging data make it difficult to train deep models, while many GAN approaches require extensive data from paired domains. The paper proposes an α-GAN-based 3D model with WGAN-GP to generate brain MRI from random vectors. It reports realistic, diverse whole-brain images across normal and diseased types and modalities, outperforming alternative structures in distribution capture and sample diversity.
Problem
Medical image analysis lacks sufficient data, and image-to-image GAN methods require extensive datasets from different domains.
Method
The model adapts α-GAN for 3D MRI generation, combining VAE and GAN components with a code discriminator and WGAN-GP losses.
Results
The model outperforms alternative structures in capturing real data distributions and generating diverse samples across multiple brain MRI types and modalities.
Takeaways & Limitations
The results suggest that the model can support efficient augmentation of 3D brain MRI data for applications such as disease diagnosis.
Takeaways & Limitations
Future work is needed to generate difficult disease images such as Alzheimer’s and demonstrate improved diagnostic performance using generated images.
Abstract
from arXiv · showhide
As deep learning is showing unprecedented success in medical image analysis tasks, the lack of sufficient medical data is emerging as a critical problem. While recent attempts to solve the limited data problem using Generative Adversarial Networks (GAN) have been successful in generating realistic images with diversity, most of them are based on image-to-image translation and thus require extensive datasets from different domains. Here, we propose a novel model that can successfully generate 3D brain MRI data from random vectors by learning the data distribution. Our 3D GAN model solves both image blurriness and mode collapse problems by leveraging alpha-GAN that combines the advantages of Variational Auto-Encoder (VAE) and GAN with an additional code discriminator network. We also use the Wasserstein GAN with Gradient Penalty (WGAN-GP) loss to lower the training instability. To demonstrate the effectiveness of our model, we generate new images of normal brain MRI and show that our model outperforms baseline models in both quantitative and qualitative measurements. We also train the model to synthesize brain disorder MRI data to demonstrate the wide applicability of our model. Our results suggest that the proposed model can successfully generate various types and modalities of 3D whole brain volumes from a small set of training data.
1 Introduction
Medical image analysis benefits from deep learning but remains constrained by the limited, costly supply of medical images. The paper addresses this gap with a 3D GAN that generates brain MRI from random vectors rather than relying on extensive paired domains.
- Medical image datasets are laborious and costly to obtain, limiting CNN-based medical image analysis.
- Geometric augmentation produces outputs that remain highly dependent on the original data.
- Image-to-image GAN translation can produce realistic medical images but requires extensive training data and inherits attributes such as shape from the source data.
- The proposed 3D GAN generates 3D brain MRI from random vectors using an adapted α-GAN and WGAN-GP loss.
- The model is trained on normal, tumor, and stroke MRI to generate realistic whole-brain images across types and modalities from small datasets.
2 Methods
The method adapts α-GAN to 3D brain MRI generation, combining auto-encoding and adversarial components to address blurriness and mode collapse. WGAN-GP losses and a large latent space are used to improve training stability and image variety.
- Model Architecture: α-GAN combines VAE and GAN components with a code discriminator to address VAE blurriness and GAN mode collapse.
- Model Architecture: The 3D adaptation uses convolutional encoder and discriminator networks and resize-convolution in the generator to reduce parameters and checkerboard artifacts.
- Loss Function: A latent vector size of 1000 is used because vectors of approximately 100 lacked sufficient capacity to reflect image variety.
- Loss Function: WGAN-GP gradient penalties replace gradient clipping to maintain the 1-Lipschitz condition during training.
- Loss Function: The loss comprises four network-specific terms, including reconstruction loss for the generator and gradient penalties for the discriminator and code discriminator.
- Training: Training jointly optimizes encoder-generator, discriminator, and code discriminator networks, with the generator updated twice per step.
3 Dataset and Preprocessing
The study uses preprocessed 3D MRI datasets spanning normal, tumor, and stroke cases. Images are stripped of non-brain regions, trimmed, and resized to standardized 64×64×64 volumes.
- Normal MRI Data: The normal MRI task uses 991 control-normal T1 structural images from the ADNI dataset.
- Normal MRI Data: Non-brain areas are removed, redundant all-zero planes are trimmed, and images are resized to 64×64×64.
- Diseased MRI Data: Tumor MRI data include T1, T1ce, T2, and FLAIR modalities, with FLAIR and T2 selected for tumor generation.
- Diseased MRI Data: The ATLAS dataset contributes 220 T1w images containing diverse stroke lesions.
- Diseased MRI Data: Stroke and tumor images undergo non-brain removal, redundant-plane trimming, and resizing to 64×64×64.
4 Experiments and Results
Experiments compare the proposed 3D-α-WGAN-GP model with alternative generators using quantitative distribution and diversity measures, PCA visualizations, and diseased MRI synthesis. The model most closely matches real-data distributions, produces diverse samples, and generates plausible tumor and stroke images across modalities.
- Quantitative Results: The proposed model achieves the lowest MMD scores, indicating the closest generated-sample distribution to real data.MMD measures distributional similarity between generated and real batches.
- Quantitative Results: 3D-WGAN-GP and 3D-VAE-GAN show mode collapse, whereas α-GAN variants generate more diverse samples with similarity scores closer to real data.The model with an overly small latent vector also fails to escape mode collapse.
- Qualitative Results: PCA shows the proposed model has the most similar distribution range to real data among the compared baseline models.3D-WGAN-GP covers only a small range, 3D-VAE-GAN has limited variety, and 3D-α-GAN remains distributionally mismatched.
- Latent Vector Size: A latent vector size of 100 causes severe mode collapse, while a moderately large size provides the best results.Very large latent vectors can produce samples outside the real-data distribution.
- Diseased MRI Generation: The model generates realistic tumor MRI samples with lesions at varied positions while reflecting different modalities.Tumor-T2 and Tumor-FLAIR datasets are used for severe tumor cases.
- Diseased MRI Generation: For stroke MRI, the model generates plausible T1w samples with damaged areas in varied positions, whereas other models produce severely deteriorated images.The stroke cases contain large lesions and differ substantially from tumor features.
5 Conclusion
The paper presents an auto-encoder-based GAN that generates realistic and diverse 3D brain MRI from small training sets across data types and modalities. The authors suggest its use for data augmentation, while diagnostic-performance improvement remains future work.
- Conclusion: The proposed auto-encoder-based GAN generates realistic 3D brain MRI data from a small number of training samples.The conclusion reports realistic generation across the studied settings.
- Conclusion: The model outperforms alternative structures in capturing the real-data distribution and generating diverse samples.The conclusion summarizes both distribution matching and sample diversity as demonstrated outcomes.
- Conclusion: The model generates images that reflect the attributes of different data types and can support efficient 3D brain MRI data augmentation.The authors identify disease diagnosis as one possible application.
- Future Work: Improvement in diagnostic performance using generated images is left for future work.The authors also plan to generate disease images that are difficult to obtain through image-to-image translation.
Supplementary Material
The supplementary figures compare generated brain MRI samples with real data across normal, diseased, tumor-FLAIR, tumor-T2, and stroke-T1w settings. Views span sagittal, coronal, and axial planes, while PCA assesses distributional similarity for tumor samples.
- Normal brain MRI: Normal-brain slices compare real data, Ours, 3D-α-GAN, 3D-VAE-GAN, and 3D-WGAN-GP across sagittal, coronal, and axial planes.Numbers below the images indicate slice numbers.
- Diseased brain MRI: Diseased-brain center-cut views show severe deterioration for 3D-α-GAN and 3D-VAE-GAN, while 3D-WGAN-GP produces little variance from mode collapse.
- Tumor MRI: PCA plots compare tumor-FLAIR and tumor-T2 samples from real data, baseline models, and Ours.The plots report that Ours has the closest distribution to real data.
- Tumor MRI: Tumor-FLAIR samples from real data and Ours are shown along sagittal, coronal, and axial planes.
- Tumor MRI: Tumor-T2 samples from real data and Ours are shown along sagittal, coronal, and axial planes.
- Stroke MRI: Stroke-T1w samples from real data and Ours are shown along sagittal, coronal, and axial planes.