Source-linked AI summary
Medical Image Synthesis for Data Augmentation and Anonymization using Generative Adversarial Networks
Hoo-Chang Shin, Neil A Tenenholtz, Jameson K Rogers, Christopher G Schwarz, Matthew L Senjem, Jeffrey L Gunter, Katherine Andriole, Mark Michalski
TL;DR
Medical imaging suffers from rare abnormalities, class imbalance, limited variability, and costly three-dimensional annotation, while patient-data restrictions hinder sharing. The paper trains a GAN on public brain-MRI datasets to generate varied synthetic tumor images from segmentation masks for augmentation and anonymization. Synthetic data improve segmentation performance and, with sufficient synthetic volume plus limited real-data fine-tuning, can achieve performance comparable to training on real data.
Problem
Rare abnormalities, class imbalance, limited variability, costly three-dimensional annotation, and patient-data restrictions limit medical-imaging training data and sharing.
Method
The paper uses an image-to-image translation GAN to generate varied abnormal multi-parametric brain MRIs from segmentation masks using public ADNI and BRATS data.
Results
Synthetic data improve segmentation performance, while more than five times as much synthetic data followed by fine-tuning on 10% real training data achieves performance comparable to training on real data.
Takeaways & Limitations
The synthetic images provide a potential source of diverse augmentation data and anonymized training data that can be shared outside an institution.
Takeaways & Limitations
The GAN operates at half resolution and trains on only T1-weighted sequences, which may limit segmentation gains; higher resolution and additional sequences remain avenues for improvement.
Abstract
from arXiv · showhide
Data diversity is critical to success when training deep learning models. Medical imaging data sets are often imbalanced as pathologic findings are generally rare, which introduces significant challenges when training deep learning models. In this work, we propose a method to generate synthetic abnormal MRI images with brain tumors by training a generative adversarial network using two publicly available data sets of brain MRI. We demonstrate two unique benefits that the synthetic images provide. First, we illustrate improved performance on tumor segmentation by leveraging the synthetic images as a form of data augmentation. Second, we demonstrate the value of generative models as an anonymization tool, achieving comparable tumor segmentation results when trained on the synthetic data versus when trained on real subject data. Together, these results offer a potential solution to two of the largest challenges facing machine learning in medical imaging, namely the small incidence of pathological findings, and the restrictions around sharing of patient data.
1 Introduction
The paper addresses scarce, imbalanced, and costly-to-annotate medical imaging data by generating synthetic brain MRI with tumors using a GAN. It positions this approach as both data augmentation and anonymization, while extending prior medical image synthesis work toward tumor segmentation.
- Motivation: Abnormal findings are uncommon, and manually segmenting a three-dimensional abnormality can require upwards of fifteen minutes per study.Traditional crops, translations, and rotations can mitigate some data limitations but produce highly correlated training data.
- Approach: The paper generates synthetic abnormal multi-parametric brain MRIs from brain-anatomy and tumor segmentation masks using a GAN.Changing the input labels can vary tumor size or location, or place a tumor in an otherwise healthy brain while retaining the corresponding annotation.
- Benefits: Synthetic images provide an additional form of data augmentation and can serve as an anonymization method for sharing hospital-trained data outside the institution.The proposed use is intended to provide an automatable, low-cost source of diverse training data.
- Related work: Prior work evaluated synthetic images with MSE, SSIM, and PSNR, but not directly on diagnostic quality, and related studies used different datasets or retinal images.The paper uses publicly available ADNI and BRATS data for multi-parametric MRI synthesis.
- Novelty: The authors identify no existing literature combining synthetic medical-image anonymization and data augmentation for tumor segmentation tasks.This frames the paper’s stated novelty within the reviewed GAN literature.
2 Data
The study uses two publicly available brain-MRI datasets, ADNI and BRATS, with preprocessing and split procedures tailored to their differing imaging characteristics. BRATS supplies multi-series tumor studies and voxel-wise annotations, while ADNI supports brain-anatomy processing.
- Datasets: The study uses two publicly available brain-MRI datasets: ADNI and BRATS.
- ADNI: ADNI combines serial MRI, PET, biological markers, and clinical and neuropsychological assessments to study mild cognitive impairment and early Alzheimer’s disease.The paper uses an established approach to segment white matter, gray matter, and cerebrospinal fluid in ADNI T1-weighted images.
- BRATS: BRATS contains multi-institutional pre-operative MRIs focused on gliomas, with T1, T2, contrast-enhanced T1, and FLAIR series plus voxel-wise annotations.The annotations cover edema, enhancing tumor, and non-enhancing tumor.
- Preprocessing and split: BRATS 2015 provides 264 studies, split into 80% training and 20% testing, with the test set evaluated once for each assessed algorithm and setting.The GAN training data are cropped to central slices and resampled to 128×128×54 because of memory and compute constraints.
3 Methods
The method uses conditional GANs to translate between MRI and anatomical or tumor labels, then modifies label maps to generate varied synthetic tumor MRIs. These images support augmentation and anonymization, while image quality and brain-segmentation generalization remain constrained.
- GAN architecture and tasks: Pix2pix conditional GANs translate T1-weighted MRI into brain labels and labels into synthetic MRI, with a separate MRI-to-tumor segmentation task.The generator and discriminator are trained for forward and inverse image-label translation tasks.
- Label construction: ADNI-derived brain anatomy labels are combined with BRATS tumor labels to create complete brain-with-tumor segmentations.The ADNI model segments normal anatomy, while BRATS supplies tumor annotations.
- Synthetic-image generation: Tumor size, location, and placement on tumor-free brains are altered before label-to-MRI synthesis to generate controlled variation.The workflow can shift, enlarge, or shrink tumors and place tumor labels on ADNI brain labels.
- Limitations: The study did not evaluate brain-segmentation performance, and image quality still requires higher resolution, more non-T1 data, and cross-dataset generalization analysis.The authors also identify synthetic tumor boundaries as an area needing improvement.
- Image quality: Generated T1 images were qualitatively difficult to distinguish from originals, whereas T2 and FLAIR images were easier to distinguish from scanner-acquired images.The authors attribute this modality difference to the fundamentally different T2 and FLAIR acquisition processes and note that larger modality-specific training sets may help.
- Data augmentation: Synthetic image-label pairs expand training data with desired tumor characteristics without labor-intensive manual annotation.The approach complements conventional augmentation methods such as cropping, rotation, translation, and elastic deformation.
4 Experiments and Results
Experiments evaluate tumor segmentation with real, synthetic, and combined training data. Synthetic data improves performance, but resolution and sequence limitations may constrain gains relative to conventional augmentation.
- 4.1 Data Augmentation using Synthetic Data: Synthetic data substantially improves segmentation performance when conventional augmentation is omitted.The improvement is observed for the GAN-based model and also benefits the comparison model when synthetic data is added.
- 4.1 Data Augmentation using Synthetic Data: Synthetic data produces only a small additional increase when conventional augmentation is already used.The comparison includes cropping, rotation, translation, and elastic deformation.
- 4.1 Data Augmentation using Synthetic Data: The proposed model uses half-resolution volumes because of GPU memory limits, and down-/up-sampling may lose very small tumors.The authors hypothesize that limited resolution and using only T1-weighted training sequences contributed to the relatively small performance increase versus usual augmentation.
- 4.2 Training on Anonymized Synthetic Data: Training on synthetic data alone is sub-optimal at the real-data training-set size.The evaluation uses synthetic data without including any original training data.
- 4.2 Training on Anonymized Synthetic Data: More than five times as much synthetic data, followed by fine-tuning on 10% of the real training data, achieves performance comparable to training on real data.Synthetic data functions as pre-training in this setting.
5 Conclusion
The paper proposes a GAN that generates varied synthetic abnormal brain-tumor MRI images from segmentation masks. These images improve segmentation across multiple algorithms and support training on anonymized data for sharing.
- 5 Conclusion: A generative algorithm produces synthetic abnormal multi-parametric brain-tumor MRI images from corresponding segmentation masks.The method uses an image-to-image translation GAN.
- 5 Conclusion: Altering the input label map introduces high variation into the generated images and improves segmentation performance across multiple algorithms.
- 5 Conclusion: The same algorithms can be trained on completely anonymized data sets, allowing training data to be shared.The paper connects this capability with smaller institution-specific data sets.