Source-linked AI summary
Synthetic Medical Images from Dual Generative Adversarial Networks
John T. Guibas, Tejpal S. Virdi, Peter S. Li
TL;DR
Medical imaging research is constrained by scarce, private, and insufficiently annotated data. The paper proposes a two-stage GAN pipeline that separates segmentation geometry from photorealistic rendering, and reports synthetic-data segmentation performance close to DRIVE-trained performance while creating larger privacy-preserving datasets.
Problem
Privacy and proprietary restrictions, consent requirements, and limited dataset size and annotation restrict public medical imaging research.
Method
The pipeline uses Stage-I to generate varied segmentation geometries and Stage-II, a conditional GAN, to translate masks into photorealistic images.
Results
An F1 accuracy of 0.8877 for a synthetically trained u-net compared with 0.8988 for a DRIVE-trained u-net showed a negligible difference.
Takeaways & Limitations
The pipeline produces larger quantities of original synthetic images that retain the real dataset’s general statistical distribution without depicting real patients.
Takeaways & Limitations
Using the pipeline on different datasets may require hyperparameter tuning, and more varied data would help develop natural backgrounds.
Abstract
from arXiv · showhide
Currently there is strong interest in data-driven approaches to medical image classification. However, medical imaging data is scarce, expensive, and fraught with legal concerns regarding patient privacy. Typical consent forms only allow for patient data to be used in medical journals or education, meaning the majority of medical data is inaccessible for general public research. We propose a novel, two-stage pipeline for generating synthetic medical images from a pair of generative adversarial networks, tested in practice on retinal fundi images. We develop a hierarchical generation process to divide the complex image generation task into two parts: geometry and photorealism. We hope researchers will use our pipeline to bring private medical data into the public domain, sparking growth in imaging tasks that have previously relied on the hand-tuning of models. We have begun this initiative through the development of SynthMed, an online repository for synthetic medical images.
1 Introduction
Medical imaging research needs larger public datasets, but privacy, consent, proprietary access, and limited annotations restrict available data. The paper proposes synthetic medical images and SynthMed as a public repository to address this constraint.
- Privacy and proprietary restrictions limit public access to medical images, while available datasets often lack sufficient size and expert annotations.
- ImageNet’s error rate fell from 28.2% to 6.7% between 2010 and 2014, illustrating the value of large, accurate public datasets.
- Medical imaging research has often hand-tuned models instead of addressing limited data access, restricting broader field development.
- The proposed pipeline generates synthetic medical images for a public, extensive dataset without privacy concerns, implemented through SynthMed.
2 Related Works
Prior synthetic-data work brought private scalar data into the public domain, while this paper explores synthetic images using GAN-based generation. Its pipeline is illustrated and applied to retinal fundi images.
- Earlier synthetic-data applications primarily publicized private scalar quantities, whereas this paper investigates synthetic data for images.
- GANs offer realistic image generation but can produce artifacts and noise because of training instabilities.
- The authors train Stage-I on DRIVE vessel segmentations and Stage-II on MESSIDOR masks paired with photorealistic images.
- The proposed two-GAN pipeline generates segmentation masks with varied geometries, then translates those masks into photorealistic images.
- The paper illustrates the pipeline using retinal fundi images.
5 Generative Adversarial Network
A GAN trains a generator and discriminator in competition: the discriminator judges realism while the generator learns to produce data classified as real. The architecture is used to generate varied realistic medical images while upholding privacy.
- A GAN consists of a discriminator and generator that compete during training.
- The discriminator classifies generated data as realistic or unrealistic, while the generator produces data that the discriminator will classify as real.
- The generator begins with random noise, upsamples it through learned nonlinear transformations, and can produce more images than the original dataset.
- The paper applies this architecture to generate a wider variety of realistic medical images while upholding patient privacy.
6 Stage-I GAN
Stage-I uses a DCGAN to generate varied retinal vessel segmentation masks from noise. Its adversarial losses train the generator and discriminator competitively.
- Stage-I generates varied segmentation masks representing the dataset’s variable geometries using a DCGAN architecture.
- The generator converts a noise vector into a synthetic image through multiple strided convolutions.
- The Stage-I discriminator is trained with cross-entropy loss.
- The Stage-I loss notation defines D as discriminator, G as generator, m as mini-batch size, z as noise, x as image, and i as image index.
- The generator and discriminator continually compete to minimize their respective connected loss functions.
7 Stage-II GAN
Stage-II GAN translates segmentation masks into corresponding photorealistic medical images using a conditional GAN trained on paired examples.
- Stage-II GAN translates segmentation masks into corresponding photorealistic images.
- The conditional GAN conditions its generator and discriminator on an input image and vector representing the mapping between segmentation masks and photorealistic images.
- The model learns the mask-to-image mapping from corresponding real fundi images and segmentation masks.
8 U-net
The study uses a U-net to evaluate whether synthetic medical images support segmentation, comparing predictions from synthetic-data and DRIVE-trained models against ground truth.
- A U-net segmentation network creates a segmentation mask from a photorealistic medical image and is suited to biomedical applications.
- Synthetic-data training targets medical image segmentation tasks such as outlining relevant structures and abnormalities.
- The synthetic and real datasets are evaluated using DRIVE test images and ground-truth segmentations to calculate F1 scores.
- KL divergence measures variance between the synthetic and real datasets, while F1 score is the harmonic mean of precision and recall used for segmentation comparison.
10 Quantitative Results
Quantitative evaluation found similar segmentation performance for synthetic- and DRIVE-trained U-nets, while KL divergence indicated that synthetic data differed from the original distribution.
- 0.8877 F1 accuracy for the synthetically trained U-net compared with 0.8988 for the DRIVE-trained U-net.The paper describes the difference between these scores as negligible.
- 4.759 KL-divergence score for synthetic versus real data, compared with 4.212 x 10^-4 for two random real-data subsets.The higher synthetic-data score indicates that the synthetic data does not simply copy the original distribution.
- Pixel-intensity distributions are presented for the real and synthetic datasets.
- Generated segmentation masks are shown alongside their closest masks in the training dataset.
- Examples from the DRIVE dataset are shown alongside examples from the synthesized dataset.
12 Pipeline Validation
A second-dataset test applied the dual GAN pipeline to 35 rat smooth muscle cell images, producing realistic images and corresponding segmentations despite limited data.
- 35 rat smooth muscle cell images and segmentations from the BU-BIL database were used to test pipeline flexibility.
- The pipeline produced realistic smooth muscle cell images and corresponding segmentations despite substantial variation in subject shape and position.
- The pipeline generated segmentation masks with Stage-I GAN and translated them into photorealistic smooth muscle cells using Stage-II GAN.
- Examples from BUBIL and a corresponding synthesized dataset are displayed.
- The validation used an extremely small dataset of 35 images for both stages.The authors report that greater data variety would help develop the natural background found in the original images.
13 Discussion
The discussion argues that a dual-GAN hierarchy produces more realistic, varied, and privacy-preserving medical images than a single GAN pipeline. It also frames SynthMed as a public repository supporting broader access to synthetic medical data.
- A single GAN captures general color, shape, and lighting but fails to represent complex retinal structures such as vessel trees and dark spots.
- The dual architecture separates geometry generation from photorealistic rendering, allowing each GAN to solve a more focused task.Stage I generates segmentation geometries, while Stage II generates colors, lighting, and textures from that geometry.
- The pipeline produces varied synthetic images that preserve the real dataset’s general statistical distribution while avoiding patient-privacy concerns.
- The proposed end-to-end system generates segmentation masks and photorealistic images that are not images of real patients.
- SynthMed provides an online repository intended to make synthetic medical imaging data publicly available.
15 Future Work
Future work considers extending the two-stage pipeline beyond medical images and improving its representations and architectures. Applying the pipeline across datasets remains constrained by data requirements and tuning needs.
- The two-stage pipeline may extend to scene generation by separating simple scene features from learned visual details.
- Alternative Stage-I representations, including bezier curves, 2D point clouds, and skeletons, could reduce dimensionality, computation time, and artifacts.
- Using the pipeline on different datasets may require hyperparameter tuning for greater effectiveness.
- The pipeline relies on accurate, high-variance data, including access to private research collections for broader medical-image applications.