Generative AI for Synthetic Data Across Multiple Medical Modalities: A Systematic Review of Recent Developments and Challenges

1 Introduction

• Healthcare AI needs large, diverse, high-quality datasets, but privacy, sharing restrictions, acquisition complexity, annotation costs, and limited real-world diversity often constrain their availability.Synthetic data generation uses generative models to create artificial data resembling real-world datasets.Source: §1 · p. 3
• Synthetic data can support data sharing while protecting privacy, dataset augmentation, and fairness and equity in AI applications.The survey examines synthesis purpose, generation techniques, and evaluation methods, and highlights personalization and standardized evaluation as gaps.Source: §1 · p. 3

1.1 Related work

• Earlier surveys often focus on one data type, one modality–technology pairing, or data augmentation, while this survey spans imaging, tabular EHRs, physiological signals, and clinical text.It also considers models beyond GANs and emphasizes conditional generation and a recent-paper scope.Source: §1.1 · p. 3

1.2 Review methods and protocol

• The review protocol examines synthesis applications beyond augmentation, recent generative models, and evaluation protocols and trade-offs across evaluation dimensions.These are the review’s stated objectives.Source: §1.2 · p. 3

diffusion models

• The review covers EHRs, physiological signals, clinical text, and medical images including dermatoscopy, mammography, ultrasound, CT, MRI, OCT, and X-rays.It considers GANs, VAEs, diffusion models, and LLMs.Source: diffusion models · pp. 3–4
• The search queries Scopus, PubMed, and ArXiv for papers and preprints published from January 2021 to November 2023, excluding reviews.Searches are repeated for each modality using terms related to synthetic data generation, conditional generation, and generative models.Source: diffusion models · p. 4

1.3 Search results

• 249 papers are included in the review, with papers distributed across modalities.The literature screening follows PRISMA guidelines.Source: §1.3 · p. 4

2.1 Synthesis applications and purpose of synthesis

• Synthesis applications cover four EHR formats, cross- and within-modality image/signal translation, class/attribute/text conditioning, and clinical-text augmentation, NLP enhancement, and de-identification.Image and signal synthesis can be conditioned on patient attributes such as age, sex, or BMI; text-based generation can use clinical reports as input.Source: §2.1 · p. 4

2.2 Generation techniques

• Across data types, GANs, VAEs, diffusion, and language models are used, with autoencoder latents, mixed discrete/continuous diffusion, and recurrent networks addressing data-specific constraints.Vanilla GANs and diffusion models do not directly generate time series; RNN, GRU, or LSTM components are used to model sequential data.Source: §2.2 · pp. 4–5

2.2.2 GANs

• GANs use adversarial generator–discriminator training; PGGAN grows resolution incrementally, while conditional GANs and translation variants enable controlled generation and paired or unpaired image mapping.CycleGAN uses cycle consistency for unpaired translation, whereas Pix2Pix uses paired datasets; paired approaches are often preferred for medical data generation.Source: §2.2.2 · pp. 5–7

2.2.4 Language models:

• Language models progress from statistical and recurrent approaches to pretrained transformers and large language models, which support clinical text augmentation, summarization, de-identification, and question answering.Medical-domain models are trained on literature and clinical records to handle specialized text; FLAN-T5 is trained to treat NLP tasks as text-to-text problems.Source: §2.2.4 · pp. 8–9

2.3 Evaluation

• Synthetic-data evaluation spans utility, fidelity, diversity, qualitative assessment, clinical validation, and privacy, including downstream-model training, augmentation, validation, and testing.Privacy assessment for EHR synthesis includes membership-inference and attribute-inference risks.Source: §2.3 · p. 10
• NNAA assesses privacy risk by comparing synthetic records with real individuals, while differential privacy adds randomness to protect individuals in the source data.A high nearest-neighbor similarity raises re-identification risk; differential privacy is described as a rigorous privacy framework.Source: §2.3 · p. 10
• Fidelity evaluates resemblance to real data at sample and distribution levels, while diversity measures variability; the review treats them as complementary metrics.Diversity evaluation checks whether models capture broad real-world variation rather than only average or common scenarios.Source: §2.3 · p. 10
• FID and IS may not reflect subtle medical-image variations and noise because their ImageNet-pretrained feature extractor targets natural images.The review notes that fidelity metrics often were designed for other data types and do not account for typical medical-image characteristics and statistics.Source: §2.3 · p. 10
• Clinical evaluation checks whether synthetic data represents anatomy and clinical features consistently with real-world scenarios and medical guidelines.It can also assess whether synthetic data supports medical decision-making, patient care, and clinical outcomes, which fidelity metrics may not capture.Source: §2.3 · p. 10
• Qualitative evaluation can ask medical experts to distinguish synthetic from real data or visually compare their distributions using t-SNE or PCA.Expert review is also described as a visual Turing Test.Source: §2.3 · p. 10

• The survey distinguishes EHR, physiological signals, medical images, and clinical notes as separate dimensions of patient health data.Source: §3 · p. 10
• EHR formats include longitudinal, time-dependent, aggregated, and snapshot records, spanning specific care episodes to comprehensive patient summaries.Source: §3 · p. 10
• Physiological signals capture real-time data such as ECG and EEG, while medical images include ultrasound, mammography, dermoscopy, MRI, CT, X-ray, and OCT.Imaging modalities provide distinct anatomical and pathological insights and may also be combined.Source: §3 · pp. 10–11
• Clinical notes describe observations, diagnoses, treatments, and other details that contextualize a patient’s condition and healthcare journey.Source: §3 · p. 11
• Across modalities, review tables summarize synthesis applications, generation models, evaluation frameworks and metrics, and code availability, using a consistent structure where feasible.Source: §3 · p. 11

3.1 EHR

• EHRs are represented as longitudinal, time-dependent, aggregated, or snapshot records, with differing temporal, dimensionality, and mixed-feature challenges.Longitudinal records retain visit sequences, aggregated records flatten codes and lose sequence, and snapshots capture selected attributes at one time.Source: §3.1 · p. 11, Figure 10 · p. 13

3.1.1 Generative models- Longitudinal format

• Longitudinal models preserve temporal correlations across visits, with also preserving within-visit correlations, but none explicitly models missingness.Evaluation generally covers downstream utility and privacy; the diffusion model lacks fidelity evaluation, while language models use perplexity.Source: §3.1.1 · p. 12

3.1.2 Generative models- Time-dependent format

• Time-dependent models handle up to 714 attributes, while model missingness and conditions on demographics for underrepresented groups.By contrast, uses outcome labels but does not extend to granular patient-specific EHR conditioning; evaluation also lacks time-specific metrics.Source: §3.1.2 · pp. 12, 14

3.1.3 Generative models- Aggregated format

• Aggregated models handle high dimensionality but omit missingness and joint static-temporal features; some support demographic or class conditioning without targeting uncommon diseases.Diffusion approaches include latent-space SDE generation and mixed-type generation; privacy evaluation also identifies challenges for TabDDPM.Source: §3.1.3 · p. 14

3.1.4 Generative models- Snapshot format

• Snapshot models do not preserve temporal correlations or model missingness; handles 324 features, while ADS-GAN explicitly minimizes patient identifiability.CTGAN variants are also used for augmentation and underrepresented sub-population modeling.Source: §3.1.4 · pp. 14–15

3.1.5 Diverse papers from diverse formats

• Cross-format EHR work extends evaluation to fairness, uses federated GAN training across organizations, and compares four tabular GANs on four medical datasets.Source: §3.1.5 · p. 15

3.2.1 ECGs

• A dual-GAN approach models AF intervals and morphology separately, while uses a second dimension to capture spatial and physiological correlations across ECG leads.Source: §3.2.1 · p. 15
• Pulse2Pulse outperformed WaveGAN for 12-lead ECG generation, while both methods treat signal generation with architecture choices tailored to ECG data.Source: §3.2.1 · p. 15
• Diffusion model outperformed SSSD-ECG and earlier GAN works, including WaveGAN and Pulse2Pulse, on fidelity and utility.Separately, SSSD-ECG outperformed prior GAN-based models such as WaveGAN and Pulse2Pulse.Source: §3.2.1 · p. 16
• ECG conditioning includes HRV and pulse morphology, while Auto-TTE incorporates text descriptors and patient-specific inputs such as age and gender.Conditional VAEs also use subject characteristics and cardiac-MRI-derived information.Source: §3.2.1 · p. 16
• ECG evaluation emphasizes fidelity and utility, gives limited attention to diversity, and does not explicitly report privacy evaluation in the reviewed studies.Generic metrics are common, while temporal alignment metrics appear in only a handful of studies; some include expert assessment or clinical validation.Source: §3.2.1 · pp. 16–17

3.2.2 EEGs and other signals

• EEG and other signal work includes latent diffusion for 30-second EEG windows, recurrent VAEs modeling variable relationships, and diffusion conditioned on EEG spectrograms.Other approaches generate signals from extracted EEG features or assess whether dataset-specific signal features are reproduced.Source: §3.2.2 · p. 17

3.3.1 Dermatoscopy

• Dermatoscopy uses GANs and diffusion models for augmentation, imbalance correction, and classification, including text-conditioned generation that benefits underrepresented Fitzpatrick skin types.GANs may amplify prominent training-data biases while mitigating less common ones; mixed real and synthetic training can improve resilience. Few studies assess authenticity qualitatively, and only one considers privacy.Source: §3.3.1 · pp. 17–18

3.3.2 Mammography

• Mammography synthesis addresses limited labeled FFDM data, single-view screening, and dense-breast detection through DFM-to-FFDM and single-to-two-view translation and dense-image augmentation.Methods also include unconditional, class-conditioned, and text-conditioned generation; evaluation emphasizes utility and fidelity, with limited clinical validation, privacy, diversity, and model comparisons.Source: §3.3.2 · pp. 18–19

3.3.3 Ultrasound

• Ultrasound studies use hybrid, GAN, and VAE approaches for synthesis and translation across organs and modalities, while evaluation relies mainly on utility, fidelity, and small specialist assessments.DSR-GAN outperformed Transformer-based TB-GAN on FID; clinical validation is uncommon, and comparisons often use off-the-shelf models not optimized for medical modalities.Source: §3.3.3 · pp. 19–20

3.3.4 MRI

• MRI synthesis targets missing or corrupted contrasts, longitudinal data, and labeled segmentation; proposed methods include multi-source GANs, recurrent GANs, and mask-conditioned generation.Brain-SPADE segmentation models trained on synthetic data performed comparably to real-data-trained models and generalized to unseen distributions, whereas nnU-Net augmentation outperformed Med-DDPM.Source: §3.3.4 · p. 21
• MRI studies often lack downstream usability assessment, with limited evaluation of synthetic-image diversity and privacy.Some studies focus only on qualitative synthesis results without testing effects on segmentation or classification; text- and image-conditioned prostate MRI generation is among the described applications.Source: §3.3.4 · p. 21
• Radiotherapy motivates synthetic CT generation from MRI or CBCT for dose calculation, while CBCT artifacts and low-dose CT image-quality tradeoffs drive restoration research.MRI lacks the tissue-attenuation information needed for CT-equivalent dose calculations, and CBCT scatter and truncated projections can produce reconstruction artifacts.Source: §3.3.5 · pp. 21–22
• MRI/CT synthesis spans paired and unpaired translation, label- and mask-conditioned generation, and diffusion methods, with a DDPM–Swin-transformer model reporting state-of-the-art image quality and diversity.The review notes that unsupervised MRI-to-CT methods remain rare, while diffusion has prompted interest in unsupervised and text-to-image generation; 3D synthesis remains complex and resource-intensive.Source: §3.3.5 · pp. 22–23
• MRI/CT evaluation often omits downstream utility and comprehensive assessment of newer diffusion or transformer models; fidelity metrics may miss anatomy, and visual tests can be biased by small samples and 2D views of 3D data.The review points to Hounsfield-unit, dose-distribution, organ-contour, and anatomical-measurement comparisons as clinically relevant evaluation approaches.Source: §3.3.5 · pp. 23–24
• CT privacy evaluation is uncommon and needs quantified guarantees, whereas synthetic tumor data in continual-learning validation improved segmentation over real-only data and helped mitigate overfitting.The validation approach used synthetic tumors in an expanding dataset to support model selection and early cancer detection.Source: §3.3.5 · pp. 24–25

3.3.6 X-Ray

• X-ray synthesis targets pathological and COVID-19 cases, using unconditional GANs and conditional methods for augmentation and downstream tasks, but text-to-image work still lacks comprehensive clinical evaluation and comparisons.Text-conditioned approaches include latent and semantic diffusion; reported concerns include false positives, limited exploration of augmentation utility, and uncertainty about image quality and clinical applicability.Source: §3.3.6 · p. 25
• Human evaluations found synthetic X-rays could be mistaken for real images, including abnormal cases, but fine anatomical details, report alignment, and clinical-task assessment remain concerns.A Turing test of 400 images found radiologists struggled to distinguish synthetic from real CXRs, especially abnormal cases; other studies reported false positives, misdiagnoses, and limited privacy evaluation.Source: §3.3.6 · pp. 25–26

3.3.7 OCT

• OCT counterfactual GANs generated identity-preserving images and longitudinal series, with expert assessments finding realism, but disease–age/sex correlations and unseen subject groups remain limitations.Unpaired cycle-consistent approaches also support cross-site OCT harmonization, though instability and contrast inversion limit reliability.Source: §3.3.7 · pp. 26–27
• Over 95% of synthetic SD-OCT images conveyed clinically relevant information and predicted biomarker presence accurately, while the method also improved therapeutic-effect prediction up to 12 months [320].The biomarkers-aware asymmetric bibranch GAN used Adaptive Memory Batch Normalization to transfer knowledge from large-scale data to a target branch trained on small paired data.Source: §3.3.7 · p. 27
• OCT synthesis improved downstream performance when tailored to tasks and modality, including 94.83% accuracy for DME-fluid detection and a 15.8% improvement in plaque-detection accuracy.The studies used a Pix2Pix U-net-based generator for DME detection and conditional GANs with class encoding for plaque detection.Source: §3.3.7 · pp. 26–27
• [68] translated between FA data and OCT macular-thickness maps for diabetic macular edema assessment, while generated glaucoma-related OCT from fundus images and reported gains from incremental training across hospitals.[68] combined FA physiological information with OCT structural information to assess treatment response.Source: §3.3.7 · pp. 27–28
• OCT studies assessed treatment-response prediction, biomarkers, counterfactual retinal aging, and downstream grading or measurement, but evaluation of diversity, quality, clinical utility, and privacy remains limited.Clinician review of generated post-therapy images supported prediction of short- and long-term treatment effects; other studies addressed foveal hypoplasia, visual acuity, macular thickness, retinal volume, and despeckling.Source: §3.3.7 · p. 28

3.3.8 Multiple Modalities

• [61] generated 3D synthetic data across four anatomical regions, CT and MRI, and three resolutions without dataset-specific hyperparameter fine-tuning, while used edge-guided diffusion across ultrasound, spleen CT, and prostate MRI.[334] used RadImageNet pretraining and task-specific fine-tuning to adapt generation to each dataset.Source: §3.3.8 · p. 28
• Across modalities, [331] conditioned GANs on segmentation masks, used class-conditional LDMs, and found label- or attribute-conditioned DDPM augmentation improved classification accuracy and fairness for under-represented groups.[333] included hospital or sensitive attributes such as ethnicity and sex as conditioning information.Source: §3.3.8 · p. 28
• [335] adapted pretrained Stable Diffusion to medical modalities through textual inversion, optimizing a text-embedding vector while freezing the rest of the model using a small set of example images.The method addresses limited availability of large medical image-caption datasets.Source: §3.3.8 · p. 28
• [338] used two GANs to generate hybrid synthetic records combining chest X-rays with non-image clinical data, compressing images with a pretrained GAN encoder.The survey identifies [338] as its only study generating synthetic hybrid image and non-image data.Source: §3.3.8 · p. 28
• Few multi-modality studies measured diversity, and qualitative assessments relied on small expert groups; used nearest-SSIM differences, while compared SSIM across synthetic-image pairs.The reported expert groups numbered two for, three for, and four for.Source: §3.3.8 · p. 28
• [330] found diffusion models more susceptible than GANs to memorizing training images, based on pairwise correlations between 100 synthetic images and the training set.The analysis compared each selected synthetic image with the entire training set.Source: §3.3.8 · p. 29

3.4 Medical text

• Medical text generation targets NLP enhancement, augmentation where clinical notes are limited or privacy-sensitive, and replacement of sensitive information; inputs can also include EHR data, diagnostic images, and clinical-trial reports.Reported text types include discharge summaries, guidelines, case reports, histories of present illness, and chief complaints.Source: §3.4 · p. 29
• Medical-text generation lacks accepted metrics for fidelity, accuracy, and clinical utility; hallucination was not assessed for [341], and [342]'s diabetic-ketoacidosis claims lacked sufficient systematic quantitative evaluation.Text evaluation also raises truthfulness, guideline adherence, and human readability concerns.Source: §3.4 · pp. 29–30
• BERT and GPT are common model families, while MIMIC clinical notes, i2b2, and n2c2 are commonly used training datasets.Source: §3.4 · p. 29
• The LLaMA-based generator outperformed Alpaca, MedAlpaca, and ChatDoctor, and models trained on actual versus synthetic notes showed no significant performance difference.It was open-source and clinician-validated, but limited to discharge summaries and one-turn instruction following; hallucination was not analyzed.Source: §3.4 · pp. 29–30
• NER models trained on synthetic HPI text performed slightly better than those trained on real text, while models trained on real-plus-synthetic text outperformed real-only models [343].The study generated a 500-subsection synthetic corpus from 570 HPI descriptions using CharRNN, SegGAN, GPT-2, and CTRL.Source: §3.4 · p. 30
• Privacy-focused methods combine sensitive-token masking with similar-document substitution [344] or add Gaussian noise during transformer training [345], but [345] did not sufficiently evaluate generated-text privacy.The masking method uses customizable blacklists and whitelists; the differential-privacy study evaluated utility at word, document, and corpus levels.Source: §3.4 · pp. 30–31
• Across 39 Dutch healthcare organizations, annotation distributions and generated-text diversity significantly affected downstream performance, while non-English training faced additional challenges [346].The study compared LSTM- and GPT-2-based models and argued that realism is not required for downstream utility, while mathematical privacy protection remains necessary.Source: §3.4 · p. 31

4.1 Synthesis applications and purpose of synthesis

• Conditional synthesis supports clinically relevant translation across modalities, while synthetic medical text also serves NLP tasks and privacy-oriented replacement beyond augmentation.Examples include missing or corrupted MRI contrasts, mammography and ultrasound translation, entity extraction, and question answering.Source: §4.1 · pp. 31–32
• Although most approaches target training-data scarcity, studies rarely establish which synthetic samples improve outcomes; selected challenging cases and underrepresented groups have helped in some studies.Examples include dense mammograms, severe lung nodules, overlapping benign and malignant ultrasound nodules, and underrepresented groups.Source: §4.1 · p. 32

4.2 Generation techniques

• Text-guided conditioning can supply structured clinical or demographic descriptions and medical reports, but some modalities have yet to adopt it.The text identifies text conditioning as an emerging direction across modalities.Source: §4.2 · p. 32
• Conditional models using text, clinical knowledge, or other inputs are reported to produce higher-quality, more realistic data, yet clinical knowledge and patient characteristics are rarely leveraged.The discussion identifies patient-specific synthesis, fairness, and inclusivity as areas needing attention.Source: §4.2 · p. 32
• Medical image generators often reuse natural-image designs, while patch-based generation, computational limits, and gaps in medical-text reasoning constrain current approaches.The discussion also describes modality-specific challenges in EHR and signal generation, including mixed static and sequential features and signal morphology.Source: §4.2 · p. 32

4.3 Evaluation

• Comparisons and benchmarks are scarce, and studies often rely on small private datasets, limiting understanding of methods’ relative strengths and generalizability.Training commonly uses up to several hundred samples, with substantially fewer for validation or testing.Source: §4.3 · p. 32
• Generic fidelity metrics may miss medical-data characteristics, while clinical validation is often poorly implemented or limited to small subsets.The discussion also notes the absence of accepted medical-text fidelity metrics and the risk of misleading conclusions from generic metrics alone.Source: §4.3 · pp. 32–33
• Human assessments often involve only one to three experts and unclear sample selection; diversity metrics lack shared definitions, and privacy evaluation remains uneven across modalities.Diffusion approaches also lack thorough utility, clinical, qualitative, and fidelity evaluation, while image privacy discussions leave identity disclosure and memorization questions unresolved.Source: §4.3 · p. 33

5 Recommendations

• Tailor generative approaches to medical data and downstream tasks, whose requirements differ across segmentation, classification, registration, and denoising.The recommendation emphasizes matching both the data specifics and the intended task.Source: §5 · p. 33
• Use synthetic data for validation and testing as well as augmentation and downstream training, and consider it for developing explainability mechanisms.The recommendation associates explainability mechanisms with confidence in proposed analysis models and clinicians’ confidence in automated methods.Source: §5 · p. 33
• Prioritize fairness and inclusivity through synthesis personalized to subject characteristics and patient demographics.The recommendation presents this as important in light of data scarcity.Source: §5 · p. 33
• Further investigate interpretability in diffusion models’ understanding and translation of radiological reports to images.Source: §5 · p. 33
• Use established benchmarks and open-source datasets to clarify technique strengths and weaknesses and guide choices for clinical applications.The recommendation links these resources with openness and collaboration.Source: §5 · p. 33
• Build holistic evaluation around specific fidelity metrics, clinical validation, human assessment, diversity metrics, and established privacy practices.The recommendation especially emphasizes privacy-evaluation gaps across some data types.Source: §5 · p. 33

6 Conclusion

• The review analyzes 249 studies published from January 2021 to November 2023 across imaging, text, time-series, and tabular medical data, considering synthesis purposes, generation techniques, and evaluation.It covers GANs, VAEs, large language models, and diffusion models, based on searches of Scopus, PubMed, and arXiv.Source: §6 · p. 34
• The surveyed work spans diverse clinically valid synthesis applications, but synthetic data remains underused beyond augmentation and how it improves outcomes is often insufficiently analyzed.Source: §6 · p. 34
• Text conditioning offers a direction for generation, while clinical knowledge, patient context, and medical-data-specific tailoring remain limited.Source: §6 · p. 34
• Evaluation challenges include scarce benchmarking and comparisons, small private datasets, limited frameworks, insufficient assessment of recent diffusion and transformer approaches, and privacy disparities.Source: §6 · p. 34
• The conclusion calls for personalized synthesis, standardized and clinically relevant evaluation, and more benchmarking and comparative studies.The authors frame these as gaps and challenges for the field.Source: §6 · p. 34

10.1 Evolution of the usage of generation techniques over years

• GANs lead publication counts across half-year periods, while diffusion-model publications rise notably in late 2022 and 2023; LLMs and transformers show periodic spikes.Source: §10.1 · p. 42

10.2.1 Privacy Metrics

• The appendix identifies commonly used EHR privacy-risk metrics and risks in Table 15.The supplied text does not enumerate the table’s metrics.Source: §10.2.1 · p. 42, Table 15 · p. 42

10.2.2 Fidelity Metrics

• Fidelity evaluation spans imaging, EHR, signals, and text; the appendix lists modality-specific metrics, with only frequently used imaging metrics displayed.Table 16 covers imaging, while Table 17 covers EHR, signals, and text.Source: §10.2.2 · p. 42, Table 16 · p. 44, Table 17 · p. 45
• EHR fidelity metrics compare individual-feature distributions, joint distributions, and inter-feature relationships, with additional measures for latent similarity, uncommon diseases, and time-series properties.Examples include feature-level statistical tests, classifier-based propensity scores, correlation measures, uncommon-disease coverage, autocorrelation, and patient trajectories.Source: §10.2.2 · p. 43