A Multifaceted Benchmarking of Synthetic Electronic Health Record Generation Models

INTRODUCTION

• Synthetic EHR data could support clinical decision-support development and research workflows while enabling broader sharing that mitigates patient-privacy concerns.Source: INTRODUCTION · p. 3
• Prior benchmarking lacks consensus on metrics, systematic comparisons, use-case-specific evaluation, and repeated runs despite GAN training instability.Source: INTRODUCTION · p. 4
• The framework generates multiple synthetic datasets, assesses and ranks them by metric, then uses use-case-specific metric weights to recommend a top model.In the experiments, it generates three datasets per model and combines ten metric-ranking lists.Source: Figure 1 · p. 5, INTRODUCTION · p. 6
• The framework combines complementary utility and privacy metrics with a rank-based scoring mechanism that enables tradeoffs between competing evaluation criteria.Source: INTRODUCTION · p. 6
• Designed for reuse, the framework accommodates categorical and continuous data, different synthesis paradigms, and inconsistent quality across GAN-generated datasets.It is applied to structured EHR data and state-of-the-art GAN models using data from two large U.S. academic medical centers and contextualized use cases.Source: INTRODUCTION · p. 6
• No model is unequivocally best across all criteria for every use case and dataset, underscoring the need for systematic, use-case-specific assessment.Source: INTRODUCTION · p. 6

Benchmarking framework

• Utility covered feature distributions, outcome prediction, and clinical readability, while privacy metrics assessed attribute inference, membership inference, and meaningful identity disclosure.Privacy assessments assumed attackers could access synthetic data but not the generative models.Source: Benchmarking framework · p. 7
• To capture GAN training variability, the framework generated five datasets per model and benchmarked the three that best preserved real-data dimension-wise distributions.The selection filtered out datasets that poorly captured individual features’ first-moment statistics.Source: Benchmarking framework · p. 9
• Models were ranked by average performance across three generated datasets per metric, then scored using a weighted sum tailored to each use case.The framework calculated rank-derived scores for ten metrics.Source: Benchmarking framework · p. 10
• The sampling baseline outperformed the five evaluated GAN models in practical use cases.The evaluated models were medGAN [25], medBGAN [34], EMR-WGAN [20], WGAN [34], and DPGAN [35].Source: Benchmarking framework · p. 10

Datasets

• The benchmark used UW data from 188,743 patients with 2,665 features and VUMC data from 20,499 COVID-19-positive outpatient patients with 2,592 features.UW data covered January 2007 to February 2019; the supplied passage identifies the VUMC data as collected from March 2020 onward but ends before giving the end date.Source: Datasets · p. 10
• Baseline had the lowest APD among synthesis models, while EMR-WGAN’s binary-feature prevalences closely followed the real-data diagonal for both UW and VUMC.WGAN’s APD resembled EMR-WGAN’s on VUMC but was substantially higher on UW; DPGAN deviated more from real-data marginal distributions. All models except WGAN and DPGAN had higher APD on UW than VUMC.Source: Figure 2 · p. 12, Datasets · p. 13
• 4 of 6 VUMC utility metrics favored EMR-WGAN, while WGAN led feature selection and clinical knowledge violation; EMR-WGAN led every UW utility metric except dimension-wise distribution.Baseline had the worst average utility for all six VUMC metrics and was among the two worst models for five UW metrics; DPGAN had the lowest UW utility for column-wise correlation, latent cluster analysis, and feature selection. When utility varied substantially across generated datasets, average metric performance could disagree with rank-derived scores: medGAN had higher average AUROC than medBGAN on TSTR, but medBGAN ranked better.Source: Figure 3 · p. 13, Datasets · p. 14
• 0.152 for attribute inference, 0.276 for membership inference, and 0.015 for meaningful identity disclosure were EMR-WGAN’s highest privacy risks, although all synthetic datasets had lower risk than real data.Under the cited 0.333 low-risk cutoff, all three values are low; relative to the passage’s cited thresholds, only EMR-WGAN on VUMC had membership-inference risk slightly above the 0.242 threshold.Source: Figure 4 · p. 15, Datasets · p. 16

Utility Privacy Tradeoff

• Higher-utility models generally had lower privacy scores across both datasets; EMR-WGAN illustrated this pattern, while Baseline on VUMC and DPGAN on UW paired lower utility with higher privacy scores.The passage says this pattern generally held for all tested models.Source: Utility Privacy Tradeoff · p. 16, Figure 5 · p. 17
• Only 2 of 45 metric pairs were strongly correlated (correlation coefficient > 0.85): column-wise correlation with latent cluster analysis (0.89), and membership inference with meaningful identity disclosure (0.88).Attribute inference was weakly correlated with the other privacy metrics, indicating that the metrics were complementary overall.Source: Utility Privacy Tradeoff · p. 17, Figure 5 · p. 17

Model selection in the context of use cases

• The framework applies different utility and privacy metric weights to education, medical AI development, and systems development according to their typical needs.Education emphasizes first-moment statistics, medical concept correlations, and patient-level clinical knowledge; medical AI development emphasizes model development and interpretability; systems development emphasizes privacy risks and data sparsity.Source: Model selection in the context of use… · pp. 17–18
• EMR-WGAN best supported education, WGAN medical AI development, and sampling-based Baseline systems development; DPGAN never ranked better than WGAN across six use-case-by-dataset scenarios.In systems development, EMR-WGAN and DPGAN had the lowest rankings for VUMC and UW, respectively.Source: Table 3 · p. 18, Model selection in the context of use… · p. 19

Synthesis paradigms

• The framework accommodates different synthesis paradigms for outcome variables because paradigm selection can affect synthetic data utility and privacy.The study tested combined and separate synthesis paradigms.Source: Synthesis paradigms · p. 19
• The separate synthesis paradigm outperformed the combined paradigm for all six models in all use cases, except for DPGAN in education and medGAN in systems development.It tended to improve dimension-wise distribution and outcome prediction utility without greater privacy risks.Source: Synthesis paradigms · p. 19

DISCUSSION

• The framework helps determine which EHR synthesis models suit a use case for a given dataset and can guide more consistent evaluation in developing new models.The authors note multiple aspects of the framework for discussion.Source: DISCUSSION · p. 20

Model comparison

• EMR-WGAN performed well on most utility metrics but had the greatest privacy risks, while Baseline had low privacy risks and low utility except for dimension-wise distribution.Baseline’s sampling strategy neglects the real data’s joint distribution.Source: Model comparison · p. 20
• DPGAN, which adds a small amount of noise to WGAN, ranked similarly or worse than WGAN, suggesting little benefit from differential privacy in the settings studied.Source: Model comparison · p. 20
• For APD, WGAN and DPGAN performed well on VUMC but not UW; separate synthesis improved feature-distribution resemblance, though dataset complexity or synthesis paradigms may explain the difference.Source: Model comparison · p. 20

Utility-privacy tradeoff

• No synthesis model was best across all metrics, use cases, or datasets; model rankings differed by use case, and utility and privacy metrics were generally negatively correlated.Baseline ranked worst for Medical AI development but best for System development.Source: Utility-privacy tradeoff · p. 21
• Column-wise correlation strongly tracked latent cluster analysis, while membership inference strongly related to identity disclosure; these metrics remain distinct and correlations below 0.9 were imperfect.The authors caution that these relationships may not generalize to other GANs or non-GAN models.Source: Utility-privacy tradeoff · p. 21

Knowledge violation

• Over 50% of DPGAN-generated records with a prostate cancer diagnosis code had a generated “Female” gender, and every model showed some knowledge violations.Violations may be partly addressed by post-hoc editing, while further work could embed violation constraints as a learning penalty [21].Source: Knowledge violation · p. 22

Limitations

• The framework uses representative rather than exhaustive utility and privacy metrics, but can be extended as new measures emerge.The authors avoided related metrics they considered redundant.Source: Limitations · p. 22
• The handful of use cases may not represent all applications or justify the chosen metric weights; privacy thresholds lack consensus, while data augmentation prioritizes utility.For data augmentation, privacy metrics can be assigned zero weight and a real-training, real-testing metric can be added; legal and policy deliberation is also needed.Source: Limitations · pp. 22–23
• The framework targets structured EHRs; longitudinal events, clinical notes, and images need data-specific metrics, although the ranking strategy may be reusable.Source: Limitations · p. 23
• The demonstration covers only a subset of GANs, omitting models designed for application-specific EHR synthesis, including MC-medGAN, CorGAN, and HGAN.These models address multi-category features, correlations between adjacent features, and constraints between features, respectively.Source: Limitations · p. 23
• Because the study datasets differed from those used to develop the GANs, its parameter settings may not be optimal, and more experiments are needed to test generalizability.The authors sought consistent parameterizations when implementations shared a technical mechanism while respecting each method’s original implementation.Source: Limitations · pp. 23–24

METHODS

• The study used a UW enterprise EHR cohort of 188,743 patients and a VUMC cohort of COVID-19-positive patients for a 21-day admission prediction task.Source: METHODS · p. 25
• After preprocessing, UW features were binary, while VUMC included eight continuous features; both datasets were split 70:30 for model training and evaluation.Categorical diagnoses, procedures, medications, and demographics were standardized and features with more than 20 occurrences retained.Source: METHODS · pp. 25–26

• The benchmark’s depicted architectures covered medGAN, medBGAN, WGAN, DPGAN, and EMR-WGAN, while synthesis paradigms included combined and separate synthesis.Source: Figure 6 · p. 27, GAN models for benchmarking · p. 27
• medGAN [16] used a pretrained autoencoder to map discrete EHR data [25] into a compact continuous space, plus batch normalization and short connections, while optimizing JS divergence.Source: GAN models for benchmarking · p. 28
• medBGAN [34] retained medGAN’s architecture but replaced its discriminator loss with boundary-seeking loss, directing generation near the decision boundary, particularly for categorical data.Source: GAN models for benchmarking · p. 28
• WGAN [34] replaced JS divergence with Wasserstein divergence [55] to address diminishing gradients, using parameter clipping to satisfy the discriminator’s Lipschitz constraint.The implementation clipped discriminator parameters to the range -0.01 to 0.01.Source: GAN models for benchmarking · p. 28
• DPGAN [35] made WGAN differentially private using DP-SGD [56], replacing gradient clipping with parameter clipping and setting ϵ to 104 to avoid very low-utility synthetic datasets.Source: GAN models for benchmarking · pp. 28–29
• EMR-WGAN [20] removed the autoencoder, added discriminator layer normalization, and used gradient penalty to enforce the Lipschitz constraint and reduce parameter clipping’s negative impact.Its design addressed training barriers associated with autoencoders and advanced distribution-distance measures.Source: GAN models for benchmarking · p. 29

Model training

• Training used normalized continuous features and shared hyperparameters, while model-specific loss trajectories guided endpoint selection and utility-based epoch checks.For medGAN, medBGAN, and DPGAN, the endpoint was set before loss growth or fluctuation; for EMR-WGAN and WGAN, the top three datasets from the converged-loss region were selected for higher dimension-wise utility.Source: Model training and data selection · pp. 29–30

Dimension-wise distribution

• Utility spans more than prediction value; dimension-wise distribution was measured with APD for binary features and normalized AWD for continuous features.For mixed-feature datasets, absolute prevalence differences and normalized Wasserstein distances were averaged across features, then multiplied by 1000 for readability.Source: Model training and data selection · pp. 30–31

Additional utility metrics

• Correlation distance, latent deviation, and clinical knowledge violation assess feature-association fidelity, latent-space resemblance, and record-level clinical consistency, respectively.Correlation distance averages absolute differences between real and synthetic Pearson-correlation matrices; latent deviation uses PCA-retained dimensions covering 80% of variance and K-means with K=3. Clinical violation is based on opposite-gender occurrence odds for selected gender-specific phecodes, with higher values indicating poorer clinical-knowledge representation.Source: Model training and data selection · pp. 31–32

Prediction Performance and important features

• Prediction utility compared LightGBM performance across synthetic-to-real and real-to-synthetic evaluation scenarios and measured overlap in important features ranked by SHAP.The important features were the top features retaining 90% of real-data performance: 20 for VUMC and 25 for UW.Source: Model training and data selection · pp. 32–33

Attribute inference

• Attribute inference modeled an adversary using KNN and majority voting to infer unknown attributes, with risk aggregated across attributes using information-entropy weights.Binary-attribute risk used F1 scores, while continuous-attribute risk used accuracy under a closeness threshold; the stated setup used k=1 and a 0.1 threshold.Source: Model training and data selection · pp. 34–35

Membership inference

• Membership inference20,30 tested whether an adversary with target-record attributes could identify real-dataset membership by matching records within a distance threshold.The risk measure was the F1 score; continuous attributes were normalized, and the main-experiment threshold was 2.Source: Model training and data selection · pp. 35–36

Meaningful identity disclosure

• Meaningful identity disclosure29 assessed linkage of synthetic records to an identified population dataset and whether the linkage could reveal identity and new sensitive information.The evaluation used an El Eman et al. risk metric and a specified population-dataset adversary for each center; the VUMC and UW setups differed in population size, quasi-identifiers, and the threshold for a successful attack.Source: Model training and data selection · pp. 36–38

Ranking mechanism

• Models were ranked by weighted sums of metric-level average rank-derived scores, with tied datasets assigned the average of their tied rank positions.Figure 7 illustrates the ranking mechanism; for example, a tie across ranks 3, 4, and 5 receives adjusted rank 4.Source: Figure 7 · p. 38, Model training and data selection · p. 39

• Use-case weights adjust metric-level results from equal default weights of 0.1, with all weights summing to 1; detailed profiles appear in Supplementary Table C.1.Source: Use case description · p. 39

Education

• For education, privacy metrics each receive weight 0.05, dimensional-wise distribution 0.25, column-wise correlation and clinical knowledge violation 0.15 each, and remaining metrics 0.1.The use case targets students in health informatics or entry-level health data analytics, with relatively small privacy risks due to access controls, audit logs, and data-use agreements.Source: Use case description · p. 39

Medical AI development

• Medical AI development evaluates 21-day hospital admission after positive COVID-19 testing for VUMC and six-month mortality for UW, prioritizing prediction-related utility and privacy.Privacy is prioritized because synthetic data are open to the broad data science community; the passage does not provide the model-performance weight value.Source: Use case description · pp. 39–40

System development

• System development weights dimension-wise distribution at 0.25 and privacy metrics at 0.5, with all remaining weights set to 0.05.The use case requires sufficiently large, realistic data that preserve real-data size and sparsity for function, workflow, and computational-resource testing.Source: Use case description · p. 40

Synthesis paradigms

• Combined synthesis treats the outcome as another feature, whereas separate synthesis trains a generative model independently for each outcome; the paradigms are orthogonal to models and use cases.Both are embedded in the synthetic EHR data generation phase.Source: Synthesis paradigms · p. 40
• The comparison used UW only: combined synthesis was applied to both datasets, but VUMC had too few positive records to support separate GAN training.All results reported up to that point were based on the combined strategy.Source: Synthesis paradigms · pp. 40–41
• 6 × 3 × 2 = 36 synthetic datasets were evaluated for UW, spanning six candidate models, three datasets per model, and two synthesis paradigms.Synthetic datasets were sized to match the corresponding real dataset.Source: Synthesis paradigms · p. 41

DATA AVAILABILITY

• Study data are available upon request from the corresponding authors, subject to approval from the institutions’ respective IRBs.Source: DATA AVAILABILITY · p. 42

Supplementary A: Comparison of synthesis paradigms

• Supplementary Figure A.1 compares real and synthetic feature prevalence under combined and separate synthesis for UW, showing three generated datasets per candidate model.Features on the dashed diagonal indicate perfect prevalence replication.Source: Supplementary A: Comparison of… · p. 53
• Supplementary Figure A.2 presents UW utility comparisons across six measures, including correlations, cluster analysis, predictive performance, shared top features, and gender-specific phecode violations.It contrasts separate synthesis in panels A–F with combined synthesis in panels G–L.Source: Supplementary A: Comparison of… · p. 54
• Supplementary Figure A.3 compares UW synthetic-data attribute inference, membership inference, and meaning identity disclosure risks under combined and separate synthesis.Real-data risk is shown in red bars, with 95% confidence intervals.Source: Supplementary A: Comparison of… · p. 55
• Supplementary Tables A.1 and A.3 report UW model rank-derived scores by metric for separate synthesis and for both synthesis paradigms, respectively.Best and worst metric scores are marked; Table A.3 uses “_com” and “_sep” suffixes for combined and separate synthesis.Source: Supplementary A: Comparison of… · pp. 56, 58
• Supplementary Table A.2 gives overall model ranks and framework scores for each use case, noting that medGAN and DPGAN’s System Development score match reflects precision loss, not a true tie.Source: Supplementary A: Comparison of… · p. 57

DPGAN_sep

• Supplementary Table A.4 ranks generative models across UW use cases using benchmarking-framework scores for combined and separate synthesis paradigms.The captions define “_com” as combined synthesis and “_sep” as separate synthesis; they do not show the actual ranks or scores.Source: DPGAN _sep · p. 59

Supplementary B: Detailed results for evaluation metrics.

• Dimension-wide distribution uses APD for UW and APD combined with average variable-wise Wasserstein distances for VUMC, while separate tables report column-wise correlation and latent deviation.The passages identify the metrics covered but do not provide their results.Source: Supplementary B: Detailed results for… · pp. 60–62
• The combined- and separate-paradigm UW TSTR tables compare synthetic-data training with a real-data baseline of AUC 0.921 [0.916,0.926].TSTR trains on UW synthetic data and tests on UW real data; the captions do not give synthetic-model AUC values.Source: Supplementary B: Detailed results for… · p. 63
• For UW training on real data and testing on combined- or separate-paradigm synthetic data, the real-data baseline is AUC 0.911 [0.907,0.915].The baseline trains on 30% and tests on 70% of UW real data; the captions do not report synthetic-test model AUC values.Source: Supplementary B: Detailed results for… · p. 64
• Additional evaluation tables cover model performance on VUMC, feature-selection overlap, and clinical-knowledge violations for UW combined and separate synthesis and VUMC synthetic data.The VUMC real-data baselines are AUC 0.802 [0.772,0.830] for training on 70% and testing on 30%, and AUC 0.773 [0.752,0.796] for the reverse split.Source: Supplementary B: Detailed results for… · pp. 65–69

EMR-WGAN WGAN DPGAN Real

• The evaluation includes attribute inference, membership inference, and meaningful identity disclosure tables, with captions defining k as the neighbor count and the identity-disclosure ratio as correctly inferred attributes in a successful attack.The membership-inference caption defines the threshold as the Euclidean distance between two records.Source: Supplementary B: Detailed results for… · p. 70, EMR-WGAN WGAN DPGAN Real · pp. 72–73

Supplementary C: Metric weight profiles for use cases.

• The metric-weight profiles for use cases rely on TSTR results to characterize model utility in prediction performance.Source: Supplementary C: Metric weight… · p. 74