Source-linked AI summary
General and specific utility measures for synthetic data
Joshua Snoke, Gillian Raab, Beata Nowok, Chris Dibben, Aleksandra Slavkovic
TL;DR
The paper evaluates general and specific utility measures for judging synthetic data against original data. It develops extensions to general utility measures and finds that general measures can complement specific utility when specific utility gives misleading results.
Problem
The paper addresses how to evaluate whether synthetic data adequately represent original data for external dissemination, comparing general and specific utility measures.
Method
The paper develops extensions to general utility measures for synthetic data and evaluates whether the synthesizing model is correct.
Results
Specific utility gave misleading results in one example, while general utility provided a different perspective and aided a more complete utility evaluation.
Takeaways & Limitations
General measures should contribute to evaluating synthetic-data utility alongside specific measures rather than being considered separately.
Takeaways & Limitations
More experience with further synthesized data is needed before the results can be generalized.
Abstract
from arXiv · showhide
Data holders can produce synthetic versions of datasets when concerns about potential disclosure restrict the availability of the original records. This paper is concerned with methods to judge whether such synthetic data have a distribution that is comparable to that of the original data, what we will term general utility. We consider how general utility compares with specific utility, the similarity of results of analyses from the synthetic data and the original data. We adapt a previous general measure of data utility, the propensity score mean-squared-error (pMSE), to the specific case of synthetic data and derive its distribution for the case when the correct synthesis model is used to create the synthetic data. Our asymptotic results are confirmed by a simulation study. We also consider two specific utility measures, confidence interval overlap and standardized difference in summary statistics, which we compare with the general utility results. We present two examples examining this comparison of general and specific utility to real data syntheses and make recommendations for their use for evaluating synthetic data.
1 Introduction
Synthetic data offer a way to expand access to sensitive records while reducing disclosure risk, but their validity depends on how well synthesis models represent the original data. This paper evaluates general and specific utility measures for synthetic data, extends a propensity-score-based measure, and compares the measures across syntheses.
- Motivation: Statistical agencies seek to disseminate data, but sensitivity can restrict access to raw records; synthetic values reduce disclosure risk by replacing sensitive observations.Synthetic datasets replace some or all observed data with values generated from models based on the original data.
- Motivation: Synthetic-data inferences are valid only when the synthesis models correspond to models that could have generated the original data.Staff therefore need to assess how well this condition is fulfilled using utility measures.
- Utility measures: General utility measures summarize distributional differences, whereas specific utility measures compare results from particular analyses.Specific measures can be limited because agencies generally cannot know all analyses future users will perform.
- Motivation: Generative models that adhere too closely to a proposed utility model can conceal other interactions, motivating general measures that assess agreement across analyses.The paper argues that general utility could help assess how final inferences might compare with results from unchanged data beyond a gold-standard analysis.
- Contributions: The paper evaluates and recommends extensions to global and specific utility measures, extends a propensity-score-based measure with two synthetic-data statistics, and compares general with specific utility across synthesis methods.Two data examples examine differences in how the measures evaluate alternative syntheses.
2 Brief Review of Synthetic Data Methodology
Synthetic data are generated by fitting a conditional model to observed data and sampling new values, possibly across multiple synthetic datasets. For exploratory use followed by gold-standard analysis, simple averages and plug-in variance estimates consistently approximate corresponding original-data quantities under stated conditions.
- Synthetic data generation: Synthesis replaces sensitive data Y while retaining data X, using an estimated conditional generative model f(Y |X, θ).All data may be synthesized when X is empty.
- Synthetic data generation: New Y values are generated by fitting f(Y |X, θ) to observed data, obtaining ˆθ, and sampling from f(Y |X, ˆθ).In practice, synthesis is typically approximated with a sequence of conditional models.
- Intended use and inference: The paper focuses on exploratory synthetic-data use followed by a gold-standard analysis on the original data after models are chosen.The goal is to approximate the original-data estimate ˆQ and variance-covariance matrix V ˆQ.
- Intended use and inference: Under correct-model and asymptotic conditions, the average synthetic-data estimate ¯qm consistently estimates ˆQ, while the plug-in estimator ¯vm consistently estimates V ˆQ.Neither multiple syntheses with combining rules nor posterior sampling of Y is required for these quantities.
- Utility assessment: Specific utility is assessed by comparing synthetic-data confidence intervals and standardized coefficients with equivalent original-data quantities calculated from ¯qm and ¯vm.The same estimator applies whether all observations, selected variables, or selected data values are synthesized.
3 General Utility Measures for Masked Data
General utility measures assess how closely masked data match the original data’s distribution. This section extends propensity-score utility measurement to synthetic data, using classification distinguishability and pMSE to quantify similarity.
- General utility: General utility measures assign higher utility to masked datasets that are more distributionally similar to the original data.Examples include empirical CDF distance and Kullback-Liebler (KL) divergence.
- Prior measures: Earlier comparisons across disclosure-control methods evaluated propensity scores as the most effective general utility measure.The comparisons covered microaggregation, additive noise, swapping, and resampling methods.
- Propensity-score method: The pMSE is the mean squared difference between estimated propensity scores and the masked-data proportion, with lower values indicating better utility.For m > 1 synthetic datasets, pMSE is calculated for each dataset and averaged overall.
4 General Utility for Synthetic Data
The section extends propensity-score general utility measures to synthetic data by deriving the null pMSE distribution under Correct Synthesis and standardizing departures from it. It also evaluates CART-based propensity models and resampling methods for complex settings where theoretical null results are unavailable.
- CART models and resampling: CART propensity models are promising for complex datasets, but their theoretical null pMSE results do not hold because the parameter count is unknown.The section proposes resampling to approximate the null distribution; simulations agreed with theoretical results and showed increasing pMSE ratios for less-correct synthesis models.
- Simulation evidence: pMSE values increase as the model generating synthetic data diverges from the model generating the original data.Simulations confirmed the theoretical null expressions under Correct Synthesis and showed increasing pMSE departures with increasing model differences.
5 Specific Utility Measures for Synthetic Data
Specific utility evaluates synthetic data by comparing results of the same analyses on original and synthetic datasets. The section describes confidence-interval overlap, ellipsoid overlap, and standardized differences as complementary measures of analysis fidelity.
- Specific utility: Specific utility compares results from identical analyses conducted on original and synthetic data, with closer agreement indicating higher utility.These measures are also described as fidelity measures because they assess trustworthiness relative to analyses of unreleased data.
- Confidence-interval overlap: Confidence-interval overlap (IO) compares intervals for corresponding coefficients or summary statistics, with higher IO indicating greater utility.IO can be averaged across statistics and becomes negative when intervals do not overlap.
- Ellipsoid overlap: Ellipsoid overlap (EO) estimates overlap between joint parameter distributions and accounts for correlations among parameter estimates.EO is more satisfactory for this reason but is more burdensome to compute and may be less easily understood.
- Standardized difference: The standardized difference is |β_orig − β_syn|/s.e.(β_orig), comparing coefficients from the same model fitted to real and synthetic data.The denominator is the estimated standard error of the coefficient from the original data.
- Implementation: The examples use both IO and standardized difference, implemented in synthpop’s compare.fit.synds() function for comparing synthetic results with a gold-standard original-data analysis.For models with many coefficients, these measures can be summarized by their mean or median and range.
6 Data Examples
The two real-data examples show that general and specific utility measures can disagree, and that specific utility may be misleading when tested models resemble synthesis models. The authors therefore recommend using utility measures together with visualizations and marginal distribution checks.
- Scottish Health Survey: For the Scottish data, logistic pMSE with interactions rejects sampling, ranks parametric synthesis above CART, and main-effects logistic pMSE shows little discrimination.CART-based pMSE gives similar ratios for CART and parametric synthesis, while sampling performs worse but less severely than under logistic pMSE with interactions.
- 1901 Scottish Census: The 1901 Scottish Census example finds disagreement between CART and logistic propensity models, although both rate all synthesis methods poorly and rank CART synthesis best.The logistic model fails to identify the poor normal synthesis that generates impossible negative numeric values, whereas the CART propensity model detects this problem.
- 1901 Scottish Census: In the census example, parametric syntheses have high specific utility for closely represented models but much lower utility for model (3).These results reflect relationships built into the parametric synthesis rather than generally accurate data generation.
- Interpretation: The census example shows that specific utility can be artificially increased by testing models related to the synthesis process, even when the synthesis is completely wrong.The authors argue that general measures can improve evaluation when specific utility is misleading and highlight the potential value of non-parametric propensity-score models.
7 Conclusions
The paper develops general utility measures for synthetic data, including standardized pMSE-based statistics designed for interpretability and sample-size-independent comparison. Comparisons with specific utility show that general measures complement analysis-specific assessments and can support both synthesis-method selection and broader access to confidential data.
- General utility measures: The proposed pMSE ratio and standardized pMSE rescale utility statistics by their null expectations and standard deviations for greater interpretability.The measures assess whether the synthesizing model is correct for the original data rather than comparing identical data matrices.
- General utility measures: The standardized measures are easier to compare because they do not depend on sample size.The proposed functions are being implemented in the synthpop package for R.
- Propensity-score models: CART models may be useful for estimating propensity scores, particularly with complex data, whereas higher-order parametric interactions may be computationally infeasible with many categorical variables.The authors identify further exploration of higher-order interactions as warranted.
- General and specific utility: General and specific utility measures showed reasonable concordance in one example, but specific utility was misleading in another.The authors conclude that general measures provide a different perspective and contribute to a complete utility evaluation.
- Practical implications: The measures can help staff tailor synthesis methods to produce data fit for users’ purposes and help researchers assess whether exploratory analyses were misleading.Their implementation in synthpop is intended to support acceptance and use of synthetic data and widen access to confidential data.
9 Appendix I
The appendix derives the pMSE distribution under correct synthesis, extends it to incomplete synthesis and comparisons of two synthetic datasets, and identifies assumptions limiting theoretical expectations. It also notes that CART-based variation is dominated by differences between fitted models rather than synthesis randomness.
- pMSE derivation: The pMSE is computed by fitting a logistic regression of the original-versus-synthetic indicator on predictors built from both datasets.The predictor matrix includes an intercept, original and synthetic values, interactions, product terms, and other functions.
- pMSE derivation: Under correct synthesis, the difference between synthetic and original predictor means converges to zero, with variance V/n2, yielding a quadratic-form pMSE distribution.The quadratic form has dimension k −1, and the appendix derives the corresponding expected value and standard deviation.
- Incomplete synthesis: For incomplete synthesis, the same pMSE distribution applies with k∗ replacing k, where k∗ counts predictors involving synthesized variables, including relevant interactions.Predictors based only on unchanged variables contribute zero mean differences.
- Two synthetic datasets: 2(k −1)(1 −c)2c/N is the expected pMSE for two synthetic datasets, twice the null expected pMSE for a sample of size n2.The result assumes independent syntheses whose variation arises from differences in their predictor matrices and was confirmed for logistic models.
- Two synthetic datasets: For CART methods, theoretical expectations do not apply reliably because between-synthesis variation is dominated by differences between selected final CART models.The appendix concludes that mean between-pair differences estimate the null pMSE for CART methods.
10 Appendix II
Appendix II evaluates CART-based propensity-score models in complete and incomplete synthesis simulations. CART produces larger pMSEs and introduces overfitting-related variation, while excessive tree complexity limits discrimination between poor syntheses.
- Simulation setup: The appendix replaces logistic propensity-score models with non-parametric CART models for complete and two-variable incomplete synthesis simulations.Null distributions are estimated by resampling pairs of synthetic datasets from the same generative model; permutation gives similar results for complete data.
- Complete syntheses: CART pMSEs are much larger than logistic-model pMSEs, with model selection introducing an over-fitting component of variation.This reflects the greater flexibility of CART models for assessing more complex datasets.
- Complete syntheses: For correct synthesis, ratio and standardized pMSE values remain constant across correlations, although the expected value changes because CART model parameters are not fixed.The null has slight upward bias, producing slight downward bias in ratio statistics, but the effect is considered small for practical use.
- Tree complexity: Ratios for incorrect synthesis increase more slowly because tree size raises pMSE scores for both observed and null data.This is a tradeoff for CART’s flexibility and makes overly large trees undesirable when distinguishing worse syntheses.
- Tree complexity: As effective parameters approach the number of observations, the expected pMSE ratio is limited at 2 regardless of synthesis quality.The appendix attributes this ceiling to a maximum pMSE of 0.25 and a null expectation of 0.125 when datasets have equal row counts.
- Incomplete syntheses: The incomplete-synthesis CART simulation exhibits the same patterns as the complete-data simulation, and resampling estimates the null pMSE when not all variables are synthesized.Table 11 reports results for multivariate Normal data with only two of ten columns synthesized.