PaperlayerView exact revision on arXiv

Source-linked AI summary

How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models

Ahmed M. Alaa, Boris van Breugel, Evgeny Saveliev, Mihaela van der Schaar

arXiv:2102.08921v2cs.LGstat.ML

TL;DR

Generative-model evaluation lacks domain- and model-agnostic metrics that diagnose diverse failures and detect memorization. The paper introduces α-Precision, β-Recall, and Authenticity as sample- and distribution-level measures of fidelity, diversity, and generalization, with auditing as a post-hoc use case. The metrics diagnose failure modes, identify copied training samples, and support rejection of low-quality outputs without modifying the model.

  • Problem

    Existing likelihood and image-focused metrics inadequately diagnose generative-model failures across broader domains and may overlook overfitting to training data.

  • Method

    The paper combines minimum-volume-set precision-recall analysis with sample-level classification to estimate α-Precision, β-Recall, and Authenticity.

  • Results

    The metrics diagnose outliers, distributional mismatches, mode collapse, mode invention, and memorization, while achieving optimal values only when real and generative distributions are identical.

  • Takeaways & Limitations

    Sample-level metrics enable auditing that rejects low-fidelity or unauthentic outputs and improves generated datasets post hoc without modifying the model.

  • Takeaways & Limitations

    The paper leaves an in-depth study of using the metrics to evaluate fairness and representation of patient subgroups for future work.

Abstract

from arXiv · show

Devising domain- and model-agnostic evaluation metrics for generative models is an important and as yet unresolved problem. Most existing metrics, which were tailored solely to the image synthesis setup, exhibit a limited capacity for diagnosing the different modes of failure of generative models across broader application domains. In this paper, we introduce a 3-dimensional evaluation metric, ($α$-Precision, $β$-Recall, Authenticity), that characterizes the fidelity, diversity and generalization performance of any generative model in a domain-agnostic fashion. Our metric unifies statistical divergence measures with precision-recall analysis, enabling sample- and distribution-level diagnoses of model fidelity and diversity. We introduce generalization as an additional, independent dimension (to the fidelity-diversity trade-off) that quantifies the extent to which a model copies training data -- a crucial performance indicator when modeling sensitive data with requirements on privacy. The three metric components correspond to (interpretable) probabilistic quantities, and are estimated via sample-level binary classification. The sample-level nature of our metric inspires a novel use case which we call model auditing, wherein we judge the quality of individual samples generated by a (black-box) model, discarding low-quality samples and hence improving the overall model performance in a post-hoc manner.

1. Introduction

The paper introduces a domain- and model-agnostic evaluation framework that measures fidelity, diversity, and generalization at both sample and distribution levels. Its metrics diagnose failure modes, detect copying, and support post-hoc auditing of generated samples.

  • Likelihood-based evaluation can scale poorly in high dimensions and collapse distinct model failures into one uninterpretable score.
  • The framework classifies individual synthetic samples and represents model performance in three dimensions: Fidelity, Diversity, and Generalization.Fidelity concerns sample quality, Diversity concerns coverage of real-sample variability, and Generalization concerns copying training data.
  • α-Precision and β-Recall generalize precision and recall across density level sets, producing curves that diagnose fidelity and diversity failures.The metrics evaluate all α, β ∈ [0, 1] after embedding real and synthetic data into hyperspheres.
  • Authenticity adds an independent generalization dimension that identifies synthetic samples copied from training data.This is especially relevant when modeling sensitive data with privacy requirements.
  • The proposed metrics address outlier robustness and distributional mismatch while enabling diagnostics such as mode collapse and mode invention.Optimal metric values occur only when the real and generative distributions are identical.
  • The metrics support model auditing by judging individual outputs, rejecting low-fidelity or unauthentic samples, and improving outputs post hoc without modifying the model.The paper demonstrates this use case for synthesizing clinical data for COVID-19 patients.

2. Evaluating and Auditing Generative Models

The paper seeks a metric that evaluates generative-model performance and audits individual outputs by disentangling fidelity, diversity, and generalization. Its three-dimensional metric supports sample-level quality decisions and post-hoc curation of synthetic datasets.

  • Evaluation goals: The evaluation metric should distinguish failure modes of the generative distribution and determine whether individual synthetic samples are low quality.These requirements support both model evaluation and sample-level auditing.
  • Three qualities: The metric quantifies fidelity, diversity, and generalization, where generalization excludes synthetic samples that merely copy training data.Fidelity concerns resemblance to real data, diversity concerns coverage of real-data variability, and generalization concerns avoiding training-data copies.
  • Metric design: The proposed evaluation is a three-dimensional metric whose components are α-Precision, β-Recall, and Authenticity.The components correspond respectively to fidelity, diversity, and generalization.
  • Pipelines: The pipeline embeds real and synthetic data into a feature space before evaluating the metric, while auditing discards generated samples with low authenticity and/or precision.With direct model access, auditing repeatedly samples from Pg and accepts samples meeting the desired quality criteria.

3. α-Precision, β-Recall and Authenticity

The metric refines precision-recall analysis with density-aware α-supports and adds Authenticity to detect memorization. Its curves and integrated scores diagnose fidelity, diversity, and distributional mismatch more completely than standard precision and recall.

  • Definitions: An α-support is the minimum-volume subset containing probability mass α, separating typical high-density samples from outliers.The construction applies analogously to β-supports and enables density-aware evaluation.
  • α-Precision and β-Recall: α-Precision measures the probability that a synthetic sample lies in the α-support of the real distribution, while β-Recall measures real samples covered by the β-support of the generative distribution.Sweeping α and β changes which samples count as typical and produces full Pα and Rβ curves.
  • Density-aware analysis: Pα and Rβ account for distributional densities, assigning greater importance to denser regions and reducing standard metrics’ sensitivity to outliers and failure to diagnose distributional differences.Standard precision and recall correspond to the full-support case, P1 and R1.
  • Theoretical guarantee: Pα and Rβ are jointly optimal for every α and β if and only if the generative and real densities are equal, Pg = Pr.Equivalently, the normalized curves have unity slopes; matching supports alone is insufficient when densities differ.
  • Integrated scores: The integrated metrics IPα and IRβ summarize curve deviation in [0, 1], and both equal 1 only when Pg = Pr.Together they separate fidelity and diversity without requiring a shared support or an additional f-divergence.
  • Authenticity: Authenticity independently measures whether generated samples are new rather than copied, modeled as a mixture of innovation and noisy resampling of training data.The mixture assigns probability A to innovation and 1 − A to pulling a training sample with small added noise.

4. Estimating the Evaluation Metric

The estimation procedure converts the sample-level metric into binary decisions for synthetic and real samples. An evaluation embedding makes α- and β-support estimation tractable, while an authenticity classifier tests proximity to training examples.

  • Sample-level estimation: The estimated metric assigns binary scores to individual synthetic and real samples, then averages those scores to estimate Pα, Rβ, and Authenticity.Three classifiers produce the sample-level scores: fP, fR, and fA.
  • Precision and recall classifiers: The precision and recall classifiers test whether samples lie within estimated α- and β-supports.Their construction follows the definitions of Pα and Rβ.
  • Support estimation: Directly computing α- and β-supports is difficult because it requires solving a minimum-volume optimization problem, even when the underlying distributions are known.The method addresses this difficulty through an evaluation embedding.
  • Evaluation embedding: The evaluation embedding uses a one-class neural network to map real data into a minimum-volume hypersphere centered at cr, making support estimation reducible to concentric-radius thresholds.The radius, embedding parameters, output dimensions, center, and ν are treated through the stated optimization and hyperparameterization.
  • Authenticity classifier: The authenticity classifier tests whether a synthetic sample is non-memorized using a likelihood-ratio construction based on its nearest training sample and that sample’s nearest neighbor distance.The classifier labels a sample unauthentic when it is closer to a training point than any other training sample.

5. Experiments and Use Cases

Experiments apply the metrics across clinical tabular data, MNIST mode dropping, time-series synthesis, and image generation. They use the metrics for model ranking, privacy–utility analysis, post-hoc auditing, and diagnosis of fidelity–diversity trade-offs.

  • COVID-19 predictive modeling: ADS-GAN, WGAN-GP, VAE, and GAN were ranked by predictive-model AUC on real COVID-19 data, with ADS-GAN performing best.Predictive models were trained on synthetic data and tested on real data.
  • COVID-19 predictive modeling: Distribution-comparing metrics, including the proposed metrics, PW, and FID, accurately ranked the four generative models.The results emphasize accounting for distribution densities rather than supports alone.
  • COVID-19 predictive modeling: Post-hoc auditing improved audited ADS-GAN predictive-model AUC from 0.76 to 0.78 while producing nearly optimal precision and authenticity.Auditing discards unauthentic or imprecise samples; the improvement was statistically significant at p < 0.005.
  • MNIST mode dropping: As MNIST mode dropping increased, β-Recall declined while α-Precision remained unchanged, whereas standard precision and recall detected only the extreme case.Statistical distances increased but did not disentangle fidelity from diversity.
  • Hide-and-Seek time-series data: In the Hide-and-Seek competition, the winning Gaussian-noise model was among the least authentic despite competitive precision and recall.The metrics identified its samples as noisy copies of real data posing patient-privacy risk.
  • Image generation benchmarks: For CIFAR-10, DDPM achieved higher α-Precision while StyleGAN2-ADA achieved higher recall, revealing different distributional and diversity strengths than FID alone.FID ranked StyleGAN2-ADA higher, while the proposed metrics provided a more nuanced comparison.

6. Conclusions

The paper concludes that its metric evaluates fidelity, diversity, and generalization at both sample and distribution levels across varied applications. It highlights clinical and fairness applications while leaving in-depth fairness analysis for future work.

  • Contributions and scope: The metric characterizes generative-model fidelity, diversity, and generalization through sample-level and distribution-level evaluation.It is grounded in minimum volume sets and demonstrated across a wide variety of use cases and application domains.
  • Applications and future work: The authors identify clinical applications and synthetic-data fairness as important settings for the metrics, especially where individual sample quality and subgroup representation matter.They leave an in-depth study of fairness as future work.

Appendix A: Literature Review

The literature review contrasts statistical divergence with precision-recall metrics and identifies shortcomings in existing approaches. It motivates sample-level, multi-dimensional diagnostics that account for density, failure modes, and support estimation.

  • Metric classes: Existing generative-model metrics fall into statistical divergence and precision-recall classes.Divergence metrics measure distributional distance, whereas precision-recall metrics classify real and generated samples by coverage.
  • Statistical divergence metrics: Likelihood-based evaluation is model-dependent, inaccessible for some models, and inadequate for high-dimensional sample quality.GANs lack explicit likelihoods, while energy-based models involve difficult normalization constants; high-dimensional likelihood can obscure sample quality.
  • Statistical divergence metrics: Statistical divergence measures collapse distinct generative failures into a single score, limiting diagnosis of mode dropping, mode collapse, and poor coverage.The review explicitly identifies this loss of diagnostic resolution as a limitation.
  • Precision and recall metrics: Standard precision and recall can miss density mismatches because they primarily assess distributional support rather than actual probability densities.A model may score perfectly when it matches the support even if real and generated distributions assign different densities.
  • Precision and recall metrics: The proposed α-Precision and β-Recall evaluate density level sets, improving robustness to outliers and enabling more detailed failure diagnostics.Their support estimates use neural networks rather than nearest-neighbor estimates to reduce overestimation of generated-sample coverage or quality.

Appendix B: Proof of Theorem 1

The appendix proves that equality of the proposed precision and recall quantities across density levels implies equality of the real and generative distributions. The proof proceeds by comparing probability masses over progressively finer support regions.

  • Proof structure: The proof begins by establishing the theorem’s two required statements.The supplied proof then develops the precision direction before applying an analogous argument to recall.
  • Proof structure: When Pg = Pr, the proof uses this equality to establish the relevant condition for all β.This completes one direction of the theorem’s argument.
  • Precision direction: If Pα = α for every α, probability masses match on infinitesimally thin regions of the α-support as Δα approaches zero.This establishes equality over subsets of the real distribution’s full support.
  • Recall direction: Applying the same reasoning to recall yields equality over the generative support and therefore Pg = Pr.The proof combines both support-wise arguments to conclude distributional equality.

Appendix C: Alternative approach for estimating the support of synthetic data & code snippets

The appendix compares a separate one-class representation for each synthetic sample with the adopted k-NN support estimator. Although the approaches perform similarly, the k-NN approach is chosen to avoid model-comparison bias.

  • Alternative estimator: A separate one-class representation could estimate the generative support for each synthetic sample.This is presented as an alternative to the k-NN approach.
  • Estimator choice: The k-NN estimator is adopted because separate representations may induce bias when comparing generative models.The two approaches were reported to perform rather similarly.

.1. Data

The experiments evaluate the metrics across tabular, image, and time-series data using modality-specific embeddings and several generative models. The setup includes baseline comparisons, authenticity evaluation, and support-estimation choices for the reported tables.

  • Datasets: The study evaluates tabular, image, and time-series datasets to assess metric versatility across modalities.The datasets include SIVEP-GRIPE, MNIST, and AmsterdamUMCdb-based intensive-care time series.
  • Embedding strategy: Image and time-series data are embedded into static representations because their original spaces are non-Euclidean for most metrics.These embeddings support baseline metric computation and One-Class representations.
  • Image representation: MNIST images use an InceptionV3 ImageNet embedding, with very similar results obtained using VGG-16.Random VGG-16 embeddings did not produce stable baseline or proposed-metric results in preliminary experiments.
  • Time-series representation: Time-series representations use a sequence-to-sequence LSTM autoencoder whose encoder hidden and cell states are concatenated into one vector per example.The autoencoder is trained with the input sequence as the reversed target sequence, and the resulting representation is used at inference.
  • Metric comparisons: Table 3 compares Wasserstein distance, Fréchet Distance, likelihood, standard precision-recall metrics, density, coverage, and the proposed integrated metrics and authenticity.The table reports results for multiple generated versions of the SIVEP-GRIPE tabular dataset.
  • Support estimation: The β-recall support estimator selects the smallest k covering α held-out real samples, with k = 5 optimal in most experiments.The same value is used broadly for consistency in precision and recall comparisons.

.5. Toy experiments

Two toy experiments compare the proposed metrics with prior precision-recall and density-coverage measures, focusing on robustness to outliers and resolution of nearby modes. The proposed α-Precision and β-Recall detect failures that the compared metrics miss or handle poorly.

  • Toy experiments: The toy experiments compare the proposed metrics with improved precision and recall, and with density and coverage.The comparison targets metrics from Kynk¨a¨anniemi et al. (2019) and Naeem et al. (2020).
  • Robustness to outliers: The outlier-robustness experiment compares original X ∼ N(0, I) with synthetic Y ∼ N(µ, I) over µ ∈ [−1, 1].The setup uses samples in R^d and evaluates all metrics across the specified mean shifts.
  • Robustness to outliers: A single outlier can cause dramatic effects for conventional precision and recall, whereas IPα and IRβ remain unaffected unless α or β is large.The outlier does not belong to the α-support or β-support at smaller parameter values.
  • Mode resolution: In the mode-resolution experiment, the original distribution is a two-Gaussian mixture separated by µ, while the synthetic distribution uses a single mode.The experiment evaluates µ ∈ [0, 5].
  • Mode resolution: Neither P&R nor D&C detects that synthetic data contain one mode instead of the original two, but α-Precision captures the missing separation for small α.For small α, the original α-support excludes the space between the two modes, where the synthetic mode lies.
Loading 2102.08921v2…