PaperlayerView exact revision on arXiv

Source-linked AI summary

Assessing Generative Models via Precision and Recall

Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, Sylvain Gelly

arXiv:1806.00035v2stat.MLcs.LG

TL;DR

Generative-model evaluation lacks reliable ways to distinguish different failure cases because common metrics yield one-dimensional scores. The paper defines distributional precision and recall, derives an efficient sample-based algorithm, and shows that it separates generated-sample quality from target-distribution coverage across generative models and datasets.

  • Problem

    Common evaluation metrics cannot distinguish realistic but incomplete samples from broader but lower-quality samples, while implicit high-dimensional distributions make likelihood-based assessment difficult.

  • Method

    The paper defines precision and recall for distributions, proves desirable theoretical properties, and develops an efficient algorithm based on samples from reference and learned distributions.

  • Results

    The proposed metric quantitatively captures precision and recall, including mode dropping and mode inventing, across real-world image and text generation and comparisons of GANs and VAEs.

  • Takeaways & Limitations

    Precision and recall provide a two-dimensional evaluation that separates generated-sample quality from coverage of the target distribution.

  • Takeaways & Limitations

    The PRD trade-off curves describe a model relative to its target distribution; they do not represent different models or distributions.

Abstract

from arXiv · show

Recent advances in generative modeling have led to an increased interest in the study of statistical divergences as means of model comparison. Commonly used evaluation methods, such as the Frechet Inception Distance (FID), correlate well with the perceived quality of samples and are sensitive to mode dropping. However, these metrics are unable to distinguish between different failure cases since they only yield one-dimensional scores. We propose a novel definition of precision and recall for distributions which disentangles the divergence into two separate dimensions. The proposed notion is intuitive, retains desirable properties, and naturally leads to an efficient algorithm that can be used to evaluate generative models. We relate this notion to total variation as well as to recent evaluation metrics such as Inception Score and FID. To demonstrate the practical utility of the proposed approach we perform an empirical study on several variants of Generative Adversarial Networks and Variational Autoencoders. In an extensive set of experiments we show that the proposed metric is able to disentangle the quality of generated samples from the coverage of the target distribution.

1 Introduction

Evaluating deep generative models is difficult because implicit high-dimensional distributions lack efficient likelihood evaluation, while common one-dimensional metrics cannot distinguish different failure cases. The paper proposes distributional precision and recall to separate sample quality from target-distribution coverage and quantify mode dropping and mode inventing.

  • Motivation: Implicit high-dimensional distributions make likelihood-based evaluation difficult, often requiring subjective visual or qualitative assessment.Even tractable likelihood can be inadequate or misleading for high-dimensional problems.
  • Motivation: One-dimensional metrics cannot distinguish models that produce realistic but incomplete samples from models that cover more modes with lower quality.Figure 1 reports similar FIDs—32/29 on MNIST and 65/62 on CelebA—despite different sample characteristics.
  • Contribution: The proposed approach separates distributional divergence into precision, measuring sample quality, and recall, measuring the proportion of the reference distribution covered.It computes both quantities from samples of the reference and learned distributions.
  • Contribution: The paper introduces a theoretically sound precision-and-recall definition, an efficient computation algorithm, and connections to total variation, Inception Score, and FID.The contributions also include empirical comparisons across generative models and real-world image and text datasets.

2 Background and Related Work

Prior evaluation methods include likelihood-based approaches, classifier proxies, Inception Score, and FID, but commonly used metrics generally produce a single score. Their limitations motivate metrics that separately characterize generated-sample quality and distributional coverage.

  • Likelihood-based evaluation: Likelihood comparisons can reward high likelihood despite low image quality, while kernel density estimation is difficult in high-dimensional spaces.These concerns have motivated alternative evaluation methods for deep generative models.
  • Existing metrics: Inception Score evaluates image samples through classifier-based label distributions but requires labeled data and can provide weak guidance for model comparison.It favors low conditional-label entropy and high entropy across the dataset.
  • Existing metrics: FID embeds samples, fits multivariate Gaussians, and computes a mean-and-covariance distance without requiring labeled data.The metric is sensitive to both spurious modes and mode dropping.
  • Existing metrics: Classifier-based quality proxies require training a classifier for each model and may overreact to a single subtle dimension of discrepancy.This can assign lower quality because of barely visible artifacts even when broader sample characteristics differ less.
  • Motivation: Common evaluation metrics are one-dimensional, while earlier manifold-distance precision and recall estimates were not computable for complex datasets.This leaves a need for practical metrics that distinguish different generative-model failure cases.

3 PRD: Precision and Recall for Distributions

The paper defines precision and recall as a trade-off over shared distributional components, separating sample quality from target-distribution coverage. The resulting PRD set has theoretical properties and an efficient sample-based computation procedure.

  • 3.1 Derivation: Precision measures the share of Q attributable to P, while recall measures the share of P covered by Q.The formulation represents the remaining components as precision and recall losses.
  • 3.1 Derivation: The mixture decomposition assigns 1 − α to Q's noise component and 1 − β to the part of P missed by Q.These components are denoted νQ and νP in the formal definition.
  • 3.2 Formal definition: PRD(Q, P) contains all attainable precision–recall pairs, exposing trade-offs that a single score cannot represent.Varying the common component µ produces different precision–recall pairs without changing the underlying model or distributions.
  • 3.2 Formal definition: PRD(Q, P) equals {(0, 0)} exactly for disjoint supports, contains (1, 1) exactly when Q = P, and is monotone and dual.The maximum precision and recall also equal Q(supp(P)) and P(supp(Q)), respectively.
  • 3.2 Formal definition: The PRD set distinguishes equal-support distributions with opposite quality–coverage trade-offs, including high precision/low recall versus low precision/high recall.These noisy cases are illustrated as extensions of the simpler mode-dropping and mode-inventing examples.
  • 3.3 Algorithm: An equivalent characterization computes PRD(Q, P) as a union of line segments parameterized by λ, using α(λ) and β(λ) over an equiangular grid.The resulting curves can be plotted for comparing multiple distributions, with interpolation used to approximate the full sets.

4 Application to Deep Generative Models

The proposed PRD approach evaluates generative models from samples by separating sample quality from target-distribution coverage. Experiments on image and text data show that it distinguishes mode dropping, mode inventing, and precision–recall trade-offs more clearly than one-dimensional metrics.

  • Evaluation approach: The algorithm embeds samples, clusters the combined feature space, and compares cluster histograms to estimate precision and recall across modalities.A meaningful embedding is sufficient; the paper applies the approach to image and text generation.
  • Evaluation approach: PRD quantitatively disentangles low-quality generation from mode dropping, avoiding reliance on visual inspection.The metric is designed to separate precision-related and recall-related failure cases.
  • Adding and dropping modes: FID cannot distinguish mode dropping from mode inventing: Q4 and Q6 have similar FID, whereas PRD assigns dropping to recall loss and fake modes to precision loss.Adding correct modes increases recall, while adding fake modes decreases precision.
  • Relation to IS: The proposed PRD metric can be applied to unlabeled data, unlike IS, which requires a classifier and labeled data set.The paper also reports Spearman correlations of -0.83 for precision and 0.89 for recall between the approaches.
  • GANs and VAEs: Across Fashion-MNIST models, VAEs generally achieve lower precision and/or higher recall than GANs, consistent with lower sample quality and less mode collapse.F8 emphasizes recall, while F1/8 emphasizes precision in the plotted trade-off.

5 Conclusion

The paper argues that one-dimensional scores cannot capture distinct generative-model failure cases and proposes theoretically grounded precision and recall instead. An efficient sample-based algorithm captures these dimensions across image and text data, with experiments supporting the expected VAE–GAN differences.

  • Conclusion: One-dimensional scores are insufficient because they cannot distinguish different failure cases of generative models.The motivating examples contrast realistic but incomplete samples with low-quality samples covering more modes.
  • Conclusion: The paper introduces theoretically sound distributional precision and recall, connects them to total variation, FID, and IS, and computes them efficiently from samples.The proposed algorithm is intended for evaluating deep generative models.
  • Conclusion: Experiments on image and text data show that the method captures precision and recall properties of generative models.The evaluation uses real-world data sets and sample-based computation.
  • Conclusion: VAEs produce lower-quality samples but are less prone to mode collapse than GANs in the reported empirical evidence.This conclusion is stated as empirical support for an existing folklore claim.

A Proofs

The proof reduces membership in the PRD set to the existence of an auxiliary distribution satisfying pointwise lower-bound constraints. This equivalence supports both directions of the characterization.

  • Auxiliary lemma: For probability distributions P and Q on a finite state space, (α, β) belongs to PRD(Q, P) exactly when an auxiliary distribution µ satisfies the corresponding pointwise inequalities.The lemma establishes the equivalence for α, β ∈ (0, 1].
  • Proof: The forward implication follows directly from the PRD definition and non-negativity of νP and νQ.The same auxiliary distribution µ is used in the resulting inequality.
  • Proof: Conversely, the inequalities define the distributions needed by the PRD definition, establishing membership in PRD(Q, P).The proof invokes the defining relations after constructing the distributions.

A.1 Proof of Theorem 1

The theorem’s properties characterize PRD through equality, support overlap, maximal precision and recall, monotonicity, and duality. These properties follow from the auxiliary-distribution characterization and the definition.

  • Basic properties: PRD(Q, P) contains (1, 1) exactly when P = Q.The proof uses Definition 1 in both directions.
  • Basic properties: PRD(Q, P) is larger than {(0, 0)} exactly when P and Q have overlapping supports.A shared support point constructs a feasible PRD pair, while the converse follows from the lemma.
  • Extremal properties: Every feasible precision α is at most Q(supp(P)), yielding the maximum-precision bound.The auxiliary distribution’s support lies within supp(P), which gives the bound.
  • Extremal properties: The maximum-recall property follows by applying the established property (vi) to property (iii).
  • Structural properties: PRD is monotone: reducing either α or β preserves feasibility.The same auxiliary distribution satisfies the weaker pointwise inequalities for α′ ≤ α and β′ ≤ β.
  • Structural properties: PRD has a duality property obtained by switching α with β, P with Q, and νP with νQ.

A.2 Proof of Theorem 2

The proof characterizes the PRD set as scaled pairs generated by λ and θ, establishing both inclusions through explicit distribution constructions.

  • Every PRD point can be represented as (θα(λ), θβ(λ)) for some λ ∈ (0, ∞) and θ ∈ [0, 1].The proof handles the zero point separately and otherwise selects λ = α′/β′ and θ = β′/β(λ).
  • For any nonzero PRD point, α′ > 0 exactly when β′ > 0, ensuring the selected λ lies in (0, ∞).
  • Conversely, when β(λ) = 0, the claimed representation is immediate because (0, 0) belongs to PRD(Q, P).
  • When β(λ) ≠ 0, defining µ using min(λP(ω), Q(ω)) yields the required point after scaling by θ.The construction satisfies the pointwise bounds needed for Lemma 1 because θ ∈ [0, 1].

B Further figures

The appendix figures illustrate PRD’s ability to distinguish variation and coverage among generated samples and compare it with supervised and feature-space analyses.

  • Two MNIST GANs that both produce only 1s receive different PRD recall values because the right model generates more varied shapes, including tilted 1s.
  • Clustering real and generated samples into 10 feature-space centers across four datasets produces meaningful, though imperfect, groupings of GAN samples.
  • PRD precision and recall correlate with supervised measures across GANs and VAEs, with Spearman coefficients of -0.83 and 0.89, respectively.
  • Figures 11 and 12 extend the corresponding comparisons to MNIST, Fashion-MNIST, CIFAR-10, and CelebA.
Loading 1806.00035v2…