PaperlayerView exact revision on arXiv

Source-linked AI summary

MAUVE Scores for Generative Models: Theory and Practice

Krishna Pillutla, Lang Liu, John Thickstun, Sean Welleck, Swabha Swayamdipta, Rowan Zellers, Sewoong Oh, Yejin Choi, Zaid Harchaoui

arXiv:2212.14578v2cs.LGcs.AIcs.CL

TL;DR

Evaluating generative models requires tractable measures of how closely model and target distributions match, including both unrealistic and missing data mass. The paper introduces MAUVE as f-divergence-frontier summaries with quantization, nearest-neighbor, and classifier estimators, finding useful alignment with human judgments and established image metrics. It recommends careful embedding choices and vector quantization while identifying dimensionality and cross-model-scale evaluation boundaries.

  • Problem

    Automatic evaluation must measure gaps between generated and target distributions, including unrealistic model mass and target mass the model misses, for complex text and image data.

    Source: §1 · p. 3

  • Method

    MAUVE summarizes f-divergence frontiers and estimates them using vector quantization, nearest-neighbor, or classifier-based methods, with statistical bounds for quantization.

    Source: §1 · p. 3

  • Results

    MAUVE measures correlate with human judgments and known text properties, extend to images, and align with Fréchet distance; vector quantization is recommended as a default estimator.

    Source: §1 · p. 3, §7.3.4 · p. 46

  • Takeaways & Limitations

    MAUVE provides a practical distribution-comparison recipe across text and image modalities, but its empirical usefulness depends strongly on informative embeddings.

    Source: §1 · p. 3, §7.5.5 · p. 54

    Nonparametric and parametric estimates require substantial dimensionality reduction, with monotonicity potentially failing above 100 embedding dimensions; reliable fine-grained metrics across model scales remain open.

    Source: §7.3.1 · p. 43, §7.7.2 · p. 58

Abstract

from arXiv · show

Generative artificial intelligence has made significant strides, producing text indistinguishable from human prose and remarkably photorealistic images. Automatically measuring how close the generated data distribution is to the target distribution is central to diagnosing existing models and developing better ones. We present MAUVE, a family of comparison measures between pairs of distributions such as those encountered in the generative modeling of text or images. These scores are statistical summaries of divergence frontiers capturing two types of errors in generative modeling. We explore three approaches to statistically estimate these scores: vector quantization, non-parametric estimation, and classifier-based estimation. We provide statistical bounds for the vector quantization approach. Empirically, we find that the proposed scores paired with a range of $f$-divergences and statistical estimation methods can quantify the gaps between the distributions of human-written text and those of modern neural language models by correlating with human judgments and identifying known properties of the generated texts. We demonstrate in the vision domain that MAUVE can identify known properties of generated images on par with or better than existing metrics. In conclusion, we present practical recommendations for using MAUVE effectively with language and image modalities.

1 Introduction

The paper introduces MAUVE, distribution-comparison measures based on f-divergence frontiers, to quantify unrealistic generated mass and missing realistic mass. It develops estimation methods and theory, then evaluates MAUVE across text and images.

1 Introduction

  • MAUVE summarizes f-divergence frontiers between model distribution Q and target distribution P, capturing unrealistic model mass and missing target mass.The frontier uses interpolations Rλ = λP + (1 −λ)Q and generalizes earlier divergence-frontier constructions.Source: §1 · p. 3, §1.1 · p. 4

1.1 Contributions

  • The paper proposes three scalar summaries: transformed-frontier area, an integral over frontier costs, and the midpoint cost.The exponential transform accommodates unbounded divergences in the area summary.Source: §1.1 · p. 4
  • Three estimators use joint embedding-space quantization, nearest-neighbor likelihood-ratio estimation, or classifier-based likelihood-ratio estimation.These methods address the lack of direct access to large-support text and image distributions.Source: §1.1 · p. 5
  • The quantization analysis separates statistical and quantization errors, handles long tails through missing-mass bounds, and achieves quantization error O(1/k).Add-constant smoothing improves empirical estimation when the quantization size k is large.Source: §1.1 · pp. 5–6
  • All estimators identify expected quality trends and correlate with human evaluations, but nearest-neighbor and classifier methods are weaker for selecting hyperparameters; vector quantization is recommended by default.Parametric Gaussian estimation requires extreme dimensionality reduction.Source: §1.1 · p. 6
  • MAUVE variants based on f-divergences correlate perfectly, whereas some optimal-transport summaries miss expected trends.Embedding choice is crucial: language-model and shallow embeddings work, while string-kernel and embedding-free methods fail to capture expected trends.Source: §1.1 · pp. 6–7
  • In images, MAUVE recovers expected effects of sampling algorithms and model scale and correlates perfectly with Fréchet distance.The work extends prior text-focused papers with broader summaries, estimators, embedding studies, and vision experiments.Source: §1.1 · p. 7

2 Background and Setup

The setup frames open-ended generation as comparing a model distribution with human data rather than matching one reference output. It introduces autoregressive modeling, decoding-induced distribution shifts, and f-divergences for this comparison.

2.1 Language Modeling and Open-Ended Text Generation

  • An autoregressive language model predicts the next-token distribution ˆP(· | x1:t) from each prefix and generates sequences by ancestral, left-to-right sampling.The model also induces an estimate of a sequence’s joint probability.Source: §2.1 · pp. 7–8
  • Even when next-token predictions are accurate on human prefixes, generated prefixes can drift from the human distribution, causing the model–human gap to grow with generation length.This motivates evaluating the induced sequence distribution rather than only conditional predictions.Source: §2.1 · p. 8

Decoding Algorithms.

  • Decoding algorithms reshape the model’s next-token distribution: temperature below one makes it more peaked, while top-K and nucleus sampling restrict probability mass to selected tokens.Top-K ranges from ancestral sampling at K = |V| to greedy decoding at K = 1.Source: Decoding Algorithms · pp. 9–10

2.2 Comparing Generative Models

  • Open-ended generation is evaluated by comparing model distribution Q with target distribution P because multiple outputs can be correct for one prompt.The goal is to measure how close Q is to P for text or images.Source: §2.2 · p. 10

2.3 Information Divergences

  • The evaluation framework uses f-divergences, defined from a convex generator f with f(1) = 0, to quantify discrepancies between distributions.The setup reviews KL, interpolated KL, Jensen–Shannon, and interpolated χ2 divergences.Source: §2.3 · pp. 10–11

3 Generalizing Divergence Frontiers with f-Divergences

This section generalizes divergence frontiers to f-divergences and introduces three scalar summaries for comparing target and model distributions. It establishes their construction and properties, including boundedness, symmetry, and identity conditions.

3.1 Tradeoff Curves to Evaluate Generative Models

  • The frontier captures two evaluation costs: unrealistic model outputs under P and realistic target mass that Q fails to cover.The Pareto frontier is carved out by the mixture distributions Rλ, so no alternative R improves both KL objectives simultaneously.Source: §3.1 · p. 12
  • f-divergence frontiers extend divergence-frontier analysis beyond KL using mixtures Rλ = λP + (1 −λ)Q and retain tractable frontier structure.The conditions f(0) < ∞ and f∗(0) = ∞ ensure finite interior coordinates while permitting endpoint growth when supports differ.Source: §3.1 · pp. 12–13

3.2 Scalar Summaries of Divergence Frontiers

  • The paper defines area, integral, and mid-point summaries that respectively integrate a transformed frontier, average linearized costs over λ, and evaluate λ = 1/2.MAUVE uses a monotonic transformation because f-divergence frontiers can be unbounded; its scaling constant changes values but not model ordering.Source: §3.2 · p. 14

3.3 Properties of Divergence Frontier Summaries

  • MAUVEf lies between 0 and 1, with larger values indicating greater similarity, and equals 1 exactly when P = Q for strictly convex f.The integral summary is also bounded for common divergences: FIKL ≤ 1 and FIχ2 ≤ 2; both integral and mid-point summaries are symmetric.Source: §3.2 · p. 14, §3.3 · pp. 15–16

3.2 Scalar Summaries of Divergence Frontiers

  • For KL and χ2 generators, the mid-point summary recovers Jensen-Shannon and Le Cam divergences, respectively, and is close to the area summary.Figure 3 relates MAUVEf to Midf by comparing the area under the transformed frontier with an orange-rectangle approximation.Source: §3.2 · p. 14, Figure 3 · p. 15

3.3 Properties of Divergence Frontier Summaries

  • Under twice differentiable strictly convex generators, the integral and mid-point summaries are zero if and only if P = Q.The integral and mid-point summaries are themselves f-divergences, and the integral summary is symmetric in P and Q.Source: §3.3 · pp. 15–16

4 Practical Computation of the Divergence Frontier and its Summaries

This section develops practical, sample-based estimators for MAUVE and divergence-frontier summaries, covering vector quantization, nearest-neighbor and kernel-density methods, and classification. It also addresses their statistical and quantization-error guarantees, computational procedures, and conditions relevant to high-dimensional, long-tailed, or poorly separated distributions.

4 Practical Computation of the Divergence Frontier and its Summaries

  • MAUVE and related summaries are estimated from i.i.d. samples using vector quantization, nearest neighbors, classifiers, or parametric approximations.Closed-form f-divergences are generally intractable for high-dimensional text and image distributions with enormous supports; the parametric approach has no theoretical guarantees.Source: §4 · p. 17

4.1 Estimation via Vector Quantization

  • Vector quantization replaces P and Q with multinomial distributions over partition cells, then estimates the reduced frontier with a plug-in estimator.Data-dependent schemes such as k-means or lattice quantization can be applied to dense image or text representations.Source: §4.1 · p. 18

4.1.1 Estimation Error Bounds

  • Smoothing reduces absolute error under missing mass, with Good-Turing and Krichevsky-Trofimov often strongest, while the best estimator remains problem-dependent.The authors recommend Krichevsky-Trofimov as a rule of thumb because it performs well when k/n is large and remains competitive when k/n is small.Source: §4.1 · p. 18, §4.1.1 · pp. 23–24
  • The total frontier-estimation error combines statistical estimation error and quantization error, with the theory balancing orders O(k/n) and O(1/k).The guarantees require smoothness and tail assumptions on the f-divergence generator, which hold for FIKL, MidKL, and coordinates of the KL and χ2 frontiers.Source: §4.1.1 · pp. 18–19
  • The bounds apply to countable supports, adapt to tail behavior without depending on p*, and provide O(n−1/2) convergence up to logarithmic factors.The resulting distribution-free sample complexity is roughly k/ε2, ignoring constants and logarithmic factors; the frontier is consistently estimable by specializing the divergence bound.Source: §4.1.1 · pp. 20–22
  • Vector-quantization estimation balances statistical and quantization errors with k = Θ(n^1/3), while modern deep-neural vector quantizers enable scaling to real data.Nonparametric vector quantization suffers from the curse of dimensionality, but nonparametric estimators can instead estimate f-divergences directly.Source: §4.1.1 · p. 25

4.1.2 Towards a Practical Algorithm

  • The practical vector-quantization algorithm jointly clusters samples in a pretrained embedding space, applies Krichevsky-Trofimov smoothing with b = 1/2, discretizes λ, and numerically integrates the frontier.Closed-form summaries can be used for KL and χ2 divergences; k-means quantization dominates complexity at O(Tknd).Source: §4.1.2 · pp. 25–26

4.2 Estimation via Nearest Neighbors

  • Nearest-neighbor estimation computes f-divergences from k-neighbor counts in an embedding-space metric, interpreting normalized neighbor-count ratios as likelihood-ratio estimates.The method constructs each divergence-frontier coordinate from the corresponding estimated f-divergence.Source: §4.2 · pp. 26–27

4.2.1 Estimation Error Bounds

  • After Gaussian smoothing in embedding space, nearest-neighbor estimation always underestimates f-divergences, with equality for discrete data and injective embeddings.The guarantee follows from the data-processing inequality.Source: §4.2.1 · p. 27
  • Under bounded-density-ratio, Hölder-continuity, and Lipschitz assumptions, the nearest-neighbor error suffers the curse of dimensionality, balancing at k = n^γ/(d+γ) with rate n^-2γ/(d+γ).The smoothed distributions must satisfy the stated density regularity assumptions.Source: §4.2.1 · p. 28

4.2.2 Towards a Practical Algorithm

  • Nearest-neighbor computation requires PCA to d′ < 50 dimensions in practice, then estimates discretized frontier coordinates and MAUVE, FIf, and Midf by quadrature or closed forms.PCA costs O(dn2+ d′d2), while tree-based nearest-neighbor search costs O((d′ + k)n log n).Source: §4.2.2 · pp. 28–29

4.2.3 Extensions and Variants

  • Kernel density estimation provides an alternative embedding-space estimator, using bandwidth-controlled kernels and plug-in f-divergences before applying the remainder of the nearest-neighbor procedure.The Gaussian kernel is given as a typical choice.Source: §4.2.3 · p. 29

4.3 Estimation via Classification

  • Classification estimates likelihood ratios from classifier odds, but the method uses linear models on frozen embeddings because pixel-space neural classifiers can perfectly separate generated images and yield poor ratios.Training separates distributions on one split and evaluates likelihood ratios on a validation split.Source: §4.3 · p. 30

5 Related Work

The paper situates MAUVE within divergence-frontier, text-evaluation, image-evaluation, and statistical-estimation research, extending prior work with general estimation methods, theoretical bounds, and empirical recommendations.

5.1 Divergence Frontiers for Generative Models

  • Earlier divergence-frontier work modeled two generative errors but lacked statistical bounds for vector quantization, f-divergence frontier analysis, and open-ended text applications.Prior quantization studies identified optimistic bias from small supports and pessimistic error from small samples.Source: §5.1 · p. 31
  • MAUVE extends divergence-frontier methods with a general framework for estimating frontiers and statistical summaries, including quantitative bounds for statistical and quantization error.The framework also addresses missing mass through smoothed estimators and evaluates several alternative estimation procedures.Source: §5.1 · p. 31
  • MAUVE’s usefulness is not uniquely driven by vector quantization: alternative nonparametric, parametric, and classifier estimators can be nearly as effective with suitable hyperparameters.Vector quantization remains attractive for its simplicity and fast open-source implementations, while modern transformer and simple GloVe embeddings work well; string-kernel and direct language-model-probability estimates fail on known trends.Source: §5.1 · pp. 31–32
  • The original MAUVE score has been adopted for diverse language-generation applications, including decoding, controllable generation, architectural changes, and private language generation.Subsequent work also reported favorable comparisons with other automatic metrics and correlations with human judgments.Source: §5.1 · p. 32

5.2 Divergence Measures for Text

  • For text evaluation, reference-based metrics suit targeted generation but are unsuitable for open-ended generation, while language-modeling metrics observe only one direction of the distributional comparison.Statistics-based metrics often target individual properties, whereas MAUVE summarizes the overall gap between human and model distributions.Source: §5.2 · pp. 32–33

5.3 Divergence Measures for Images

  • For images, Fréchet- and Kernel-Inception distances do not capture trade-offs between error types, and Fréchet distance adapted to text misses text-length effects that MAUVE captures.The paper notes that sequential temporal effects are absent in images and leaves other sequential modalities for future work.Source: §5.3 · p. 34

5.4 Statistical Estimation of Information Divergences

  • Statistical-divergence research motivates smoothing for missing mass, while MAUVE combines vector-quantization estimation with empirical comparisons against nearest-neighbor, kernel, classifier, and Gaussian approaches.The paper’s experiments also rely on data-dependent vector quantizers.Source: §5.4 · p. 34

6 Experiments: Setup

The experiments evaluate MAUVE across open-ended web, news, and story generation, varying models, decoding algorithms, baselines, human-evaluation procedures, and score hyperparameters.

6.1 Task Domains and Models

  • The setup covers web text, news, and story continuation using size-based transformer models and human continuations as target-distribution samples.Prompts are short relative to generations, which are typically hundreds of tokens long.Source: §6.1 · p. 35, §6 · p. 35

6.2 Decoding Algorithms

  • The study varies greedy, ancestral, nucleus, beam, locally typical, and adversarial perplexity decoding to test quality, diversity, and known degeneration behaviors.Nucleus sampling is evaluated across multiple truncation parameters, while adversarial perplexity sampling deliberately targets low-quality text with human-like perplexity.Source: §6.2 · p. 36

6.3 Baseline Metrics

  • Baselines include generation perplexity, Zipf coefficient, repetition frequency, Distinct-4, Self-BLEU, and discriminator accuracy, each targeting different properties or human distinguishability.Except discriminator accuracy, property metrics are compared through their gaps from corresponding human-text statistics.Source: §6.3 · p. 37

6.4 Human Judgements and Evaluation of Automatic Metrics

  • Human evaluation uses pairwise ratings of human-likeness, interestingness, and sensibility across model-decoder settings, converted to rankings with a Bradley-Terry model.The web-text study includes 36 setting pairs, 90 ratings per pair, and 214 crowd-workers, with completions capped at 256 tokens.Source: §6.4 · p. 38
  • Automatic metrics are compared with human rankings using worst-case Spearman correlation over metric means and standard deviations.Higher correlations indicate closer agreement with human quality judgments, while the protocol explicitly acknowledges potential inconsistency in human evaluation.Source: §6.4 · pp. 38–39

6.5 Hyperparameters

  • The default score is MAUVE_KL using k-means vector quantization with 500 buckets and Krichevsky–Trofimov add-1/2 smoothing, contrasted with the original empirical-estimator version.The experiments separately vary estimation methods and divergence-frontier summaries.Source: §6.5 · p. 39

7 Experimental Results

This section evaluates MAUVE across text and image generation, examining its alignment with human judgments, sensitivity to model and sampling changes, and performance under different estimation procedures, embeddings, and divergence summaries. It also studies estimation behavior and practical choices, including the reliability of vector quantization and the limitations of some alternative approaches and bounds.

7.1 Comparison to Human Evaluation

  • MAUVE correlates more strongly with human judgments than competing automatic metrics, reaching 0.857 for human-like, 0.714 for interesting, and 0.762 for sensible text.It also has the highest correlations with discriminator accuracy: 0.956 for news and 0.905 for stories, without requiring separate discriminator training when using vector quantization.Source: §7.1 · p. 40

7.2 Quantifying the Effect of Model Size, Decoding, Text Length

  • MAUVE captures expected generation-quality trends: it increases with model size, ranks greedy ≺ ancestral ≺ nucleus decoding, and decreases monotonically as generation length grows.For the GPT-2 xl model, the decoding scores are 0.016, 0.882, and 0.940 for greedy, ancestral, and nucleus sampling, respectively; alternative metrics often miss these relationships.Source: §7.2 · pp. 40–42, Figure 8 · p. 42

7.3.1 Comparison of Vector Quantization with Other Approximations

  • All estimation methods recover most experimental trends, but non-parametric and parametric estimates can violate divergence monotonicity when projections exceed 100 dimensions.Using m = 5 or 10 principal components captures model-size, decoding, and text-length trends, whereas the parametric approximation deteriorates steeply at m = 100.Source: §7.3.1 · p. 43, Hyperparameters · p. 44

Hyperparameters.

  • The tested estimators have Spearman correlations of 0.95 to 1.0 with default vector quantization and at least 0.857 with human evaluations at their best hyperparameters.Vector quantization is recommended as the reliable default because it is relatively inexpensive, robust to its quantization size, and requires less dimensionality tuning.Source: Hyperparameters · p. 44

7.3.2 Effect of Smoothing on Vector Quantization-Based Estimation

  • Krichevsky–Trofimov smoothing preserves vector-quantization rankings exactly, with Spearman correlation 1.0, while addressing empty bins that range from about 2%–5% for sampling to around 30% for greedy decoding.This supports using smoothed distribution estimators even with data-dependent vector quantization.Source: §7.3.2 · pp. 44–45, Table 8 · p. 45

7.3.3 Direct Estimation from Model Probabilities

  • Direct estimation from model probabilities fails key quality trends, including the expected ancestral-versus-nucleus ordering, and correlates only 0.430 with default MAUVE and 0.371 in worst case with human evaluations.It identifies the model-size trend for nucleus sampling but not the ancestral-sampling trend.Source: §7.3.3 · p. 45, Figure 10 · p. 46

7.3.4 Summary and Discussion

  • The experiments indicate that embedding-based vector quantization is not essential for MAUVE’s empirical success, whereas a reliable vector embedding is key to its strong correlation with human judgments.All considered estimation procedures can produce useful divergence-frontier summaries at suitable hyperparameters, but direct estimation fails.Source: §7.3.4 · p. 46

7.4.1 Divergence Frontier Summaries and Other f-Divergences

  • Divergence-frontier summaries are nearly interchangeable, with Spearman correlations of at least 0.99, while FIKL correlates best with human evaluations at 0.93 versus 0.85 for the other measures.FIKL also distinguishes the human-preferred comparison that the other measures, including MAUVEKL, cannot distinguish within one standard deviation.Source: §7.4.1 · pp. 46–47, Table 9 · p. 47

7.4.2 Variants based on Optimal Transport

  • Quantized optimal-transport costs and frontier summaries capture the correct effects of text length, model size, and decoding, unlike plug-in transport costs.The plug-in estimator rates a medium model worse than GPT-2 small under nucleus sampling (1499 ± 5 vs. 1473 ± 4) and incorrectly suggests longer generations approach humans.Source: §7.4.2 · p. 49

7.5.1 Reusing a Generative Model For Embeddings

  • MAUVE scores from GPT-2 and Grover embeddings agree strongly that Grover generations are closer to RealNews than GPT-2 generations, with Spearman correlation 0.971 and no evidence of reuse bias.Agreement holds across model sizes and decoding algorithms, although the embeddings reveal minor differences in model-size and nucleus-sampling trends.Source: §7.5.1 · pp. 50–51, Figure 13 · p. 50

7.5.2 Masked Language Model Embeddings

  • Masked-language-model embeddings correlate strongly with left-to-right-language-model embeddings and quantify model-size and decoding trends, with Spearman correlation 0.962.RoBERTa and GPT-2 both identify greedy ≺ ancestral ≺ nucleus decoding, but differ on some sampling and model-size trends.Source: Figure 14 · p. 50, §7.5.2 · p. 51

7.5.3 Learned Shallow GloVe Embeddings

  • GloVe embeddings identify key trends in model size, decoding, and text length, and their worst-case human-evaluation correlation is marginally above GPT-2 embeddings, 0.93 vs. 0.86.Because GloVe is a bag-of-words representation, it ignores word order, unlike GPT-2 embeddings; these results demonstrate MAUVE’s usefulness with shallow embeddings.Source: §7.5.3 · p. 52, Table 11 · p. 52, Figure 15 · p. 52

7.5.4 String-based Kernel Embeddings

  • String-kernel embeddings identify model-size, decoding, and text-length trends only weakly and unreliably, with worst-case human correlations always below 0.5.For example, subsequence-kernel MAUVE is 0.879 ± 0.013 at 512 tokens, 0.889 ± 0.010 at 256, and 0.871 ± 0.005 at 1024, all within one standard deviation.Source: Table 11 · p. 52, Figure 15 · p. 52, §7.5.4 · p. 54

7.5.5 Summary and Discussion

  • Overall, MAUVE’s usefulness depends critically on informative embeddings, but it can exploit left-to-right, masked-language-model, and shallow non-contextual representations.Poor performance from string kernels and direct model probabilities contrasts with the versatility of language-model and shallow embeddings.Source: §7.5 · p. 50, §7.5.5 · p. 54

7.6 Comparison to Generative Precision and Recall

  • Generative precision and recall fail to reproduce expected natural-language trends, whereas MAUVE identifies the expected effects of model size, decoding algorithm, and text length.Precision incorrectly improves for larger models, favors p = 1 over p ∈[0.9, 0.95], and suggests longer text is more human-like.Source: §7.6 · p. 55, Figure 17 · p. 55

7.7.1 Effect of the Sampling

  • For StyleGAN images, MAUVE and Fréchet distance identify the same sampling ordering, 1 ≻ 1.2 ≻ 0.7 ≻ 0.3 ≻ 0, while reflecting the quality–diversity tradeoff.MAUVE therefore summarizes both image quality and diversity, and the image experiments compare StyleGAN architectures using 5,000 samples per metric against FFHQ references.Source: Figure 18 · p. 56, §7.7 · p. 56, §7.7.1 · p. 57

7.7.2 Effect of the Model Scale and Architecture

  • StyleGAN-XL achieves the best MAUVE⋆ and Fréchet-distance scores, while StyleGAN2-ADA and StyleGAN3 are statistically indistinguishable within one standard deviation.Both metrics favor more recent models overall, but generative precision instead ranks the oldest StyleGAN highest, failing visual inspection.Source: §7.7.2 · p. 58
  • StyleGAN2-ADA (ψ = 1) outperforms NCSN++ by a large margin on both MAUVE and Fréchet distance, with diffusion samples showing facial artifacts and asymmetries.The comparison uses DINOv2 ViT-L/14 embeddings to avoid the Inception-space bias reported against diffusion models; Table 12 summarizes the metrics and Figure 20 shows samples.Source: §7.7.2 · p. 58, Table 12 · p. 59, Figure 20 · p. 59

7.7.3 Summary

  • MAUVE with f-divergence frontiers and pretrained deep-net embeddings provides a powerful general recipe for approximating gaps between complex high-dimensional distributions.The summary covers results from the preceding experimental sections as well as the image experiments reported here.Source: §7.7.3 · p. 59

7.8 Tightness of the Statistical Error Bounds

  • The oracle bound matches the estimated FIKL error rate in some small-sample or large-quantization regimes, whereas the distribution-independent bound better matches the slope for large n.The study uses real text and image data, varying sample size n and quantization size k; Figure 21 reports Monte Carlo estimates and bounds scaled by 30.Source: §7.8 · pp. 59–60, Figure 21 · p. 60
  • Theorem 10 is not tight in two settings, but Theorem 12 remains a valid upper bound and the experiments overall demonstrate favorable statistical error properties for MAUVE.For text with k = 64, the Monte Carlo error decreases faster than the bound rather than more slowly, preserving validity.Source: §7.8 · p. 60, Figure 21 · p. 60

8 Empirical Recommendations

The authors provide practical guidance for using divergence-frontier summaries when matching both the quality and diversity of real data matters, emphasizing relative comparisons and uncertainty.

8 Empirical Recommendations

  • Use divergence-frontier summaries when the goal is to match both the quality and diversity of the target distribution.They are ill-suited when only low Type I error matters, because a model can lose diversity and incur a large Type II error.Source: §8 · p. 61
  • Compare MAUVE values relatively rather than interpreting individual absolute scores, which vary with approximations, hyperparameters, and sample counts.The paper evaluates rankings induced by MAUVE through Spearman rank correlations.Source: §8 · p. 61
  • Account for standard deviations across multiple runs because stochastic decoding and random k-means initialization affect MAUVE estimates.For subtle differences, increase the sample size or number of random seeds to reduce estimation uncertainty.Source: §8 · p. 61
  • Use at least 1000 samples per distribution because smaller sample sizes bias scores toward optimism and increase standard deviation.The measures better capture gaps between texts of at least 256 tokens, preferably 512, than between shorter texts.Source: §8 · p. 61

Appendix

  • The appendix supplies proofs, approximation details, additional experiments, and human-evaluation details supporting the paper’s main sections.Appendix A covers divergence-frontier properties, Appendix B quantization bounds, Appendix C parametric approximation, Appendix D experiments, and Appendix E human evaluations.Source: Appendix · p. 62

Appendix A. Properties of the Divergence Frontiers

The appendices derive analytical properties of divergence-frontier summaries, including a closed-form KL Frontier Integral and bounds for the linearized cost.

Appendix A. Properties of the Divergence Frontiers

  • A closed-form expression is provided for the Frontier Integral of the KL divergence frontier.Source: Appendix A · p. 62
  • The KL Frontier Integral is characterized as an f-divergence generated by a convex function.Source: Appendix A · p. 62
  • The appendix establishes the KL expression by integrating a pointwise function over a dominating measure and handling equal, zero, and unequal density cases separately.Source: Appendix A · pp. 62–63
  • For finite-support probability measures, Proposition 19 bounds the linearized cost Lf,λ by reducing its maximum to distinct vertices of the probability simplex.The resulting expression evaluates the bound at distributions supported on different points.Source: Appendix A · p. 63

Appendix B. Proofs of Theoretical Bounds: Quantization

Appendix B organizes the complete proofs of the quantization results, covering empirical, add-constant, and quantization statistical error bounds.

Appendix B. Proofs of Theoretical Bounds: Quantization

  • The appendix provides complete proofs for the quantization results from Section 4.1.Source: Appendix B · p. 63
  • Its subsections prove the empirical estimator bound, add-constant estimator bound, and quantization error bound.Source: Appendix B · p. 63

B.1 Statistical Error Bound

The statistical-error proof combines an approximate Lipschitz argument with missing-mass control to bound empirical f-divergence estimation errors.

B.1 Statistical Error Bound

  • The proof handles atoms with positive empirical mass through approximate Lipschitzness and zero empirical mass through missing-mass bounds.These two cases are treated for P and analogously for Q.Source: §B.1 · pp. 63–64

B.1.1 Approximate Lipschitz Property

  • The approximate Lipschitz lemma bounds f-divergence estimation error using total variation distance, up to a logarithmic factor, when probabilities are not too small.Source: §B.1.1 · pp. 64–65
  • The plug-in estimator’s expected total variation error is further bounded using Jensen’s inequality and finite-support properties.Source: §B.1.1 · pp. 65–66

B.1.2 Missing Mass Computation

  • The missing-mass analysis controls the probability that an atom is absent from the empirical measure through (1 − P_a)^n.Source: §B.1.2 · p. 66
  • The k/n missing-mass bound is tight up to a constant factor when the support size is finite.Source: §B.1.2 · p. 66

B.1.3 Full Proof of the Statistical bound

The proofs derive statistical bounds for the plug-in divergence estimator and extend them to interpolated divergences through prior theorems and symmetry arguments.

B.1.3 Full Proof of the Statistical bound

  • The proof controls estimator discrepancies using the triangle inequality, the approximate Lipschitz lemma, and Theorem 21.These steps handle terms involving zero estimated probabilities and logarithmic factors in n.Source: §B.1.3 · p. 67
  • Theorem 10 establishes the statistical inequality, with the T2 term following by symmetry and adjusted constants.For finite k, inequality (11) additionally follows by invoking Theorems 22 and 21.Source: §B.1.3 · p. 67
  • Theorem 11 follows directly from Theorem 10 after rewriting D_f(P∥R_λ) as D_f^λ(P∥Q) using f_λ.The proof then reuses the bounds established for Theorem 10.Source: §B.1.3 · pp. 67–68

B.2 Statistical Error Bound with Smoothing

This section analyzes add-constant smoothing for estimating f-divergences, exploiting its positive lower bounds to avoid separate missing-mass control.

B.2 Statistical Error Bound with Smoothing

  • Add-constant smoothing estimates f-divergences while studying the resulting statistical error.The smoothed estimators use a common constant b for simplicity, although b could depend on symbol counts.Source: §B.2 · p. 68
  • Because every smoothed entry is bounded below by (n+kb)^−1 or (m+kb)^−1, the approximate Lipschitz lemma applies without controlling missing mass.The remaining total-variation term is bounded between the smoothed estimator and its population counterpart.Source: §B.2 · p. 68
  • Theorem 12 combines bounds for both distribution terms, yielding a logarithmic error dependence on n/b + k.The T2(a) term follows analogously with adjusted constants and sample size.Source: §B.2 · pp. 69–70

B.3 Quantization Error

The paper bounds the error from quantizing distributions into finite partitions and supplies technical properties used by these statistical analyses.

B.3 Quantization Error

  • Finite-partition quantization approximates f-divergences for general measurable spaces, with bounds applying to the linearized cost and frontier integral.The quantized distributions are formed from partitions of X with at most k cells.Source: §B.3 · p. 70
  • The quantization argument is related to prior results showing that f-divergences can be approximated by quantized counterparts (Györfi and Nemetz, 1978, Theorem 6).The cited result motivates the finite-partition analysis developed here.Source: §B.3 · p. 70, §B.4 · p. 72
  • The proof partitions the regions X1 and X2 separately, groups points by likelihood-ratio values, and combines the resulting bounds using the triangle inequality.The construction also accounts for the set where f((dP/dQ)(x)) equals f(0).Source: §B.3 · p. 71

B.4 Properties and Technical Lemmas

  • The technical lemmas establish properties of convex generators, including the sign pattern of f′ around x=1 and bounds for auxiliary expressions.These results support later statistical estimates under the stated assumptions on f.Source: §B.4 · p. 72

Appendix C. Estimation of Divergences via Parametric Approximations

The appendix describes a parametric Gaussian approximation for divergence estimation, analyzes its cost, and demonstrates that it can be arbitrarily inaccurate without theoretical guarantees.

Appendix C. Estimation of Divergences via Parametric Approximations

  • The parametric approach maps data through an embedding, approximates each embedded distribution with a multivariate Gaussian, and estimates the divergence by Monte Carlo integration.The Gaussian parameters are empirical means and covariance matrices, and the integration uses samples from the estimated Gaussian.Source: Appendix C · p. 73
  • The method has no theoretical guarantee, and its performance can become arbitrarily bad depending on P and Q.A simple Gaussian-mixture example shows that the approximation can be extremely inaccurate even in one dimension.Source: Appendix C · pp. 73–74
  • The approach has overall time complexity O(nd^2 + Bd^2 + d^3).The terms cover moment estimation, Monte Carlo evaluation with B samples, and covariance inversion.Source: Appendix C · p. 74

Appendix D. Experiments: Additional Results

  • Appendix D provides additional experiments covering model size, decoding, text length, alternative frontier summaries, and hyperparameter tuning.The appendix organizes these results across subsections D.1 through D.4.Source: Appendix D · p. 74

D.1 Results for Other Domains: News and Stories

Across news, story, length, divergence, and tuning analyses, MAUVE tracks expected generation-quality patterns and supports comparisons across models, decoding methods, and divergence choices.

D.1 Results for Other Domains: News and Stories

  • MAUVE rates larger models and nucleus sampling better in news generation, with one exception: Grover large favors ancestral sampling over nucleus sampling.Source: §D.1 · p. 74
  • SP, JS, and ε-PPL can favor degenerate decoding choices and are unsuitable for comparing generated text with human text.JS favors greedy decoding, ε-PPL favors ancestral sampling, and SP does not distinguish ancestral from nucleus sampling.Source: §D.1 · p. 75, Table 15 · p. 76

D.2 Effect of Text Length

  • As maximum text length increases, MAUVE is the only comparison measure that consistently shows the expected degradation across models and decoding algorithms.Source: §D.2 · p. 75

D.3 Comparison with Other Divergences and Optimal Transport

  • f-divergence summaries correlate perfectly, whereas some optimal-transport distances fail to capture expected trends.Source: Table 16 · p. 76

D.4 Use of MAUVE for Hyperparameter Tuning

  • Figure 23 uses MAUVE to compare nucleus-sampling p values and top-K K values, illustrating MAUVE’s utility for hyperparameter tuning across various hyperparameter settings.Source: §D.4 · p. 76

Appendix E. Human Evaluation: Protocol and Full Results

The appendix details how human pairwise preferences are converted into rankings and documents the evaluation materials, protocol, and comparison measures used alongside MAUVE.

Appendix E. Human Evaluation: Protocol and Full Results

  • The evaluation documentation includes MAUVE and optimal-transport comparison tables, sampling figures, evaluator interfaces, and annotated examples.Source: Appendix E · pp. 76–77, Figure 22 · p. 77, Figure 23 · p. 78
  • The human evaluation compares text sources across interestingness, sensibility, and human-likeness using Bradley-Terry scores derived from pairwise preferences.Source: §E.1 · p. 79

E.1 From Pairwise Preferences to Ranking: the Bradley-Terry Model

  • The Bradley-Terry model represents pairwise outcomes with a sigmoid of score differences, assumes comparison independence, and normalizes scores to mean zero.Source: §E.1 · p. 79
  • Table 18 reports fitted Bradley-Terry scores for human-like, interesting, and sensible judgments, whose rank correlations range from 0.917 to 0.967.Source: Table 18 · p. 80

E.2 Full Results of the Human Evaluation

Human evaluations compare human text with eight GPT-2 model-and-decoding combinations using multiple quality criteria, filtered prompts, and controlled annotation procedures.

E.2 Full Results of the Human Evaluation

  • Human evaluators compare nine players—human text plus eight GPT-2 model-and-decoding pairs—through pairwise judgments on interestingness, sensibility, and human-likeness.Source: §E.2 · p. 80, §E.3 · pp. 80–81
  • GPT-2 xl with nucleus sampling loses to human text with probability 0.578, while GPT-2 small with nucleus sampling loses with probability 0.679.The authors interpret these probabilities as evidence that model-generated text still has substantial room for improvement.Source: §E.2 · p. 80
  • Both human evaluations and MAUVE favor nucleus over pure sampling and larger models, except that GPT-2 small beats medium under pure sampling.Source: §E.2 · p. 80

E.3 Additional Details

  • The protocol filters web prompts, requires at least 200 BPE tokens per generation, truncates outputs at 256 tokens, and uses Mechanical Turk evaluators with quality-control exclusions.Source: §E.3 · pp. 81–82

3. Human-like: “Which continuation is more likely to be written by a human?”

Loading 2212.14578v2…