PaperlayerView exact revision on arXiv

Source-linked AI summary

MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers

Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, Zaid Harchaoui

arXiv:2102.01454v3cs.CL

TL;DR

Evaluating open-ended generation requires measuring both implausible model text and missing coverage of plausible human text despite intractable, high-dimensional discrete distributions. MAUVE addresses this by comparing human and machine text through divergence frontiers computed in a quantized embedding space. It identifies generation-quality differences across tasks and settings, correlates with human judgments, and has documented scope and implementation limitations.

  • Problem

    Evaluating open-ended generation requires capturing both implausible model text and failure to cover diverse human text, but the distributions are high-dimensional and discrete.

  • Method

    MAUVE compares human and machine text using a divergence curve over mixture distributions, approximated by KL divergences in a jointly quantized embedding space.

  • Results

    MAUVE identifies expected quality differences across decoding methods, correlates highly with human judgments, and captures effects of generation length, decoding, and model size.

  • Takeaways & Limitations

    MAUVE offers an efficient scalar summary of the gap between neural and human text that captures distribution-level error trade-offs.

  • Takeaways & Limitations

    Closed-form MAUVE computation is intractable over the full token-sequence space, and the method rewards text resembling human-authored text despite risks from human-mimicking systems.

Abstract

from arXiv · show

As major progress is made in open-ended text generation, measuring how close machine-generated text is to human language remains a critical open problem. We introduce MAUVE, a comparison measure for open-ended text generation, which directly compares the learnt distribution from a text generation model to the distribution of human-written text using divergence frontiers. MAUVE scales up to modern text generation models by computing information divergences in a quantized embedding space. Through an extensive empirical study on three open-ended generation tasks, we find that MAUVE identifies known properties of generated text, scales naturally with model size, and correlates with human judgments, with fewer restrictions than existing distributional evaluation metrics.

1 Introduction

Open-ended generation produces a distribution of text intended to resemble human-written language, but evaluating this resemblance requires addressing both erroneous and missing coverage of human text. MAUVE is introduced as a practical comparison measure that captures these errors and supports broad empirical evaluation.

  • Open-ended generation samples from a model-induced distribution over natural-language sequences, aiming to resemble human-written text.
  • Evaluation must capture Type I errors from implausible high-probability text and Type II errors from failing to cover diverse human text.The underlying distributions are high-dimensional and discrete, making principled, tractable measurement difficult.
  • MAUVE compares neural and human text distributions using divergence frontiers and computes divergences in a quantized, low-dimensional embedding space.Its end-user interface takes neural text and human text and returns a scalar gap measure.
  • MAUVE quantifies properties related to text length, model size, and decoding, correlates better with human quality judgments, and is often robust to quantization, embeddings, and scaling.The authors also release a pip-installable Python package.

2 MAUVE

MAUVE measures the gap between human and machine text by softly balancing Type I and Type II errors across a divergence curve. It makes this comparison tractable by estimating distributions in a quantized embedding space and summarizing the curve by its area.

  • Open-ended Text Generation: Open-ended generation has no single correct output; coherence, creativity, and fluency are the main evaluation criteria.
  • Open-ended Text Generation: The model distribution Q is determined jointly by an autoregressive language model and its decoding algorithm.Temperature rescaling and truncation are examples of decoding modifications.
  • Sources of Error in Text Generation: The evaluation gap comprises Type I errors, where Q assigns high mass to text unlikely under P, and Type II errors, where Q misses plausible human text.The gap is small only when both errors are small.
  • Quantifying the Errors: KL(Q|P) measures Type I error and KL(P|Q) measures Type II error, but either divergence can be infinite when P and Q have different supports.This support mismatch makes raw KL divergence unsuitable as an evaluation metric.
  • Quantifying the Errors: MAUVE replaces raw KL comparisons with KL(Q|Rλ) and KL(P|Rλ), where Rλ = λP + (1 − λ)Q, to measure both errors softly.Varying λ produces a divergence curve that encodes the trade-off between error types.
  • Summarizing the Errors with a Divergence Curve: MAUVE(P, Q) is the area under the divergence curve, lies in (0, 1], and equals 1 if and only if Q = P.Larger MAUVE values indicate that the model distribution is closer to the human distribution.
  • Computing MAUVE for Open-Ended Text Generation: The exact closed-form computation is intractable because text distributions span the prohibitively large space of all token sequences.
  • Computing MAUVE for Open-Ended Text Generation: The estimator samples human and machine text, embeds both with an external language model, jointly quantizes embeddings, and forms discrete histograms.The resulting k-element distributions make each curve coordinate a tractable KL divergence.

3 Related Work

Existing evaluation metrics capture different aspects of machine–human text similarity, but many reduce distributional comparison to references, statistics, or single likelihood values. MAUVE instead summarizes divergence frontiers in a joint embedding space for open-ended generation.

  • Reference-based measures: Recent reference-based metrics capture distributional semantics in high-dimensional spaces, including Moverscore, which uses optimal transportation distance.Moverscore computes the minimum cost of transforming generated text into reference text using vector representations of n-grams.
  • MAUVE: MAUVE summarizes all points along the divergence curve using comparisons in a joint embedding space rather than independent per-distribution statistics.It extends information divergence frontiers to open-ended text generation and large-scale neural language-model outputs.
  • Reference-based measures: Reference-based metrics compare generated text with a small set of references, making them unsuitable when open-ended prompts allow multiple plausible continuations.They are better suited to targeted tasks such as translation and summarization, where matching references is central.
  • Statistics-based measures: Statistics-based measures compare scalar statistics of model and human distributions, so they focus on selected properties rather than the overall gap.Generation perplexity cannot trade off Type I and Type II errors because its statistic is scalar.
  • Language modeling metrics: Language-modeling metrics evaluate human text under the model distribution, but do not assess how likely model samples are under the human distribution.They also cannot be used for decoding algorithms such as beam search that do not define a token-level distribution.

4 Experiments

The experiments evaluate whether MAUVE captures known generation-quality properties, remains robust to approximation choices, and agrees with human judgments across open-ended text tasks.

  • Experimental setup: MAUVE is evaluated on web text, news, and stories using completion pairs, GPT-2 language models, and multiple decoding algorithms.The experiments vary generation length, decoding algorithm, model size, embeddings, quantization, and scaling settings.
  • Generation length: MAUVE decreases monotonically as generation length increases, whereas Fréchet distance improves, perplexity is non-monotonic, and sparsemax remains constant.The expected decline reflects deteriorating generation quality as longer outputs become more likely to wander or lose coherence.
  • Decoding algorithm: MAUVE ranks greedy decoding lowest (.016), ancestral sampling next (.882), and nucleus sampling highest (.940), matching the expected quality ordering.It also identifies adversarial and beam-search degeneracy, while generation perplexity is vulnerable to the adversarial decoder.
  • Model size: MAUVE increases with model size and agrees with human quality judgments, while generation perplexity incorrectly rates the large model’s text as best.Other language-modeling metrics capture size-quality differences but fail on length or decoding quality in the reported experiments.
  • Human judgments: MAUVE correlates highly with human judgments of human-likeness (0.952), interestingness (0.810), and sensibility (0.857), outperforming other automatic measures.Generation perplexity correlations are 0.12 to 0.17 lower, suggesting MAUVE may serve as an automatic surrogate for costly human evaluations.

5 Conclusion

MAUVE measures the gap between neural and human text for open-ended generation by summarizing a divergence curve, and its empirical performance supports its use as an automatic evaluation measure. The paper also identifies robustness, implementation details, and future directions, while acknowledging risks from rewarding human-like machine text.

  • 5 Conclusion: MAUVE measures the area under a divergence curve that summarizes errors between neural-generated and human-written text.The curve captures a spectrum of phenomena present in machine- and human-generated text.
  • 5 Conclusion: MAUVE correlated with human judgments and identified quality differences due to generation length, decoding algorithm, and model size.The paper states that prior metrics struggle to capture these differences.
  • Scope and broader impacts: The authors identify extensions to closed-ended tasks as promising future work and encourage development of systems that distinguish human from machine text.They note risks from rewarding systems that mimic human-authored text.
  • Divergence frontiers: The divergence curve encodes a Pareto frontier over KL divergences, with mixture distributions Rλ providing the frontier points.The supporting identity shows that Rλ minimizes the relevant weighted divergence objective.
  • Quantization: Quantization approximates distributions over encoded text representations as multinomial histograms, using schemes such as k-means and lattice quantization.The paper describes the quantized distribution as a piecewise-constant approximation in a Euclidean embedding space.
  • Method and implementation: MAUVE computes divergences after embedding sampled text, quantizing the embeddings, forming quantized distributions, and estimating the divergence curve numerically.The implementation uses an external language model for embeddings, clustering for quantization, and numerical quadrature for the area estimate.

C.1 Task Domains

The experiments cover open-ended generation in three domains: web text, news, and stories. Each domain uses a distinct prompt and model setup while comparing machine continuations with corresponding human text.

  • C.1 Task Domains: The study considers web text, news, and story continuation as three open-ended generation domains.The task settings differ across domains, as summarized by the paper.
  • Web text Generation: Web text generation extends the first 35 tokens of 5000 Webtext test articles with pretrained GPT-2 models of various sizes.Human-written continuations from the test set provide the comparison text.
  • News Generation: News generation produces article bodies from headlines and metadata using pretrained Grover architectures of various sizes.The metadata includes publication domain, date, and author names.
  • Story Continuation: Story continuation extends the first 50 tokens of 5000 WritingPrompts test samples using a GPT-2 medium model fine-tuned for one epoch.Machine generations may reach 512 tokens and are compared with truncated human-written test examples.

C.2 Training and Decoding Hyperparameters

The experiments use size-based Transformer models with human-text prompts and compare multiple decoding strategies, embedding choices, quantization methods, and automatic metrics. MAUVE’s ordering is preserved across scaling constants, while its k-means implementation is substantially faster than the alternatives described.

  • Training setup: The experiments use pretrained or fine-tuned size-based Transformer models and explore continuation generation conditioned on human-written prompts.The models are implemented with HuggingFace Transformers.
  • Story Continuation Finetuning: Story continuation fine-tunes GPT-2 medium on WritingPrompts for one epoch using cross-entropy loss and an effective batch size of 32.The stated optimizer is Adam with a learning rate of 5 × 10^-5.
  • Decoding Hyperparameters: The decoding comparison includes ancestral sampling, greedy decoding, and nucleus sampling across task-specific p values.Nucleus sampling uses p ∈ {0.9, 0.92, 0.95, 0.99} for web and stories, and p ∈ {0.9, 0.92, 0.94, 0.96, 0.98} for news.
  • MAUVE hyperparameters: MAUVE varies the scaling constant, embedding model, and quantization algorithm, including the size of the quantized distribution.The paper uses GPT-2 large embeddings and reports similar results with representations from models such as RoBERTA.
  • MAUVE hyperparameters: A scaling constant changes MAUVE’s numerical value but preserves the relative ordering between models.The authors choose c = 5 throughout their experiments.
  • Quantization methods: The study compares k-means, DRMM, and lattice quantization algorithms for constructing MAUVE representations.The k-means method applies PCA and normalization before clustering, while lattice quantization uses a learned four-dimensional representation and 744 bins.
  • Evaluation time: MAUVE-k-means takes 1–3 minutes for 5000 generations on one CPU core, compared with 1.75 hours for MAUVE-DRMM and about 5 minutes for MAUVE-Lattice.These timings use the hardware and cached representations described by the authors.

D Experiments: Additional Results

The additional-results materials organize fuller analyses across model size, decoding, text length, approximations, and hyperparameter tuning, alongside tables for web-text and news-generation comparisons. The supplied passages primarily identify these analyses and table contents rather than report their numerical outcomes.

  • Additional analyses: Appendix D.1 provides full results across model size and decoding, while Appendix D.2 provides full results across text length.These appendices elaborate on the analyses introduced in Section 4.1.
  • Web text results: Table 6 compares measures across web-text model sizes and decoding approaches, with human Bradley-Terry scores included for comparison.For nucleus sampling, the table reports the hyperparameter selected by MAUVE.
  • News results: Table 7 evaluates news generation across Grover model sizes and decoding approaches, including discrimination accuracy from a Grover mega model.The table reports the nucleus-sampling hyperparameter selected by MAUVE.
  • Additional analyses: Appendix D.3 studies approximations in MAUVE, and Appendix D.4 includes miscellaneous plots such as MAUVE for hyperparameter tuning.Section 4.3 is elaborated in Appendix E.

D.1 Comparison of Measures Across Model Size and Decoding

Across model sizes, decoding algorithms, and text lengths, MAUVE more consistently reflects expected generation-quality differences than alternative measures. It also distinguishes decoding quality and remains comparatively inexpensive for evaluating model–decoding pairs.

  • Model size: Across news and story domains, MAUVE generally rates larger models and nucleus sampling higher, with Grover large as an exception.Statistics-based measures instead prefer smaller Grover models in these comparisons.
  • Alternative measures: Language-modeling measures can mis-rank decoding methods because they do not directly compare model-generated text with human text.JS favors greedy decoding, ε-PPL favors ancestral sampling, and SP cannot distinguish ancestral from nucleus sampling.
  • Alternative measures: Self-BLEU can prefer GPT-2 medium over larger GPT-2 models, indicating that word/token statistics do not fully capture generated-text quality.This provides a concrete example of an alternative measure behaving unlike the expected model-size trend.
  • Computational practicality: MAUVE strongly correlates with discrimination accuracy across discriminator choices while being less expensive than fine-tuning a separate discriminator for each model–decoding pair.The comparison is especially relevant for larger models such as Grover mega.
  • Text length: MAUVE is the only measure in Figure 6 that consistently shows generation quality degrading as maximum text length increases.Fréchet distance sometimes decreases, Gen. PPL. is non-monotonic in some settings, and sparsemax is independent of generated samples.

D.3 Effect of Approximations of MAUVE

The study examines how MAUVE changes with embedding, quantization, cluster-count, and sample-size choices. Rankings are highly stable across several approximations, but small samples and extreme cluster counts distort scores.

  • Embedding model: Using RoBERTa large embeddings produces similar MAUVE trends to the default GPT-2 large embeddings.The two feature representations have a Spearman rank correlation of 0.993, although RoBERTa is restricted to 256 BPE tokens.
  • Quantization: MAUVE rankings remain nearly identical across k-means, DRMM, and lattice quantization choices.Spearman correlations with k-means at k = 500 are at least 0.99, while numerical MAUVE values can vary.
  • Sample size: Smaller sample sizes yield optimistic MAUVE estimates and larger variance.Figure 7 uses k = n/10 when plotting MAUVE against sample size n.

E Human Evaluation: Protocol and Full Results

The human evaluation compares human and machine continuations through pairwise judgments and Bradley–Terry scores. It measures interestingness, sensibility, and human-likeness across model and decoding conditions.

  • Evaluation protocol: Annotators compare continuations of the same prompt and independently judge which is more interesting, sensible, and human-like.Each question may have a different answer, allowing the three quality dimensions to diverge.
  • Compared sources: The evaluation includes one human source and eight GPT-2 model–decoding sources spanning four model sizes and two decoding algorithms.The evaluated machine sources use pure sampling or nucleus sampling.
  • Statistical model: Pairwise preferences are converted into Bradley–Terry scores, with each text source treated as a player.A higher Bradley–Terry score indicates stronger preference from human annotators.
  • Human-judgment relationships: Human-like and interesting scores correlate at 0.917, human-like and sensible scores at 0.917, and interesting and sensible scores at 0.967.Each reported correlation has p-value ≤5 × 10−4.
  • Statistical model: The Bradley–Terry model represents the probability that one source beats another with a sigmoid of their score difference.The scores assume independent head-to-head outcomes and are mean-normalized for uniqueness.

E.3 Full Results of the Human Evaluation

Human judgments generally favor nucleus sampling and larger models, and MAUVE tracks these preferences closely. The evaluation also reveals remaining gaps between machine generations and human text, plus some specific disagreements between MAUVE and annotators.

  • Human versus machine text: GPT-2 xl with nucleus sampling loses to human text with probability 0.578, while GPT-2 small with nucleus sampling loses with probability 0.679.These Bradley–Terry predictions indicate that machine-generated text remains distinguishable from human text in the evaluation.
  • Overall preferences: Human evaluations and MAUVE generally agree that nucleus sampling is better than pure sampling and larger models are better for the same decoding algorithm.Both identify GPT-2 small as better than GPT-2 medium under pure sampling as an exception.
  • Caveats: MAUVE disagrees with human evaluations on specific comparisons, including preferring GPT-2 medium nucleus sampling over pure sampling from GPT-2 large and xl.The paper leaves detailed study of this phenomenon to future work.
  • Evaluation setup: The evaluation compares nine sources using 90 comparisons between each pair, based on continuations prompted by the first 35 tokens of human text.The sources include human text and eight GPT-2 model–decoding combinations.
  • Evaluation setup: Prompts are manually filtered to 1831 well-formed English sentences from 5000 webtext test prompts.Generations are included only when each continuation reaches at least 200 BPE tokens and are truncated at 256 BPE tokens.

F Interpreting the Quantization

The quantization analysis finds that MAUVE’s clusters can capture semantically meaningful structure in human and generated news text, including geographic and source-related groupings.

  • Cluster analysis: MAUVE analyzes 150 news-source domains through cluster memberships computed from human text P and Grover Mega text Q.The generations use nucleus sampling with p = 0.96.
  • Cluster analysis: Some clusters are dominated by geographically similar or source-similar web domains.Among the 21 clusters containing at least 20 samples each, several show concentrated domain membership.
  • Cluster analysis: 7 clusters contain exactly one or two web domains each.
  • Geographic groupings: Cluster 254 groups Australian web domains, while Cluster 51 primarily groups Canadian domains with one Baltimore outlier.
  • Source groupings: Cluster 391 groups UK domains, and Cluster 421 contains three financial or investment domains.
  • Interpretation: These results show that some clusters form groupings humans recognize as meaningful without being given domain metadata beforehand.

G Example Generations

The examples contrast decoding behaviors in generated continuations, ranging from repetitive degeneration to incoherent text and more plausible but still imperfect continuations.

  • Decoding artifacts: Greedy decoding and adversarial perplexity sampling produce degenerate text outputs.
  • Semantic inconsistency: The examples also include continuations that introduce semantically inconsistent material, such as fabric-cleaning advice followed by unrelated food-storage instructions.
  • Decoding conditions: Nucleus sampling with p = 0.95 is presented as a distinct decoding condition for comparison.
  • Repetition: The greedy-style continuation repeatedly reproduces “a small, sharp knife” and “put the fabric in the hole.”
  • Overall contrast: Together, the examples illustrate that generated text can depart from coherent human-like discourse through repetition, incoherence, or abrupt topic shifts.
  • Incoherence: Another continuation becomes incoherent after a plausible opening, combining unrelated words and fragments.
Loading 2102.01454v3…