PaperlayerView exact revision on arXiv

Source-linked AI summary

Unifying Human and Statistical Evaluation for Natural Language Generation

Tatsunori B. Hashimoto, Hugh Zhang, Percy Liang

arXiv:1904.02792v1cs.CLcs.AIstat.ML

TL;DR

Natural language generation evaluation must capture both output quality and diversity, but human and statistical measures each cover only one dimension. The paper introduces HUSE, which estimates an optimal reference-versus-model discrimination error using human judgments and model probabilities, and shows across generation tasks that it exposes diversity defects and quality–diversity trade-offs missed by human evaluation alone.

  • Problem

    Human evaluation captures quality but misses diversity, while statistical evaluation captures diversity but inadequately measures quality; existing combinations can therefore be misleading.

  • Method

    HUSE combines human judgment scores approximating reference probabilities with model probabilities and estimates twice the leave-one-out error of a k-nearest neighbor discriminator.

  • Results

    Across four generation tasks, HUSE reveals deficiencies that human evaluation alone misses, and annealing increases model–reference distinguishability through diversity loss.

  • Takeaways & Limitations

    Models that score well under HUSE are evaluated as both high-quality and diverse because the metric uses human judgments together with model probabilities.

  • Takeaways & Limitations

    The ideal optimal discriminator is unavailable, and HUSE relies on crowdworker annotations, although substantially fewer annotations suffice for low-quality models.

Abstract

from arXiv · show

How can we measure whether a natural language generation system produces both high quality and diverse outputs? Human evaluation captures quality but not diversity, as it does not catch models that simply plagiarize from the training set. On the other hand, statistical evaluation (i.e., perplexity) captures diversity but not quality, as models that occasionally emit low quality samples would be insufficiently penalized. In this paper, we propose a unified framework which evaluates both diversity and quality, based on the optimal error rate of predicting whether a sentence is human- or machine-generated. We demonstrate that this error rate can be efficiently estimated by combining human and statistical evaluation, using an evaluation metric which we call HUSE. On summarization and chit-chat dialogue, we show that (i) HUSE detects diversity defects which fool pure human evaluation and that (ii) techniques such as annealing for improving quality actually decrease HUSE due to decreased diversity.

1 Introduction

The paper argues that evaluating generated text requires measuring both quality and diversity, because human and statistical evaluation each miss one of these dimensions. It proposes HUSE, which combines human judgments and model probabilities to expose such deficiencies and evaluates models across several generation tasks.

  • Human evaluation captures quality but fails to detect under-diverse models that can pass the quality bar while plagiarizing training data.Such models may have zero generalization ability despite producing human-acceptable text.
  • Statistical evaluation through perplexity captures diversity but inadequately measures quality, since occasional low-quality outputs can be heavily penalized.
  • Combining perplexity with human evaluation can misleadingly make one model appear both diverse and high-quality when the assessments use different generation procedures.Sampling supports the diversity estimate, whereas beam search supports the human-quality estimate.
  • HUSE defines evaluation as twice the optimal discriminator’s error when distinguishing reference sentences from model-generated sentences, thereby targeting both quality and diversity.Low quality makes generated text distinguishable, while low diversity makes reference text distinguishable.
  • HUSE estimates this ideal metric by combining model probabilities with human judgment scores in a two-dimensional space and applying a k-nearest neighbor classifier.The metric is defined as twice the classifier’s leave-one-out error.
  • Across language modeling, chitchat dialogue, story generation, and summarization, HUSE reveals model deficiencies that human evaluation alone misses, including diversity losses from annealing.

2 Optimal Discriminator

The paper formulates evaluation as discriminating between samples from a reference distribution and a model distribution. This ideal metric captures distributional similarity but is unattainable directly because the optimal discriminator is unavailable and human discriminators lack full knowledge of model behavior.

  • The generation setup samples a context x from p(x), then compares outputs drawn from the model distribution pmodel(y | x) and reference distribution pref(y | x).The reference distribution is generally human-generated but may instead be defined by experts.
  • The discriminator predicts whether each sampled output came from the reference or model distribution using an indicator variable z.
  • L* is twice the lowest classification error over all discriminators attempting to identify the source of x and y.
  • L* measures similarity between the model and reference distributions: it is 0 for disjoint distributions and 1 for identical distributions.
  • The ideal metric is unattainable because computing the optimal discriminator is infeasible, while human discriminators cannot fully know which sentences a model cannot generate.
  • A human discriminator can assign the same error to a perfect model and a model producing only one reference-supported sentence, failing to reveal the latter’s low diversity.
  • Learned discriminators are difficult to train on high-dimensional text and may not generalize reliably as evaluation metrics.

3 Human Unified with Statistical Evaluation (HUSE)

HUSE approximates the optimal human-versus-model discrimination score using model probabilities and human judgments as a two-dimensional feature map. Human typicality judgments provide a practical surrogate for reference probabilities, while the resulting score is at least as informative as human evaluation alone.

  • 3.1 Optimal feature map: The optimal discriminator depends on sentences only through their reference and model probabilities, yielding an exactly optimal two-dimensional feature map.The discriminator predicts reference when pref(y | x) exceeds pmodel(y | x).
  • 3.2 HUSE features: HUSE replaces the unavailable reference probability with human typicality judgments while retaining the directly computable model probability.The HUSE feature construction uses human judgments as a surrogate for log pref and normalizes model probability by sentence length.
  • 3.2 HUSE features: Human typicality judgments correlate strongly with Reddit sentence frequency (r = 0.92), supporting their use as a surrogate for log pref.The judgments average ratings from 20 crowdworkers, and the figure reports 90% confidence intervals.
  • 3.3 Guarantees derived from HUSE: HUSE is always tighter than human-only evaluation because it incorporates human evaluation together with model information.The paper also states that HUSE can provide approximation guarantees when human judgments approximate reference probabilities.
  • 3.3 Guarantees derived from HUSE: A low HUSE score is sufficient to show that a model is far from the reference distribution, although the score does not certify that a model is good.Feature-map scores provide optimistic upper bounds on the optimal score, so they can detect poor models without certifying quality.

4 Evaluating models with HUSE

HUSE is estimated by comparing reference and model outputs in feature space with leave-one-out classification. Its score can be decomposed into quality and diversity components, enabling analysis of quality-diversity tradeoffs.

  • 4.1 Estimating HUSE: HUSE estimation samples reference and model outputs for fixed contexts, computes feature vectors, and classifies each example by source.The classifier predicts whether each sentence comes from the reference distribution or the model.
  • 4.1 Estimating HUSE: A k-nearest neighbors classifier estimates the error rate in HUSE’s two-dimensional feature space using scaled L2 distances and k = 16.Nearest neighbors require no training and can asymptotically represent arbitrary continuous decision boundaries.
  • 4.2 Quality-diversity decomposition: HUSE-Q measures quality through the human-evaluation feature, while HUSE-D captures the additional discrimination attributable to limited diversity.The diversity interpretation relies on humans detecting quality defects, so increased error after removing model probability is attributed to diversity deficiency.
  • 4.2 Quality-diversity decomposition: HUSE-D = 1 indicates no diversity defects, whereas HUSE-D = 0 indicates perfect discrimination caused by lack of diversity.Under non-degenerate discriminators, the component scores lie in [0, 1] and satisfy the stated decomposition.

5 Experiments

The experiments evaluate HUSE across four generation tasks and examine quality–diversity tradeoffs under different generation schemes. HUSE reveals deficiencies that human quality judgments alone miss, including diversity losses from annealing and retrieval.

  • Experimental setup: HUSE evaluates language modeling, story generation, chit-chat dialogue, and summarization across diverse generation-quality tradeoffs.The experiments include neural models, temperature annealing, retrieval baselines on selected tasks, and human judgments from crowdworkers.
  • Experimental setup: The measured task–generation combinations are not comprehensive, though they generally cover available quality–diversity tradeoffs for conditional tasks.Retrieval was omitted for chit-chat and annealing was omitted for language modeling for cost or experimental-design reasons.
  • Overall results: HUSE scores vary widely: language modeling reaches HUSE = 0.86, while summarization and dialogue are highly distinguishable at HUSE ≈0.5 when sampled at t = 1.0.The table defines HUSE as overall goodness-of-fit, with the implied discriminator error equal to HUSE/2.
  • Overall results: Summarization quality rises from HUSE-Q 0.58 to 0.92 and dialogue from 0.56 to 0.92 at t = 0.7, while diversity falls from 0.95 to 0.34 and 1.0 to 0.57, respectively.The resulting quality improvement degrades overall HUSE because annealing causes severe diversity losses.
  • Overall results: All story-generation schemes are inadequate: the original model has HUSE-Q = 0.15, while retrieval improves quality to HUSE-Q = 0.47 at the expense of diversity.Direct sampling at t = 1.0 is always diverse, suggesting human evaluation is appropriate for directly sampled systems.
  • Model error analysis with HUSE: At t = 1.0, human judgment alone distinguishes many summarization outputs, while lower temperatures require pmodel to separate model and reference distributions.The two-dimensional classifier also identifies high-quality model successes, low-quality outputs, and under-diverse reference sentences.
  • HUSE stability: Accurate HUSE estimates often require 20 crowdworkers per example, although 50 test-set examples can suffice and story generation may need only 10 workers.The required number of annotations depends heavily on task distinguishability; summarization requires more than 20 workers.

6 Related work

Prior NLG evaluation commonly mixes human judgments with perplexity or n-gram diversity measures, but these practices use heterogeneous and potentially misleading measurements.

  • The current state of NLG evaluation: Among 26 ACL 2018 NLG papers, six used only human evaluation, fourteen combined human evaluation with diversity metrics, and six used no human judgments.The reported diversity metrics included perplexity and n-gram diversity.
  • The current state of NLG evaluation: Perplexity and n-gram counts can evaluate diversity in principle, but their practical implementations have serious drawbacks.The supplied related-work passage begins describing separate evaluation of human judgments and perplexity.

diversity and quality terms.

The paper situates HUSE among evaluation approaches that measure diversity and quality, arguing for a single measure applicable across generation tasks. It contrasts HUSE with n-gram, adversarial, and distributional methods that have important limitations or assumptions.

  • HUSE differs from prior diversity evaluations by using a single measure that captures quality and diversity across generation tasks.
  • N-gram diversity can remain high when a model emits one memorized utterance per context, because it counts unique n-grams across contexts.
  • HUSE addresses real-data settings where the reference distribution is unavailable by treating human judgments as an acceptable proxy for reference probabilities.
  • Unlike image-generation evaluations relying on accurate Fréchet Inception Distance estimates, HUSE directly leverages human judgments without that assumption.
  • Earlier optimal-classification methods classify sentences directly, whereas HUSE combines human evaluation with model probabilities.

7 Discussion

The discussion presents HUSE as a general evaluation strategy that combines model probabilities and human judgments to assess both quality and diversity. It reports that HUSE exposes under-diverse models that human evaluation alone fails to penalize.

  • 7 Discussion: HUSE addresses the failure of human evaluation to penalize under-diverse models by combining human judgments with model sampling probabilities.
  • 7 Discussion: HUSE can be applied to any model for which sampling probabilities can be calculated and is an upper bound on optimal classification error.
  • 7 Discussion: The framework is described as a “superhuman version” of the Turing Test because it approximates an optimal classifier using information about the model and reference distribution.

A.1 Relationship between total variation distance and optimal discriminator error

This appendix relates total variation distance between model and reference distributions to the error of an optimal discriminator. The proof partitions outcomes according to which distribution assigns higher probability and aggregates the resulting error contributions.

  • A.1 Relationship between total variation distance and optimal discriminator error: The total variation distance equals one minus the optimal discriminator error: ∥pmodel − pref∥TV = 1 − L∗.
  • A.1 Relationship between total variation distance and optimal discriminator error: The proof defines the set S where pmodel assigns higher probability than pref and aggregates pref over that set.
  • A.1 Relationship between total variation distance and optimal discriminator error: On S, the optimal discriminator predicts the model, yielding an error contribution based on the reference probability assigned to S.
  • A.1 Relationship between total variation distance and optimal discriminator error: The complementary region contributes the remaining error, and the total optimal error follows by summing the two contributions.

A.2 Approximation error from φ features

The appendix analyzes approximation error when HUSE uses feature-based classification rather than the optimal discriminator. It then describes the human-judgment procedure and replication requirements used to estimate the features in practice.

  • A.2 Approximation error from φ features: Theorem 1 bounds the error of a classifier using feature map φ relative to the optimal classification error L∗ through conditional mutual information.
  • A.2 Approximation error from φ features: When φ is an invertible function of the optimal feature map, the conditional mutual information is zero and the feature-based error equals L∗.
  • A.3 Amazon Mechanical Turk for human judgments: Human judgment scores HJ are estimated by averaging Amazon Mechanical Turk ratings of sentence typicality on a 0–5 scale.
  • A.3 Amazon Mechanical Turk for human judgments: At least twenty replicates are needed for difficult classification tasks such as summarization, whereas story generation requires five to ten replicates.
Loading 1904.02792v1…