PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating the Evaluation of Diversity in Natural Language Generation

Guy Tevet, Jonathan Berant

arXiv:2004.02990v3cs.CL

TL;DR

NLG lacks a principled way to evaluate output diversity, especially across different aspects such as form and content. The paper introduces correlation-based decoding and content tests, showing that humans outperform automatic metrics for content diversity while decoding parameters mainly affect form. It also releases the McDiv benchmark and limits its analysis to form–content differences.

  • Problem

    NLG has no principled, standard method for evaluating diversity metrics, and it remains unclear whether humans can reliably judge diversity.

  • Method

    The paper evaluates metrics by correlating their scores with diversity parameters in decoding and human content-diversity tests.

  • Results

    Humans substantially outperform automatic metrics at detecting content diversity, while decoding parameters predominantly control text form rather than content.

  • Takeaways & Limitations

    The framework, best practices, and McDiv benchmark provide a platform for comparing diversity metrics and developing content-diversity-aware metrics.

  • Takeaways & Limitations

    The paper limits its scope to differences between form and content diversity, leaving other aspects such as sentiment diversity for future work.

Abstract

from arXiv · show

Despite growing interest in natural language generation (NLG) models that produce diverse outputs, there is currently no principled method for evaluating the diversity of an NLG system. In this work, we propose a framework for evaluating diversity metrics. The framework measures the correlation between a proposed diversity metric and a diversity parameter, a single parameter that controls some aspect of diversity in generated text. For example, a diversity parameter might be a binary variable used to instruct crowdsourcing workers to generate text with either low or high content diversity. We demonstrate the utility of our framework by: (a) establishing best practices for eliciting diversity judgments from humans, (b) showing that humans substantially outperform automatic metrics in estimating content diversity, and (c) demonstrating that existing methods for controlling diversity by tuning a "decoding parameter" mostly affect form but not meaning. Our framework can advance the understanding of different diversity metrics, an essential step on the road towards better NLG systems.

1 Introduction

NLG diversity lacks a standard evaluation method, making metric comparisons difficult. The paper proposes correlation-based tests, finds humans better at detecting content diversity than automatic metrics, and releases the McDiv benchmark.

  • There is no standard metric for measuring NLG diversity, so different studies evaluate diversity differently and are difficult to compare.
  • The framework evaluates a diversity metric by measuring its correlation with a parameter that controls diversity in generated response sets.The tester may be a neural model or a human, and high correlation indicates that the metric captures the parameter's effect on output diversity.
  • The paper tests n-gram-based, neural, and human diversity metrics using decoding and content-diversity tests.
  • Humans substantially outperform automatic metrics at detecting content diversity, while n-gram-based metrics correlate well with decoding parameters but mainly reflect text form.
  • The authors construct and release the McDiv benchmark to support research on content-diversity metrics.

2 Background: Diversity Evaluation

Prior work uses varied metrics for NLG diversity, but there is no consensus about what these metrics measure. Existing approaches include perplexity, n-gram and embedding-based metrics, and human evaluation, each with limitations.

  • Perplexity does not purely measure diversity, and high perplexity does not entail low diversity.Perplexity also evaluates a language model rather than the full NLG system, whose diversity depends partly on decoding.
  • N-gram-based metrics: N-gram-based metrics include distinct n-grams, Shannon entropy over n-grams, and Self-BLEU, which measures similarity among generated sentences.
  • Embedding-based metrics: Embedding-based approaches evaluate diversity in a latent space using clustering inertia or the volume induced by embedded sentences.
  • Human evaluation: Prior human-evaluation studies assess diversity in essays or conversations, whereas this paper focuses on diversity across responses to the same context.
  • There is no consensus on how to evaluate diversity or what existing diversity metrics actually measure.

3 Evaluating Diversity Metrics

The framework evaluates diversity metrics by correlating their scores with a parameter that controls an aspect of generated-text diversity. It instantiates this approach in decoding and content tests using machine or human testers.

  • Framework: The framework covers distinct diversity facets, including form diversity and content diversity.The authors note that sets can differ in content while sharing similar form, or differ in form despite similar content.
  • Framework: The framework measures a diversity metric’s sensitivity by correlating its scores on response sets with a deterministic diversity parameter.The tester generates response sets conditioned on a context and parameter; testers may be neural models or humans.
  • Framework: The framework evaluates metrics across machine- and human-generated response sets to test whether they capture the intended diversity aspect.A hypothetical sentiment-diversity test illustrates how correlations between metric scores and the parameter measure sensitivity to that aspect.
  • Decoding test: In the decoding test, a language model generates responses while a decoding parameter such as temperature serves as the diversity parameter.The test uses automatically generated data and examines metric correlations with decoding parameters.
  • Content test: In the content test, crowdsourcing workers generate response sets with similar form diversity but low or high content diversity controlled by a binary parameter.The parameter d ∈ {0, 1} corresponds to low or high content diversity.

4 Human Diversity Score

The human-diversity-score experiments investigate whether people can reliably judge diversity and which elicitation practices produce useful judgments. The authors use multiple task variations and report practical constraints from pilot studies.

  • Motivation: The study asks whether humans can reliably evaluate diversity, a question whose importance is established by the role of human judgment in quality evaluation.Workers rate the diversity of response sets shown with a context.
  • Evaluation practices: The authors compare multiple Human Diversity Score variations to establish practices for eliciting diversity judgments.Variations include rating set diversity, inferring diversity from pairwise similarity, and comparing sets.
  • Limitations: The comparison of whether humans could judge one response set more diverse than another produced no conclusive result.The authors describe this experiment in Appendix C.
  • Pilot findings: Pilot studies found that perceived quality biases human diversity ratings, so workers were separately asked to assess quality and ignore it during diversity judgments.This procedure was intended to neutralize quality bias.
  • Pilot findings: Pilot studies also prompted workers to generate a sentence before rating diversity and limited response sets to no more than 10 responses.The first step was used to ensure workers read the context; larger sets were difficult to evaluate.

5 Diversity to Similarity Reduction

The paper reduces sentence similarity metrics to diversity metrics by negating mean pairwise similarity across response sets. This enables diversity evaluation using existing n-gram and neural similarity methods.

  • Reduction: A diversity metric can be constructed from any symmetric two-sentence similarity metric by negating the mean similarity across all unordered response pairs.The reduction converts pairwise similarity into a set-level diversity score.
  • Applications: The reduction makes it straightforward to define diversity metrics from prior sentence-similarity research.The authors apply it to both n-gram-based and neural semantic similarity metrics.

6 Experiments

The experiments evaluate diversity metrics through decoding and content tests across several NLG tasks. They find that automatic metrics track decoding changes better than human scores, while humans best detect content diversity and n-gram metrics mainly capture form.

  • Experimental setup: The experiments cover story completion, dialog response generation, and prompt generation, using distinct n-gram, neural, and human diversity metrics.The decoding test uses neural generators and temperature or other decoding parameters; the content test uses human-generated response sets with controlled content diversity.
  • Decoding test: The decoding test measures correlations between metric scores and the softmax temperature parameter for neural-model-generated response sets.Each task uses sets of 10 responses per context across a 100-value temperature sweep from 0.2 to 1.2.
  • Decoding test: N-gram metrics generally capture temperature-induced diversity better than neural and human metrics, although Pearson and Spearman correlations can disagree.For respGen, cosine similarity reaches 0.6 Pearson correlation, compared with 0.75 for BERT-STS and 0.77 for absHDS; cosine similarity is highly rank-correlated but nonlinear.
  • Decoding test: Temperature changes predominantly affect text form rather than content, and Top-p correlates with temperature across tasks whereas Top-k does not.The decoding results also indicate that rating an entire response set is more useful than averaging pairwise ratings, since simHDS performs worse despite costing 3x more.
  • Content test: In the content test, n-gram metrics perform poorly, neural metrics perform better, and humans substantially outperform all automatic metrics.Human HDS separates high- and low-content-diversity classes more clearly, while n-gram metrics saturate both classes near maximal values.
  • Content test: The McDiv benchmark contains 6K context-response-set pairs, including 3K McDivnuggets examples that neutralize form diversity for content-metric evaluation.On McDivnuggets, neural metrics perform substantially worse than on McDiv, and the gap between sent-BERT and absHDS increases from 0.04 to 0.1 in Spearman’s ρ for respGen.

7 Aspects of Diversity

The paper distinguishes content diversity, or what is said, from form diversity, or how it is said, and notes that these aspects can interact. Form can vary while content remains similar, whereas diversifying content generally also changes form.

  • Content diversity concerns what responses say, while form diversity concerns how those responses are expressed.
  • Content diversity can involve varying sentiment, relevance, and other response characteristics.
  • High form diversity with low content diversity is relatively easy to produce, but diversifying content without changing form is almost impossible.

8 Conclusions

The work presents a framework for evaluating diversity metrics as a step toward standardized evaluation, while limiting its scope to form and content diversity. It also reports crowdsourcing procedures and costs for human diversity judgments.

  • The framework evaluates diversity metrics as a step toward standardized evaluation.
  • The study limits its scope to differences between form and content diversity, while leaving other aspects for future work.
  • Human Diversity Score judgments were collected from qualified English-speaking Amazon Mechanical Turk workers.
  • The questionnaires included diversity questions for absolute, aspect-based, ranking, and similarity-based Human Diversity Score variants.
  • $0.18 was the cost for a single rating for HDS metrics requiring one query per response set.
  • $0.056 was the cost of a single similarity-based rating, making that experiment three times more expensive at $1120.

B Data Samples

The appendix provides data samples for decoding and content tests across several generation tasks. Decoding samples vary decoding parameters, while content samples contrast low- and high-content-diversity sets.

  • Decoding-test samples cover storyGen, respGen, and promptGen with neural testers and three decoding methods.
  • Each decoding response set uses a different value of softmax temperature, Nucleus sampling, or Top-k.
  • Content-test samples cover storyGen, respGen, and promptGen with human testers.
  • For each content-test context, one response set belongs to the low-content-diversity class and one to the high-content-diversity class.

C.1 Decoding Test (decTest)

The decoding test compares diversity metrics with decoding parameters across tasks and evaluation designs. Top-k is less reliably represented than other decoding methods, while human ranking performs better than absolute scoring but does not separate automatic metrics.

  • StoryGen produces noisier decoding-test scores across metrics, yielding lower ρ values and higher variance than other tasks.
  • Top-k consistently yields lower ρ than other decoding parameters, especially for storyGen, implying less reliable diversity representation.
  • The ranking experiment compares response sets generated with different temperature values by sweeping temperature differences.
  • Ranking performance is measured with Spearman’s ρ and accuracy for predicting which set has the higher temperature.
  • Humans rank better than they score absolutely and perform as well as automatic metrics, whose scores also improve and become harder to distinguish.

C.2 Metrics for Content Diversity (McDiv)

McDiv and its form-neutralized McDivnuggets subset are used to test whether diversity metrics track content diversity rather than surface variation. On McDivnuggets, n-gram metrics become uninformative, while neural metrics degrade relative to McDiv and remain behind human judgments.

  • Dataset construction: McDivnuggets was subsampled by sorting examples by distinct-n score, grouping them into sets of 40, and balancing low- and high-content-diversity samples within each group.Each group contributes equal class counts up to the size of its smaller class.
  • Results: On McDivnuggets, n-gram metrics score near-zero correlation, making low- and high-content-diversity classes nearly indistinguishable from surface features.The result is intentional: form diversity is neutralized in this subset.
  • Results: Neural metrics perform strictly worse on McDivnuggets than on McDiv, indicating reduced performance when surface-level diversity is controlled.The comparison is reported across the three NLG tasks.
  • Results: For 200 respGen samples from McDivnuggets, the gap between sent-BERT and absHDS was 0.1 in Spearman’s ρ, compared with 0.04 in the earlier comparison.The larger gap favored human diversity judgments.

D Additional Reproducibility Details

The paper documents its released data, code, datasets, model configurations, runtime, and supplementary examples for reproducibility. These materials cover both decoding-parameter tests and content-diversity tests across storyGen, respGen, and promptGen.

  • Released resources: Collected data, per-sample metric scores, and code for running and visualizing decTest and conTest are publicly available.The collection procedures are described in Section 6.
  • Datasets: The original datasets include ROC Stories for storyGen, Reddit comment-response data for respGen, and CMDC movie-script responses for promptGen.ROC Stories and Reddit data were used without further processing; CMDC contexts were reduced to the first three words.
  • Models: MASS was fine-tuned separately for storyGen and respGen, while GPT-2 large was used without fine-tuning for promptGen.MASS fine-tuning used 200K examples over 30 epochs; GPT-2 large had 774M parameters.
  • Runtime: Each decTest and conTest run with 200 samples takes less than a minute on a standard Intel i7 CPU when metric scores are available.This runtime refers to executing the tests, not generating model outputs.
  • Supplementary materials: Supplementary tables provide decTest samples across temperatures, nucleus-sampling p values, and Top-k values for all three generation tasks.Additional tables show conTest samples for storyGen, respGen, and promptGen, while figures document the human-judgment interfaces.
Loading 2004.02990v3…