Source-linked AI summary
Trading Off Diversity and Quality in Natural Language Generation
Hugh Zhang, Daniel Duckworth, Daphne Ippolito, Arvind Neelakantan
TL;DR
Open-ended language generation lacks consensus on how to compare decoding algorithms across quality and diversity, despite the need to balance both. The paper evaluates decoders with a multi-objective framework and studies selective sampling; it finds similar performance when diversity is prioritized but a nucleus-sampling advantage when quality matters more.
Problem
Open-ended generation needs diverse, high-quality responses, but there is no consensus on the best decoder or criteria for comparing methods across those goals.
Source: §1 · pp. 1–2Method
Results
Decoders perform similarly when diversity is prioritized, while nucleus sampling outperforms the other evaluated methods when quality matters more; the paper also confirms the likelihood trap.
Source: §1 · p. 2Takeaways & Limitations
Decoder comparisons should span the quality–diversity frontier, since their relative performance depends on how much quality is valued against diversity.
Source: §1 · p. 2, §6 · p. 8The study leaves extension to more language models and image or music generation, and adaptive prompt-specific cutoffs for selective sampling, for future work.
Source: §6 · p. 9
Abstract
from arXiv · showhide
For open-ended language generation tasks such as storytelling and dialogue, choosing the right decoding algorithm is critical to controlling the tradeoff between generation quality and diversity. However, there presently exists no consensus on which decoding procedure is best or even the criteria by which to compare them. We address these issues by casting decoding as a multi-objective optimization problem aiming to simultaneously maximize both response quality and diversity. Our framework enables us to perform the first large-scale evaluation of decoding methods along the entire quality-diversity spectrum. We find that when diversity is a priority, all methods perform similarly, but when quality is viewed as more important, the recently proposed nucleus sampling (Holtzman et al. 2019) outperforms all other evaluated decoding algorithms. Our experiments also confirm the existence of the `likelihood trap', the counter-intuitive observation that high likelihood sequences are often surprisingly low quality. We leverage our findings to create and evaluate an algorithm called \emph{selective sampling} which tractably approximates globally-normalized temperature sampling.
1. Introduction
Open-ended generation requires diverse, high-quality responses, but evaluating quality depends on costly human judgments and decoder comparisons have been limited. The paper frames decoding as a quality–diversity tradeoff, evaluates methods across that spectrum, and studies likelihood and selective sampling.
1. Introduction
- Human judgments are the credible measure of generated-text quality, but their cost makes direct optimization prohibitively expensive.The paper describes training to imitate human-written text and inference-time decoding as a two-stage process.Source: §1 · p. 1
- The paper frames generation as multi-objective optimization over quality and diversity, accommodating tasks that assign them different weights.The framework is described as applicable from machine translation and summarization to storytelling.Source: §1 · p. 2
- When diversity is highly valued, decoders perform similarly; when quality matters more, nucleus sampling outperforms the other evaluated algorithms.The comparison spans the quality–diversity spectrum and uses over 38,000 ratings on almost 10,000 samples.Source: §1 · p. 2
- Human quality judgments generally rise with model likelihood, then decline beyond an inflection point, so the highest-likelihood sentences can be surprisingly low quality.The paper calls this counter-intuitive pattern the likelihood trap.Source: §1 · p. 2
- Selective sampling draws from a global temperature-adjusted model distribution using rejection sampling without computing its partition function, but performs poorly in evaluation.The authors suggest local token-by-token decoders may capture structure that a global decoder does not.Source: §1 · p. 2
2. Framework
The framework evaluates language generation by combining human-rated quality with diversity, whose relative importance depends on the task. Because direct optimization relies on human judgments, the paper instead relates this objective to decoding procedures.
2. Framework
- Autoregressive language models factor sequence likelihood left-to-right, and decoders can post-process token probabilities to favor high-likelihood tokens.Random sampling instead draws tokens sequentially from the model's conditional distribution.Source: §2 · p. 2
- Quality is the expected human quality judgment for sentences drawn from the model.The framework defines quality for an individual sentence through human judgment HJ(x).Source: §2 · p. 2
- Diversity is measured using Shannon entropy.Source: §2 · p. 2
- The objective combines model quality and diversity as G(p) = Q(p) + λH(p), with λ setting their task-specific relative importance.Larger λ suits open-ended tasks such as dialogue; smaller values suit summarization or machine translation.Source: §2 · p. 3
- Directly optimizing the combined objective is infeasible because it depends on human judgments; prior work instead optimizes a proxy before decoding warps the model distribution.The paper next relates its objective to existing decoders and considers global rather than token-by-token normalization.Source: §2 · p. 3
3. Selective Sampling
The section shows that maximizing sequence likelihood can select low-quality responses, then develops selective sampling to approximate globally normalized temperature sampling. It also explains why local temperature sampling cannot reproduce every global-temperature distribution.
3.1. The Likelihood Trap
- Response quality rises with log pmodel(x) until −58.09, then falls, so maximizing sequence likelihood can yield suboptimal quality—the likelihood trap.The examples describe very high-likelihood sentences as tending toward extreme repetition, while crowdworkers rated overall quality.Source: §3.1 · p. 3, Table 1 · p. 4
3.2. Global Temperature Sampling
- The framework uses a proxy objective combining expected quality and entropy to trade off quality against diversity.The relative weight λ is task-specific; larger values matter for open-ended tasks that prioritize variety.Source: §3.2 · p. 3
- P(A) = 0.5763 and P(B) = 0.4237 at global temperature τ = 0.5, although local temperature sampling must keep P(A) = P(B).The example illustrates that some globally temperature-adjusted joint distributions cannot be matched by any local temperature setting.Source: Figure 2 · p. 4, §3.2 · p. 4
- Selective sampling uses rejection sampling to draw from the globally temperature-adjusted distribution without explicitly computing its partition function.The model distribution serves as the proposal distribution.Source: §3.2 · pp. 3–4
- The method sets cutoff α at the human-judgment likelihood inflection point and discards generations above it, whose total probability mass is less than 0.5% in the experiments.The cutoff also permits a tighter envelope bound that increases acceptance probabilities by several orders of magnitude.Source: §3.1 · p. 3, §3.2 · p. 5
4. Experiments
The experiments compare decoding configurations across entropy levels and evaluate selective sampling. Quality differences emerge mainly at low entropy, where nucleus sampling leads among autoregressive methods, while selective sampling underperforms.
4. Experiments
- The study sweeps temperature, top-k, top-p, and selective-sampling configurations; hyperparameter extremes converge to greedy decoding and random sampling.Source: §4 · p. 5
4.1. Setup
- Across 48 GPT-2 prompts, 20 configurations generated ten samples each; 146 qualified raters scored 30-token outputs, yielding nearly 10,000 samples and over 38,000 pairwise ratings.Source: §4.1 · p. 5, §4.2 · p. 6
4.2. Results
- At low entropy, nucleus sampling outperforms top-k, which outperforms temperature; across a wide range of matched entropy, autoregressive methods are comparable.Source: Figure 4 · p. 6, §4.2 · p. 6
- Selective sampling consistently underperforms all other evaluated decoding algorithms, including more computationally efficient strategies.Source: §4.2 · p. 6, Figure 5 · p. 7
4.3. Selective Sampling
- Across strategies, lower diversity tends to yield higher-judged equal-log-likelihood pairs; temperature sampling at t = 0.7 is preferred to model samples.Source: §4.3 · p. 6, Figure 6 · p. 7
- Unlike locally normalized autoregressive decoders, selective sampling lacks their additional preference beyond high joint likelihood.Source: §4.3 · p. 7
- A single global α may have little effect for some prompts but eliminate nearly half the samples for others, leaving only lower-quality samples.Source: §4.3 · p. 7
5. Related Work
Prior work explores both decoding changes and training-based approaches to diversity, but comparisons leave the quality–diversity tradeoff central. This paper situates its contributions alongside prior accounts of the likelihood trap and generative-model objectives.
Encouraging Diversity
- Diversity strategies either modify decoding or change training, while training-based approaches add complexity when adapting existing models.Examples include diverse beam search, training objectives that discourage repetition, and adversarial learning.Source: §5 · pp. 7–8
- Prior comparisons found the quality–diversity tradeoff made it nearly impossible to identify one decoding method as ubiquitously best.The paper contrasts decoding changes that encourage beam diversity with sampling modifications aimed at reducing diversity to increase quality.Source: §5 · p. 8
- Top-k, nucleus, and temperature sampling each control the quality–diversity tradeoff with one hyperparameter, using different token-selection rules.Top-k restricts sampling to the most likely tokens; nucleus sampling adaptively selects tokens by cumulative probability mass; temperature rescales logits.Source: §5 · p. 8
Oversampling and Ranking
- Oversampling and ranking generations predates this work; its contribution connects selective sampling to reverse KL and competing-objective maximization and estimates sentence densities.Such density estimates are typically unavailable for oversampling algorithms.Source: §5 · p. 8
Likelihood Trap
- Earlier work observed likelihood-trap effects, while this paper measures likelihood against human quality across the model probability space, including low-quality generations at both extremes.Holtzman et al. (2019) reported repetitive generations under likelihood maximization.Source: §5 · p. 8
- A proposed explanation attributes the likelihood trap to low-quality training examples; carefully screening a smaller dataset avoided it, but such screening may not scale to increasingly large datasets.Ott et al. (2018) present the training-data explanation and the smaller-dataset result.Source: §5 · p. 8
Frameworks
- Unlike frameworks that seek to reproduce the training distribution, human-judgment-based evaluation can capture superhuman performance without requiring models to imitate human errors.Some tasks, including representation learning and the Turing Test, do require indistinguishability as an ultimate goal.Source: §5 · p. 8
6. Conclusion
The paper evaluates decoding methods across the quality–diversity frontier, reports when methods perform similarly or differently, and proposes selective sampling. It identifies extensions to other models, modalities, and sampling choices as future work.
6. Conclusion
- Existing decoders are broadly interchangeable when diversity is prioritized, while nucleus sampling performs best when quality is valued more.The paper also reports evidence for the likelihood trap and an algorithm for tractably estimating globally normalized temperature sampling.Source: §6 · p. 8
- The paper proposes a framework for decoder evaluation across the quality–diversity frontier and selective sampling to estimate globally normalized temperature sampling.The conclusion describes the evaluation as the first large-scale study across that frontier and selective sampling as the first tractable estimator.Source: §6 · p. 8
- Future work includes additional language models and modalities such as image and music generation, as well as adaptive cutoffs and proposal distributions beyond random sampling.The paper leaves these selective-sampling extensions for future discovery.Source: §6 · p. 9
A.1. Proof of Proposition 1
Choosing K* permits the Lagrangian dual to be written in the displayed form; the proof then relates temperature sampling to local maxima and minima.
A.1. Proof of Proposition 1
- Choosing K* permits the Lagrangian dual of the constrained optimization problem to be written in the displayed form.The expression is λ log P(x) + log Q(x) + 1 + µ = 0.Source: §A.1 · p. 11
- The proof identifies 1/τ with temperature sampling.It then uses positive temperatures for local maxima and negative temperatures for local minima to complete the proof.Source: §A.1 · p. 11
- Positive temperatures give local maxima, while negative temperatures give local minima.The passage presents this observation as the final step completing the proof.Source: §A.1 · p. 11
A.2. Experimental Design
The appendix details crowdsourced rating collection, pairwise-preference analysis, and repeatability checks. In the five-rater follow-up, the main decoding-method trends persisted despite low inter-rater agreement.
A.2. Experimental Design
- Each task paired a 20-word news context with five 30-word-piece continuations, rated from High Quality to Terrible after label descriptions and examples.The contexts came from GPT-2’s WebText dataset; the sample task shows five continuations sharing a context.Source: §A.2 · pp. 11–12, Figure 8 · p. 12, Figure 9 · p. 13
- Only workers rating all five qualification continuations correctly or nearly correctly qualified; 136 of 550 surveyed workers met the criterion.Nearly correct allowed at most one label error, and that error could be at most one level from the correct label.Source: §A.2 · p. 11
- The analysis converted five absolute ratings into 20 directed pairwise preferences, scoring ties 0 and higher-rated continuations +1 or -1 by order.The two directions for each pair received opposite scores when ratings differed.Source: §A.2 · p. 11
- In the A/A check, two rater pools evaluated the same 150 tasks, and their results showed consistent preferences for top-p over top-k and local temperature sampling.The analysis also found severe disapproval of random model sampling, consistent with the full-scale experiment.Source: §A.2 · p. 11, Figure 10 · p. 13
- The five-rater follow-up yielded Fleiss’s Kappa of 0.1964, indicating some correlation among raters while showing that tasks remained far from unambiguous.The figure caption likewise characterizes agreement as low while noting consistency between the two experiments’ Kappa results.Source: §A.2 · p. 11, Figure 11 · p. 13
- A/A Cohen’s Kappa was 0.19578, nearly identical to the inter-rater experiment; the similarity supports repeatability despite task ambiguity.The authors present this as evidence for the repeatability of the experimental design.Source: §A.2 · pp. 11–12
- Top-p remained preferred to all other methods; increased diversity correlated with lower human-judgment scores, while random model sampling scored lowest by a large margin.The five-rater follow-up reports these trends as consistent with the full-scale experiment.Source: §A.2 · p. 12, Figure 12 · p. 14