The Curious Case of Neural Text Degeneration

1 INTRODUCTION

• Likelihood-maximizing decoding such as beam search produces generic, repetitive, awkward text even with GPT-2 Large, while human token probabilities exhibit greater variance.Models generally score well-formed text highly, but their highest-scoring longer texts can nevertheless be degenerate.Source: §1 · p. 2, Figure 2 · p. 2
• Pure sampling produces incoherent, contextually unrelated text because its unreliable tail contains tens of thousands of low-probability candidate tokens that are collectively over-represented.Pure sampling draws directly from the model’s predicted probabilities without truncating this tail.Source: §1 · p. 2
• Nucleus Sampling dynamically expands and contracts its candidate pool to sample from the top-p probability mass rather than a fixed top-k.The nucleus typically contains one to a thousand candidates. Temperature scaling alone does not sufficiently suppress the unreliable tail.Source: §1 · p. 2
• Maximization and top-k sampling produce overly probable text with insufficient diversity, whereas pure sampling produces text less likely than human references and lower in quality.The comparisons examine generated-text perplexity and the likelihood of entering repetition against the reference distribution.Source: §1 · p. 2
• Nucleus Sampling can match reference perplexity by tuning p, avoiding incoherence associated with the large k needed for top-k sampling to match human statistics.Vocabulary usage and Self-BLEU (Zhu et al., 2018) require high k values to match human statistics, but these settings often produce high variance in likelihood.Source: §1 · p. 2
• Human Unified with Statistical Evaluation (HUSE; Hashimoto et al., 2019) identifies Nucleus Sampling as the best overall decoding strategy for quality and diversity.HUSE jointly assesses quality and diversity, which neither human nor automatic evaluation alone can capture.Source: §1 · p. 2

2.1 TEXT GENERATION DECODING STRATEGIES

• The paper pursues model-agnostic decoding, proposing Nucleus Sampling as complementary to diversity-promoting search and potentially to training-time solutions.Diverse beam search uses task-specific scores or constraints separating hypotheses (Li et al., 2016a; Vijayakumar et al., 2018; Kulikov et al., 2019; Pal et al., 2006), but still requires appropriate decoding. Welleck et al. (2020) instead address degeneration through an unlikelihood loss targeting repeated tokens, implicitly reducing gradients on frequent tokens.Source: §2.1 · p. 3
• Maximization-based generation tends toward high grammaticality but low diversity (Kulikov et al., 2019; Holtzman et al., 2018; Fan et al., 2018).GAN-generated text does not outperform language-model generations when quality and diversity are considered jointly (Caccia et al., 2018; Tevet et al., 2019; Semeniuta et al., 2018).Source: §2.1 · p. 3

2.2 OPEN-ENDED VS DIRECTED GENERATION

• Directed generation tightly constrains outputs through inputs and typically uses beam search, making repetition and genericness less problematic.Examples include translation, data-to-text generation, and summarization, typically using encoder-decoder architectures. Problems nevertheless occur with large beams (Koehn & Knowles, 2017) and exact inference, despite their more comprehensive probability-maximizing search.Source: §2.2 · p. 3
• Open-ended generation permits substantial freedom in plausible continuations despite contextual constraints, making this freedom the paper’s focus.Examples include conditional story generation and contextual text continuation; goal-oriented dialog may fall between open-ended and directed generation.Source: §2.2 · p. 3

3 LANGUAGE MODEL DECODING

• Maximization-based decoding assumes higher-probability text is better, yet yields low-quality open-ended generations (Fan et al., 2018; Holtzman et al., 2018).Exact sequence maximization is intractable for recurrent models and Transformers (Chen et al., 2018), so beam search is commonly used (Li et al., 2016b; Shen et al., 2017; Wiseman et al., 2017).Source: §3 · pp. 3–4

3.1 NUCLEUS SAMPLING

• Nucleus Sampling samples from the smallest set of highest-probability tokens whose cumulative probability mass exceeds a chosen threshold p.The retained distribution is renormalized before sampling, and the set changes dynamically at each step. For high p, this nucleus contains most probability mass within a small vocabulary subset.Source: §3.1 · pp. 4–5

3.2 TOP-k SAMPLING

• Top-k sampling improves text quality over beam search and full-distribution sampling, but a fixed k is suboptimal across varying contexts.It renormalizes the k highest-probability tokens, retaining probability mass that can vary widely between steps. Small k risks bland text in flat distributions; large k admits inappropriate candidates whose sampling probabilities increase through renormalization.Source: §3.2 · pp. 5–6

3.3 SAMPLING WITH TEMPERATURE

• Lowering temperature improves generation quality at the cost of diversity (Caccia et al., 2018; Hashimoto et al., 2019).Setting t in [0, 1) favors high-probability events and reduces tail mass. Low temperature can also reshape distributions before top-k sampling to partially alleviate its problems (Fan et al., 2018).Source: §3.3 · p. 6

4.1 EXPERIMENTAL SETUP

• 5,000 passages are generated using the 762M-parameter GPT2 Large model, trained on 40GB of WebText.Unless otherwise noted, generation is conditioned on the initial paragraph, restricted to 1–40 tokens, of held-out WebText documents. Passages end at an end-of-document token or 200 tokens.Source: §4.1 · p. 6

4.2 PERPLEXITY

• Nucleus Sampling achieves perplexity closest to human text among the tested decoding strategies.Perplexity is evaluated against gold text using the generating model; minimizing it can sacrifice diversity and cause repetition. Pure sampling produces worse perplexity than gold by oversampling unlikely tokens, while lowering temperature creates diversity and repetition issues.Source: §4.2 · p. 6, Table 1 · p. 7

4.3 NATURAL LANGUAGE DOES NOT MAXIMIZE PROBABILITY

• Natural text has lower average per-token probability than beam-search output and favors lower-probability, informative tokens over sustained high-probability sequences.Natural language also avoids repetition loops, although the model tends to assign them high probability; these observations challenge search error as the explanation for maximization’s failures.Source: §4.3 · p. 7
• The authors conjecture that human language’s preference for informativeness makes larger models or improved architectures unlikely to solve degeneration under standard per-word objectives.Grice’s Maxims of Communication describe people avoiding the obvious. The authors argue that word-at-a-time language models without global text modeling struggle to capture this property.Source: §4.3 · p. 7

5.1 ZIPF DISTRIBUTION ANALYSIS

• Pure sampling most closely matches human vocabulary frequencies, followed by Nucleus Sampling, but slightly overuses rare words, likely contributing to higher perplexity.Vocabulary rank–frequency distributions are compared using the Zipfian coefficient s, with s = 1 representing a theoretically perfect curve. Lower-temperature sampling avoids rare tail words (Fan et al., 2018).Source: §5.1 · pp. 7–8, Figure 7 · p. 8

5.2 SELF-BLEU

• p ∈ [0.9, 1) yields human-like Self-BLEU for Nucleus Sampling; top-k and temperature sampling need very high settings that produce unnaturally high perplexity.Self-BLEU (Zhu et al., 2018) measures similarity against other generations; lower scores indicate greater diversity. The evaluation samples 1000 generations, comparing each against all 4999 other generations.Source: Figure 8 · p. 8, §5.2 · pp. 8–9

5.3 REPETITION

• Only Nucleus Sampling satisfies all evaluated distributional criteria, while both Nucleus Sampling and top-k sampling show the least repetition at reasonable parameter settings.Overly low tuning parameters over-truncate all stochastic methods, inducing repetition resembling greedy search. High-temperature sampling reduces repetition at the cost of coherence.Source: Figure 9 · p. 9, §5.3 · p. 9
• Below 0.9, temperature sampling severely increases repetition; wider beam search appears less repetitive largely because it produces shorter generations.The repetition evaluation examines the first 200 tokens, counting a phrase of minimum length 2 when it repeats at least three times at the generation’s end.Source: Figure 9 · p. 9

6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)

• HUSE (Hashimoto et al., 2019) combines human and statistical evaluation to assess both generation quality and diversity.Statistical metrics inadequately measure coherence, while purely human evaluation omits diversity. HUSE trains a human-versus-model discriminator using two features: language-model probability and human judgments of generation typicality.Source: §6.1 · p. 9
• 0.1 interpolation mass from the original probability distribution smooths top-k and Nucleus Sampling distributions when generating text for HUSE evaluation.Direct application initially gave both methods scores near zero because truncation assigns zero probability to most potential tokens, despite human preference for these methods.Source: §6.1 · p. 9
• 200 generations per decoding algorithm receive 20 typicality annotations each from different annotators, totaling 4000 annotations per decoding scheme.Source: §6.1 · pp. 9–10
• Nucleus Sampling achieves the highest HUSE score, with top-k sampling ranking second.HUSE was computed using a KNN classifier with k = 13 neighbors, selected for higher discrimination accuracy.Source: §6.1 · p. 10

6.2 QUALITATIVE ANALYSIS

• Representative generations show failures across decoding methods, including semantic confusion in Nucleus Sampling and unrelated topic shifts in top-k sampling.Beam search becomes trapped in repetition, while full sampling produces the hardest-to-understand output and invents a word. Nucleus Sampling confuses whales with birds, while top-k sampling with temperature 0.7 devolves into repetition.Source: §6.2 · p. 10

7 CONCLUSION

• Nucleus Sampling is proposed as a solution that effectively captures the region of confidence of language models.Source: §7 · p. 10
• Likelihood-maximizing decoding causes repetition and overly generic language in open-ended generation.Source: §7 · p. 10
• Sampling without truncation risks selecting from the low-confidence tail of a model’s predicted distribution.Source: §7 · p. 10
• Future work aims to characterize the region of confidence dynamically and incorporate a more semantic utility function to guide decoding.Source: §7 · p. 10

• Beam Search produces extremely few distinct trigrams compared with human text.Source: Figure 10 · p. 13
• Average generation length decreases linearly as beam width increases.Source: Figure 10 · p. 13
• The number of distinct trigrams remains constant across beam widths.Source: Figure 10 · p. 13

• The section provides additional generation examples for qualitative comparison.Source: B EXAMPLE GENERATIONS · p. 14
• Only Pure Sampling and Nucleus Sampling escape the repetition loop in the Figure 12 examples.Source: Figure 12 · p. 15
• Nucleus Sampling’s generation is far closer in style to the ground truth in Figure 12.Source: Figure 12 · p. 15