Source-linked AI summary
Neural Text Generation with Unlikelihood Training
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, Jason Weston
TL;DR
Neural text generation suffers from dullness, repetition, and token-distribution mismatch because likelihood training does not adequately shape the probabilities used during generation. The paper proposes token- and sequence-level unlikelihood training, which lowers probabilities for overly likely negative candidates while retaining likelihood updates for true tokens. It reports less repetitive, less dull text with maintained perplexity and stronger human-evaluated generations under standard decoding.
Problem
Likelihood training and decoding produce dull, repetitive text with excessive frequent-word probabilities, while decoding-only fixes leave the model’s underlying probabilities uncorrected.
Method
Unlikelihood training combines likelihood updates for true tokens with penalties on overly probable negative candidates collected from next-token predictions or generated sequences.
Results
Token- and sequence-level unlikelihood training reduce degeneration while maintaining perplexity, and beam-search generations outperform likelihood-trained alternatives in human evaluations.
Takeaways & Limitations
Unlikelihood training provides a strong alternative to decoding-only methods for producing less repetitive, less dull generations with standard greedy or beam search.
Takeaways & Limitations
Token-level unlikelihood training is limited to training-distribution prefixes, leaving a mismatch with model-generated sequences that motivates sequence-level objectives.
Abstract
from arXiv · showhide
Neural text generation is a key tool in natural language applications, but it is well known there are major problems at its core. In particular, standard likelihood training and decoding leads to dull and repetitive outputs. While some post-hoc fixes have been proposed, in particular top-$k$ and nucleus sampling, they do not address the fact that the token-level probabilities predicted by the model are poor. In this paper we show that the likelihood objective itself is at fault, resulting in a model that assigns too much probability to sequences containing repeats and frequent words, unlike those from the human training distribution. We propose a new objective, unlikelihood training, which forces unlikely generations to be assigned lower probability by the model. We show that both token and sequence level unlikelihood training give less repetitive, less dull text while maintaining perplexity, giving superior generations using standard greedy or beam search. According to human evaluations, our approach with standard beam search also outperforms the currently popular decoding methods of nucleus sampling or beam blocking, thus providing a strong alternative to existing techniques.
1 INTRODUCTION
The paper argues that likelihood training and decoding produce dull, repetitive text because the objective poorly shapes both top-ranked token probabilities and sequence generation. It introduces unlikelihood training to penalize overly probable negative candidates while preserving likelihood updates for true tokens.
- Motivation: Likelihood training produces dull, repetitive generations with excessive high-frequency tokens and insufficiently interesting content words.These problems persist despite more training data, and decoding-only fixes leave the underlying sequence probabilities unchanged.
- Why likelihood fails: The likelihood objective optimizes the entire next-token distribution rather than prioritizing the top-ranked token used by greedy or beam decoding.Thus, maximizing ground-truth-token likelihood does not ensure that the ground-truth token receives rank one.
- Why likelihood fails: Likelihood training optimizes next-token prediction rather than sequence generation, so prediction imperfections accumulate during decoding.This sequence-level error accumulation is not addressed by likelihood training.
- Proposed approach: Unlikelihood training combines likelihood updates for true tokens with penalties that lower the probability of overly likely negative candidates.Negative candidates can be collected during next-token prediction or from generated sequences, supporting token- and sequence-level training.
- Reported outcome: Token- and sequence-level unlikelihood training improve dullness and repetition metrics while maintaining perplexity and token accuracy relative to maximum likelihood.Human evaluations also compare favorably with likelihood-trained models under beam search and with beam blocking or nucleus sampling.
2 RELATED WORK
Prior work documents repetition and token-distribution mismatch in neural generations and mainly addresses them by changing decoding. This paper’s approach instead belongs to learning methods that increase scores for desired outputs and decrease scores for negative candidates.
- Neural text degeneration: Neural text degeneration includes excessive frequent-word use, insufficient rare-word use, and repetition relative to human text.These mismatches appear in dialogue and language modeling and are not remedied simply by increasing training data.
- Improved decoding algorithms: Earlier decoding methods reduce repetition by selecting candidates unlike previously chosen tokens or by blocking repeated content.These approaches modify beam search rather than the model’s learned probabilities.
- Improved decoding algorithms: Top-k and nucleus sampling restrict generation to a subset of tokens and randomize selection, often reducing duplicate tokens even when beam candidates repeat.Because the underlying model is unchanged, sampling does not directly correct its probability distribution.
- Improved learning algorithms: Unlikelihood training is related to structured prediction methods that raise scores for true examples and lower scores for negative examples.Related approaches include structured perceptrons, energy-based models, reflective likelihood, and applications to dialogue and translation.
3 NEURAL TEXT GENERATION
The paper frames language modeling as learning a distribution over variable-length token sequences and studies sequence completion as conditional generation from a human-written prefix. Because optimal continuation search is intractable, it reviews deterministic and stochastic decoding strategies.
- Language modeling: A neural language model seeks a distribution pθ(x) resembling the human text distribution p∗(x) over variable-length token sequences.The sequences contain vocabulary tokens xt ∈ V.
- Language modeling: Training conventionally finds parameters θ by maximizing the log-likelihood of finite samples from the human text distribution.This is the de facto training approach described before the paper’s proposed alternative.
- Sequence completion: Sequence completion conditions on a human-sampled prefix and decodes a continuation intended to resemble a sample from the human distribution.The setting covers story generation, contextual completion, language modeling, and dialogue modeling.
- Decoding: Finding the optimal continuation is intractable, so practical systems use approximate deterministic or stochastic decoding.The paper studies these strategies as methods for generating continuations.
- Deterministic decoding: Greedy search selects the highest-probability token at each step, whereas beam search retains a fixed-size set of partially decoded hypotheses.Both are deterministic decoding approaches.
- Stochastic decoding: Stochastic decoding samples tokens from a model-dependent distribution, commonly restricting candidates to a vocabulary subset.Top-k uses the k most probable tokens, while nucleus sampling uses the smallest subset whose probability mass exceeds threshold p.
4 NEURAL TEXT DEGENERATION
The paper characterizes neural text degeneration as excessive repetition and mismatch with human token distributions. These problems occur in both sequence generation and next-token prediction under conventional maximum-likelihood models.
- Repetition: Maximum-likelihood language models frequently generate repetitive continuations, particularly with deterministic decoding.GPT-2 examples show degeneration under greedy, top-k, and nucleus sampling for some settings, with greedy decoding worst.
- Repetition: 43% of n-grams were repeated in model continuations with greedy decoding, compared with 0.5% in human text.The comparison uses average repeated n-grams in Transformer language-model continuations.
- Repetition: Neural language models also repeat preceding tokens more often in next-token prediction than human text.This extends prior analysis beyond sequence-level repetition.
- Repetition: 62% of Transformer next-token predictions appeared in the preceding 128 words, versus 49% in ground-truth text.The finding is notable because maximum-likelihood training directly optimizes next-token conditional distributions.
- Token distribution mismatch: Greedy continuations and next-token predictions have token distributions different from human text, overusing high-frequency tokens and underusing low-frequency tokens.Frequency is defined by the human token distribution.
5 THE UNLIKELIHOOD TRAINING OBJECTIVE
Unlikelihood training augments likelihood training by lowering the probability of selected negative candidates, operating at token and sequence levels to target repetition. Sequence-level variants fine-tune on decoded continuations, while experiments show practical reduction of degeneration with fewer than 1,500 updates.
- 5 THE UNLIKELIHOOD TRAINING OBJECTIVE: Unlikelihood training penalizes tokens assigned too much probability while retaining likelihood updates for true target tokens.Negative candidates can be collected during next-token prediction or from generated sequences, enabling token- and sequence-level training.
- 5.1 UNLIKELIHOOD TRAINING: Previous-context tokens serve as negative candidates, reducing the probability of incorrect repeats and frequent tokens without additional supervision.These candidates are efficient to compute because they come directly from the preceding context.
- 5.1 UNLIKELIHOOD TRAINING: The unlikelihood gradient responds to the negative candidate’s probability, increasing the ground-truth token’s probability and decreasing the candidate’s probability.Its behavior differs from the likelihood gradient through a term governed by the hyper-parameter α and the candidate probability.
- 5.2 SEQUENCE-LEVEL UNLIKELIHOOD TRAINING: Token-level unlikelihood is limited to training-distribution prefixes, motivating sequence-level unlikelihood on decoded continuations.Sequence-level training addresses the mismatch between training prefixes and generated sequences by constructing negative candidates over model-generated text.
- 5.2 SEQUENCE-LEVEL UNLIKELIHOOD TRAINING: Sequence-level training penalizes tokens that participate in repeating n-grams and can be fine-tuned from either an MLE baseline or a token-level unlikelihood model.The two variants are LUL-seq and LUL-token+seq, with sequence-level and originating token-level updates mixed equally.
- 5.2 SEQUENCE-LEVEL UNLIKELIHOOD TRAINING: Under 1,500 fine-tuning updates, sequence-level unlikelihood substantially reduces degeneration despite potentially high decoding costs.The result is presented as making the objective practical for modern large-scale neural models.
6 EXPERIMENTS
Experiments on Wikitext-103 evaluate token- and sequence-level unlikelihood training across repetition, diversity, language-modeling, and human-preference measures. Unlikelihood reduces degeneration, preserves or improves language-modeling quality, and outperforms baseline decoding methods in human evaluations.
- Experimental setup: Experiments use Wikitext-103 with a 16-layer Transformer and evaluate sequence completions using deterministic and stochastic decoding.Sequence-level fine-tuning uses 1,500 updates; evaluation reports sequence-level and token-level metrics, plus human judgments.
- Baseline: The maximum-likelihood baseline has 25.64 test perplexity but substantially more repetition and fewer unique tokens than human text.Greedy baseline seq-rep-4 is .442 versus .006 for human text, while unique tokens are 11.8k versus 19.8k.
- Token-level objective: Token-level unlikelihood reduces wrong next-token repetition from .352 to .311 and increases unique next-tokens from 11.8k to 12.7k, with similar perplexity and accuracy.With greedy search, it also reduces seq-rep-4 by 36% and generates roughly 22% more unique tokens than the baseline.
- Sequence-level objective: Sequence-level fine-tuning further reduces seq-rep-4 by 97% versus greedy maximum likelihood and increases beam-search unique tokens by 77%.Relative to token-level unlikelihood, fine-tuning improves repetition and diversity while slightly improving perplexity from 26.91 to 26.72.
- Additional findings: Unlikelihood-trained models retain language-modeling quality under top-k and nucleus sampling while improving repetition, and sequence-level fine-tuning substantially reduces degeneration within 1,500 updates.These results indicate that the objectives remain useful with stochastic decoders and can be practical for large-scale models despite decoding costs.
- Human evaluation: Human evaluations prefer all proposed models over the baseline, with sequence-level training improving win rates; the best model beats nucleus sampling and beam blocking.Expert annotators also find LUL-token+seq significantly better than both nucleus sampling and beam blocking.
7 CONCLUSION
The paper identifies neural text degeneration in likelihood-trained models and proposes unlikelihood training to reduce it. Token- and sequence-level variants substantially reduce degeneration and outperform likelihood-trained models under multiple decoding methods and human evaluation.
- Likelihood-trained state-of-the-art models exhibit neural text degeneration, characterized by repetition and token-distribution mismatch.
- The likelihood objective is insufficiently constrained because models with identical perplexity can have very different generation quality.
- Token- and sequence-level unlikelihood training substantially reduce degeneration according to automatic metrics.
- Unlikelihood-trained models outperform likelihood-trained models with various decoding methods in human evaluation.
A.1 DERIVATION
The derivation analyzes the token-level unlikelihood gradient for true targets, negative candidates, and other tokens, then extends the formulation to multiple candidates. The supplied table captions situate evaluations across sequence- and token-level metrics.
- The token-level loss combines likelihood treatment of the true next-token with an unlikelihood penalty for a negative candidate.
- The gradient is examined separately when a token is the true target, a negative candidate, or another token.
- The objective generalizes from one negative candidate to multiple candidates using the same gradient form with candidate-specific weighting.
- Table 5 reports token-level objectives and sequence-level fine-tuning using sequence-level and token-level metrics on Wikitext-103 validation data.
- Table 6 reports stochastic-decoding results using sequence-level and token-level metrics on Wikitext-103 test data.
B STOCHASTIC DECODING RESULTS
Unlikelihood-trained models retain next-token quality under top-k and nucleus sampling while reducing repetition. The improvements are especially large at lower sampling settings, although high settings can produce more unique tokens than humans.
- Unlikelihood-trained sampling models maintain next-token accuracy comparable to likelihood-trained counterparts while exhibiting fewer repetitions.
- 0.277 reduced to 0.0041 for 4-gram sequence repetitions using top-p-0.3 with unlikelihood training.
- Sampling next-token accuracy is lower than deterministic decoding accuracy, with top-k-50 at 0.302 versus greedy MLE at 0.394.
- At higher p and k values, all methods produce continuations with more unique tokens than humans, suggesting those settings may be too high.
- GPT-2 evaluation uses 50 BPE-token prefixes and 100 BPE-token continuations for likelihood and sequence-level unlikelihood fine-tuning.
D SEQUENCE-LEVEL RANDOM CANDIDATES
Random-seq candidates penalize randomly selected generated tokens during sequence-level fine-tuning. Penalizing 10% of tokens improves sequence repetition metrics, while n-gram candidates can improve them further.
- D SEQUENCE-LEVEL RANDOM CANDIDATES: Random-seq candidates are randomly selected tokens from a generated continuation and are penalized by the sequence-level loss.
- D SEQUENCE-LEVEL RANDOM CANDIDATES: The candidate-selection rate is controlled by p_penalize, a fixed hyper-parameter in [0, 1].
- D SEQUENCE-LEVEL RANDOM CANDIDATES: Penalizing 10% of generated tokens substantially improves seq-rep-4 for greedy and beam search relative to baseline.
- D SEQUENCE-LEVEL RANDOM CANDIDATES: 41% improvement is reported for LUL-seq greedy and 73% for LUL-tok+seq greedy under random-seq candidates.
- D SEQUENCE-LEVEL RANDOM CANDIDATES: N-gram repetition candidates yield further improvements than random-seq candidates on seq-rep-4.
E.3 FULL HUMAN EVALUATION RESULTS
Table 9 presents the full human evaluation results, including comparisons omitted elsewhere and raw win–loss counts.
- Table 9 reports additional human-evaluation comparisons omitted for brevity.
- The table includes the raw number of wins for each comparison.
- The table includes the raw number of losses for each comparison.