Source-linked AI summary
Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training
Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, Jason Weston
TL;DR
Generative dialogue models produce copying, repetition, frequent-word overuse, and logical flaws that maximum likelihood training does not address. The paper extends unlikelihood loss to penalize these behaviors using distribution-matching objectives and negative supervision, yielding more consistent dialogue across several axes and improved coherence for contradiction. Its scope depends on supervised data containing positive and negative examples of coherent behavior.
Problem
Maximum likelihood training does not address dialogue models' copying, repetition, frequent-word overuse, and logical-coherence problems.
Method
The paper extends unlikelihood loss to control dialogue statistics and lower the probability of incoherent or contradictory generations using negative candidates and NLI supervision.
Results
The approach yields more consistent dialogue across several axes and measurably improves coherence with respect to contradiction.
Takeaways & Limitations
Unlikelihood provides a framework for reducing repetition, contextual copying, frequent-word overuse, and contradiction in generative dialogue models.
Takeaways & Limitations
The contradiction approach assumes supervised data containing positive and negative examples of coherent behavior.
Abstract
from arXiv · showhide
Generative dialogue models currently suffer from a number of problems which standard maximum likelihood training does not address. They tend to produce generations that (i) rely too much on copying from the context, (ii) contain repetitions within utterances, (iii) overuse frequent words, and (iv) at a deeper level, contain logical flaws. In this work we show how all of these problems can be addressed by extending the recently introduced unlikelihood loss (Welleck et al., 2019) to these cases. We show that appropriate loss functions which regularize generated outputs to match human distributions are effective for the first three issues. For the last important general issue, we show applying unlikelihood to collected data of what a model should not do is effective for improving logical consistency, potentially paving the way to generative models with greater reasoning ability. We demonstrate the efficacy of our approach across several dialogue tasks.
1 Introduction
Open-ended dialogue exposes copying, repetition, frequent-word overuse, and logical incoherence that standard objectives do not address. The paper generalizes unlikelihood training to control these mismatches and improve dialogue consistency across several axes.
- 1 Introduction: Open-ended dialogue reveals repetitive copying, frequent-word overuse, and failures to maintain logical coherence.These failures indicate that the learning objective does not match simple training-data statistics and that models can lack understanding of their generations.
- 1 Introduction: GPT-2 345M can assign high probability to contradictory generations, illustrating a deeper coherence failure.
- 1 Introduction: The paper generalizes unlikelihood training to remedy copying, repetition, vocabulary-usage, and logical-consistency problems.The method adds an extra objective term that lowers the probability of repetitions and other undesirable events.
- 1 Introduction: Dialogue loss functions control contextual copies, within-utterance repeats, and vocabulary usage by measuring corresponding training-distribution statistics.The approach reports improved metrics on several tasks.
- 1 Introduction: NLI supervision supplies negative examples for training models to assign low probability to incoherent and contradictory text.The paper uses existing natural language inference data as supervision against poor-quality generations.
- 1 Introduction: Overall, the approach yields more consistent dialogue models across several axes and provides a framework for further advances.
2 Dialogue Unlikelihood Training
Dialogue unlikelihood training generalizes unlikelihood loss to control copying, repetition, vocabulary mismatch, and contradictory generations. It combines likelihood for desired behavior with penalties for selected negative candidates or incoherent responses.
- Dialogue generation predicts an utterance from initial context sentences and preceding dialogue turns, but likelihood-based decoding produces statistics unlike human text.
- Unlikelihood adds candidate-specific penalties to selected tokens while likelihood preserves probability for gold tokens.The candidate set and scale β determine which tokens are penalized and how strongly.
- 2.1 Repetition and Copying: Repeating n-grams from the context or generated label define negative candidates for reducing copying and within-utterance repetition.The two losses can also be mixed to mitigate both issues.
- 2.2 Vocabulary Usage: Vocabulary unlikelihood penalizes tokens when the model’s estimated unigram probability exceeds the human distribution.The model distribution is estimated from tokens in recent generated sequences.
- 2.3 Contradictions: Contradictory responses are trained as negative examples so unlikelihood lowers the probability of generating their target tokens.The approach assumes labeled positive and negative examples of coherent behavior, such as NLI data.
3 Related Work
Prior work improves dialogue generation through negative training, controllable generation, sampling, and discriminative reranking. This paper instead integrates improvements directly into generative-model training.
- Negative training has been studied extensively for dialogue retrieval and used to prevent generic or malicious responses in dialogue generation.
- Control methods and nucleus sampling target repetition, specificity, generic responses, or repetitive utterances, with nucleus sampling incurring an expense.
- NLI has improved retrieval coherence and supported multitask or discriminative-rescoring approaches, whereas this work integrates improvements into model training.
4 Experiments
Across dialogue and question-answering tasks, unlikelihood training reduces copying, repetition, and frequent-word overuse while improving human evaluations and coherence-related selection.
- Repetition and Copying: Unlikelihood reduced ConvAI2 context repetitions by 69% (.0352 vs .1131) and label repetitions by 89% (.0023 vs .0210), while keeping perplexity essentially constant.The reductions brought both repetition metrics closer to human levels.
- Repetition and Copying: On ELI5, label-unlikelihood reduced label repetition by 91% (.055 vs .617) and increased F1 from .130 to .182.The strongest repetition problem in ELI5 was label repetition.
- Repetition and Copying: The α parameter smoothly controls label and context repeats, with increased perplexity occurring only at very high values.Figures 2 and 3 evaluate perplexity against the two repetition types as α varies.
- Human Evaluation: Human evaluations found statistically significant improvements over MLE for label unlikelihood on ELI5 and vocabulary unlikelihood on ConvAI2.The ELI5 comparison evaluated readability and accuracy, while the ConvAI2 evaluation asked which system humans preferred for a long conversation.
- Vocabulary Unlikelihood: Vocabulary unlikelihood shifted probability mass from over-represented frequent words toward medium and rare words, reducing common-token overuse by 9 points and increasing rare-token production by 3 percentage points.These distributional changes came at a small cost to perplexity and F1.
- Contradictions: For contradiction-sensitive dialogue evaluation, unlikelihood produced large perplexity differences between contradictory and entailing utterances and improved selection accuracy across both tasks.On the two-utterance task, contradicting versus entailing perplexity was 248.9 vs. 9.1; selection accuracy rose from 18% to 78% for neutral statements and from 37.4% to 69.8% on full dialogue.
5 Conclusion
The paper defines unlikelihood objectives to reduce copying, repetition, and frequent-word overuse while improving coherence by lowering the probability of inconsistent dialogue. Supervised labels for incoherent utterances measurably improve contradiction-related coherence.
- Unlikelihood objectives make dialogue models repeat themselves less, copy context less, and use rarer vocabulary, bringing outputs closer to human statistics.
- Supervised coherent and incoherent utterances improve coherence with respect to contradiction when used with unlikelihood training.
- The same approach could be applied to other supervised data targeting causal or commonsense reasoning errors.
A Repetition Control with Beam Search
Beam-decoding experiments reproduce the main repetition and copying trends, although the MLE baseline shows more repetition than before. The authors suggest sequence-level training and beam-specific hyperparameters as possible improvements.
- Beam decoding with size 5 shows trends similar to greedy decoding for the four evaluated models.The same unlikelihood models were evaluated without retraining for beam search.
- The baseline model suffers more repetition with beam decoding, consistent with a known effect.
- Sequence-level unlikelihood training with beam search and beam-specific hyperparameters may produce better results.
B Nucleus Sampling for Vocabulary control
Vocabulary unlikelihood shifts probability mass from frequent toward medium and rare words, improving distributional alignment with human vocabulary at some cost to perplexity and F1. Nucleus sampling can achieve similar distributional alignment but with a larger F1 trade-off.
- 9 points: vocabulary unlikelihood reduces overuse of common tokens while shifting probability mass toward medium and rare words.The effect strengthens as α increases, with a small cost to perplexity and F1.
- Nucleus sampling restricts generation to the smallest token set whose probability mass exceeds p, with larger p producing distributions closer to human vocabulary usage.
- Stronger α terms shift probability mass from Frequent words to Medium and Rare words at a small cost to PPL and F1.The token classes are defined by cumulative masses in human-generated text.
- p = 0.5 matches the distribution of unlikelihood with α = 102, but F1 is 0.160 versus 0.190.
C Human Evaluation
Human evaluations compare model-generated conversations or answers using pairwise judgments and quality-controlled annotation. The reported results favor repetition-controlled and some vocabulary-unlikelihood models over the MLE baseline.
- 252 model-human conversations were collected for each model before pairwise preference evaluation.
- Approximately 200 preferences per model comparison were collected, with annotators failing quality control filtered out.
- ELI5 evaluators judged which response answered the question better across five questions, including two quality-control items.
- The repetition-controlled ELI5 model and two vocabulary-repetition models significantly outperform the MLE baseline under a two-tailed binomial test (p < .01).
- Baseline conversations and α = 102 conversations are contrasted, with the latter more frequently employing rarer words.
- Figure 10 reports complete human-evaluation results, while Figure 9 shows the evaluator interface.