PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Decoding: Open-ended Text Generation as Optimization

Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, Mike Lewis

arXiv:2210.15097v2cs.CLcs.AIcs.LG

TL;DR

Open-ended generation faces a trade-off between repetitive outputs from probability maximization and incoherent or drifting outputs from sampling. The paper proposes contrastive decoding, which contrasts large and small language models under a plausibility constraint; across domains and model scales, it improves coherence over prevalent decoding methods without additional training.

  • Problem

    Probability maximization can produce short, repetitive text, while sampling can produce incoherent continuations and topic drift in open-ended generation.

  • Method

    Contrastive decoding maximizes the difference between expert and amateur LM log-probabilities subject to an expert-based plausibility constraint.

  • Results

    CD improves coherence over nucleus, top-k, typical, and SimCTG decoding while improving or maintaining fluency across teacher–student combinations, domains, and model scales.

  • Takeaways & Limitations

    Contrasting frozen language models provides a zero-training decoding approach that produces higher-quality text than decoding from the larger LM alone.

  • Takeaways & Limitations

    The paper focuses on open-ended generation; applying scale-based contrasting to task-oriented generation is not directly applicable and requires further study of amateur-model properties.

Abstract

from arXiv · show

Given a language model (LM), maximum probability is a poor decoding objective for open-ended generation, because it produces short and repetitive text. On the other hand, sampling can often produce incoherent text that drifts from the original topics. We propose contrastive decoding (CD), a reliable decoding approach that optimizes a contrastive objective subject to a plausibility constraint. The contrastive objective returns the difference between the likelihood under a large LM (called the expert, e.g. OPT-13B) and a small LM (called the amateur, e.g. OPT-125M), and the constraint ensures that the outputs are plausible. CD is inspired by the fact that the failures of larger LMs (e.g., repetition, incoherence) are even more prevalent in smaller LMs, and that this difference signals which texts should be preferred. CD requires zero additional training, and produces higher quality text than decoding from the larger LM alone. It also works across model scales (OPT-13B and GPT2-1.5B) and significantly outperforms four strong decoding algorithms (e.g., nucleus, top-k) in automatic and human evaluations across wikipedia, news and story domains.

1 Introduction

Open-ended generation must balance coherence and diversity: probability maximization can yield repetitive text, while sampling can drift or become incoherent. Contrastive decoding addresses this by contrasting differently sized LMs under a plausibility constraint and improves coherence across evaluations.

  • Sampling-based generation can become incoherent or drift from the prompt as unlucky choices accumulate over long sequences.
  • Maximum-probability decoding often produces short, repetitive, and tedious continuations.
  • Contrastive decoding selects text by maximizing the expert's log-probability advantage over an amateur LM while restricting tokens to those plausible under the expert.
  • The contrastive objective downweights failure modes common in smaller LMs and emphasizes desirable outputs receiving more probability from stronger models.
  • CD requires no additional training and can produce higher-quality text than decoding from the larger LM alone, with larger expert–amateur scale gaps performing better.
  • Across teacher–student combinations, CD significantly improves coherence over four decoding baselines while improving or maintaining fluency in human and automatic evaluations.

2 Problem Statement

Open-ended generation feeds a prompt to an autoregressive language model and seeks a fluent, coherent continuation. Decoding may sample likely tokens or search for the most likely sequence, with search methods tending toward repetition.

  • The input is a prompt xpre = x1 ··· xn of n tokens, and the decoder generates a continuation xcont of length m.
  • Autoregressive decoding generates one token at a time by conditioning on the preceding context.
  • At each step, pLM(xi | x<i) represents the next-token distribution, with subscripts distinguishing language models.
  • The amateur and expert models are denoted pAMA and pEXP, respectively, with GPT-2 small and GPT-2 XL as examples.
  • Nucleus and top-k sampling draw from truncated candidate sets, whereas greedy decoding and beam search seek the most likely sequence but can produce repetitive, tedious outputs.

3 Contrastive Decoding

Contrastive decoding is a search-based method that contrasts large and small language models while restricting generation to plausible tokens. It addresses false positives and false negatives through an adaptive plausibility constraint and uses beam search over token-level scores.

  • 3 Contrastive Decoding: Contrastive decoding maximizes a novel contrastive objective subject to an adaptive plausibility constraint.The method combines expert–amateur contrast with expert-based filtering of candidate tokens.
  • 3 Contrastive Decoding: The objective rewards patterns favored by the expert and penalizes patterns favored by the amateur, downweighting repetition, topic drift, and self-contradiction.These undesirable behaviors are described as more prevalent in smaller language models.
  • 3.2 Vhead: Adaptive Plausibility Constraint: The constraint also preserves correct tokens that both models favor, avoiding false negatives caused by indiscriminate amateur penalization.The paper illustrates this with the subword “#orn” in “unicorn,” whose contrast is small despite high probability under both models.
  • 3.2 Vhead: Adaptive Plausibility Constraint: The plausibility constraint filters implausible tokens that can receive high unconstrained contrastive scores.For example, “Net-Message” receives high contrast despite extremely low expert probability.
  • 3.3 Full Method: CD factors an intractable sequence-level objective into token-level scores and applies beam search after plausibility filtering.Remaining tokens are scored by log p_EXP(x_i | x_<i) − log p_AMA(x_i | x_<i).
  • 3.4 Choice of Amateur: The amateur is chosen to expose behaviors to downweight, with smaller same-family models, temperature adjustment, and restricted context as design options.Examples include OPT-125M for OPT-13B and GPT-2 small for GPT-2 XL; the main experiments use α = 0.1.

4 CD as Pragmatic Communication

The paper interprets contrastive decoding through pragmatic communication, modeling the expert as a knowledgeable speaker and the amateur as a less-informed listener. The objective balances high-quality language with informativeness relative to the listener.

  • 4 CD as Pragmatic Communication: Pragmatic communication frames language generation as a tradeoff between high-quality language and informativeness to a listener.The cited pragmatic formalisms describe communication as a cooperative process.
  • 4 CD as Pragmatic Communication: The expert and amateur LMs model a knowledgeable speaker and a less-informed listener, respectively.This interpretation motivates contrasting their token probabilities.
  • 4 CD as Pragmatic Communication: Expert probabilities and plausibility constraints favor fluent, relevant tokens, while amateur probabilities suppress language predictable to the listener.The paper connects this division to outputs that are both high quality and informative.
  • 4 CD as Pragmatic Communication: Using a uniform amateur reduces CD to maximizing expert log-probabilities, while an n-gram amateur yields soft n-gram blocking.With a very low amateur temperature, the latter approaches forbidding repeated n-grams.
  • 4 CD as Pragmatic Communication: Using the same expert and amateur with a restricted amateur context makes the method equivalent to an MMI decoding objective.That objective maximizes pointwise mutual information between the prompt and continuation.

5 Experimental Setup

The experiments evaluate open-ended generation across news, Wikipedia, and stories using fixed prompts and continuations, automatic metrics, human judgments, and multiple decoding baselines across model families and sizes.

  • 5 Experimental Setup: The evaluation covers news, Wikipedia, and story domains using Wikinews, WikiText-103, and the BookCorpus Project Gutenberg split.The first 32 words provide prompts, followed by 256-token continuations.
  • 5.1 Datasets and Metrics: Automatic evaluation measures diversity, MAUVE distribution similarity, and coherence between prompt and continuation sentence embeddings.Coherence uses cosine similarity with pre-trained SimCSE embeddings.
  • 5.2 Human Evaluation: Human evaluation asks Mechanical Turk workers to compare paired continuations for fluency and coherence.Fluency concerns grammatical natural flow, while coherence concerns staying on topic and avoiding topic drift.
  • 5.3 Baselines: CD is compared with nucleus, top-k, typical, greedy, and contrastive search decoding.The study uses p = 0.95 for nucleus sampling, k = 50 for top-k, and τ = 0.95 for typical decoding.
  • 5.4 Models and Hyperparameters: Experiments use GPT-2 XL, OPT-6.7B, and OPT-13B as experts, paired with GPT-2 small or OPT-125M amateurs.The pairings test generalization across LM families and sizes.
  • 5.4 Models and Hyperparameters: The main results use α = 0.1, amateur temperature 1.0 for OPT and 0.5 for GPT-2, with beam size 5.The paper reports α = 0.1 as robust across domains.

6 Main Results

Contrastive decoding outperforms the evaluated baselines on MAUVE and coherence across domains and model sizes, while human evaluators also prefer it for coherence and fluency. Its diversity is competitive with sampling and improves relative to nucleus sampling as model size increases.

  • 6.1 Automatic Evaluation: CD outperforms all other decoding baselines in MAUVE and coherence across news, Wikipedia, stories, and 1.5B and 13B models.Its diversity is comparable to or slightly worse than nucleus and typical sampling, but substantially better than other search methods.
  • 6.1 Automatic Evaluation: 0.83 vs. 0.94 is the diversity comparison between CD and nucleus sampling for stories with the 1.5B model.The gap shrinks to 0.89 vs. 0.93 when model size increases to 13B.
  • 6.1 Automatic Evaluation: CD outperforms all baselines by a large margin in coherence, followed by greedy decoding.The paper attributes this to objective optimization, implicit coherence rewards, and restricting the amateur’s context.
  • 6.2 Human Evaluation: 2.6x and 6.4x are the average coherence-preference ratios for CD over nucleus and typical decoding across settings.For fluency, CD is preferred 1.4x more than nucleus sampling and 3.5x more than typical decoding.
  • 6.3 Qualitative Examples: Nucleus sampling exhibits topic and style drift in a qualitative example, whereas CD stays on topic and elaborates coherently on the prompted game.The nucleus output shifts from a video game to music, includes email formatting, and changes narrative style.
  • 6.3 Qualitative Examples: A larger expert–amateur scale gap improves generation quality in the GPT-2 family experiments.The OPT-family results appear in the appendix.

7 Ablation Studies

The ablations show that contrastive decoding benefits from a larger expert, a suitably capable smaller amateur, an appropriately chosen temperature, search rather than sampling, and plausibility constraints. These design choices improve generation quality across automatic and human evaluations.

  • 7.1 Size of Amateur and Expert LMs: Using the largest model as expert and the smallest model as amateur yields the highest diversity and MAUVE among model pairings.Identical expert–amateur pairs produce highly repetitive text, while reversing their sizes leads to inferior quality.
  • 7.1 Size of Amateur and Expert LMs: A trigram amateur produces low-quality text with a MAUVE score of only 0.73.The n-gram model’s mistakes do not effectively highlight the expert model’s failure modes.
  • 7.2 The Impact of Amateur Temperature: τ in [0.5, 1.5] attains good and robust coherence and fluency, while large τ makes repetition no longer penalized.The main results use τ = 0.5 for GPT-2 and τ = 1.0 for OPT.
  • 7.3 Sampling v.s. Search: Sampling from the contrastive objective produces lower-quality text than searching under the same objective.Automatic and human evaluations consistently rank CD (search) above CD (sample) in fluency and coherence.
  • 7.4 Plausibility Constraints: Removing the plausibility constraint causes severe fluency problems, with CD(-Vhead) achieving a MAUVE score of 0.01.The ablation directly tests the role of Vhead in restricting the search space.
  • 7.5 Amateur Context: Conditioning the amateur on the entire prompt context slightly hurts coherence and fluency compared with conditioning it only on the last context token.Table 5 compares CD(-context), which uses the entire context for both models, with CD(full), which uses only the last context token for the amateur.

8 Related Work

Prior decoding work includes search and sampling methods, while prior contrastive text-generation methods use trained models or task-specific anti-experts. Contrastive decoding instead contrasts off-the-shelf language models of different scales without additional training.

  • Decoding Methods: Search methods can work well for goal-driven tasks but tend to produce tedious and repetitive text in open-ended generation.Sampling methods such as nucleus, top-k, and typical decoding produce more diverse and interesting text.
  • Contrast in Text Generation: The closest contrastive approach, DExpert, contrasts a trained expert with a trained anti-expert for controllable text generation.This paper focuses on domain- and task-agnostic anti-experts obtained by using a smaller language model.
  • Contrast in Text Generation: Contrastive decoding contrasts off-the-shelf language models of different scales to produce high-quality text without training.Its contrastive setup is applied to open-ended text generation rather than requiring separately trained models.

9 Conclusion and Future Work

The paper concludes that contrastive decoding improves open-ended generation over prevalent decoding methods in fluency and coherence. It also identifies broader model-contrasting setups as future work.

  • Conclusion: Contrastive decoding is a search-based approach that contrasts language models of different scales.The method is presented as the paper’s central decoding approach for open-ended generation.
  • Conclusion: Contrastive decoding improves over prevalent methods such as nucleus sampling in both fluency and coherence.The conclusion reports this result for the evaluated open-ended generation setting.
  • Future Work: Future work could contrast an early language-model checkpoint with a later checkpoint of the same model.The authors suggest expanding the expert–amateur idea to additional setups.

Limitations

The paper’s evidence is concentrated on open-ended text generation, with task-oriented generation remaining an unresolved scope boundary. It also notes that stronger generation quality may increase risks from hard-to-detect disinformation.

  • Scope boundary: Contrastive decoding is demonstrated for open-ended text generation, while its effectiveness for summarization and machine translation remains an open question.The authors state that contrasting differently scaled models is not directly applicable to task-oriented settings and identify amateur-model properties as future work.
  • Scope boundary: A smaller summarization model used as the amateur produced lower ROUGE than a uniform amateur distribution.The uniform distribution is equivalent to beam search based on log-probabilities in the reported comparison.
  • Interpretation: The method’s formal objective interprets selected continuations as text distinguishable as originating from the expert rather than the amateur.This interpretation connects the contrastive score to pointwise mutual information with an expert-source indicator.
  • Societal impact: Higher-quality generated text could enable more powerful disinformation that is difficult to distinguish from human-written text.The paper suggests watermarking generated outputs as a possible mitigation that need not affect quality.

M Variant of CD: Training the Amateur LM

This variant trains an amateur model to mimic the expert’s degenerate, repetitive behavior, then uses it in contrastive decoding. The resulting beamprefix variant outperforms nucleus sampling in human evaluation.

  • Training the amateur: The authors train an amateur to mimic degenerate expert behavior by beam-searching the expert on randomly sampled Wikipedia prompts.The generated training data comes from prompts of different lengths.
  • Training the amateur: Beamprefix uses prefix-tuning to adapt GPT-2 with a learned soft prompt of length 10, which then serves as the amateur during decoding.The adaptation is described as lightweight because only the soft prompt is learned and stored.
  • Evaluation: Human evaluation finds that CD (beamprefix) significantly outperforms nucleus sampling.The comparison is reported for the human-evaluation results in Table 18.
Loading 2210.15097v2…