PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Decoding Improves Reasoning in Large Language Models

Sean O'Brien, Mike Lewis

arXiv:2309.09117v2cs.CLcs.AI

TL;DR

Reasoning tasks commonly use greedy decoding, while open-ended generation uses truncated sampling, leaving a split the paper seeks to address. It applies Contrastive Decoding, which contrasts stronger and weaker model scores, and reports improved reasoning performance over greedy decoding across benchmarks. The authors conclude that it is a general-purpose generation method, while noting limitations in factual recall and evidence beyond LLaMA models.

  • Problem

    Language-model generation commonly uses truncated sampling for open-ended text and greedy decoding for reasoning, an undesirable split associated with more reasoning errors during open-ended generation.

    Source: §1 · p. 1

  • Method

    Contrastive Decoding scores strings using expert and amateur logits, masking low-probability expert tokens and penalizing amateur scores.

    Source: §2.1 · p. 3

  • Results

    Contrastive Decoding outperforms greedy decoding on reasoning problems, improving various LLaMA models by up to 8 absolute percentage points on GSM8K and outperforming LLaMA 2 there.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results strengthen the case for Contrastive Decoding as a simple, general-purpose method for eliciting more desirable behavior from large language models.

    Source: §7 · p. 9

    The study focuses mainly on LLaMA models, and further work is needed to establish CD’s effect on larger tuned models.

    Source: §6 · p. 9

Abstract

from arXiv · show

We demonstrate that Contrastive Decoding -- a simple, computationally light, and training-free text generation method proposed by Li et al 2022 -- achieves large out-of-the-box improvements over greedy decoding on a variety of reasoning tasks. Originally shown to improve the perceived quality of long-form text generation, Contrastive Decoding searches for strings that maximize a weighted difference in likelihood between strong and weak models. We show that Contrastive Decoding leads LLaMA-65B to outperform LLaMA 2, GPT-3.5 and PaLM 2-L on the HellaSwag commonsense reasoning benchmark, and to outperform LLaMA 2, GPT-3.5 and PaLM-540B on the GSM8K math word reasoning benchmark, in addition to improvements on a collection of other tasks. Analysis suggests that Contrastive Decoding improves over existing methods by preventing some abstract reasoning errors, as well as by avoiding simpler modes such as copying sections of the input during chain-of-thought. Overall, Contrastive Decoding outperforms nucleus sampling for long-form generation and greedy decoding for reasoning tasks, making it a powerful general purpose method for generating text from language models.

1 INTRODUCTION

The paper applies Contrastive Decoding to reasoning tasks to address the split between sampling for open-ended generation and greedy decoding for reasoning. It reports gains over greedy decoding across reasoning benchmarks and analyzes possible mechanisms and boundaries of those gains.

1 INTRODUCTION

  • The usual split between truncated sampling for open-ended generation and greedy decoding for reasoning is undesirable because it increases reasoning errors during open-ended generation.The paper notes that likely strings tend to be short and uninteresting, while sampling for reasoning risks sampling errors.Source: §1 · p. 1
  • Contrastive Decoding searches for strings maximizing a weighted likelihood difference between a stronger expert and weaker amateur, avoiding modes that are likely under both models.It was previously shown to outperform existing methods for open-ended text generation.Source: §1 · p. 2
  • On GSM8K, Contrastive Decoding improves various LLaMA models by up to 8 absolute percentage points and outperforms LLaMA 2, while CD answer ranking lets LLaMA outperform all existing models except GPT-4 on HellaSwag.Figure 2 reports significant improvement on HellaSwag; the paper also finds general gains on arithmetic reasoning and multiple-choice ranking tasks, including models as large as LLaMA-65B.Source: Figure 2 · p. 1, §1 · p. 2
  • CD produces less surface-level prompt copying and misses fewer reasoning steps than greedy decoding, suggesting it reduces repetitive or other undesirable distribution modes.The paper reports mixed commonsense reasoning results and a slight factual-retrieval degradation, motivating further refinement.Source: §1 · p. 2
  • The authors present CD as a generation algorithm that improves LLM accuracy across benchmarks and achieves state-of-the-art results in both reasoning and text generation.They describe it as a more unified method for improving generation across tasks.Source: §1 · p. 2

2 CONTRASTIVE DECODING

The formulation combines expert and amateur logits, with expert-based α-masking and an amateur penalty controlled by β. Its probabilistic interpretation describes CD probabilities as adjustments to expert probabilities and explains how the penalty changes with β.

2.1 SIMPLIFIED FORMULATION

  • The formulation works directly in logit space, refactoring the original choice of α and amateur temperature τa while fixing expert temperature τe = 1.The paper presents the refactor as a way to make hyperparameter choice more interpretable and simplify the algorithm.Source: §2.1 · p. 2
  • The simplified formulation uses α to mask tokens below a proportion of the expert model’s maximum probability and β to set the amateur penalty strength.The expert logits receive a leading (1 + β) coefficient, decoupling contrastive strength from expected output-logit scale; this matches DExperts.Source: §2.1 · pp. 2–3

2.2 PROBABILISTIC INTERPRETATION

  • Without α-masking, CD’s post-softmax probabilities are normalized adjustments to the expert probabilities; as β approaches zero the penalty disappears, while as β grows the distribution collapses to the amateur argmax.The limiting behavior is described as the original formulation from Li et al. (2022).Source: §2.2 · p. 3

3 EXPERIMENTS

Across these experiments, contrastive decoding generally improves arithmetic reasoning and yields strong results on HellaSwag, while commonsense performance is mixed. Its outcomes depend on the amateur model, contrastive strength, and task.

3.1 EXPERIMENTAL SETUP

  • Experiments use untuned LLaMA-family models, generally with LLaMA-65B as expert and a 1.5B amateur, and 8-shot chain-of-thought prompts for generation tasks.Greedy decoding is the default except in self-consistency experiments, which sample at τ = 0.7; evaluations span math, commonsense and multiple-choice benchmarks.Source: §3.1 · pp. 3–4

3.2 HYPERPARAMETER SELECTION

  • A 1B-parameter amateur helps reasoning performance, whereas a 7B amateur harms it; results are fairly insensitive to α when β is below 1.The authors use α = 0.1 unless otherwise stated. β = 0.5 works well on GSM8K and PIQA, while values that are too high can harm performance, especially with a larger expert-amateur gap.Source: §3.2 · p. 4, Figure 4 · p. 4
  • 57.7 on GSM8K with LLaMA-65B and β = 0.25 exceeds PaLM-540B (56.5), LLaMA-2 (56.8) and GPT-3.5 (57.1).GPT-3.5 is reported with 5-shot evaluation; the other models use 8-shot prompting.Source: Table 1 · p. 4, §3.2 · p. 4, §3.3 · p. 4

3.3 ARITHMETIC REASONING

  • Contrastive decoding tends to help arithmetic reasoning with chain-of-thought, but MATH remains challenging for both standard and contrastive decoding.The authors conjecture it cannot help when a task is beyond the expert model’s ability; sampled self-consistency also improves with CD across mathematical and commonsense reasoning.Source: §3.3 · p. 4

3.4 COMMONSENSE REASONING

  • CommonsenseQA and StrategyQA results are mixed: CD harms smaller models, with the harm easing for the 65B model and under self-consistency.Table 3’s caption characterizes performance as evening out with a larger expert-amateur gap.Source: §3.4 · p. 5, Table 3 · p. 5

3.5 CONTRASTIVE RANKING

  • LLaMA-65B scores 88.0 on HellaSwag with contrastive ranking, exceeding LLaMA-2 (85.3), GPT-3.5 (85.5) and PaLM 2-Large (86.8).The task scores candidate completions rather than generating open-ended answers.Source: §3.5 · p. 6, Table 4 · p. 6

4 ADDITIONAL STUDIES

The additional studies examine how Contrastive Decoding affects reasoning errors and prompt copying, its efficiency and prompting conditions, and alternative amateur models. They also find limitations in factual recall and some arithmetic settings.

4.1 EFFECTS OF CONTRASTIVE DECODING

  • CD makes more arithmetic errors but fewer omitted steps and semantic misunderstandings in a manual analysis of 100 GSM8K examples.The analysis categorizes errors as arithmetic, missing steps, semantic misunderstanding, or degeneration.Source: Table 5 · p. 6, §4.1 · p. 6
  • CD systematically reduces token-level copying from the prompt, while sampled responses remain similar in length and template use.The comparison analyzes 26,000 generations; automatic ROSCOE evaluation of greedy generations finds no significant differences in its metrics.Source: §4.1 · pp. 6–7, Table 6 · p. 6, Figure 6 · p. 6
  • CD slightly degrades performance on OpenBookQA and TriviaQA, and may slightly degrade exact-match accuracy on a synthetic arithmetic-expression task.The factual-recall tests do not use chain-of-thought; the synthetic task uses 1,000 multiplication and subtraction equations with operands up to four digits.Source: §4.1 · pp. 6–7, Table 7 · p. 7, Table 8 · p. 7
  • A 1.5B amateur with a 65.2B expert increases FLOPs by 3.25%, and CD is more efficient than self-consistency.Self-consistency requires several additional full generation loops.Source: §4.1 · p. 7, Figure 9 · p. 8

4.2 ABLATION STUDIES

  • With multiple sampled paths, α-masking alone provides no benefit, suggesting the contrastive objective is the primary source of improved self-consistency results.The ablation sets β = 0 and α = 0.1, masking candidates without applying the contrastive objective.Source: §4.2 · p. 7, Figure 7 · p. 8
  • Without chain-of-thought prompting, CD provides no consistent improvement on GSM8K; the authors suggest the expert-amateur gap may be too small to exploit.Removing chain-of-thought lowers performance for both standard and contrastive decoding.Source: §4.2 · p. 7, Figure 8 · p. 8
  • CD provides a slight GSM8K boost with FLAN-T5-XXL as expert and FLAN-T5-Small as amateur, though GSM8K is among FLAN-T5’s finetuning tasks.The study uses 8-shot random prompts and leaves more extensive experiments on other model families for future work.Source: §4.2 · pp. 7–8, Table 9 · p. 8
  • Negative prompting weakens CD relative to a small amateur, while partially trained 7B checkpoints improve GSM8K performance where a fully trained 7B amateur harms it.The authors caution that partial-checkpoint results may vary with training-data order and did not perform extensive hyperparameter sweeps.Source: §4.2 · p. 8, Table 10 · p. 9

5 RELATED WORK

The paper situates Contrastive Decoding among sampling and contrastive generation methods, reasoning-specific steering, and prompting approaches. It distinguishes its work through reasoning benchmarks, amateur-model and hyperparameter studies, and analysis of chain-of-thought and self-consistency.

Steering methods for reasoning

  • GRACE trains an external step-level discriminator to select among sampled reasoning steps; the paper relates this to FUDGE through expert-versus-amateur distinguishability.The paper describes its formulation as close to FUDGE, whose binary predictor estimates whether a generated token came from the expert rather than the amateur.Source: §5 · p. 9

Sampling methods; Contrastive Generation Methods

  • Truncated sampling methods reduce repetition in open-ended generation, while sampling can introduce logical-chain errors, motivating greedy decoding for reasoning tasks.The paper also relates its formulation to DExperts and Contrastive Input Decoding.Source: §5 · p. 9

Related work

  • Unlike Li et al. (2022), this work tests standard reasoning benchmarks, explores β and amateur types, and studies combinations with chain-of-thought prompting and self-consistency.The paper notes concurrent DoLA work and reports broader reasoning-task coverage and larger gains from a 1.5B rather than 7B amateur.Source: §5 · p. 9

6 LIMITATIONS

The investigation is mainly limited to LLaMA models; further work is needed to establish CD’s effect on larger, tuned models.

6 LIMITATIONS

  • The investigation is mainly limited to LLaMA models, leaving CD’s effect on larger, tuned models to be established.The authors report continued benefit on larger LLaMA models but call for further work on larger tuned models.Source: §6 · p. 9

7 CONCLUSION

Contrastive decoding can improve chain-of-thought reasoning in large language models, though factual recall remains a challenge. Public reproducibility is limited for results relying on the 1.5B-parameter amateur model because its weights are unavailable.

7 CONCLUSION

  • Contrastive decoding can improve chain-of-thought reasoning in large language models, while factual recall remains a challenge.The authors describe it as a simple, general-purpose method for eliciting more desirable behavior from language models.Source: §7 · p. 9

REPRODUCIBILITY STATEMENT

  • Results relying on the 1.5B-parameter LLaMA amateur are less publicly reproducible because its weights are unavailable.Its training process and architecture are public; results on FLAN-T5 and the negative-prompting and 7B-amateur studies use open-source models and data.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 CODE IMPLEMENTATION

The appendix provides two PyTorch implementations of contrastive decoding: an original probability-space formulation and a reformulation operating directly on logits.

A.1 CODE IMPLEMENTATION

  • Algorithm 1 softmaxes expert and temperature-scaled amateur logits, subtracts their log probabilities, then masks tokens below an expert-probability cutoff.The cutoff is α times the expert model's maximum probability.Source: §A.1 · p. 13
  • Algorithm 2 computes contrastive logits as (1 + β) times expert logits minus β times amateur logits, then masks below a logit cutoff.Its inputs include expert and amateur logits; α sets the masking threshold and β the expert-amateur tradeoff.Source: §A.1 · p. 13

A.2 EQUIVALENCE - MASKING

The appendix shows that the original probability-based mask can be implemented as a logit threshold, then relates the contrastive score to a linear combination of expert and amateur logits. It also explains the parameterization used to separate the contrastive tradeoff from sampling diversity.

A.2 EQUIVALENCE - MASKING

  • Thresholding expert logits at log α plus the maximum expert logit produces the same valid-token set as the original probability mask.The equivalence is shown for single-token generation and follows from the monotonicity of exponentiation and the logarithm.Source: §A.2 · p. 14
  • With an expert temperature, the mask depends on the combined quantity τ log α; the authors fix τ to 1 and use β in the reformulation.The resulting mask then depends solely on α.Source: §A.2 · p. 14

A.3 EQUIVALENCE - LOGIT COMBINATION

  • The contrastive score is equivalent to linearly combining expert and amateur logits, with temperatures extending the combination.The appendix defines the expert-to-amateur probability ratio and introduces the CD-assigned logit before expanding the expression.Source: §A.3 · p. 14
  • The β, τout parameterization separates sampling diversity from the expert-amateur tradeoff, and equally weighting the models performed worse than downweighting the amateur.The authors prefer this formulation because β does not scale the final logits in expectation; τout affects generation diversity while β controls the tradeoff.Source: §A.3 · p. 15

B MULTIPLE-CHOICE BETA SWEEP RESULTS

The appendix includes beta-sweep plots for multiple-choice tasks and estimates the computational overhead of contrastive decoding, including the effect of longer generations.

B MULTIPLE-CHOICE BETA SWEEP RESULTS

C FLOP ESTIMATES

  • 2.30% additional FLOPs per forward pass for a 65B expert model rises to 3.25% when increased generation length is included.The estimate uses an approximation that treats the cost of generating one token as constant regardless of context length.Source: C FLOP ESTIMATES · p. 17

Related papers

Loading 2309.09117v2…