PaperlayerView exact revision on arXiv

Source-linked AI summary

DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models

Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, Pengcheng He

arXiv:2309.03883v2cs.CLcs.AIcs.LG

TL;DR

LLMs remain prone to hallucinating content that departs from facts, motivating methods that improve reliable generation. DoLa contrasts later- and earlier-layer logits during decoding without external retrieval or fine-tuning. It improves truthfulness across multiple tasks, while relying on internal knowledge and focusing primarily on factuality.

  • Problem

    LLMs persistently hallucinate by generating content that deviates from real-world facts observed during pretraining, limiting trustworthy deployment.

  • Method

    DoLa dynamically selects premature layers and contrasts their logits with mature-layer logits to surface factual knowledge during decoding without retrieval or fine-tuning.

  • Results

    DoLa significantly improves truthfulness across multiple tasks, including multiple-choice, reasoning, and open-ended generation, with only a small additional decoding latency.

  • Takeaways & Limitations

    DoLa provides a decoding strategy for making transformer-based LLMs more factual and potentially safer and more reliable without changing pretrained parameters.

  • Takeaways & Limitations

    DoLa relies on internal model knowledge, so it cannot correct misinformation acquired during training, and its evaluation focuses on factuality rather than other dimensions.

Abstract

from arXiv · show

Despite their impressive capabilities, large language models (LLMs) are prone to hallucinations, i.e., generating content that deviates from facts seen during pretraining. We propose a simple decoding strategy for reducing hallucinations with pretrained LLMs that does not require conditioning on retrieved external knowledge nor additional fine-tuning. Our approach obtains the next-token distribution by contrasting the differences in logits obtained from projecting the later layers versus earlier layers to the vocabulary space, exploiting the fact that factual knowledge in an LLMs has generally been shown to be localized to particular transformer layers. We find that this Decoding by Contrasting Layers (DoLa) approach is able to better surface factual knowledge and reduce the generation of incorrect facts. DoLa consistently improves the truthfulness across multiple choices tasks and open-ended generation tasks, for example improving the performance of LLaMA family models on TruthfulQA by 12-17% absolute points, demonstrating its potential in making LLMs reliably generate truthful facts.

1 INTRODUCTION

LLMs can hallucinate because language modeling may favor plausible patterns over facts, creating deployment risks. DoLa contrasts layer outputs to surface factual knowledge and improve truthfulness without retrieval or fine-tuning.

  • Motivation: LLM hallucinations generate content that deviates from real-world facts observed during pretraining, creating a bottleneck for high-stakes deployment.The paper highlights clinical and legal settings as examples where trustworthy generation is crucial.
  • Motivation: Maximum-likelihood training can assign probability to sentences inconsistent with training knowledge, encouraging superficial pattern recognition over factual generation.The paper identifies mass-seeking behavior as a possible mechanism behind this failure mode.
  • Motivation: Transformer layers appear to distribute knowledge hierarchically, with earlier layers encoding lower-level information and later layers encoding more semantic or factual information.Prior work cited by the paper locates knowledge neurons and editable factual knowledge in upper or specific feedforward layers.
  • Approach: In the illustration, “Seattle” remains probable across layers while the factual answer “Olympia” gains probability in higher layers, making layer contrasts informative.This example motivates using differences between layers to sharpen probabilities toward factually correct outputs.
  • Approach: DoLa contrasts logits from higher and lower transformer layers, emphasizing later-layer knowledge while downplaying lower-layer linguistic knowledge.The method is designed to amplify factual knowledge during next-token decoding.
  • Results: Experiments across TruthfulQA, FACTOR, StrategyQA, GSM8K, and open-ended chatbot evaluation report higher truthfulness and only a small additional decoding latency.The reported gains cover multiple-choice, reasoning, and open-ended generation settings.

2 METHOD

DoLa contrasts predictions from mature and dynamically selected premature layers to amplify factual knowledge during decoding. It selects the premature layer using divergence from the final layer and applies contrastive scoring with a mature-layer probability filter.

  • 2 METHOD: DoLa obtains next-token predictions by contrasting higher-layer and lower-layer information rather than applying the vocabulary head only to the final layer.The method subtracts premature-layer log probabilities from mature-layer log probabilities to amplify higher-layer information.
  • 2.2 DYNAMIC PREMATURE LAYER SELECTION: Dynamic premature-layer selection chooses the early layer with maximum Jensen-Shannon divergence from the mature layer among candidate layers.The candidate layers are grouped into buckets, with the best bucket selected using a validation set.
  • 2.2 DYNAMIC PREMATURE LAYER SELECTION: DoLa-static selects one premature layer through brute-force validation, whereas dynamic selection reduces layer search and is intended to be more robust across data distributions.DoLa-static requires more hyperparameter-search runs and in-distribution validation sets because its best layers are data-sensitive.
  • 2.3 CONTRASTING THE PREDICTIONS: The contrastive distribution restricts candidates to tokens with sufficiently high mature-layer probabilities, assigning zero probability to the rest.This head-token filtering limits implausible tokens that could receive unstable high scores after contrast.
  • 2.3 CONTRASTING THE PREDICTIONS: DoLa can repeat previously generated sentences, especially during long chain-of-thought sequences, so decoding uses a repetition penalty with θ = 1.2.The authors motivate the penalty by an observed tendency toward repetition in the resulting DoLa distribution.

3 EXPERIMENTS

Experiments evaluate DoLa across factuality, reasoning, and instruction-following tasks using LLaMA models and several decoding baselines. DoLa improves performance across these settings, with especially clear gains in TruthfulQA, StrategyQA, GSM8K, and Vicuna QA.

  • 3.1 SETUP: Experiments cover TruthfulQA, FACTOR, StrategyQA, GSM8K, and Vicuna QA across four LLaMA model sizes, comparing DoLa with original decoding, Contrastive Decoding, and ITI.The evaluations include multiple-choice, open-ended, chain-of-thought, and GPT-4-rated chatbot tasks.
  • 3.2 MULTIPLE CHOICES: DoLa significantly improves TruthfulQA performance for four LLaMA sizes, with higher layers consistently selected during two-fold validation.The only reported exception is LLaMA-33B on MC1; MC2 and MC3 are described as more stable metrics.
  • 3.2 MULTIPLE CHOICES: DoLa outperforms baselines by 2-4% on FACTOR, while candidate layers are consistently lower than those selected for TruthfulQA.The paper attributes this difference to FACTOR’s longer sentence choices containing more non-factual tokens.
  • 3.3 OPEN-ENDED TEXT GENERATION: 12-17%: DoLa improves TruthfulQA %Truth∗Info across four models while keeping informative scores above 90% and rejection below 10%.It reaches the performance level of ITI, which uses supervised training with labels.
  • 3.3 OPEN-ENDED TEXT GENERATION: DoLa boosts StrategyQA accuracy by 1-4% and improves GSM8K accuracy by 2% for most LLaMA sizes, except 7B.Both tasks use lower candidate layers, consistent with the FACTOR setting.
  • 3.3 OPEN-ENDED TEXT GENERATION: GPT-4-rated Vicuna QA shows that DoLa notably outperforms baseline LLaMA, especially for the 13B and 33B models.The result extends the reported gains to open-ended chatbot instruction-following.

4 ANALYSIS

The analysis studies fixed versus dynamic premature-layer selection, decoding cost, and qualitative TruthfulQA behavior. Dynamic DoLa is more robust across validation subsets, adds little latency, and can correct some—but not all—factual errors.

  • 4.1 PREMATURE LAYER SELECTION STRATEGY: DoLa maintains high scores across both GSM8K subsets and requires only 2-4 bucket tests, nearly 10x fewer than the 16-40 tests for fixed-layer enumeration.Using subset #1’s optimal fixed layer on subset #2 decreases performance.
  • 4.1 PREMATURE LAYER SELECTION STRATEGY: Random layer selection performs worse than original decoding, supporting the importance of JSD-based dynamic premature-layer selection.The comparison is reported as an appendix analysis of the selection strategy.
  • 4.2 LATENCY & THROUGHPUT: 1.01 to 1.08: DoLa increases greedy decoding time by these factors, suggesting negligible latency cost.Memory and inference details are provided in the appendices.
  • 4.3 QUALITATIVE STUDY: GPT-4 evaluation and experiments on MPT-7B indicate that DoLa’s improvements extend beyond the LLaMA family.These results are reported in the appendices.

5 RELATED WORK

Prior work addresses hallucinations through training and inference-time interventions, while related studies examine how transformer layers encode syntax, semantics, and factual knowledge. Contrastive decoding methods differ in their goals and implementation, with DoLa targeting factuality using layers from the same model.

  • Hallucinations are linked to generated content that is not based on training data or facts, with mitigation strategies including reinforcement learning and inference-time checks.
  • Transformer studies associate earlier layers with syntax and later layers with semantics, while other work identifies middle, topmost, and specific layers or heads in factual prediction.
  • Contrastive Decoding contrasts expert and amateur language models to improve fluency and coherence, without specifically addressing factuality.
  • DoLa dynamically selects early layers and contrasts them with later layers from the same model, avoiding training, smaller amateur models, and multiple forward models.
  • Context-aware decoding emphasizes input context, whereas Autocontrastive Decoding targets diversity and coherence in small language models and requires fine-tuning early-layer prediction heads.

6 CONCLUSION AND LIMITATIONS

The paper presents DoLa as an inference-time decoding strategy that dynamically contrasts transformer-layer logits to improve factuality without retrieval or fine-tuning. It reports improved truthfulness across multiple tasks but notes that the method relies on internal model knowledge and focuses only on factuality.

  • DoLa dynamically selects layers and contrasts their logits to improve factuality during decoding without external information retrieval or model fine-tuning.
  • Experiments show significantly improved truthfulness across multiple tasks, supporting DoLa as a decoding strategy for safer and more reliable language models.
  • DoLa focuses on factuality and does not explore other dimensions such as reinforcement learning from human feedback.
  • Because DoLa relies on internal model knowledge without external retrieval, it cannot correct misinformation acquired during training.

A PRELIMINARY QUANTITATIVE STUDY TO SUPPORT FIGURE 2

The preliminary studies examine which layers differ from the final layer and compare contrastive-decoding configurations. Results indicate that critical layers vary with token type, while teacher forcing introduces a potential source of noise and DoLa remains stronger than tested alternatives.

  • Preliminary quantitative study: The critical-layer analysis uses 3.25K CoNLL-2003 validation examples and measures Jensen-Shannon divergence from the final layer.
  • Preliminary quantitative study: 75% of non-entity predictions identify layer 0 as the critical layer, compared with 35% for entity predictions, where over 50% occur at higher layers.
  • Limitations: Teacher forcing may add noise because ground-truth sentences are not generated by LLaMA, creating mismatches between predicted token types and actual next tokens.
  • Contrastive decoding baseline: Contrastive Decoding with a 1.3B amateur model can outperform its 7B-amateur configuration, but most scores only match baseline and remain below DoLa.
  • Implementation: For TruthfulQA multiple choice, omitting post-softmax processing improved performance, although both implementations exceeded baseline scores.
  • Contrastive decoding baseline: TruthfulQA DoLa variants contrasting with the word-embedding layer or early even-numbered layers improve performance but still trail the proposed DoLa.

D GPT-4 EVALUATION ON TEXT GENERATION QUALITY

The GPT-4 evaluation separates text quality from factuality by rating grammaticality and cohesiveness. DoLa outperforms vanilla decoding for smaller LLaMA models and remains approximately equivalent for the largest model.

  • GPT-4 rates grammaticality and cohesiveness, not factuality, on a 1-to-10 scale using pairwise comparisons.
  • Scores average evaluations of 80 Vicuna QA questions, with higher values indicating better text-generation quality.
  • DoLa achieves better grammaticality and cohesiveness than vanilla decoding for LLaMA 7B, 13B, and 33B models.
  • For the 65B model, DoLa's text-generation quality is almost the same as vanilla decoding.

E MEMORY OVERHEAD

DoLa adds only a small memory cost over vanilla decoding, while its dynamic layer selection is presented as more robust and generalizable than static selection. Experiments also suggest potential generalization beyond LLaMA to MPT-7B.

  • Memory overhead: 1.1% is the LLaMA-7B memory-overhead difference between vanilla decoding and DoLa.For 13B/30B/65B models, the difference is smaller than 1%.
  • Memory overhead: DoLa's memory-overhead difference from vanilla decoding is described as negligible.The comparison uses occupied GPU memory before the first forward pass and peak GPU memory during forward passes.
  • Generalization: DoLa shows gains on most datasets when applied to MPT-7B, suggesting potential generalization across transformer LLMs.The experiments divide candidate layers into buckets, excluding MPT-7B's word-embedding layer because it shares weights with the LM prediction head.
  • Dynamic versus static layer selection: DoLa-static is slightly better on most TruthfulQA and FACTOR cases, whereas DoLa consistently outperforms it on StrategyQA and GSM8K.The authors select DoLa as the main method because they consider it more robust and generalizable with a very small hyperparameter search space.
  • Dynamic versus static layer selection: The random layer-selection baseline performs worse than the original baseline, highlighting the importance of JSD-based layer selection.DoLa dynamically selects the premature layer using a distributional distance measure, implemented with Jensen-Shannon Divergence.

K THE EFFECTS OF REPETITION PENALTY

DoLa can repeat content during long chain-of-thought generation, so a repetition penalty is applied to mitigate this behavior. The penalty improves DoLa on StrategyQA and GSM8K but hurts the baseline.

  • Effect of repetition penalty: Repetition penalties improve DoLa's performance on StrategyQA and GSM8K.The penalty is motivated by DoLa's observed tendency to repeat previously generated sentences, especially during long chain-of-thought reasoning.
  • Effect of repetition penalty: The same repetition penalty hurts baseline performance on StrategyQA and GSM8K.For contrastive decoding, the penalty provides slight gains but remains less effective than the baseline.

M QUALITATIVE STUDY FOR PAIRWISE COMPARISON BY GPT-4

GPT-4-based qualitative comparisons report that DoLa often produces more detailed and factually accurate responses than baseline decoding. However, DoLa can sacrifice relevance, and smaller GPT2 models do not gain factuality on multiple-choice tasks.

  • Qualitative comparison: DoLa can generate more detailed answers or correct results in Vicuna QA examples judged by GPT-4.The examples are presented as evidence of factual accuracy, depth, and improved understanding.
  • Limitations: DoLa sometimes produces detailed factual information that is less relevant to the question than the baseline answer.The authors identify improving instruction following alongside factuality as future work.
  • Small language models: DoLa cannot improve GPT2 Medium's factuality on multiple-choice tasks.GPT2 Medium has 335M parameters; the authors conjecture that its limited capacity yields insufficient factual knowledge and layer distinctions.
Loading 2309.03883v2…