PaperlayerView exact revision on arXiv

Source-linked AI summary

Sources of Hallucination by Large Language Models on Inference Tasks

Nick McKenna, Tianyi Li, Liang Cheng, Mohammad Javad Hosseini, Mark Johnson, Mark Steedman

arXiv:2305.14552v2cs.CLcs.AI

TL;DR

LLMs are expected to perform NLI for applications such as question answering and summarization, but the sources of their apparent inference ability remain unclear. Using controlled behavioral studies across LLaMA, GPT-3.5, and PaLM, the paper identifies pretraining-derived attestation and relative-frequency biases, which produce hallucinations and poorer performance on bias-adversarial samples. It concludes that these biases should be controlled in evaluation and further addressed before robust language reasoning can be trusted.

  • Problem

    The paper examines why LLMs can appear capable of NLI while producing hallucinated information in downstream language-understanding applications.

  • Method

    The authors conduct controlled behavioral studies that modify existing NLI datasets and measure prompted decisions across LLaMA, GPT-3.5, and PaLM.

  • Results

    Across models, performance reflects two pretraining-derived biases: affirming attested hypotheses and wrongly affirming entailment when the premise is less frequent than the hypothesis.

  • Takeaways & Limitations

    The paper proposes controlling for these biases in LLM evaluation and concludes that further attention to alleviating them is needed before robust language reasoning can be trusted.

  • Takeaways & Limitations

    The paper examines two prominent hallucination sources but acknowledges that the search is not exhaustive and that residual NLI performance remains unexplained.

Abstract

from arXiv · show

Large Language Models (LLMs) are claimed to be capable of Natural Language Inference (NLI), necessary for applied tasks like question answering and summarization. We present a series of behavioral studies on several LLM families (LLaMA, GPT-3.5, and PaLM) which probe their behavior using controlled experiments. We establish two biases originating from pretraining which predict much of their behavior, and show that these are major sources of hallucination in generative LLMs. First, memorization at the level of sentences: we show that, regardless of the premise, models falsely label NLI test samples as entailing when the hypothesis is attested in training data, and that entities are used as ``indices'' to access the memorized data. Second, statistical patterns of usage learned at the level of corpora: we further show a similar effect when the premise predicate is less frequent than that of the hypothesis in the training data, a bias following from previous studies. We demonstrate that LLMs perform significantly worse on NLI test samples which do not conform to these biases than those which do, and we offer these as valuable controls for future LLM evaluation.

1 Introduction

The paper investigates hallucination in LLM natural language inference by testing whether pretraining-derived memorization and corpus-frequency patterns substitute for robust reasoning. Across LLaMA, GPT-3.5, and PaLM, these biases produce misleadingly high scores on aligned samples and sharply degraded performance on adversarial ones.

  • Motivation: LLMs hallucinate when information is incorrect or inappropriate but presented as fact, creating risks for downstream language-understanding tasks.The paper focuses on natural language inference, which it describes as critical for applications such as summarization and question answering.
  • Approach: The study uses controlled behavioral experiments that alter existing NLI datasets and measure prompted decisions across LLaMA, GPT-3.5, and PaLM.The experiments target two proposed sources of performance: attestation of hypotheses in training text and relative term frequency.
  • Attestation bias: 1.9, 2.2, and 2.0 times more likely to wrongly predict Entail, respectively, LLaMA-65B, GPT-3.5, and PaLM-540B were when the hypothesis was attested.The models also use named entities as identifying indices for recalling propositional memory, despite their irrelevance to predicate logic.
  • Relative frequency bias: 1.6, 1.8 and 2.0 times more likely to wrongly affirm entailments, the three LLMs were when the premise had lower term frequency than the hypothesis.This corpus-statistic bias is especially examined when propositional memory is unavailable.
  • Impact: NLI performance was misleadingly high on samples consistent with the biases but severely degraded on samples adversarial to them.When labels opposed attestation, models could become poor or near-random classifiers; relative-frequency violations caused substantial decreases across all LLMs.

2 Related Work

Prior work documents artifacts, generalization failures, memorization, and identifier effects across NLI and language models. This paper extends those concerns to larger LLMs, linking memorized inference behavior to named entities and contrasting factual recall with hypothetical generalization.

  • NLI artifacts: NLI datasets contain artifacts that supervised models can learn from hypotheses alone, motivating controls that separate dataset idiosyncrasies from model memory.This paper applies a similar hypothesis-only test to LLMs without training and conditions on hypothesis attestation.
  • Generalization: Supervised NLI models have shown generalization failure across datasets, while smaller language models have relied on dataset artifacts during directional predicate inference.The present work studies much larger language models with stronger reported performance across NLP tasks.
  • Memorization: Prior studies find that larger LLMs memorize more data and attend early to numbers and nouns as unique identifiers for training sentences.The paper extends this line of work by showing that memories used in language inference are tied to specific named entities.
  • Entity effects: Entity frequency correlates with factual recall but is anti-correlated with hypothetical generalization performance in this paper.The comparison distinguishes remembering facts about entities from generalizing hypothetical inferences involving them.
  • Scope of claims: The paper does not disprove generalization, but reports that GPT-4 exhibits the same hallucinations in Appendix F.This contrasts with claims that GPT-4 understands language beyond memorization.

3 Experimental Design

The experiments probe two biases in LLM natural-language inference: reliance on memorized hypotheses and corpus-relative predicate frequencies. Controlled dataset transformations and comparisons across NLI settings test how these factors alter model decisions.

  • The study uses behavioral experiments that modify NLI datasets to measure how targeted changes affect predictions across LLaMA, GPT-3.5, and PaLM.The experiments examine propositional-memory effects, corpus frequency, and their impact on real performance.
  • 3.1 Two Biases in Inference Predictions: The Attestation Bias (Λ) is greater willingness to affirm an NLI hypothesis when the statement is likely attested in training data.Attestedness is measured by prompting the model to classify the hypothesis as true, false, or unknown.
  • 3.1 Two Biases in Inference Predictions: The Relative Frequency Bias (Φ) affirms entailment when the premise eventuality is less frequent in training corpora than the hypothesis eventuality.The comparison ignores named entities and uses corpus statistics to determine relative frequency.
  • 3.2 Datasets: The Levy/Holt dataset uses premise-hypothesis pairs with shared entity arguments, making directional predicate entailment the central inference problem.The directional subset contains entailments that hold in one direction but not both.
  • 3.2 Datasets: The study excludes world-knowledge datasets and includes RTE-1 only for demonstrating the biases' impact on general NLI.These choices target reasoning about natural-language predicate semantics rather than memorized facts.
  • 3.3 Dataset Transformations: Three transformations remove or alter targeted information: randomized premise predicates, generic arguments, and type-constrained randomized arguments.The transformations preserve or control entailment labels while testing reliance on premises, entities, and extraneous information.
  • 3.3 Dataset Transformations: The Random Premise task relabels samples No-Entail because randomly paired predicates are highly unlikely to entail the hypothesis.Manual inspection found 86.6% successfully non-entailing entries, 3.8% undecided cases, and 9.6% unintended true entailments.
  • 3.3 Dataset Transformations: Generic and randomized entity replacements mask identities or vary entity frequency while preserving entity types and entailment labels.IGenArg uses typed identifiers, while IRandArg uses real entities sampled from low- or high-frequency NewsCrawl groups.

4 Querying Models with Prompts

The authors query selected LLMs with few-shot NLI prompts after comparing model families and training stages. They use a minimal four-example setup, map responses to entailment labels, and derive confidence scores for threshold-based evaluation.

  • 4.1 Models: The study evaluates LLaMA-65B, text-davinci-003 (GPT-3.5), and PaLM-540B as comparisons across model families and training stages.LLaMA and PaLM are pretrained base models, whereas GPT-3.5 is pretrained and instruction- and RLHF-fine-tuned.
  • 4.1 Models: The authors omit models superseded by LLaMA and products closed to scientific review, while analyzing GPT-4 separately in an appendix.The stated aim is to identify shared hallucination sources rather than rank model superiority.
  • 4.2 Prompt Design and Evaluation: Each NLI query inserts a premise and hypothesis into a natural-language prompt followed by Entailment, Neutral, and Contradiction choices.The three-way format follows the typical NLI setup.
  • 4.2 Prompt Design and Evaluation: The test prompt template is selected by the highest development-set AUC among eight candidate templates.The candidates include templates used in prior NLI work.
  • 4.2 Prompt Design and Evaluation: Zero-shot prompting produces severely degraded, even near-random performance, so the experiments use hand-annotated few-shot prompts.The few-shot prompts prepend four examples with explanations, chosen to evoke positive responses rather than maximize dataset scores.
  • 4.2 Prompt Design and Evaluation: Scoring maps choice A to Entail and collapses choices B and C into No-Entail for behavioral experiments.All models selected one of A/B/C on every development question, showing compatibility with the QA format.
  • 4.2 Prompt Design and Evaluation: For confidence-threshold analysis, letter choices are converted into entailment probabilities using a linear mapping involving the indicator function and token probability.The transformation preserves confidence ordering, which is sufficient for calculating a precision-recall curve.

5 Experiment 1: Attestation Bias

Experiment 1 tests whether LLM entailment predictions depend on whether the hypothesis is attested in pretraining data. Across original and random-premise tasks, models more readily predict Entail for attested hypotheses, producing false positive hallucinations when the premise cannot entail them.

  • Experimental setup: The experiment conditions entailment predictions on each model’s own prediction that the hypothesis is Attested or not.This probes reliance on propositional memory of training text.
  • Experimental setup: The random-premise task removes possible entailment while preserving the hypothesis, so predicting Entail becomes a false positive hallucination.It replaces the original premise predicate with a random one, making the samples No-Entail.
  • Results: Models assign a higher probability to Entail when the hypothesis is Attested in both original and random-premise conditions.The effect persists when the premise is related or random, indicating that memory about the hypothesis contributes to predictions.
  • Results: 1.9x, 2.2x, and 2.0x higher chances of falsely predicting Entail occur for LLaMA, GPT-3.5, and PaLM, respectively, when hypotheses are Attested.These figures apply to the random-premise task.
  • Implications: The attestation effect appears across LLaMA, GPT, and PaLM, implicating pretraining rather than instruction-tuning or RLHF.The authors characterize training-data memory as a significant contributor to inference and a possible source of hallucination.

6 Experiment 2: Entities are Indices to Memory

Experiment 2 tests whether named entities help LLMs retrieve memorized statements during entailment judgments. Replacing original entities changes behavior across models, and entity frequency further affects generalization even when statements are unattested.

  • Experimental setup: The experiment replaces original arguments with generic arguments or random infrequent and frequent entities while preserving entailment labels.The transformations are designed to remove identifying information while maintaining type constraints.
  • Results: All three models significantly change behavior when original entities are replaced by entity types or random real entities.This aligns with the hypothesis that named entities function as retrieval cues for memorized statements.
  • Results: GPT-3.5 recall is 66.5 with infrequent entities versus 55.3 with frequent entities, although both conditions contain unattested statements.Across all three models, infrequent entities produce better generalization and higher recall.
  • Implications: The findings indicate that greater pretraining exposure to an entity can impede novel inferences involving it, even when detailed entity knowledge is unnecessary.The effect is consistent across models, indicating that LLM pretraining is responsible.
  • Robustness: Results replicate on RTE-1, while instructing LLMs to ignore propositional memory produces little change.These additional tests support the robustness of the observed memory-related behavior.

7 Experiment 3: Relative Frequency Bias

The experiments test whether relative term frequency biases entailment predictions, including when entity-based propositional memory is blocked. Models produce more false-positive Entail predictions when samples conform to the bias, with the contrast increasing under generic entities.

  • Experimental setup: Relative-frequency labels classify samples by whether the premise predicate is less or more corpus-frequent than the hypothesis, excluding roughly equal pairs.Φ< denotes lower premise frequency and Φ> higher premise frequency; Φ≈ samples are discarded.
  • Experimental setup: Generic-argument transformations mask entities, preventing models from recalling propositional memory for the task.Only 2 hypotheses remain predicted as Attested by GPT-3.5, compared with 6.2% for LLaMA and 3.9% for PaLM.
  • Results: Models more often predict Entail for Φ< samples even though the premise and hypothesis have no semantic relation.This pattern is evaluated as false-positive hallucination because the samples remain No-Entail.
  • Results: With entities available, the relative-frequency effect is moderate, but blocking entity-based memory lowers overall hallucination and strengthens the Φ< versus Φ> separation.Under generic entities, the separation reaches 1.6x for LLaMA, 1.8x for GPT-3.5, and 2.0x for PaLM.

8 Impact of Bias on Performance

The paper evaluates whether the two biases distort apparent NLI performance by comparing bias-consistent and bias-adversarial subsets. Performance deteriorates sharply on adversarial samples, while masking entities reduces attestation effects and intensifies relative-frequency effects.

  • Evaluation design: The evaluation compares NLI subsets where each bias agrees or disagrees with the gold entailment label.Bias-consistent samples align with the gold label, whereas adversarial samples conflict with it.
  • Evaluation design: AUCnorm summarizes discriminative performance across confidence thresholds, with 0% representing random classification and 100% perfect classification.The metric is used to compare performance on bias-consistent and bias-adversarial subsets.
  • Results: Under the standard inference task, all three LLMs deteriorate from very good classifiers to poor or near-random classifiers when samples shift from Λ-consistent to Λ-adversarial.This result indicates severe fragility associated with the attestation bias.
  • Results: Masking entities with type-identifiers reduces the performance drop associated with attestation bias.Generic arguments force models to focus more on proposition predicates.
  • Results: Generic arguments intensify the impact of the relative-frequency bias, increasing the average performance difference between consistent and adversarial subsets.The supplied passage introduces this comparison but does not provide the completed numerical difference.
  • Results: Relative-frequency subset differences are generally narrower than attestation-consistency differences, possibly because frequency estimates require generalization and rely on crude Google N-gram approximations.Reported subset-width changes include 10.1% to 16.1% for Levy/Holt and 14.8% to 16.5% for RTE-1.

9 Conclusion

Across LLaMA, GPT-3.5, and PaLM, the paper identifies memorized statements and corpus-level term-frequency patterns as important biases in NLI performance and possible sources of hallucination. It concludes that these factors arise from pretraining and that stronger controls and mitigation are needed before robust language reasoning can be trusted.

  • Conclusion: Across major LLM families and experimental settings, the paper identifies recall of memorized statements and corpus-based term-frequency biases in NLI.The authors connect these biases to hallucination in applied tasks.
  • Conclusion: Because the same factors appear across models, the paper attributes them to LLM pretraining rather than GPT-3.5 fine-tuning correcting them.The conclusion frames pretraining as the source of sentence-level memorization and corpus-level usage patterns.
  • Conclusion: The paper proposes controlling for these biases during evaluation and calls for further attention to alleviating them before trusting LLMs to reason robustly about language.The conclusion characterizes current tools for basic language understanding and inference as unsatisfactory.

Limitations

The paper limits its claims to two prominent hallucination sources rather than an exhaustive account, and reports residual unexplained NLI performance after controlling for them. It also leaves open whether other prompting techniques could reduce the observed biases.

  • Scope: The study does not exhaustively search for all sources of hallucination in NLI, leaving further sources for future work.Its conclusions concern two prominent sources identified in the paper.
  • Residual performance: Residual unexplained NLI performance remains after controlling for the discussed factors.The authors suggest undiscovered biases or generalizing inference capability as possibilities and defer investigation.
  • Prompting scope: Other novel prompting techniques might help LLMs resist the identified biases, but this remains an open question.The study compared popular prompting techniques and selected the most promising approach.

Ethical Considerations

The paper identifies hallucination risks when LLMs perform inference and discusses practical harms, while examining prompt-engineering choices used to evaluate their behavior.

  • Ethical risks: The paper warns that pretraining overlap with a request can lead to faulty judgments, while the contents of pretraining data are difficult to ascertain.
  • Ethical risks: LLM users may receive inaccurate or irrelevant answers, or answers that contradict information supplied in context.
  • Evaluation choices: The study selects prompt templates on development sets using AUC scores, then applies the best template to each corresponding test set.
  • Evaluation choices: Preliminary experiments found near-random zero-shot performance for LLaMA and PaLM, whereas four in-context examples produced non-trivial behavior.
  • Evaluation choices: The authors therefore conduct their primary experiments in the few-shot setting to better explore the models’ abilities.

B RTE-1 Results For Experiment 2: Entities are Indices to Memory

RTE-1 experiments test whether changing named entities alters entailment predictions while preserving labels. Across models, frequent entities and explicit instructions reveal persistent memory-related biases that impair generalization.

  • RTE-1 entity controls: Replacing RTE-1 entities with generic or type-constrained random arguments lowers performance, with type-constrained replacements producing the larger degradation.
  • RTE-1 entity controls: Across both datasets, high-frequency entities consistently yield worse recall than low-frequency entities, supporting impaired generalization with greater training exposure.
  • Model-specific patterns: LLaMA shows noisy RTE-1 behavior and particularly low performance, which the authors associate with the dataset’s greater linguistic variation and difficulty.
  • Model-specific patterns: PaLM’s smaller gaps across conditions remain consistent with the claims, but degraded recall for generic arguments indicates limited generalization.
  • Instruction intervention: Instructing GPT-3.5 to ignore attestation lowers Entail probabilities similarly across conditions, narrowing neither the attestedness gap nor the underlying bias.
  • Instruction intervention: GPT-3.5 recall still drops at similar scales after random entity replacement, showing that instruction fine-tuning provides only marginal improvement against these biases.
  • Robustness checks: Consensus-attestation comparisons reproduce or enlarge the individual-model gaps, attributing the performance differences to attestation bias rather than model-specific idiosyncrasies.
  • Robustness checks: GPT-4 shows the same substantial performance drop on subsets adversarial to each factor, although its evaluation uses F-1 rather than AUCnorm.
Loading 2305.14552v2…