PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts

Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, Yu Su

arXiv:2305.13300v4cs.CLcs.AI

TL;DR

LLMs can rely on external evidence to address limitations of static parametric memory, but their response to conflicting evidence remains insufficiently understood. The paper constructs and validates parametric-memory and counter-memory pairs for controlled experiments, finding receptiveness to coherent contradiction alone but confirmation bias when supportive and contradictory evidence coexist.

  • Problem

    External evidence can conflict with LLMs’ static parametric memory, but how receptive LLMs are to such counter-memory remains insufficiently understood.

  • Method

    The paper elicits parametric memory, generates coherent counter-memory, and uses entailment and consistency checks to support controlled knowledge-conflict experiments.

  • Results

    LLMs are highly receptive to coherent counter-memory when it is the only evidence, but show strong confirmation bias when supportive and contradictory evidence are both present.

  • Takeaways & Limitations

    Tool-augmented LLMs may accept coherent disinformation and may struggle to unbiasedly orchestrate conflicting evidence.

  • Takeaways & Limitations

    The study highlights potential misuse because LLMs can be instructed to generate coherent and convincing fake information.

Abstract

from arXiv · show

By providing external information to large language models (LLMs), tool augmentation (including retrieval augmentation) has emerged as a promising solution for addressing the limitations of LLMs' static parametric memory. However, how receptive are LLMs to such external evidence, especially when the evidence conflicts with their parametric memory? We present the first comprehensive and controlled investigation into the behavior of LLMs when encountering knowledge conflicts. We propose a systematic framework to elicit high-quality parametric memory from LLMs and construct the corresponding counter-memory, which enables us to conduct a series of controlled experiments. Our investigation reveals seemingly contradicting behaviors of LLMs. On the one hand, different from prior wisdom, we find that LLMs can be highly receptive to external evidence even when that conflicts with their parametric memory, given that the external evidence is coherent and convincing. On the other hand, LLMs also demonstrate a strong confirmation bias when the external evidence contains some information that is consistent with their parametric memory, despite being presented with conflicting evidence at the same time. These results pose important implications that are worth careful consideration for the further development and deployment of tool- and retrieval-augmented LLMs. Resources are available at https://github.com/OSU-NLP-Group/LLM-Knowledge-Conflict.

1 INTRODUCTION

The paper examines how LLMs respond when external evidence conflicts with their parametric memory, using a controlled framework to construct and validate such conflicts. It finds receptiveness to coherent contradictory evidence in isolation, but confirmation bias when supportive and contradictory evidence coexist.

  • Static or inaccurate parametric memory can contribute to hallucinations, motivating external information from tools and retrieval augmentation.
  • The study investigates whether LLMs accept counter-memory—external evidence that conflicts with their parametric memory—and why this matters for tool-augmented applications.
  • Prior heuristic edits often produce incoherent counter-memory, limiting conclusions that LLMs stubbornly cling to parametric memory.
  • The proposed framework elicits parametric memory, generates coherent conflicting passages, and applies entailment and consistency checks to filter examples.
  • LLMs readily accept coherent counter-memory when it is the only evidence, but show strong confirmation bias when supportive and contradictory evidence are presented together.

2 RELATED WORK

Related work describes parametric memory as pretrained internal knowledge and tool augmentation as a way to supply external, often up-to-date information. Knowledge-conflict studies simulate contradictions with heuristic edits, while the behavior of LLMs in these scenarios remains under-explored.

  • Parametric Memory in Language Models: Parametric memory is knowledge internalized in model parameters after pre-training, but it can be incomplete, outdated, or incorrect and appear as hallucinations.
  • Tool-augmented Language Models: Tool-augmented and retrieval-augmented language models use external tools such as retrievers to provide up-to-date information and improve language-model performance.
  • Tool-augmented Language Models: The behavior of LLMs in knowledge-conflict scenarios remains under-explored, despite its significance for wider tool-augmented applications.
  • Knowledge Conflict: Knowledge-conflict experiments commonly construct counter-memory from parametric memory through negation injection or entity substitution.

3 EXPERIMENTAL SETUP

The experiments use QA datasets and a controlled framework to elicit parametric memory, construct coherent counter-memory, and filter examples through entailment and consistency checks. Multiple-choice evaluation and memorization ratio quantify how often LLMs follow memory-aligned or conflicting evidence.

  • 3.1 DATASETS: The study uses POPQA, an entity-centric dataset with 14K questions, and STRATEGYQA, a multi-step fact-reasoning benchmark.STRATEGYQA requires implicit question decomposition and integration of relevant knowledge before producing a True or False answer.
  • 3.2 PARAMETRIC MEMORY ELICITATION: In closed-book QA, LLMs answer questions without external evidence to elicit their parametric memory.Examples receiving “Unknown” are discarded, while incorrect answers are retained because they also reflect information stored in model parameters.
  • 3.3 COUNTER-MEMORY CONSTRUCTION: The framework constructs counter-memory by replacing or flipping the memory answer and generating a coherent supporting passage from scratch.For POPQA, the answer entity is replaced with a same-type entity; for STRATEGYQA, the answer polarity is flipped.
  • 3.4 ANSWER-EVIDENCE ENTAILMENT CHECKING: An NLI model checks whether both parametric memory and counter-memory entail their corresponding answers, retaining only supported examples.A manual evaluation of 200 random examples found 99% NLI-model accuracy.
  • 3.5 MEMORY ANSWER CONSISTENCY: A memory answer consistency check compares closed-book answers with answers produced when the elicited parametric memory is explicitly presented as evidence.Examples with inconsistent answers are filtered out because the underlying internal belief may not be firm.
  • 3.6 EVALUATION METRICS: The evaluation converts free-form QA to multiple-choice QA and uses memorization ratio MR = f_m/(f_m + f_c) to measure reliance on parametric memory.Higher ratios indicate greater reliance on the memory answer, while lower ratios indicate more frequent adoption of counter-memory.

4 EXPERIMENTS

The experiments show that LLM responses to conflicting evidence depend strongly on counter-memory coherence and on the structure of multi-source evidence. Models resist incoherent substitutions but accept coherent conflicts, while popularity, order, and quantity shape evidence preference.

  • Single-source evidence: Entity substitution-based counter-memory leaves LLMs clinging to parametric memory, especially ChatGPT, GPT-4, and PaLM2.The substituted context remains correlated with the original entity, reducing evidence coherence.
  • Single-source evidence: Larger Llama2 and Vicuna models are more inclined than their smaller counterparts to retain parametric-memory answers.The authors suggest enhanced memorization and reasoning make larger models more sensitive to incoherent sentences.
  • Single-source evidence: Coherent generated counter-memory makes LLMs highly receptive to conflicting external evidence when it is the sole evidence.A manual check of 50 stubborn cases found most involved hard-to-override commonsense or weak direct conflicts.
  • Single-source evidence: Many generated counter-memories are disinformation that can mislead LLMs, raising concerns about susceptibility to malicious external tools.The framework therefore exposes both receptiveness to coherent evidence and a potential misuse risk.
  • Multi-source evidence: Multi-source evidence increases memorization ratios, especially for GPT-4, indicating confirmation bias toward parametric-memory-aligned evidence.This bias can hinder unbiased use of external evidence in tool-augmented LLMs.
  • Multi-source evidence: Evidence preference varies with popularity, order, and quantity: GPT-4 reaches an 80% memorization ratio for the most popular questions, while order fluctuations exceed 5% for most models.PaLM2 and Llama2-7B exceed 30% order variation; models generally choose the answer supported by the evidence majority.

5 CONCLUSION

The paper introduces a controlled framework for studying knowledge conflicts and finds that LLMs accept coherent counter-memory alone but favor parametric memory amid mixed evidence. Evidence popularity, order, and quantity influence these preferences, while generated misinformation creates ethical risks.

  • The framework elicits parametric memory, constructs counterpart counter-memory, and checks evidence quality for controlled knowledge-conflict experiments.
  • LLMs are highly receptive to coherent counter-memory presented as the only evidence, but show confirmation bias when supportive and contradictory evidence coexist.
  • Evidence popularity, order, and quantity influence LLM evidence preference, properties the paper identifies as undesirable for tool-augmented LLMs.
  • The framework demonstrates that LLMs can generate convincing misinformation, posing potential ethical risks.

ETHICS STATEMENT

The ethics statement emphasizes that LLMs can be instructed to produce coherent, convincing fake information, creating misuse risks that require safeguards and careful data distribution.

  • LLMs can be instructed to make up coherent and convincing fake information, creating potential misuse risks.
  • The authors call for robust safeguards and prevention measures and commit to distributing generated data strictly for research purposes.

REPRODUCIBILITY STATEMENT

The paper reports two contrasting knowledge-conflict behaviors: high receptiveness to sole external evidence and confirmation bias when supportive and contradictory evidence coexist. It discusses implications for tool-augmented LLM deployment and potential safeguards.

  • LLMs show high receptiveness to single external evidence and confirmation bias toward multiple external pieces of evidence.
  • High receptiveness may help remedy outdated or incorrect parametric knowledge but also leaves LLMs vulnerable to misleading third-party tool information.
  • Confirmation bias is especially undesirable for generative search engines and multi-document summarization, where unbiased orchestration of contradictory information matters.
  • The paper suggests validating and monitoring third-party tool information, reducing confirmation bias through fine-tuning or RLHF, and citing sources for generative-search answers.

A.2 ADDITIONAL KNOWLEDGE CONFLICT DISCUSSION

Additional experiments examine how evidence length, fragmentation, and competing sources affect LLM preferences and confidence. The results reinforce both receptiveness to coherent counter-memory and confirmation bias toward parametric memory.

  • Evidence length: ChatGPT tends to adopt the longer evidence side, especially in STRATEGYQA, where longer evidence generally indicates more reasoning steps.
  • Evidence length: 56.7% to 18.8%: shortening counter-memory into a counter-answer reduces its consideration, whereas shortening parametric memory from 42.7% to 43.9% does not reduce preference.
  • Information integration: After evidence fragmentation, ChatGPT shifts toward the other intact evidence in 38.2% of examples, indicating limited integration of fragmented information.
  • Model confidence: In the single-source setting, Llama2-7B gives a counter-answer with high confidence even when counter-memory conflicts with parametric memory.
  • Model confidence: 91.3% of examples have a memory-answer probability of 95% or greater when counter-memory is the sole evidence.
  • Model confidence: In the multi-source setting, 96.3% of examples show a counter-answer log probability of 95% or greater, while memory-aligned evidence is favored frequently.

B EXPERIMENTAL SETUP DETAILS

The experimental setup constructs answer and counter-memory pairs using dataset-specific strategies and ChatGPT-generated text. The process aims to produce coherent, controlled conflicting evidence.

  • ChatGPT generates counter-memory at a human-written level after the study reframes the memory answer into a counter-answer.
  • POPQA: For POPQA, wrong memory answers use provided triplets, while right answers replace object entities with same-relation entities and apply overlap filters.
  • STRATEGYQA: For STRATEGYQA, ChatGPT generates claims corresponding to “True” and “False,” then classifies them dynamically as memory answer and counter-answer.
  • After obtaining the counter-answer, the setup instructs ChatGPT to generate the counter-memory.

B.2 DATASET DETAILS

The supplementary dataset details report scale at each construction step, inconsistency-type distributions, and examples of answer inconsistency and final datasets.

  • Dataset scale is reported for each construction step, alongside the distribution of inconsistency types.
  • The supplementary material includes examples of answer inconsistency and additional examples from the final datasets.

B.3 EXAMPLES OF STUBBORNNESS IN RESPONSE TO PARAMETRIC MEMORY

Only a minimal fraction of LLMs remain stubbornly tied to parametric memory when given counter-memory alone. Ambiguity, unacceptable counter-memory, and highly suggestive questions explain most manually examined stubborn cases.

  • The authors conclude that only a minimal fraction of LLMs demonstrate stubbornness toward parametric memory.
  • Around 400 POPQA instances lacked inferential passages on Wikipedia and instead used Wikidata triples converted into natural-language text.For other POPQA instances, relevant Wikipedia passages were recovered when needed to include the ground truth.
  • STRATEGYQA facts were manually written so that each fact supports the ground truth, requiring no additional modifications.

B.5 HUMAN EVALUATION DETAIL FOR NLI MODEL ACCURACY

The study evaluates evidence quality with manual annotations and an NLI model, while documenting how evidence and prompts were prepared across datasets. Close-sourced language models also tend to become uncertain when confronting conflicting evidence.

  • Generated parametric and counter-memory examples were manually labeled as supportive or not supportive according to whether they entailed the corresponding claim.A state-of-the-art NLI model was evaluated on 200 randomly sampled generated examples using these labels, and its accuracy was calculated.
  • The three close-sourced language models tend to exhibit uncertainty when faced with knowledge conflicts.Table B.4 reports the uncertain answer ratio when models encounter multiple pieces of evidence.
  • POPQA irrelevant evidence consisted of the top three SentenceBERT-retrieved sentences from the same question type, excluding sentences containing entities from the parametric or counter-memory.This exclusion avoided affecting the arrangement of answer options.
  • For STRATEGYQA, standard-mode evidence merges the human-written facts for each sub-question, whereas Section A.2 treats each fact separately.
  • The study adopted the most stable instruction after evaluating 10 different instructions for stability in preliminary experiments.The prompts used in the study are listed in Table C.9.
Loading 2305.13300v4…