PaperlayerView exact revision on arXiv

Source-linked AI summary

Tug-of-War Between Knowledge: Exploring and Resolving Knowledge Conflicts in Retrieval-Augmented Language Models

Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, Xiaojian Jiang, Jiexin Xu, Qiuxia Li, Jun Zhao

arXiv:2402.14409v1cs.CLcs.AIcs.IR

TL;DR

RALMs can encounter conflicts between internal memories and retrieved evidence, with inaccurate or outdated memories contributing to hallucination. The paper evaluates these conflicts, documents evidence and memory preferences, and proposes CD2 for confidence calibration; its findings include persistent trust in incorrect memory and biases toward common or frequently presented evidence.

  • Problem

    RALMs may have inaccurate or outdated internal memories while integrating external sources, creating knowledge conflicts that can limit practical applicability.

    Source: §1 · p. 1

  • Method

    The paper evaluates conflicts across QA settings and proposes CD2, which uses contrastive decoding to calibrate model confidence.

    Source: §1 · pp. 2–3

  • Results

    Capable RALMs may persist in incorrect internal memories despite correct evidence, favor common knowledge, and favor evidence that is more frequent or consistent with their memories.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The reported findings show that providing correct evidence does not necessarily displace a capable RALM’s incorrect internal memory.

    Source: §7 · p. 9

    The proposed conflict-resolution approach calibrates open-source models’ logits; mitigation for black-box LLMs without accessible output logits remains unstudied.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Retrieval-augmented language models (RALMs) have demonstrated significant potential in refining and expanding their internal memory by retrieving evidence from external sources. However, RALMs will inevitably encounter knowledge conflicts when integrating their internal memory with external sources. Knowledge conflicts can ensnare RALMs in a tug-of-war between knowledge, limiting their practical applicability. In this paper, we focus on exploring and resolving knowledge conflicts in RALMs. First, we present an evaluation framework for assessing knowledge conflicts across various dimensions. Then, we investigate the behavior and preference of RALMs from the following two perspectives: (1) Conflicts between internal memory and external sources: We find that stronger RALMs emerge with the Dunning-Kruger effect, persistently favoring their faulty internal memory even when correct evidence is provided. Besides, RALMs exhibit an availability bias towards common knowledge; (2) Conflicts between truthful, irrelevant and misleading evidence: We reveal that RALMs follow the principle of majority rule, leaning towards placing trust in evidence that appears more frequently. Moreover, we find that RALMs exhibit confirmation bias, and are more willing to choose evidence that is consistent with their internal memory. To solve the challenge of knowledge conflicts, we propose a method called Conflict-Disentangle Contrastive Decoding (CD2) to better calibrate the model's confidence. Experimental results demonstrate that our CD2 can effectively resolve knowledge conflicts in RALMs.

1. Introduction

Knowledge conflicts arise between RALMs’ internal memories and external sources, and among truthful, irrelevant, and misleading evidence. The paper evaluates these conflicts, examines model preferences, and proposes CD2 to calibrate confidence.

  • The evaluation framework uses four QA datasets, eight LLMs, and correctness, faithfulness, and memorization dimensions to assess conflicts.Source: §1 · p. 2
  • RALMs show availability bias toward common knowledge, while capable models may persist in incorrect internal memories despite correct external evidence.ChatGPT often trusted incorrect internal memory more than half the time, even when correct external evidence was provided.Source: §1 · p. 2
  • When external evidence conflicts, RALMs favor more frequent evidence, prefer evidence consistent with internal memory, and struggle more as conflicting multi-hop steps increase.Source: §1 · p. 2
  • CD2 calibrates confidence by contrasting logits with and without external sources, and by using fact-aware instruction tuning with expert and amateur models.Source: §1 · p. 2
  • CD2 improved average Recall by 2.35% and 2.41% on two datasets.Source: §1 · p. 3

2. Related Work

Prior work studies retrieval-augmented models and conflicts between parametric memory and evidence. This paper extends conflict analysis to incorrect as well as correct internal memories and proposes a resolution method.

2.1. Retrieval-Augmented Language Models

  • The paper follows an in-context paradigm, using frozen LLMs to answer from retrieved passages without updating model parameters.Source: §2.1 · p. 3

2.2. Knowledge Conflicts

  • Prior conflict studies report reliance on parametric knowledge, retrieved evidence, or memory-consistent evidence depending on the conflict setup.Source: §2.2 · p. 3
  • Unlike most cited studies that examine conflicts with correct internal memory, this work separately investigates correct and incorrect memories and proposes a resolution method.Source: §2.2 · p. 3

3. Evaluation Framework

The evaluation framework uses open-book QA datasets, multiple model families, and measures of correctness, faithfulness, and memorization. It also induces model memories and constructs conflicting evidence for evaluation.

3.1. Datasets

  • The framework evaluates NQ, TriviaQA, PopQA, and MuSiQue using 500 or 1,000 sampled questions per dataset with retrievable Wikipedia evidence.Source: §3.1 · p. 3

3.2. Analyzed Models

  • The study uses seven open-source LLMs and ChatGPT, with in-context demonstrations and retrieved evidence varied across specified counts.The models include FLAN-T5, FLAN-UL2, Baichuan2, and LLaMA2; demonstrations M ∈ {4, 8, 16} and evidence K ∈ {3, 5, 10, 20}.Source: §3.2 · p. 3

3.3. Evaluation Metrics

  • Correctness is measured with Recall, faithfulness with K-Precision, and reliance on internal memory with the memorization ratio MR = fm/(fm+fs).Recall measures gold-reference tokens present in predictions; K-Precision measures prediction tokens present in external evidence; fm and fs count reliance on internal memory and external sources.Source: §3.3 · pp. 3–4

3.4. Memory Induction and Conflict Generation

  • The authors elicit internal answers with closed-book QA, generate supporting evidence, and use ChatGPT at τ = 1.0 to create counterfactual answers and conflicting evidence.Source: §3.4 · p. 4

4. Conflicts Between Internal Memory and External Sources

RALMs may switch from internal memory to conflicting external evidence, but their confidence and knowledge preferences vary with model capability and entity popularity. The section also finds that capable models can persist in incorrect internal beliefs with high confidence.

  • After conflicting evidence is provided, recall of conflicting references is much higher than recall of internal memory, indicating a preference for external knowledge.Source: §4 · p. 4
  • As model capability grows, internal-memory confidence rises, yet LLaMA2 improves confidence specifically when correcting incorrect memory with truthful evidence.The paper notes that an uncalibrated model may still answer incorrectly.Source: §4 · pp. 4–5
  • Capable LLMs’ persistent confidence in incorrect internal memory is framed as Dunning-Kruger-like (Singh et al., 2023; Slobodkin et al., 2023).The paper contrasts this persistence with lower confidence in correct internal memory.Source: §4 · pp. 5–6
  • As entity popularity rises from 1e2 to 1e6, models shift from relying on external sources toward trusting internal memory, showing an availability bias toward common knowledge.Source: Figure 3 · p. 5, §4 · p. 6

5. Conflicts Between Truthful, Irrelevant and Misleading Evidence

RALMs struggle to distinguish truthful evidence from misleading or irrelevant material, and their choices are shaped by evidence frequency and alignment with internal memory. More conflicting hops also make multi-hop reasoning harder.

  • ↓29.28% Recall for FLAN-UL2 and ↓10.95% for ChatGPT under conflicts; most RALMs also have higher K-Precision for misleading than truthful evidence.The section reports that irrelevant evidence can also distract models.Source: §5 · p. 6
  • With LLaMA2 7B, models lean toward whichever evidence appears more frequently, while excessively long context may reduce focus and accuracy.The tested truthful-to-misleading ratios were 2:0, 2:1, 2:2, 1:2, and 0:2, with K values of 5, 10, and 20.Source: §5 · pp. 6–7
  • RALMs show confirmation bias, selecting external evidence aligned with their internal memory more often, especially when that memory is supplied as external evidence.Source: §5 · p. 7, Figure 5 · p. 7
  • As the number of conflicting hops increases, recall and reasoning accuracy decline; ChatGPT also performs poorly in multi-hop settings with multiple evidence pieces.Source: §5 · p. 7, Figure 6 · p. 7

6. Conflict-Disentangle Contrastive Decoding

The paper proposes Conflict-Disentangle Contrastive Decoding (CD2) to calibrate RALM confidence under knowledge conflicts. On NQ and TriviaQA, it improves Recall across internal-memory and evidence-conflict settings, with lower misleading-evidence K-Precision in the latter.

6. Conflict-Disentangle Contrastive Decoding

  • The method addresses the problem that external evidence can shift confidence while incorrect internal memory still leads the model to answer inaccurately.Source: §6 · p. 7, §6.1 · p. 8

6.3. Results

  • 29.83% and 23.06% Recall improvement over In-context on NQ-Conf and TriviaQA-Conf, with Mis KP also decreasing; NQ-Inco and TriviaQA-Inco gains are 2.05% and 2.70%.Source: Table 3 · p. 8, §6.3 · p. 8, Table 4 · p. 8

6.1. Method

  • CD2 maximizes differences between logits to calibrate confidence; for internal-memory conflicts it contrasts logits with and without external sources, and it can run at inference without extra training.Source: §6.1 · p. 8
  • For conflicts among truthful, irrelevant, and misleading evidence, CD2 uses fact-aware instruction tuning and contrasts an expert trained for truthful answers with an amateur trained for misleading answers.Source: §6.1 · p. 8

6.2. Experiment Setups

  • Experiments use LLaMA2 7B on NQ and TriviaQA, construct Inco and Conf datasets, and compare CD2 with closed-book, in-context, and finetuned baselines.Source: §6.2 · p. 8

7. Conclusion

The paper evaluates knowledge conflicts in RALMs and reports systematic preferences across internal-memory and external-evidence conflicts. It proposes CD2, which the authors report effectively resolves these conflicts.

  • The paper presents an evaluation framework for assessing knowledge conflicts across various dimensions.Source: §7 · pp. 8–9
  • CD2 calibrates model confidence through contrastive decoding and is reported to effectively resolve knowledge conflicts in RALMs.Source: §7 · p. 9
  • Capable RALMs persistently favor incorrect internal memory despite correct evidence and also show an availability bias toward commonly known knowledge.Source: §7 · p. 9
  • RALMs lean toward evidence that appears more frequently and are more willing to choose evidence consistent with their internal memory.The paper describes these preferences as majority rule and confirmation bias, respectively.Source: §7 · p. 9

Limitations

The authors limit their investigation to model performance and confidence scores, leaving mechanistic analysis and mitigation for black-box LLMs unresolved.

  • The study examines knowledge conflicts mainly through model performance and confidence scores.Source: Limitations · p. 9
  • Understanding mechanisms, including whether conflicts affect certain neurons’ activation values, is left for future work.Source: Limitations · p. 9
  • Mitigating conflicts in black-box LLMs without accessible output logits remains an open question.Source: Limitations · p. 9
  • The authors hope their findings will promote the practical application of RALMs.Source: Limitations · p. 9

Language Resource References

The references include works on multihop question answering, question-answering datasets, parametric and non-parametric memory, and knowledge-intensive language tasks.

Loading 2402.14409v1…