PaperlayerView exact revision on arXiv

Source-linked AI summary

Is Factuality Enhancement a Free Lunch For LLMs? Better Factuality Can Lead to Worse Context-Faithfulness

Baolong Bi, Shenghua Liu, Yiwei Wang, Lingrui Mei, Junfeng Fang, Hongcheng Gao, Shiyu Ni, Xueqi Cheng

arXiv:2404.00216v2cs.CLcs.AI

TL;DR

LLMs need to produce factual answers while following input context, but factuality-enhancement methods may make them over-rely on parametric knowledge. The paper evaluates these methods on factuality benchmarks and knowledge-editing tasks, finding inconsistent factuality gains alongside declines in context-faithfulness. It analyzes hidden states and token logits to examine these declines.

  • Problem

    LLMs are expected to be both factually accurate and faithful to context, but the effects of factuality-enhancement methods on context-faithfulness require evaluation.

    Source: §1 · pp. 1–2

  • Method

    The paper evaluates popular factuality-enhancement methods on factual question-answering benchmarks and knowledge-editing tasks, then analyzes hidden states and token-wise distributions.

    Source: §1 · p. 2, §7 · p. 10

  • Results

    Across factual question-answering and knowledge-editing evaluations, factuality gains are unstable while editing accuracy declines; on ICE, the TruthX decline reaches 67.9%.

    Source: §4.2 · p. 6, §7 · p. 10

  • Takeaways & Limitations

    The results highlight trade-offs between enhancing factuality and maintaining context-faithfulness in LLMs.

    Source: §1 · p. 3, §7 · p. 10

Abstract

from arXiv · show

As the modern tools of choice for text understanding and generation, large language models (LLMs) are expected to accurately output answers by leveraging the input context. This requires LLMs to possess both context-faithfulness and factual accuracy. Extensive efforts have been made to enable better outputs from LLMs by mitigating hallucinations through factuality enhancement methods. However, they also pose risks of hindering context-faithfulness, as factuality enhancement can lead LLMs to become overly confident in their parametric knowledge, causing them to overlook the relevant input context. In this work, we argue that current factuality enhancement methods can significantly undermine the context-faithfulness of LLMs. We first revisit the current factuality enhancement methods and evaluate their effectiveness in enhancing factual accuracy. Next, we evaluate their performance on knowledge editing tasks to assess the potential impact on context-faithfulness. The experimental results reveal that while these methods may yield inconsistent improvements in factual accuracy, they also cause a more severe decline in context-faithfulness, with the largest decrease reaching a striking 69.7\%. To explain these declines, we analyze the hidden states and logit distributions for the tokens representing new knowledge and parametric knowledge respectively, highlighting the limitations of current approaches. Our finding highlights the complex trade-offs inherent in enhancing LLMs. Therefore, we recommend that more research on LLMs' factuality enhancement make efforts to reduce the sacrifice of context-faithfulness.

1 INTRODUCTION

LLMs must be both factually accurate and faithful to context, especially when retrieved information is insufficient, outdated, or conflicts with parametric knowledge. The paper argues that factuality enhancement can compromise context-faithfulness and investigates this trade-off.

1 INTRODUCTION

  • Factuality-enhancement methods edit internal representations or token logits to improve how LLMs convey learned knowledge without changing model parameters.Prior approaches include editing attention mechanisms or hidden states and modifying logits during decoding.Source: §1 · p. 2
  • The authors assess factuality enhancement on TruthfulQA, then test context-faithfulness using in-context editing on MQUAKE knowledge-editing benchmarks.ICE accuracy reflects whether models follow outdated or counterfactual context that may contradict parametric knowledge.Source: §1 · p. 2
  • Factuality enhancement may improve factuality inconsistently while substantially reducing context-faithfulness, particularly when contextual knowledge contradicts parametric knowledge.On LLAMA2-7B-CHAT, TruthX editing accuracy decreased by as much as 60.3%.Source: §1 · p. 2
  • After context injection, enhanced models’ hidden states align more closely with their original parametric outputs, while new-knowledge logits fall and parametric-knowledge logits rise.The authors analyze knowledge tokens captured by a specially designed algorithm.Source: §1 · p. 2
  • The authors recommend developing factuality enhancement that also preserves alignment with provided context to reduce hallucinations.They call for research strategies that balance faithful context use and accurate conveyance of factual information.Source: §1 · p. 3

2 PRELIMINARY

The preliminary section introduces language modeling and knowledge editing, then uses the latter to frame how context-faithfulness is assessed. Its multi-hop setting tests whether models follow edits even when new contextual knowledge conflicts with their prior knowledge.

2.1 LANGUAGE MODELING

  • LLMs process token sequences through attention and feed-forward networks to form hidden states, then transform logits with softmax into next-token probabilities for decoding.Generation proceeds iteratively until an end token or another stopping condition is reached.Source: §2.1 · p. 3

2.2 KNOWLEDGE EDITING

  • The paper assesses context-faithfulness through knowledge-editing success, including multi-hop questions whose answers depend on edits and may conflict with parametric knowledge.For example, changing who created WWE Velocity and adding a spouse fact changes the correct two-hop answer to Joan Lee.Source: §2.2 · pp. 3–4
  • Knowledge editing adjusts a model’s behavior so the edited model returns the target answer for an edit descriptor when the original model does not.The descriptor specifies a desired change using a subject, relation, and target answer.Source: §2.2 · p. 3

3 FACTUALITY ENHANCEMENT FOR LLMS

The paper reviews output-level factuality enhancement and evaluates methods on factuality benchmarks and an illustrated in-context editing task. Reported factuality gains are modest and inconsistent across metrics.

3.1 CURRENT APPROACHES TO FACTUALITY ENHANCEMENT

  • The methods fall into representation editing, which adjusts hidden states, and contrastive decoding, which compares output probabilities across models, layers, tokens, or model outputs.Both categories modify outputs rather than the underlying model structure or parameters.Source: §3.1 · p. 4

3.2 EVALUATION OF FACTUALITY

  • Factuality-enhancement methods improve most evaluated metrics, but gains are modest and inconsistent, with some metrics declining against baseline.The evaluation uses TruthfulQA and FACTSCORE on LLAMA2-CHAT models.Source: §3.2 · p. 5
  • ICE evaluates context-faithfulness with multi-hop question answering and edits, using contextual demonstrations to guide edits and align output format.Figure 2 marks the expected output and shows the additional Thoughts step used with Chain of Thought reasoning.Source: Figure 2 · p. 5

4 EVALUATION OF CONTEXT-FAITHFULNESS

The evaluation tests whether factuality enhancement compromises context-faithfulness using knowledge-editing tasks. Across ICE and MeLLo evaluations, enhanced models show substantial declines in context-faithfulness, while factuality improvements are unstable.

4 EVALUATION OF CONTEXT-FAITHFULNESS

  • The evaluation addresses the concern that greater confidence in parametric knowledge may reduce models’ compliance with conflicting contextual information.Source: §4 · pp. 5–6

4.1 EXPERIMENTAL SETUP

  • The ICE task measures context-faithfulness through question-answering accuracy when relevant editing context provides new knowledge that may conflict with internal knowledge.Experiments compared factuality-enhancement methods with LLAMA2-7B-CHAT and LLAMA2-13B-CHAT baselines on MQUAKE.Source: §4.1 · p. 6

4.2 MAIN RESULTS

  • With MeLLo, most context-faithfulness declines exceeded 40% versus baseline, and larger models experienced more pronounced decreases.MeLLo uses one-hop edits to avoid hallucinations introduced by multi-hop reasoning.Source: Table 3 · p. 6, §4.2 · p. 7
  • ICE accuracy fell by an average of over 20% across all factuality-enhancement methods, with TruthX declining as much as 67.9% versus baseline.The evaluation used LLAMA2-7B-CHAT and LLAMA2-13B-CHAT.Source: §4.2 · p. 6
  • Factuality improvements were unstable, while context-faithfulness declined more substantially in the Figure 3 comparison with baseline.The figure compares changes based on MC2 results and 3-shot ICE results.Source: Figure 3 · p. 6, §4.2 · p. 7
  • More context demonstrations improved accuracy for all methods, but adding Chain of Thought produced unstable results and declines for most enhanced methods.The authors suggest additional reasoning may prompt enhanced models to recall parametric knowledge, negating new contextual knowledge.Source: §4.2 · p. 6

5 IN-DEPTH EXPLORATION OF CONTEXT-FAITHFULNESS

The section develops a token-capturing method to distinguish knowledge supplied by context from parametric knowledge, then uses logit, hidden-state, and output analyses to examine context-faithfulness. Across these analyses, factuality-enhancement methods are associated with weaker representation and output of new contextual knowledge.

5.1 DISTINGUISHING NEW AND PARAMETRIC KNOWLEDGE IN LLM OUTPUTS

  • The token-capturing algorithm matches decoded, high-probability tokens to new and parametric knowledge strings, focusing on factual distinctions rather than repeated or meaningless text.The study applies it to 2,000 question-answer instances from MQUAKE-STUBBORN, with outputs collected before and after context injection.Source: §5.1 · pp. 7–8

5.2 ANALYSIS IN LOGITS DISTRIBUTION

  • All factuality-enhancement methods decrease new-knowledge token probabilities and increase parametric-knowledge token probabilities compared with the baseline.The probability difference correlates with editing accuracy; the authors attribute TruthX’s poor editing to parametric knowledge dominating new knowledge.Source: §5.2 · p. 8, Figure 4 · p. 8
  • For LLAMA2-CHAT models, baseline new-knowledge tokens concentrate at probabilities of 0.8 to 1.0 and ranks 1 to 5, while enhancement methods shift them toward lower probabilities.Figure 5 reports distributions of new-knowledge logits and ranks for LLAMA2-CHAT and LLAMA2-13B-CHAT.Source: §5.2 · pp. 8–9, Figure 5 · p. 8

5.3 ANALYSIS IN HIDDEN STATES

  • After context injection, ITI and TruthX hidden states for new knowledge differ less from their pre-injection states than baseline states do.The authors say TruthX most closely resembles the original range, consistent with its significant context-faithfulness decline; the analysis uses t-SNE on LLAMA2-7B-CHAT states.Source: §5.3 · p. 9, Figure 6 · p. 9

5.4 CASE STUDY

  • The case study groups context-unfaithful outputs into ignoring context, denying its new knowledge, and failing to follow its response format.Table 4 gives illustrative examples and marks unfaithful parts or reasons in red.Source: §5.4 · p. 9, Table 4 · p. 10

6 RELATED WORK

This section places the paper’s focus on the interaction between LLM parametric memory and external context alongside work on hallucinations, knowledge conflicts, and knowledge editing.

Hallucinations in LLMs

  • Hallucinations can arise from conflicts between provided context and a model’s inherent memory, as well as from failures to generate knowledge acquired during pretraining; this paper explores their interplay.Source: §6 · p. 10

Knowledge Conficts

  • External knowledge tools and retrieval-augmented methods can supply evidence, but conflicts with parametric knowledge and model overconfidence can make reliable integration difficult.The paper argues factuality enhancement may reinforce reliance on parametric knowledge and make injected context harder to follow.Source: §6 · p. 10

Knowledge Editing

  • Knowledge editing updates outdated model knowledge through internal model edits or inference-time retrieval from external memory.Source: §6 · p. 10

7 CONCLUSION AND DISCUSSION

The paper concludes that factuality-enhancement methods can trade improved factual accuracy for reduced context-faithfulness. It recommends research on balancing both properties.

  • Evaluations on factual question-answering and knowledge-editing scenarios find unstable factual-accuracy improvements alongside substantial decreases in editing accuracy.Source: §7 · p. 10
  • Logit-distribution and hidden-state analyses provide insights into the causes of the context-faithfulness decline, and the results highlight trade-offs in factuality enhancement.Source: §7 · p. 10
  • The authors recommend developing strategies to controllably balance factual accuracy with adherence to contextual cues.Source: §7 · p. 11

ETHICS STATEMENT

The ethics statement describes the datasets and models used and the authors’ stated design and research commitments.

  • The research uses open-source datasets and open-source or widely recognized models.Source: ETHICS STATEMENT · p. 11
  • The authors state that their proposed method is designed to ensure the model does not produce harmful or misleading information.Source: ETHICS STATEMENT · p. 11
  • The authors state a commitment to ethical standards, transparency, and responsible technology use.Source: ETHICS STATEMENT · p. 11

S. M Towhidul Islam Tonmoy, S M Mehedi Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman

The references span surveys of factuality and hallucination, work on knowledge editing, and methods for mitigating hallucinations in language models.

References

A DETAILS OF THE FACTUALITY ENHANCEMENT

This section outlines five factuality-enhancement approaches and gives the model and decoding settings used in their experiments.

A DETAILS OF THE FACTUALITY ENHANCEMENT

B CONTEXT PROMPTS IN KE TASK

The appendix describes MeLLo's multi-hop question workflow and provides ICE context demonstrations with expected answers.

B.1 TEMPLATE FOR MELLO

  • MeLLo decomposes multi-hop questions, elicits tentative subanswers, then checks them against edited facts retrieved from a knowledge base.Source: §B.1 · p. 16
  • The step-by-step MeLLo illustration distinguishes model-generated text in blue from retrieved facts in orange.Source: Table 5 · p. 16

B.2 CONTEXT DEMONSTRATIONS USED IN ICE

  • The ICE demonstrations mark the expected context-faithful output in green.Source: §B.2 · p. 16
  • A Canada citizenship edit yields Ottawa, while an edited New York University education location yields North America.Source: §B.2 · p. 16
  • Other demonstrations pair an Ireland music-origin edit with Leo Varadkar and a China location edit with China as the answer.Source: §B.2 · p. 17
  • The examples answer Volkswagen Group for an edited Chevrolet ownership claim and London for an edited Fitzgerald birthplace claim.Source: §B.2 · p. 17
  • The Edison birthplace edit is paired with Ottawa, and the Leonardo da Vinci birthplace edit with Pedro Sánchez.Source: §B.2 · p. 17
  • Examples give Italy for the edited Mercedes-Benz headquarters and Sony for the edited ownership of Nintendo.Source: §B.2 · p. 17

C MORE OUTPUT CASES OF FACTUALITY-ENHANCED LLMS

This appendix presents additional cases of models using answer-relevant facts rather than supplied edits in their reasoning. The examples span questions about products, people, sports, and media, and show answers drawing on alternative entities and relations.

  • For sports and geography questions, responses connect Bryce Harper to baseball and the United States, and clubs from Monaco, Serbia, and Northern Ireland to Europe.Source: C MORE OUTPUT CASES OF… · p. 18
  • Other cases use corrected facts about institutions, media, and companies to answer Milan for the University of Milan headquarters and Washington D.C. for the U.S. capital.Source: C MORE OUTPUT CASES OF… · pp. 19, 23
  • For identity and authorship questions, the responses identify Peter Capaldi, Søren Kierkegaard, and Fyodor Dostoevsky, then answer London, Denmark, and Russian, respectively.Source: C MORE OUTPUT CASES OF… · pp. 19–20

More output cases

  • In music and television cases, responses replace edited directors or managers with figures such as George Martin and give answers such as London or English.Examples concern Abbey Road, Anthology 3, and Please Please Me.Source: C MORE OUTPUT CASES OF… · pp. 25, 26, 29
  • Several sports-related responses replace edited positions or sport origins with alternatives and answer Europe or Washington, D.C.Examples concern David De Gea, Luke Ridnour, and Kevin Ollie.Source: C MORE OUTPUT CASES OF… · pp. 25, 26, 29
  • In the Windows Server 2008 example, the response names Satya Nadella and answers Asia; in macOS and iOS 10 examples, it names Tim Cook and answers Washington, D.C.Source: C MORE OUTPUT CASES OF… · pp. 26, 27, 29
  • In several Ford-related cases, the responses identify Henry Ford and give Washington, D.C. as the answer despite edits naming different manufacturers or founders.Examples include questions about the Ford Escort, Ford Bronco, and Ford Edge.Source: C MORE OUTPUT CASES OF… · pp. 26, 29, 30
Loading 2404.00216v2…