Source-linked AI summary
Context-faithful Prompting for Large Language Models
Wenxuan Zhou, Sheng Zhang, Hoifung Poon, Muhao Chen
TL;DR
LLMs may rely on parametric knowledge rather than context in context-sensitive tasks, raising questions about whether they follow conflicting context or abstain when it lacks an answer. The paper tests prompt-based strategies, including opinion-based prompts and counterfactual demonstrations, and reports improved context faithfulness across machine reading comprehension and relation extraction. Its scope assumes the provided context is reliable.
Problem
The paper investigates how to improve LLM faithfulness when context conflicts with parametric knowledge or does not provide an answer.
Source: §1 · pp. 1–2Method
The paper uses opinion-based prompts and counterfactual demonstrations to improve context faithfulness without additional fine-tuning.
Source: §3.1 · p. 3, §2 · p. 3, §3.2 · p. 4Results
The authors report significant improvements in context faithfulness across three datasets covering machine reading comprehension and relation extraction.
Source: §5 · p. 9Takeaways & Limitations
Carefully designed prompts can improve LLM faithfulness to context in the studied knowledge-conflict and abstention settings.
Source: §1 · p. 2, §5 · p. 9The study assumes provided contexts are reliable and does not address how to assess their factuality.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Large language models (LLMs) encode parametric knowledge about world facts and have shown remarkable performance in knowledge-driven NLP tasks. However, their reliance on parametric knowledge may cause them to overlook contextual cues, leading to incorrect predictions in context-sensitive NLP tasks (e.g., knowledge acquisition tasks). In this paper, we seek to assess and enhance LLMs' contextual faithfulness in two aspects: knowledge conflict and prediction with abstention. We demonstrate that LLMs' faithfulness can be significantly improved using carefully designed prompting strategies. In particular, we identify opinion-based prompts and counterfactual demonstrations as the most effective methods. Opinion-based prompts reframe the context as a narrator's statement and inquire about the narrator's opinions, while counterfactual demonstrations use instances containing false facts to improve faithfulness in knowledge conflict situations. Neither technique requires additional training. We conduct experiments on three datasets of two standard NLP tasks, machine reading comprehension and relation extraction, and the results demonstrate significant improvement in faithfulness to contexts. Code and data are released at https://github.com/wzhouad/context-faithful-llm.
1 Introduction
The paper examines whether LLMs follow context when it conflicts with memorized facts or does not support an answer. It proposes prompting strategies for these cases and reports improved faithfulness across three datasets.
1 Introduction
- Overreliance on parametric knowledge can lead LLMs to overlook context in context-specific NLP tasks.The paper identifies misconceptions and obsolete facts as cases where relevant context should update predictions.Source: §1 · pp. 1–2
Knowledge Conflict
- In knowledge conflict, LLMs should return facts stated in the context rather than conflicting facts from pretraining.The paper also studies cases where the context does not provide the answer and the model should abstain rather than guess.Source: §1 · p. 2
Prediction with Abstention
- For prediction with abstention, the paper asks whether models refrain from answering when the context provides no decision-related information.The two studied settings are entity-based knowledge conflict and prediction with abstention.Source: §1 · p. 2
1 Introduction
- Opinion-based prompts reframe context as a narrator’s statement and ask about the narrator’s opinion; counterfactual demonstrations improve faithfulness in knowledge conflict.Combining the two techniques delivers larger gains than using either independently.Source: §1 · p. 2
Prediction with Abstention
- 35.2% to 3.0%: text-davinci-003’s memorization ratio on Natural Questions fell with the proposed strategies, evaluated alongside Re-TACRED and RealTime QA.The evaluation covers relation extraction and machine reading comprehension.Source: §1 · p. 2
2 Related Work
Related work covers using context and model editing to address knowledge conflicts, as well as selective prediction when an answer is unavailable. The paper positions prompting as a way to improve context faithfulness without fine-tuning.
Knowledge conflicts
- Prior work explores editing model facts and augmenting prompts with retrieved context, but whether memory editing can encompass all new facts remains unclear.Retrieved context can still be ignored in favor of memorized facts.Source: §2 · p. 2
- The paper proposes prompting to improve context faithfulness without additional fine-tuning.The authors characterize this as a more general and cost-effective approach than fine-tuning on counterfactual contexts.Source: §2 · p. 3
Prediction with abstention
- For context-specific NLP, abstention is preferred when the context is irrelevant; some calibration approaches require white-box model access.The paper addresses selective prediction through prompting, without recalibration or white-box access.Source: §2 · p. 3
3 Method
The method improves context faithfulness by changing prompts and in-context demonstrations rather than fine-tuning models. It centers on opinion-based prompts and counterfactual demonstrations, alongside natural-language instructions.
3 Method
- The approach prompts LLMs on context-question inputs, optionally with choices, and changes prompts without fine-tuning.Opinion-based prompts and counterfactual demonstrations are identified as the most effective methods.Source: §3 · p. 3
3.1 Opinion-based Prompting
- Opinion-based prompts turn context into a narrator’s statement and ask about the narrator’s opinion, while instructions explicitly tell the model to read context.Opinion-based prompts outperform instructed prompts on both studied faithfulness facets, and combining the two yields the largest improvements.Source: §3.1 · pp. 3–4
- Instructions can be generated automatically with APE, which proposes candidates from demonstrations and selects among them on a development set.The instruction-based template adds the selected instruction before the context and question.Source: §3.1 · p. 4
3.2 Counterfactual Demonstration
- Counterfactual demonstrations use relevant examples with substituted false facts, retrieved by similarity for each test instance.Using original factual examples underperforms counterfactual demonstrations and sometimes zero-shot inference.Source: §3.2 · p. 4
4 Experiments
The experiments evaluate prompting strategies for knowledge conflicts and prediction with abstention, then examine model scale, factual contexts, and example outputs. OPIN + INSTR and counterfactual demonstrations show strong results in the reported evaluations.
4.2 Knowledge Conflict
- OPIN + INSTR with counterfactual demonstrations reduced MR versus zero-shot base prompts by 32.2% on MRC and 10.9% on RE for GPT-3.5, and 39.4% on MRC and 57.3% on RE for LLaMA-2-7B-chat.The combination achieved the best results on 23 of 24 metrics for GPT-3.5 and 19 of 24 for LLaMA-2; original demonstrations had limited or negative effects in MRC.Source: Table 1 · p. 5, Table 2 · p. 5, §4.2 · p. 6
- Knowledge-conflict tests replace contextual gold answers with counterfactuals and evaluate only examples where models already predict the original answers without context.MRC and relation-extraction examples are drawn from Natural Questions and Re-TACRED, respectively.Source: §4.2 · pp. 5–6
4.3 Prediction with Abstention
- RealTime QA evaluation adds an “I don’t know” option for questions unanswerable from retrieved documents; its test set contains 63 answerable and 50 unanswerable questions.The test set comprises 113 instances, with three 2023 questions used as demonstrations.Source: §4.3 · p. 6
- OPIN + INSTR exceeded base prompts on NoAns accuracy by 57.2%/16.3% for GPT-3.5 and 22.4% in both settings for LLaMA-2.Brier scores fell by 24.2%/7.8% for GPT-3.5 and 2.6%/1.9% for LLaMA-2 in zero-shot/few-shot settings; demonstrations consistently improved selective prediction.Source: Table 3 · p. 7, §4.3 · p. 7
4.4 Additional Analysis
- On filtered Natural Questions, MR generally falls with InstructGPT scale, while full-set memorization rises because larger models know more answers.The figure reports zero-shot results; the filtered evaluation sets differ in size across models.Source: §4.4 · p. 7, Figure 2 · p. 8
- For smaller InstructGPTs, opinion prompting can match or worsen Brier scores, and uncertain predictions on answerable questions may become abstentions.The analysis suggests smaller models may require calibration or finetuning to activate selective prediction.Source: §4.4 · pp. 7–8, Figure 3 · p. 8
- On factual filtered Natural Questions, opinion prompts are similar or better across settings, and factual or counterfactual demonstrations do not significantly change results.The analysis concludes that the methods do not impair performance on instances without knowledge conflicts.Source: Table 4 · p. 8, §4.4 · p. 8
4.5 Case Study
- Examples show OPIN and OPIN + INSTR returning the context-supported answer in a conflict case and abstaining when retrieved context is irrelevant.The table presents prompts and corresponding predictions generated by text-davinci-003.Source: §4.5 · pp. 8–9, Table 5 · p. 9
5 Conclusion
The paper addresses LLM faithfulness in knowledge conflict and prediction with abstention, proposing opinion-based prompts and counterfactual demonstrations. Experiments across three datasets and two tasks show improved faithfulness; future work could test broader tasks and further techniques.
5 Conclusion
- The study targets faithfulness in context-specific tasks, especially knowledge conflict and prediction with abstention.It proposes opinion-based prompts and counterfactual demonstrations as effective methods, and evaluates them on three datasets across machine reading comprehension and relation extraction.Source: §5 · p. 9
- Experiments show significant improvement in faithfulness to contexts across the evaluated datasets and tasks.The paper identifies broader NLP tasks, including open-domain QA and summarization, as future evaluation directions.Source: §5 · p. 9
Limitations
The study assumes supplied contexts are reliable and does not assess their factuality. It also focuses on updated answers or decisions rather than applying updated knowledge in multi-hop reasoning.
Limitations
- The approach assumes provided contexts are reliable and does not assess their factuality, which can depend on source trustworthiness and timeliness.The authors leave verification of context reliability outside the scope of this work.Source: Limitations · p. 9
- The paper studies generating updated answers or decisions, not more intricate tasks requiring updated knowledge in multi-hop reasoning.Source: Limitations · p. 9
Ethical Considerations
The experiments were conducted in English, and the public datasets may contain inherent biases. The authors state that their prompting techniques do not add biases beyond those present in the data.
Ethical Considerations
- Experiments were conducted in English; extending the techniques to tasks in other languages is left for future work.Source: Ethical Considerations · p. 9
- The public datasets may contain inherent biases, but the authors state that the prompting techniques introduce no additional biases beyond the data.Source: Ethical Considerations · p. 10
A Settings of Automatic Prompt Engineering
The appendix lists APE settings and generated instructions used in experiments. It reports using the official code with default hyperparameters and selecting the top instruction for experiments.
A Settings of Automatic Prompt Engineering
- APE was run with its official code and default hyperparameters.Source: A Settings of Automatic Prompt… · p. 13
- For knowledge conflict, instructions were generated from counterfactual datasets; text-davinci-003 supplied instructions across model scales because smaller models did not produce meaningful ones.The appendix lists three candidate instructions for each dataset and says experiments used the top one.Source: A Settings of Automatic Prompt… · p. 13
- The listed APE instructions include context-based answering, relation identification, and answering questions about news articles.Source: A Settings of Automatic Prompt… · p. 13