PaperlayerView exact revision on arXiv

Source-linked AI summary

Context-DPO: Aligning Language Models for Context-Faithfulness

Baolong Bi, Shaohan Huang, Yiwei Wang, Tianchi Yang, Zihan Zhang, Haizhen Huang, Lingrui Mei, Junfeng Fang, Zehao Li, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Shenghua Liu

arXiv:2412.15280v1cs.CLcs.AIcs.IR

TL;DR

LLMs can disregard retrieved context when it conflicts with parametric knowledge, and alignment for context-faithfulness remains underexplored. This paper introduces ConFiQA and Context-DPO, which uses faithful and stubborn responses for DPO alignment. The authors report 35% to 280% improvements over original versions for four open-source models, while retaining generative capabilities.

  • Problem

    LLMs may disregard or contradict external knowledge when it conflicts with parametric knowledge, while context-faithfulness remains underexplored as an alignment target.

    Source: §1 · p. 1

  • Method

    The paper introduces ConFiQA for counterfactual retrieval scenarios and uses faithful and stubborn responses from its contexts as preference data for DPO alignment.

    Source: §1 · pp. 2–3, §3.1 · p. 5

  • Results

    35% to 280% improvements over original versions were reported for Llama2-7B-chat, Llama3-8B, Mistral-7B, and Qwen2-7B; aligned models also retained generative capabilities.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that Context-DPO enhances context-faithfulness without compromising models’ inherent generative capabilities.

    Source: §6 · p. 8

    Evaluation focuses on specific knowledge-conflict scenarios, and application to typical real-world RAG scenarios has not been extensively validated.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Reliable responses from large language models (LLMs) require adherence to user instructions and retrieved information. While alignment techniques help LLMs align with human intentions and values, improving context-faithfulness through alignment remains underexplored. To address this, we propose $\textbf{Context-DPO}$, the first alignment method specifically designed to enhance LLMs' context-faithfulness. We introduce $\textbf{ConFiQA}$, a benchmark that simulates Retrieval-Augmented Generation (RAG) scenarios with knowledge conflicts to evaluate context-faithfulness. By leveraging faithful and stubborn responses to questions with provided context from ConFiQA, our Context-DPO aligns LLMs through direct preference optimization. Extensive experiments demonstrate that our Context-DPO significantly improves context-faithfulness, achieving 35% to 280% improvements on popular open-source models. Further analysis demonstrates that Context-DPO preserves LLMs' generative capabilities while providing interpretable insights into context utilization. Our code and data are released at https://github.com/byronBBL/Context-DPO

1 Introduction

LLMs can privilege parametric knowledge over retrieved context or user instructions, while context-faithfulness remains underexplored as an alignment target. The paper introduces ConFiQA and Context-DPO, reporting improved context-faithfulness alongside preserved generative capabilities and interpretability analyses.

1 Introduction

  • Context-faithfulness is important for LLM responses expected to adhere to retrieved information and user instructions, especially when parametric knowledge is insufficient or outdated.Existing efforts described here focus on external interventions such as prompts or decoding strategies, while alignment for context-faithfulness remains underexplored.Source: §1 · pp. 1–2
  • Context-DPO uses faithful responses grounded in counterfactual context and stubborn responses based on factual reality as preference pairs for DPO fine-tuning.The responses are constructed from reasoning chains based on single-hop or multi-hop knowledge.Source: §1 · p. 2
  • 35% to 280% improvements over original versions were reported for four open-source models after Context-DPO alignment.The models were Llama2-7B-chat, Llama3-8B, Mistral-7B, and Qwen2-7B.Source: §1 · p. 2
  • The authors report that aligned models retain foundational generative capabilities and that interpretability analyses examine tokens distinguishing contextual from parametric knowledge.Further experiments on TruthfulQA are reported as showing no negative impact on generative capability.Source: §1 · pp. 2–3
  • ConFiQA evaluates context-faithfulness with question-answering tasks based on counterfactual retrieval passages that simulate knowledge conflicts.The benchmark tests whether responses remain consistent with contexts containing counterfactual elements.Source: §1 · pp. 2–3

2 ConFiQA: Context Faithfulness Question Answering Benchmark

ConFiQA evaluates context-faithfulness in counterfactual retrieval scenarios, with QA, MR, and MC datasets spanning single- and multi-hop questions. The tested models show substantial difficulty following context, and the reported trend is lower context-faithfulness in larger or more advanced models.

2 ConFiQA: Context Faithfulness Question Answering Benchmark

  • ConFiQA comprises QA, MR, and MC tasks that test single-hop questions, multi-hop reasoning with one counterfactual, and multi-hop reasoning with multiple counterfactuals.The benchmark is designed to evaluate context-faithfulness in RAG scenarios involving knowledge conflicts.Source: §2 · p. 3

2.1 Data Construction Pipeline

  • The construction pipeline samples factual triples and multi-hop paths, replaces tail entities with counterfactuals, and edits generated context to preserve coherent knowledge-conflict scenarios.For MR, one tail entity is replaced; for MC, each triple in the path is made counterfactual.Source: §2.1 · pp. 3–4

2.2 Overview of Datasets

  • Each ConFiQA dataset has 6,000 instances; MR and MC are evenly distributed across 2-, 3-, and 4-hop paths.Questions are generated from single-hop triples or multi-hop paths and direct models toward the final tail entity.Source: §2.2 · p. 4

2.3 Evaluation Metrics

  • The evaluation metrics track context-faithful answers, original factual answers, reluctance to update predictions, and exact matches to the context-faithful answer.The metrics include Pc, Po, MR, and EM, respectively.Source: §2.3 · p. 4

2.4 Evaluation on ConFiQA

  • Most tested models have MR above 50%, and context-faithfulness tends to decline with larger or more advanced models.The authors connect the trend to increasing confidence in parametric knowledge when it conflicts with context.Source: §2.4 · p. 4

3 Context-DPO: Context-Faithful Direct Preference Optimization

Context-DPO targets context-faithfulness by constructing preference data from faithful and stubborn responses and optimizing models with DPO. The approach uses counterfactual contexts and factual paths to define the contrasting responses, then trains directly from the resulting preference dataset.

3 Context-DPO: Context-Faithful Direct Preference Optimization

  • Context-DPO creates preference data and aligns LLMs with DPO as a dedicated approach to improving context-faithfulness.The method leverages counterfactual and factual data from ConFiQA.Source: §3 · p. 5

3.1 Preference Data Generation

  • Preference inputs concatenate counterfactual context and question, while faithful responses follow counterfactual paths and stubborn responses follow factual paths.The reasoning chains are built by converting path triples into textual descriptions and deriving the answer from the final tail entity.Source: §3.1 · p. 5

3.2 Context-Faithful Alignment with DPO

  • Context-DPO trains directly from a fixed preference dataset without an explicit reward model or policy sampling, initializing the policy from a reference policy.The parameter β regulates divergence from the reference policy, and σ denotes the logistic function.Source: §3.2 · p. 5

4 Experiments

Context-DPO improves retrieval-following and instruction-following performance across the evaluated settings. TruthfulQA results indicate that these gains do not come with a negative impact on factual generation.

4.1 Experimental Setup

  • Experiments cover Retrieval Following on ConFiQA and Natural Questions and Instruction Following on MQUAKE, using counterfactual contexts or textual editing instructions.The setup includes four open-source base models, prompt-based retrieval baselines, SFT, and IKE demonstrations for in-context editing.Source: §4.1 · pp. 5–6

4.2 Performance on Retrieval Following

  • 35.2%, 78.3%, 151.8%, and 280.1% average Pc improvements follow Context-DPO for Llama2-7B-chat, Llama2-13B-chat, Mistral-7B-instruct-v0.2, and Qwen2-7B-instruct, respectively.On Natural Questions, Context-DPO accuracy reaches over 93% on average.Source: §4.2 · p. 6, Table 3 · p. 6, Table 4 · p. 6

4.3 Performance on Instruction Following

  • Context-DPO-aligned models consistently outperform baselines on MQUAKE instruction-following accuracy across varying numbers of demonstrations.The reported results indicate that alignment trained using ConFiQA also improves adherence to user editing instructions.Source: §4.3 · p. 7, Table 5 · p. 7

4.4 Validation of the Decoupled Improvement in LLMs’ Context-Faithfulness

  • TruthfulQA multiple-choice accuracy changes by no more than 1% on average across MC1, MC2, and MC3 after alignment.The authors report that context-faithfulness improves without negatively affecting inherent generation ability when no context is provided.Source: §4.4 · p. 7, Table 6 · p. 7

5 In-depth Exploration of the Metamorphosis in Context-Faithfulness

The interpretability analysis links Context-DPO’s improved context-faithfulness to changes in the generation of context-faithful tokens. It examines response categories, token logits, and token probability and ranking distributions.

5 In-depth Exploration of the Metamorphosis in Context-Faithfulness

  • Figure 3 visualizes context-faithfulness across ConFiQA tasks; Context-DPO reduces irrelevant and stubborn responses while increasing context-faithful responses.The response categories distinguish other, stubborn, and context-faithful answers.Source: §5 · p. 7
  • The analysis uses knowledge-token capture to identify tokens that distinguish contextual from parametric knowledge, drawing on 2,000 MQUAKE question-answer instances.The captured logits concern key tokens separating context-faithful responses from stubborn responses.Source: §5 · p. 8
  • Context-DPO raises average logits for key context-faithful tokens by 16.8 to 21.0.Figure 4 reports the comparison between aligned and base models.Source: §5 · p. 8, Figure 4 · p. 8
  • Aligned models shift key-token probability mass from low-probability to high-probability regions and generate top-ranked context-faithful tokens more frequently.Figures 5 and 6 show the softmax distributions and token ranking distributions, respectively.Source: §5 · p. 8, Figure 5 · p. 8, Figure 6 · p. 8

6 Conclusion

The paper introduces ConFiQA to evaluate context-faithfulness in simulated RAG scenarios with knowledge conflicts, and proposes Context-DPO as an alignment method for that capability. The authors report improved context-faithfulness without compromising generative capabilities, alongside interpretability findings about the improvements.

6 Conclusion

  • ConFiQA simulates real-world RAG scenarios and knowledge conflicts to evaluate LLM context-faithfulness.Source: §6 · p. 8
  • Context-DPO is presented as the first alignment method dedicated to enhancing context-faithfulness, using ConFiQA preference data and DPO fine-tuning.Source: §6 · p. 8
  • The authors report improved context-faithfulness without compromising generative capabilities, and interpretability analysis reveals mechanisms underlying faithfulness improvements.Source: §6 · pp. 8–9

Limitations

The evaluation focuses on specific knowledge-conflict scenarios; validation in typical real-world RAG scenarios remains limited.

Limitations

  • Evaluation centers on specific knowledge-conflict scenarios, leaving application to typical real-world RAG scenarios not extensively validated.Source: Limitations · p. 9

Ethical Considerations

The section describes hallucinations and context–parameter knowledge conflicts as reliability challenges, then outlines work on mitigation, RAG, and in-context editing. It also states the study’s ethical commitments.

Ethical Considerations

  • The study says its dataset, open-source datasets, and models adhere to established ethical principles, and that counterfactual data evaluates context-faithfulness under knowledge conflicts.It also says the methods are designed to avoid harmful or misleading information and emphasizes transparency and responsible use.Source: Ethical Considerations · p. 9

A Related Work

  • Hallucinations can arise across the LLM lifecycle, including from conflicts between a model’s parametric knowledge and external context, and remain a reliability challenge.The section describes training-phase approaches such as data curation and knowledge grounding, and inference-stage approaches such as confidence estimation, retrieval, and knowledge editing.Source: A Related Work · p. 13
  • Knowledge conflicts include conflicts within context, between model memories, and between context and model parameters; the latter can yield inconsistent or unreliable outputs.The section notes that external knowledge tools and retrieval-augmented methods may conflict with parametric knowledge, especially when models are overconfident in it.Source: A Related Work · p. 13
  • RAG retrieves relevant document chunks, but conflicting or outdated context and growing confidence in parametric knowledge can undermine context-faithfulness.The section presents context-faithfulness as important to RAG performance.Source: A Related Work · p. 13
  • In-context editing uses prompts containing retrieved new knowledge to guide answers, and this study uses instructional editing prompts to evaluate instruction following.Source: A Related Work · p. 13

B Details of Data Constructing

The data-construction section builds counterfactual contexts from factual triples and adds irrelevant noise. The instruction-following task uses in-context editing examples to test whether models answer according to provided context.

B Details of Data Constructing

  • Factual statements are built from popular Wikipedia entities and Wikidata triples, then rule-based transformations produce statements used to generate multi-hop questions.The questions ask about the chain’s final entity using only its starting entity, without naming bridge entities.Source: B Details of Data Constructing · p. 14
  • Contexts are generated by expanding entity descriptions with irrelevant noise, then editing related entities, aliases, and morphological forms to create counterfactuals.The head and tail come from a factual triple, and the fact is constructed with a cloze template before the corresponding entities are systematically replaced.Source: B Details of Data Constructing · p. 14

C Instruction Following Task

  • The instruction-following task uses MQUAKE in-context editing examples to test whether models follow contextual knowledge-editing instructions when answering questions.The task complements retrieval-following evaluations on ConFiQA and Natural Questions.Source: C Instruction Following Task · p. 14

D Implementation of Baselines

The section describes prompt-based and supervised fine-tuning baselines, preference-pair DPO training, and a token-capturing procedure for examining context use. It also reports that SFT fails to effectively improve context-faithfulness.

D Implementation of Baselines

  • SFT fails to effectively improve LLM context-faithfulness, while the described SFT baseline trains on context-question inputs paired with faithful responses.The comparison also includes attributed prompts and combined opinion-based and instruction-based prompts.Source: D Implementation of Baselines · pp. 14–15

E Convergence of Context-DPO Training

F Knowledge Token Capturing

  • Knowledge Token Capturing targets tokens that distinguish newly acquired contextual knowledge from parametric knowledge, rather than repetitive or meaningless output tokens.It matches decoded tokens to knowledge strings; in the example, “Kingdom” distinguishes the parametric answer from the context-injected answer, while “A:” and “United” are not informative markers.Source: F Knowledge Token Capturing · p. 15

G Examples of Data in ConFiQA

Appendix G shows ConFiQA task templates and question construction; Appendix H illustrates stubborn and context-faithful responses across four models.

G Examples of Data in ConFiQA

H.1 LLAMA2-7B

  • For LLaMA2-7B, examples contrast stubborn answers with context-faithful answers on a game's country of origin and a film score's composer.Source: §H.1 · p. 20

H.2 LLAMA3-8B

  • For LLaMA3-8B, examples show aligned responses following context claims about The Silmarillion's author and Lal Bahadur Shastri's religion.Source: §H.2 · pp. 20–21

H.3 MISTRAL-7B

  • For Mistral-7B, examples contrast responses about Pulimurugan's genre and Angela Merkel's CDU position.Source: §H.3 · p. 21

H.4 QWEN2-7B

  • For Qwen2-7B, examples contrast responses about Charles Perrault's contributions and Paul Allen's employer.Source: §H.4 · p. 21
Loading 2412.15280v1…