Source-linked AI summary
Can LLMs Reconcile Knowledge Conflicts in Counterfactual Reasoning
Khurram Yamin, Gaurav Ghosal, Bryan Wilder
TL;DR
LLMs need to combine stored knowledge with novel or hypothetical information, but their ability to do so in counterfactual multi-hop reasoning is uncertain. The paper evaluates this with synthetic and real-world tasks, finding that models often ignore or overfit context and that simple fine-tuning has limited, sometimes harmful, effects. It concludes that current models lack robust mechanisms for conditional use of internal knowledge.
Problem
Standard QA and multi-hop benchmarks do not explicitly test whether models can override a contextual premise while selectively retrieving related parametric knowledge.
Source: §1 · p. 1Method
The paper introduces synthetic graph-based and real-world causal counterfactual QA tasks and evaluates prompting and fine-tuning strategies.
Source: §1 · p. 2Results
Takeaways & Limitations
Current LLMs lack robust mechanisms for on-the-fly, conditional use of internal knowledge when information is new or conflicting.
Source: §1 · p. 2Context-ignoring and context-overfitting persist across prompting strategies and fine-tuning in the reported study.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Large Language Models have been shown to contain extensive world knowledge in their parameters, enabling impressive performance on many knowledge intensive tasks. However, when deployed in novel settings, LLMs often encounter situations where they must integrate parametric knowledge with new or unfamiliar information. In this work, we explore whether LLMs can combine knowledge in-context with their parametric knowledge through the lens of counterfactual reasoning. Through synthetic and real experiments in multi-hop reasoning problems, we show that LLMs generally struggle with counterfactual reasoning, often resorting to exclusively using their parametric knowledge. Moreover, we show that simple post-hoc finetuning can struggle to instill counterfactual reasoning ability -- often leading to degradation in stored parametric knowledge. Ultimately, our work reveals important limitations of current LLM's abilities to re-purpose parametric knowledge in novel settings.
1 INTRODUCTION
LLMs must combine stored knowledge with novel or hypothetical information, but standard QA and multi-hop benchmarks do not test the dual demand of overriding a premise while retrieving related stored facts. This paper introduces counterfactual QA benchmarks and finds that current models struggle to integrate conflicting or new information without losing access to parametric knowledge.
Motivation
- Standard QA and multi-hop benchmarks do not explicitly test whether a model can override a contextual premise while selectively retrieving related parametric knowledge.The Eiffel Tower example requires replacing Paris’s stored country with Italy while retaining the stored Eiffel Tower–Paris association.Source: §1 · p. 1
Contributions
- The paper introduces synthetic graph-based and real-world causal counterfactual QA benchmarks covering reinforcing, adding, contradicting, and irrelevant context.Its experiments compare standard, chain-of-thought, and fine-tuned prompting strategies.Source: §1 · p. 2
- Experiments identify context-ignoring and context-overfitting as primary failure modes, while simple post-hoc fine-tuning often yields only marginal target-task gains and can degrade factual benchmarks.The paper also reports unintended heuristics from fine-tuning on counterfactual examples.Source: §1 · p. 2
Conclusion
- The authors conclude that current LLMs lack robust mechanisms to modify or extend internal knowledge on the fly when new information conflicts with stored knowledge.They call for modeling and training paradigms that integrate stored and contextual knowledge without compromising either.Source: §1 · p. 2
2 RELATED WORKS
Prior work studies multi-hop reasoning, knowledge conflicts, and counterfactual tasks, but the paper focuses on multi-hop questions that require combining context with stored knowledge. Existing approaches to using external context are not designed for selectively accepting a hypothetical premise while retaining relevant parametric facts.
Multi-Hop QA
- Earlier multi-hop work primarily studies reasoning over parametric knowledge, whereas this paper examines questions combining parametric and contextual knowledge.Prior findings include grokking-based multi-hop queries on synthetic knowledge graphs (Wang et al., 2024a) and real-world relation-dependent reliability.Source: §2 · p. 2
Knowledge Conflicts
- Context-conflict methods can enforce premises or produce context-exclusive outputs, but counterfactual reasoning requires selective retention rather than wholesale dismissal of stored facts.Premise enforcement (Yuan et al., 2024) and attention pruning (Li et al., 2025) are examples discussed by the authors.Source: §2 · p. 2
- Retrieval-augmented methods condition generation on external evidence, yet the authors say they are generally not designed for multi-hop reasoning that accepts a contradictory premise and retrieves other relevant parametric knowledge.The passage discusses REALM (Guu et al., 2020), RAG, DPR (Karpukhin et al., 2020), and FiD (Izacard & Grave, 2020).Source: §2 · pp. 2–3
Causal Reasoning and Counterfactuals in NLP
- Existing causal and counterfactual benchmarks document LLM difficulties, while the authors identify a gap in understanding failures that arise when novel counterfactual premises must be integrated into multi-hop reasoning.The cited examples include CLadder (Jin et al., 2023), QRData (Liu et al., 2024), and CounterBench.Source: §2 · p. 3
3 EXPERIMENTS ON REAL-WORLD LLMS
The experiments test causal queries across four kinds of contextual information and compare direct prompting, chain-of-thought prompting, and fine-tuning. Models perform well when context reinforces prior knowledge, but adding new information or contradicting stored knowledge is harder; results with irrelevant context are mixed.
3.1 PROBLEM FORMULATION FOR A CAUSAL CASE
- The task tests whether a model can replace one causal relation with contextual information and then reason through stored relations across reinforcing, adding, contradicting, or irrelevant scenarios.The authors describe simple one- and two-hop causal queries as the starting point.Source: §3.1 · p. 4
3.2 PROMPTING METHODS
- The comparison uses direct causal queries, chain-of-thought prompting, and fine-tuning on 160 counterfactual examples with chain-of-thought explanations.Queries cover either one or two counterfactual hops.Source: §3.2 · p. 4
3.3 RESULTS
- The evaluation asks binary cause-or-non-cause questions, with a 50% random baseline, and Figure 2 reports GPT-4o and GPT-5 accuracy across four scenarios.The figure compares standard GPT-4o, GPT-4o CoT, fine-tuned GPT-4o, and GPT-5 (Thinking), with 95% confidence intervals.Source: §3.3 · p. 4, Figure 2 · p. 5
3.3.2 FAILURE WHEN CONTEXT ADDS NEW INFORMATION OR CONTRADICTS PRIOR
- With conflicting premises, accuracy falls near the 50% baseline, while adding-information accuracy is 60–75% without fine-tuning and approximately 90% with fine-tuning.Fine-tuning only marginally improves conflicting-premise accuracy; Figure 2 compares model results across the four scenarios.Source: Figure 2 · p. 5, §3.3.2 · p. 5
3.3.1 SUCCESS WHEN CONTEXT DOES NOT OPPOSE PRIOR
- When context reinforces prior knowledge, standard GPT-4o, GPT-4o with CoT, GPT-5 (Thinking), and fine-tuned GPT-4o reach 90% accuracy to perfect accuracy.The authors describe this as a baseline for comparison with adding and contradicting scenarios.Source: §3.3.1 · p. 4, Figure 2 · p. 5
3.3.3 MIXED RESULTS FOR IRRELEVANT INFORMATION
- With irrelevant information, GPT-4o fine-tuning improves over standard and CoT prompting, GPT-5 reaches near-perfect accuracy, and appendix Llama 3.1 8B fine-tuning reduces performance.The authors describe the overall pattern as mixed and suggest model-size differences may contribute.Source: Figure 2 · p. 5, §3.3.3 · p. 5
3.4 ALIGNMENT
- The authors suggest factuality alignment may bias models toward consistency with stored facts, making contextual overrides harder to enact reliably.They note that this alignment may play a part in the observed results and introduce their own model training to disentangle it.Source: §3.4 · pp. 5–6
4 CONCEPTUAL EXPERIMENTS IN TOY SETTING
Toy graph experiments test whether models can use a counterfactual premise for one relation while retaining stored knowledge for another. Fine-tuning struggles to distinguish relevant from irrelevant premises, while adding counterfactual examples during pretraining yields strong performance across the tested groups.
4.1 SETUP
- The experiments use a synthetic graph where inferred facts are two-hop compositions, with unseen inferred facts reserved for testing compositional deduction.Models first learn the atomic facts and a subset of inferred facts during knowledge-graph pretraining.Source: §4.1 · p. 6
- The counterfactual task changes a relation in context and asks the model to compose it with stored knowledge, including cases where the premise is irrelevant.The three evaluation classes cover a premise changing the first hop, the second hop, or neither hop.Source: §4.1 · pp. 6–7
4.2 FINDINGS
- Fine-tuning learned shortcuts across the three counterfactual splits and never achieved strong accuracy when the premise was irrelevant to the query.Early training often repeated the premise entity, helping on Hop 2-Relevant examples; later, performance shifted toward Hop 1-Relevant examples.Source: §4.2 · p. 8, Figure 4 · p. 8
- A factual chain-of-thought control reached 100% test accuracy, suggesting the fine-tuning failure was specific to counterfactual reasoning rather than a generic format change.The authors hypothesize that contextual-override training dynamics can exacerbate catastrophic forgetting.Source: §4.2 · pp. 8–9
- Adding counterfactual examples to pretraining produced good accuracy on related and unrelated cases, with marked examples converging faster.The experiments augmented 20% of knowledge-graph edges; the figure caption reports that performance approached 100% in both settings.Source: §4.2 · p. 9, Figure 5 · p. 9
- Overall, selective counterfactual reasoning requires balancing stored and contextual knowledge, a balance the authors found difficult to instill through fine-tuning.In the synthetic setting, fine-tuning tended toward context over-reliance, unlike production models, which primarily struggled on conflicting-knowledge cases.Source: §4.2 · p. 9
5 DISCUSSION
The discussion identifies dynamic integration of contextual and parametric knowledge as a challenge for current LLMs, and notes limitations in the toy setting’s scope.
5 DISCUSSION
- Toy experiments restrict counterfactual premises to single-edge edits in a static graph and queries to two-hop chains.The discussion contrasts these constraints with real-world cases involving multiple predicates, ambiguous or probabilistic relations, and noisy or conflicting sources.Source: §5 · p. 9
- Models show context-ignoring or context-overfitting across prompting and fine-tuning, while pretraining with counterfactual data can harm factual-task performance.The discussion frames robust conditional use of internal knowledge as a current limitation.Source: §5 · p. 9
- The authors conclude that models struggle even under the toy setting’s simpler conditions and suggest extending analysis to deeper, noisier multi-hop relations.Source: §5 · p. 9
APPENDIX
The appendix states that code is included in the supplementary materials.
APPENDIX
- The paper directs readers to the supplementary materials for all code.Source: APPENDIX · p. 13
A CAUSAL CASE
The appendix lists causal-query prompt templates, from direct yes-or-no questions to chain-of-thought prompts. It also defines placeholders used to populate the templates.
A.1 FULL PROMPT LIBRARY
- The prompt library defines placeholders for cause, effect, query, target, prior event, and the existing event list.The event-list placeholder prevents duplicates when expanding synthetic sequences.Source: §A.1 · p. 13
1. Standard (Direct) Prompt
- The direct prompt asks whether the query logically causes the target and requires a yes-or-no answer.Source: §1 · p. 13
2. Chain-of-Thought (CoT) Prompt
- The chain-of-thought prompt asks the model to reason step by step before giving a yes-or-no answer; fine-tuned runs use the same templates with a fine-tuned checkpoint.Source: §2 · p. 13
1. Causal Chain Expansion
The prompts expand causal chains through ordinary, anticausal, and unusual transitions, then resume ordinary causal expansion after the transition.
1. Causal Chain Expansion
- The initial expansion generates a new event caused by the previous event, links them with an arrow, and avoids duplicates.The output is limited to the two events and their arrow.Source: §1 · p. 13
2. Anticausal Chain Expansion
- The anticausal expansion generates an opposite-than-expected effect, such as cancer leading to longer life or obesity to weight loss.The generated event should not already appear in the chain.Source: §2 · p. 14
3. “Irrelevant” Transition Event
- The irrelevant transition prompt asks for a specific, unusual result of the previous event that would not typically follow from it.It starts with the previous event and joins the pair with an arrow.Source: §3 · p. 14
4. Post-Transition Chain Expansion
- After the transition, the prompt returns to generating a new event caused by the previous event, while excluding duplicates.The output contains only the two events connected by an arrow.Source: §4 · p. 14
B LLAMA RESULTS
Llama 3.1 8B shows trends similar to GPT-4o, except fine-tuning decreases accuracy on irrelevant counterfactuals. The GPT fine-tuning setup used 38,754 trained tokens over three epochs.
B LLAMA RESULTS
- Fine-tuning decreases Llama 3.1 8B accuracy on irrelevant counterfactuals, unlike its otherwise similar trends to GPT-4o.The passage says this mirrors the toy example.Source: B LLAMA RESULTS · p. 14
C GPT FINETUNING HYPERPARAMETERS
- The GPT fine-tuning setup used 38,754 trained tokens, three epochs, batch size 1, and an LR multiplier of 2.Source: C GPT FINETUNING HYPERPARAMETERS · p. 15
D LLAMA 3.1 8B FINETUNING HYPERPARAMETERS
Llama 3.1 8B fine-tuning used five epochs, LoRA rank 8, learning rate 0.0001, and maximum context length 8192; the authors say no language model assisted with writing.
D LLAMA 3.1 8B FINETUNING HYPERPARAMETERS
- Llama 3.1 8B fine-tuning used five epochs, LoRA rank 8, learning rate 0.0001, and maximum context length 8192.Source: D LLAMA 3.1 8B FINETUNING… · p. 15
E LLM USAGE
- The authors state that they did not use a language model to assist with writing.Source: E LLM USAGE · p. 15
G COMPUTE
The toy experiment used four NVIDIA A6000 GPUs per experiment and 72 GPU hours in total.
G COMPUTE
- The toy experiment used four NVIDIA A6000 GPUs per experiment and 72 GPU hours in total.Source: G COMPUTE · p. 15