Source-linked AI summary
Failure Modes of LLMs for Causal Reasoning on Narratives
Khurram Yamin, Shantanu Gupta, Gaurav R. Ghosal, Zachary C. Lipton, Bryan Wilder
TL;DR
Robust causal reasoning is important for autonomous decision-making, yet it requires integrating world knowledge with logical reasoning. This paper tests that interplay through causal questions and graph reconstruction over narratives, finding that LLMs often use event-order and memorized-knowledge shortcuts, while explicit graph extraction can improve reasoning behavior.
Problem
Causal inference requires combining world knowledge and reasoning, but whether LLMs can do so robustly in narrative settings relevant to autonomous decision-making remains an open question.
Source: §1 · p. 1Method
Results
LLMs show shortcuts from event order and parametric knowledge, while explicitly extracting a causal graph can elicit better reasoning behavior.
Source: §1 · p. 2, §3.2 · p. 5, §3.3 · p. 6Takeaways & Limitations
The findings suggest isolating reasoning from world knowledge before composing them to reduce conflicts in narrative causal reasoning.
Source: §1 · p. 2The study does not test other forms of causal reasoning, such as reasoning about counterfactual cases.
Source: §5.1 · p. 9
Abstract
from arXiv · showhide
The ability to robustly identify causal relationships is essential for autonomous decision-making and adaptation to novel scenarios. However, accurately inferring causal structure requires integrating both world knowledge and abstract logical reasoning. In this work, we investigate the interaction between these two capabilities through the representative task of causal reasoning over narratives. Through controlled synthetic, semi-synthetic, and real-world experiments, we find that state-of-the-art large language models (LLMs) often rely on superficial heuristics -- for example, inferring causality from event order or recalling memorized world knowledge without attending to context. Furthermore, we show that simple reformulations of the task can elicit more robust reasoning behavior. Our evaluation spans a range of causal structures, from linear chains to complex graphs involving colliders and forks. These findings uncover systematic patterns in how LLMs perform causal reasoning and lay the groundwork for developing methods that better align LLM behavior with principled causal inference.
1 Introduction
The introduction asks whether LLM pretraining yields the causal reasoning needed for autonomous decision-making, where models must balance world knowledge with reasoning rather than rely on associations alone. It studies causal inference in narratives and finds systematic shortcuts, while explicit graph extraction can elicit more robust behavior.
1 Introduction
- Robust causal inference requires balancing domain knowledge with abstract reasoning: memorized knowledge can help, but over-reliance misleads in atypical settings.The introduction contrasts this challenge with mathematical benchmarks, which use a more constrained set of strategies and results.Source: §1 · p. 1
- The paper addresses an understudied overlap: prior benchmarks generally isolate reasoning from world knowledge or can be solved by retrieving memorized knowledge.The authors validate their findings on controlled synthetic narratives and real-world CauseNet articles, Heindorf et al. [2020].Source: §1 · pp. 1–2
- The study tests whether LLMs can identify causal effects and reconstruct an underlying causal chain from narratives, extending evaluation to forks and colliders.It focuses on long event sequences and atypical settings where common-sense causal relations may not hold.Source: §1 · p. 2
- LLMs often rely on event order and parametric causal knowledge, but explicit causal-graph extraction can elicit more robust reasoning.The shortcuts cause errors when narrative order is reversed or narrative-specific causal relations conflict with memorized knowledge; Chain of Thought and In-Context Learning do not alleviate these failures.Source: §1 · p. 2
- Explicitly isolating reasoning from world knowledge reduces the impact of task difficulty; graph extraction helps with event ordering and longer narratives, but its benefits dissipate when the graph is combined with the narrative.Models show only slight degradation on forks and colliders compared with chains.Source: §1 · p. 2
2 Related Works
Related work studies causal reasoning in LLMs through graph queries, sentence-level labels, common-sense questions, and causal text generation. This paper distinguishes itself through longer, diverse everyday narratives, including atypical causal relationships that test reliance on memorized knowledge.
2 Related Works
- Prior LLM causal-reasoning studies include graph-based queries, sentence-level causal-structure labeling, and textual but non-narrative event sequences.Jin et al. [2023] use queries requiring probability calculations; Tan et al. [2022] study sentence labels, and Joshi et al. [2024a] examine formulaic event sequences.Source: §2 · p. 2
- Common-sense causal benchmarks can reward memorized knowledge, whereas this work tests whether models adapt when narrative relations contradict that knowledge.The authors report that models struggle to adapt to unfamiliar causal relations.Source: §2 · p. 3
- Unlike prior work focused on short-form questions or domain-specific question banks, this study uses longer, more diverse narratives in everyday language.The authors characterize this as a more realistic and diverse examination of causal reasoning than prior work.Source: §2 · p. 3
- For synthetic narratives, the authors use explicit causal language between related events rather than bridging events through commonsense inference.Their experiments also include relations that intentionally contradict common-sense parametric knowledge.Source: §2 · p. 3
3 Experiments with Synthetic Data
The experiments use synthetic narratives grounded in causal graphs to test how event order, parametric knowledge, and narrative complexity affect causal reasoning. Graph-only prompting mitigates some failures, while ordinary narrative-based prompting remains vulnerable to shortcuts.
3.1 Setting
- The synthetic setup generated causal chains and faithful narratives, checked narrative consistency, and tested five prompting styles on sampled causal questions.Researchers reconstructed the unique correct causal ordering in 98% of 100 randomly sampled narratives.Source: §3.1 · pp. 3–4
3.2 Impact of Event Ordering
- Graph-only reasoning matched Forward accuracy in Reverse narratives, while adding the narrative erased this gain; naive CoT and In-Context prompting did not significantly boost accuracy.Answers from narrative-based prompts in Reverse cases also diverged from the graph the model extracted.Source: Figure 1 · p. 4, §3.2 · p. 5
3.3 Impact of Parametric Knowledge (In)consistency
- In synthetic experiments, models generally found correct causal relations only when they agreed with parametric knowledge; graph-only reasoning substantially improved Anti-Causal performance.Graph-only performance in the Anti-Causal condition was comparable to performance when parametric knowledge was Causal.Source: Figure 2 · p. 5, §3.3 · p. 6
3.4 Impact of Narrative Complexity
- In Reverse and Anti-Causal conditions, performance tended to decline as narratives grew, while extracted-graph accuracy often stayed high across narrative sizes.Complex-graph narratives generally performed worse than simple chains, but their gap was much smaller than in the other failure modes.Source: Figure 2 · p. 5, §3.4 · p. 6
4 Experiments with Real World Causal Graphs
Real-world experiments use CauseNet causal graphs verbalized as narratives to test event ordering, parametric-knowledge conflict, and graph complexity. The results show sensitivity to ordering and atypical causal relations, while explicit graph extraction can improve reverse-order inference.
4 Experiments with Real World Causal Graphs
- The real-world experiments use GPT-4o and Llama-3.1 8B on CauseNet causal chains to examine positional bias and parametric-knowledge consistency.CauseNet relations are assembled into chains of varying lengths for the experiments.Source: §4 · p. 7, §4.1 · p. 7
4.1 Experimental Setting
- Narratives use real CauseNet graphs verbalized synthetically by an LLM or sentences selected from CauseNet, with prompts evaluating three causal-reasoning strategies.The real-world sentences are arranged so adjacent edges remain logically coherent.Source: §4.1 · p. 7
4.2 Impact of Event Ordering and Chain Length
- Forward Graph accuracy remains stable as chains lengthen, while Standard and CoT outperform their reverse variants, whose accuracy declines on longer chains.In the long-chain regime, graph extraction makes reverse-orientation inference competitive with forward inference.Source: §4.2 · pp. 7–8, Figure 4 · p. 8
4.3 Effect of Parametric Knowledge Consistency
- Roughly 5 percent of CauseNet relations violate the model’s pretraining knowledge, and accuracy on conflicting pairs is substantially lower even with CoT.Table 1 partitions average accuracy by whether cause-effect pairs conflict with parametric knowledge.Source: §4.3 · p. 8, Table 1 · p. 9
4.4 Narrative Complexity
- Complex narratives with forks and colliders underperform simple chains in both semi-synthetic and real-world settings, with a smaller gap than ordering or knowledge conflict.Accuracy also degrades with narrative length, especially when a failure mode is present; graph extraction does not significantly improve performance on complex narratives.Source: §4.4 · pp. 8–9, Figure 5 · p. 9
5 Discussion
The discussion identifies event ordering, parametric knowledge, and narrative length or complexity as sources of unreliable causal reasoning. It also notes untested forms of narrative causal reasoning and points to counterfactual reasoning as a future direction.
5 Discussion
- LLMs rely on event ordering and parametric knowledge as shortcuts, and accuracy also degrades with longer narratives and more complex causal structures.Explicitly identifying the causal graph can elicit more reliable reasoning within the settings studied.Source: §5 · p. 9
5.1 Limitations and Future Works
- The study does not test other forms of causal reasoning in narratives, such as counterfactual cases.The authors identify counterfactual reasoning as a potential direction for future work.Source: §5.1 · p. 9
- The identified failure modes could inform targeted synthetic tasks for fine-tuning, while graph-extraction findings could inform prompt engineering.Both are presented as possible directions for improving causal reasoning in language models.Source: §5.1 · pp. 9–10
A Synthetic Data Experiments
The appendix specifies synthetic data construction and prompt variants for querying causal links against a known graph.
A.1 Selected Synthetic Prompts
- The pipeline generates events, builds a ground-truth graph and narrative, extracts a predicted graph, and tests whether queried event pairs are causally related.Events can be sampled as distinct random events, then supplemented with causal or anti-causal pairs.Source: §A.1 · p. 15, §A.1.1 · p. 15, §A.1.2 · p. 15
A.1.4 Forward Topological Narrative (N)
- Forward narratives place each cause before its effect, while reverse narratives place the effect before its cause.Each sentence expresses a direct causal link, and the prompts are repeated across the graph's links.Source: §A.1.4 · p. 15, §A.1.5 · p. 15
A.1.6 Standard Prompt
- The Standard prompt asks whether A caused B from the narrative; the Chain of Thought version additionally requests step-by-step reasoning.Both prompts allow direct or indirect causation and request a Yes/No answer.Source: §A.1.6 · p. 15, §A.1.7 · p. 15, §A.1.8 · p. 16
A.1.8 In-Context Prompt
- The In-Context prompt adds worked narrative question-and-answer examples before asking whether A caused B.Its examples include both a causal chain and a link that conflicts with typical causal expectations.Source: §A.1.8 · p. 16
A.1.9 Narrative + Graph Prompt
- The Narrative + Graph prompt supplies both the narrative and an extracted causal ordering, where each listed event is treated as a cause of every later event.It then asks whether A caused B, allowing direct or indirect causation.Source: §A.1.9 · p. 16
A.2 Parametric Graph Experiment
These appendices describe how parametric and complex causal graphs are built, verbalized as narratives, and queried across synthetic, semi-synthetic, and real-world settings.
A.2 Parametric Graph Experiment
- The parametric graph setup places odd-indexed events from the parametric graph in the first half of G and even-indexed events in its second half.Figure 6 illustrates this construction.Source: §A.2 · p. 16, Figure 6 · p. 16
A.3 Complex Graph Creation
- The complex synthetic graph algorithm builds graphs containing colliders, forks, and a spanning chain to connect the graph.It samples distinct events and creates collider and fork motifs before chain-connecting remaining nodes.Source: §A.3 · pp. 16–17
B.1 Prompt templates for narrative generation
- Semi-synthetic chains are verbalized edge by edge in forward or reverse topological order, with adjacent edges checked for logical continuity.For real-world narratives, a prompt checks whether CauseNet sentences for adjacent edges express a coherent causal chain.Source: §B.1 · pp. 17–18
B.2 Eliciting Parametric Knowledge
- Parametric knowledge is elicited by asking whether one event typically causes another and whether that relationship would be atypical.A No response to the typical-effect question and a Yes response to the atypicality question mark a relationship as contradicting the model's pretraining prior.Source: §B.2 · p. 18
B.3 Semi-Synthetic and Real-World Complex Graph Algorithm
- The semi-synthetic and real-world complex-graph algorithm draws motifs and causal edges from CauseNet, then generates or retrieves a narrative for each graph edge.It uses CauseNet edges for chains and either prompts an LLM to verbalize semi-synthetic edges or finds a causal sentence in CauseNet for real-world edges.Source: §B.3 · p. 18
C.1.1 Semi-synthetic narratives
- The examples show forward and reverse narratives for chains about prices and demand, bankruptcy and anger, and pollution and extreme weather.The real-world examples concatenate source sentences, while the semi-synthetic examples are generated narratives.Source: §C.1.1 · p. 19, §C.1.2 · p. 19
C.2 Prompt templates for assessing causal reasoning
- The Direct and Chain-of-Thought prompts ask whether a cause has a direct or indirect effect on an effect, with Chain-of-Thought requesting step-by-step reasoning.A separate extraction prompt asks for a single chain graph over the supplied nodes in random order.Source: §C.2 · p. 20
C.3 Necessary Compute
The experiments used pre-trained models through cloud API calls, with roughly 50 hours of API usage overall.
C.3 Necessary Compute
- Roughly 50 hours of cloud API calls were used across the synthetic, semi-synthetic, and real-world experiments, without pretraining or significant CPU resources.The synthetic API calls took an estimated 10 hours each for ChatGPT, Anthropic, and Llama; the other experiments used roughly 10 hours each for ChatGPT and Llama.Source: §C.3 · p. 20
D.1 Forward vs Reverse Experiments Anthropic and LLama
Figure 7 compares model accuracy across chain sizes, narrative orientations, and prompting methods.
D.1 Forward vs Reverse Experiments Anthropic and LLama
- Figure 7 plots answer agreement with ground-truth G by chain size for Claude 3.5 Sonnet and Llama 3.1 8B, separating forward and reverse orientations and prompting methods.Graph uses the extracted graph, Narr-Graph uses both narrative and graph, and Standard, CoT, and In-Context use the narrative; the figure shows 95% confidence intervals.Source: Figure 7 · p. 21
D.2 Causal Vs Anti-Causal Experiments Anthropic and LLama
The experiments compare causal and anti-causal narratives and simple and complex graph structures; the LLama 3.1 8B figure reports lower accuracy in the reverse direction.
D.2 Causal Vs Anti-Causal Experiments Anthropic and LLama
- Claude 3.5 Sonnet and LLama 3.1 8B are tested on narratives that agree or disagree with parametric knowledge.Figure 8 compares causal and anti-causal narratives and shows 95% confidence intervals.Source: Figure 8 · p. 21
D.3 Complex vs Simple Graphs Anthropic and LLama
- Claude 3.5 Sonnet and LLama 3.1 8B are tested on narratives generated from complex graphs versus simple chain graphs.Figure 9 shows 95% confidence intervals.Source: Figure 9 · p. 22
E.1 Forward vs Reverse LLama
- LLama 3.1 8B accuracy is lower in the reverse direction across prompting strategies and regimes.Figure 10 covers the Semi-Synthetic and Real-World Regimes using CauseNet and shows 95% confidence intervals.Source: Figure 10 · p. 23
E.2 Parametric Experiment LLama
The LLama 3.1 8B experiments examine accuracy across prompting strategies, parametric-knowledge conflict, and graph complexity in semi-synthetic and real-world narratives.
E.2 Parametric Experiment LLama
- Table 2 reports LLama 3.1 8B average accuracy across three prompting strategies, partitioned by whether cause-effect pairs conflict with parametric knowledge.The table omits 95% confidence intervals because they are smaller than 0.3.Source: Table 2 · p. 23
E.3 Simple vs Complex LLama
- Figure 11 compares LLama 3.1 8B accuracy on complex-graph and simple-chain narratives in semi-synthetic and real-world settings.Semi-synthetic narratives appear on the left and real-world narratives on the right; 95% confidence intervals are shown.Source: Figure 11 · p. 23