Source-linked AI summary
Causal Parrots: Large Language Models May Talk Causality But Are Not Causal
Matej Zečević, Moritz Willig, Devendra Singh Dhami, Kristian Kersting
TL;DR
The paper asks whether scaling LLMs can produce genuine causal reasoning rather than textual imitation. It formalizes meta SCMs and tests whether models answer causal questions using memorized causal facts; the results favor a weak causal-parrot interpretation, while highlighting limits on generalization and inductive inference.
Problem
The paper addresses whether LLMs can perform genuine causal inference despite being trained on text rather than physical measurements and lacking explicit causal assumptions.
Method
The paper formalizes correlations of causal facts through meta SCMs and evaluates LLM answers across common-sense, known-graph, and knowledge-base embedding settings.
Results
The empirical analysis finds evidence that LLMs use causal facts when answering causal questions, favoring the causal-parrot conjecture.
Takeaways & Limitations
LLMs may provide a head start for learning and inference, but their causal answers should be understood as textual knowledge rather than general causal prowess.
Takeaways & Limitations
The conjecture’s first two proof requirements—identifying the training-data meta SCM and showing causal facts are optimal answers—appear generally infeasible.
Abstract
from arXiv · showhide
Some argue scale is all what is needed to achieve AI, covering even causal models. We make it clear that large language models (LLMs) cannot be causal and give reason onto why sometimes we might feel otherwise. To this end, we define and exemplify a new subgroup of Structural Causal Model (SCM) that we call meta SCM which encode causal facts about other SCM within their variables. We conjecture that in the cases where LLM succeed in doing causal inference, underlying was a respective meta SCM that exposed correlations between causal facts in natural language on whose data the LLM was ultimately trained. If our hypothesis holds true, then this would imply that LLMs are like parrots in that they simply recite the causal knowledge embedded in the data. Our empirical analysis provides favoring evidence that current LLMs are even weak `causal parrots.'
1 Introduction
The paper examines whether scaling foundation models advances AGI by focusing on whether LLMs can talk causality. It formalizes this question theoretically and studies current models empirically.
- Causal AI/ML is gaining prominence, but integrating causality with deep learning remains difficult.
- Debates over whether scaling models is sufficient for AGI motivate a scientific examination of their reasoning capabilities.
- The paper investigates whether current foundation models are “castles in the air.”
- It narrows the analysis to whether LLMs can talk causality, leaving other foundation models for future work.
- The study formalizes correlations of causal facts using Pearl’s language and empirically evaluates the causal prowess of current LLMs.
2 Informal Summary of the Main Idea of the Paper
The paper argues that LLMs are not causal models even when they answer causal questions correctly. It explains apparent success as recitation of correlations between causal facts encountered in textual training data.
- LLMs lack structural equations that explicitly represent causal mechanisms, unlike parameterized neural SCMs.
- The paper calls LLMs “causal parrots” because training can expose them to correlations among causal facts in textual data.
- Textual training data can contain both correct and untrue statements, helping explain why LLM causal answers are sometimes right and sometimes wrong.
- Altitude causing temperature can be learned either by induction from physical measurements or by reciting an encyclopedia’s textual representation.
- Even universal function approximation does not make an LLM causal or guarantee exposure to enough causal facts from a meta SCM.
3 Formalizing “Correlations of Causal Facts”
The paper formalizes how causal facts about one SCM can themselves be represented in another, “meta” SCM, clarifying how LLMs might answer causal queries without implementing causal mechanisms. It uses Pearl’s causal hierarchy to distinguish observational, interventional, and counterfactual knowledge and conjectures that LLM success reflects correlations of causal facts in training data.
- Illustrative example: In the chocolate–Nobel example, an alternate SCM M2 represents the unobserved GDP-related common cause as an exogenous confounder linking X and Y.This makes M2 better match both the observed data and the real-world intuition that chocolate consumption does not cause Nobel laureates.
- Pearl’s Causal Hierarchy: Pearl’s Causal Hierarchy separates observational L1, interventional L2, and counterfactual L3 queries.P(Yx = y) denotes the probability of Y under an intervention setting X to x.
- Causal facts: An SCM’s structural equations provide knowledge for answering L3 queries, while its causal graph provides knowledge for answering L2 queries.
- Meta SCMs: Meta SCMs use variables representing causal facts—knowledge about L2 and L3—instead of restricting variables to natural concepts such as chocolate consumption.They encode information about answering causal questions in another SCM.
- Correlation of Causal Facts: The CCF conjecture proposes that an LLM answers causal queries correctly when doing so minimizes training error over correlations between causal queries and their answers.The proposed explanation is that LLMs encounter causal facts in textual training data rather than acquiring structural causal mechanisms.
4 Testing for Causal Knowledge in Large Language Models
The paper evaluates whether LLMs can perform causal reasoning, recover causal graphs, and use embedded knowledge-base facts across several prompting and representation settings. Results are mixed: models sometimes answer correctly, but performance is sensitive to task structure, wording, and training-related knowledge.
- Evaluation setup: The evaluation tests three publicly accessible LLMs on common-sense reasoning, partially known causal graphs, and embeddings of knowledge-base facts.The models are GPT-3, Luminous, and OPT, and the experiments cover intuitive physics, causal chains, graph prediction, and embedding-based prediction.
- Common-sense inference: GPT-3 handles causal chains up to 5 variables but fails at n = 6, while Luminous behaves differently and OPT answers only two queries correctly.All models decline when chain order or queried subchains are randomized; only Luminous retains reasonable performance on randomized queries.
- Common-sense inference: Chain-of-thought prompting improves answer execution, but examples also impose answer formats and can bias models toward always answering “Yes” or “No”.The bias is especially pronounced for Luminous and OPT, so improved accuracy may reflect response-style adaptation rather than improved problem understanding.
- Knowledge and unknown concepts: GPT-4 gives meta answers for unknown concepts, while embedding-based predictions transfer ConceptNet facts to a causal edge correctly in 4 out of 5 sentence wordings.The ConceptNet experiment links the predicted driving-style-to-fuel-consumption edge to nearby driving and fuel-use facts.
- Causal discovery: GPT-3 generally matches ground-truth causal graph structure better than Luminous and OPT, whose denser predictions yield higher F1 scores but poorer structural alignment.GPT-3 predicts sparser graphs closer to the generally sparse ground-truth graphs, whereas Luminous and OPT predict more edges.
- Causal discovery: All models become more decisive with asymmetric query wording, but GPT-3 can change its predicted graph substantially when only the wording changes.The authors suggest that semantic differences between terms such as “causality” and “cause” may be represented unevenly in the latent space.
5 Related Work
The paper situates itself among work connecting causality and natural language processing, while distinguishing its focus on whether LLMs themselves are causal.
- The paper formalizes and empirically investigates whether LLMs themselves are causal, unlike work using foundation models as inference engines for do-calculus.
- Related research interprets interventions as semantic text manipulation and benchmarks whether NLP models respect realistic semantic interventions.
- Another benchmark studies causal discovery with off-the-shelf and fine-tuned LLMs, finding off-the-shelf models perform poorly empirically.
6 Conclusive Discussion
The paper argues that LLMs are not causal because they lack physical measurements and explicit causal assumptions, while meta SCMs explain why they can sometimes answer causal questions correctly. Empirical findings favor the causal-facts explanation, but the conjecture remains unproven and LLM performance is often inadequate.
- Conclusive Discussion: LLMs lack physical measurements, explicit causal assumptions, and the ability to infer causal structure from purely observational data.The paper contrasts textual training with induction on actual data-generating mechanisms and explicitly modeled structural equations.
- Conclusive Discussion: Meta SCMs and the correlations of causal facts conjecture explain how LLMs can answer causal questions correctly despite not being causal models.The proposed account links textual causal statements and causal questions through facts encoded across training sources.
- Conclusive Discussion: The empirical analysis searched for three components of the conjecture: identifying a meta SCM, establishing the optimality of causal-fact answers, and showing that LLMs provide those answers.The authors report evidence for the third component in cases where LLMs answer causal questions correctly.
- Conclusive Discussion: The evidence favors the causal-facts conjecture, but no definitive conclusion is possible because two components remain unproven and LLMs often underperform.The authors characterize current systems as potentially causal parrots, but weak and incomplete ones.
- Takeaway, Ethical Challenges and Societal Implications: LLMs may serve as starting points for learning and inference, but they cannot be relied upon for general causal reasoning or grounding textual facts in physical measurements.The discussion also reports bias in medical causal-graph views and notes broader ethical concerns associated with AGI.
B Technical Details and Metrics
The experiments query LLMs in natural language about causal relationships between variable pairs, while accounting for differences among query templates. Graph sparsity counts predicted half-edges relative to a fully connected graph.
- Technical Details and Metrics: The results were produced using one NVIDIA A100-SXM4-80GB GPU, while Luminous and GPT-3 were queried through their provided APIs.Querying OPT took 40 GPU minutes.
- Technical Details and Metrics: The experiments query LLMs about whether two variables in a data set have a causal relationship.Queries are expressed in natural-language templates with variable names substituted for X and Y.
- Technical Details and Metrics: Five natural-language templates ask whether X and Y are causally related, connected, causal, or whether X causes or influences Y.The templates include both symmetric and asymmetric formulations.
- Technical Details and Metrics: The templates differ in symmetry and in whether they explicitly ask about causality, so the analysis treats these distinctions carefully.“Does X cause Y?” is asymmetric, while “Does X influence Y?” does not explicitly use causal language.
- Technical Details and Metrics: Sparsity measures present edges relative to the maximum number of edges in a fully connected graph, counting A →B and B →A separately.The evaluation therefore treats directed half-edges individually.
B.3 Algorithm for ADS
The Decisiveness algorithm measures how often predicted graph edges are directed rather than symmetric. It returns the directed-edge proportion, with zero returned when no edges are predicted.
- Algorithm for ADS: Decisiveness measures an LLM’s tendency to choose directed edges when switching between asymmetric and symmetric causal queries.The metric was introduced because asymmetric queries produced more unidirected edges in the authors’ observations.
- Algorithm for ADS: The algorithm counts directed predictions as asym and symmetric predictions as sym.Each predicted edge increments exactly one of these counters according to its type.
- Algorithm for ADS: The algorithm sets count to asym + sym and returns 0.0 when count equals zero.This handles graph predictions containing no classified edges.
- Algorithm for ADS: Otherwise, Decisiveness returns asym/count, the fraction of classified predicted edges that are directed.The procedure terminates after dividing the directed count by the total classified count.
C Experimental Results
This section contains additional material for the paper’s three main experimental sections that was omitted from the main paper because of space restrictions.
- Experimental Results: The appendix collects supplementary material for all three main experimental sections that could not fit in the main paper.It is presented as additional material for readers interested in the experiments.
C.1 How to Interpret LLM Output
Luminous and OPT sometimes failed to produce sensible answers, while GPT-3 consistently produced sensible output.
- Luminous and OPT sometimes produced ambiguous or nonsensical responses.Failures included indefinite repetition and multiple-choice-style outputs in which the model selected an answer itself.
C.2 Intuitive Physics
The intuitive-physics experiments elicited varied responses across models, with GPT-4 and GPT-3 answering the showcased questions sensibly while Luminous and OPT sometimes produced problematic outputs.
- For the tilted-board question, GPT-4 and GPT-3 answered that the ball would end up in the bucket.
- Luminous answered that the ball would roll down the board, while OPT produced repeated answer choices about falling into the bucket.
- For the kilogram question, GPT-4 correctly stated that metal and feathers weigh the same.
C.3 Propositional Logic
The propositional-logic experiments tested whether models could infer causal chains and preserve the direction of stated causal relations, with inconsistent answers across models.
- For “If A causes B and B causes C,” GPT-4 affirmed that A causes C indirectly through B.
- GPT-3 gave an uncertain answer to whether A causes C, while Luminous answered no and OPT generated contradictory alternatives.
- When asked whether A causes B, GPT-4 affirmed the premise, whereas GPT-3 said there was insufficient information and Luminous answered no.
C.4 Changing the Variable Names for Prediction
The paper varied variable names and Chain-of-Thought demonstrations to examine how wording affects causal graph predictions and question answering. Results included model-specific sensitivity to renamed concepts and differences in benefit from example counts.
- Changing the Variable Names for Prediction: Changing variable names altered graph predictions in the Health dataset, with Luminous becoming sparser when “mobility” was renamed “fitness.”The experiment compared synonym-like and contextually related alternatives across three LLMs and multiple causal-question templates.
- Changing the Variable Names for Prediction: The variable-renaming experiment compared three LLMs across alternative names for age, nutrition, health, and mobility.
- Chain of Thoughts Prompting: Chain-of-Thought prompting placed exemplary question-answer pairs before the actual input, using distinct variables matched to each task’s style.
- Chain of Thoughts Prompting: Figure 7 reports the absolute number of correctly answered queries for the CoT experiments.
- Chain of Thoughts Prompting: Luminous and OPT appeared to benefit more than GPT models from fewer provided examples.
C.6 Embedding predictions
The section predicts graph edges by comparing GPT-3 embeddings of instantiated query sentences with embeddings of labeled ConceptNet causal facts. Predictions are made across datasets and five query-sentence templates.
- GPT-3 embeddings of ConceptNet causal and anti-causal facts provide labeled embeddings for graph-edge prediction.The labeled facts are used as the reference set for subsequent nearest-neighbor comparisons.
- Each dataset edge query is instantiated using five sentence templates before its text is embedded.The resulting query embeddings are compared against ConceptNet fact embeddings.
- A k-NN search compares each query embedding with all ConceptNet fact embeddings to predict whether the queried edge is present.The edge decision is based on the label associated with the most similar embedding.
- Figure 8 reports graph predictions for each dataset and each query-sentence template.