Causal Reasoning and Large Language Models: Opening a New Frontier for Causality

Emre Kıcıman; Robert Ness; Amit Sharma; Chenhao Tan

Source-linked AI summary

Causal Reasoning and Large Language Models: Opening a New Frontier for Causality

Emre Kıcıman, Robert Ness, Amit Sharma, Chenhao Tan

arXiv:2305.00050v3Apr 28, 2023cs.AIcs.CLcs.CYcs.HCcs.LGstat.ME

TL;DR

The paper asks whether LLMs can generate valid causal arguments and what causal workflows this ability can support, a question with implications for medicine, science, law, and policy. It probes LLM behavior across causal discovery and token-causality tasks using benchmark, memorization, and novel-dataset tests, finding strong performance that can complement existing causal methods. However, unpredictable failures and unresolved limits mean LLMs should not be trusted alone in critical causal applications.

Problem
The paper examines whether LLMs genuinely support causal reasoning or merely reproduce unreliable or memorized answers in societally consequential domains.
Method
The paper probes LLMs with causal prompts and evaluates benchmarks, memorization tests, novel post-cutoff datasets, and redaction or perturbation tests.
Results
LLMs show strong performance across causal discovery, counterfactual reasoning, and event-causality tasks, including 97% Tübingen accuracy and 92% counterfactual accuracy.
Takeaways & Limitations
LLMs can complement existing causal methods by contributing domain knowledge and helping reduce human effort in causal graph construction, effect inference, and attribution.
Takeaways & Limitations
Unpredictable failures, prompt sensitivity, and unresolved causal mechanisms make it imprudent to trust LLMs alone in critical causal applications.

Abstract

from arXiv · show

The causal capabilities of large language models (LLMs) are a matter of significant debate, with critical implications for the use of LLMs in societally impactful domains such as medicine, science, law, and policy. We conduct a "behavorial" study of LLMs to benchmark their capability in generating causal arguments. Across a wide range of tasks, we find that LLMs can generate text corresponding to correct causal arguments with high probability, surpassing the best-performing existing methods. Algorithms based on GPT-3.5 and 4 outperform existing algorithms on a pairwise causal discovery task (97%, 13 points gain), counterfactual reasoning task (92%, 20 points gain) and event causality (86% accuracy in determining necessary and sufficient causes in vignettes). We perform robustness checks across tasks and show that the capabilities cannot be explained by dataset memorization alone, especially since LLMs generalize to novel datasets that were created after the training cutoff date. That said, LLMs exhibit unpredictable failure modes, and we discuss the kinds of errors that may be improved and what are the fundamental limits of LLM-based answers. Overall, by operating on the text metadata, LLMs bring capabilities so far understood to be restricted to humans, such as using collected knowledge to generate causal graphs or identifying background causal context from natural language. As a result, LLMs may be used by human domain experts to save effort in setting up a causal analysis, one of the biggest impediments to the widespread adoption of causal methods. Given that LLMs ignore the actual data, our results also point to a fruitful research direction of developing algorithms that combine LLMs with existing causal techniques. Code and datasets are available at https://github.com/py-why/pywhy-llm.

1 Introduction

The paper examines whether LLMs can generate valid causal arguments across causal discovery, counterfactual, and event-causality tasks, amid debate over genuine reasoning versus unreliable memorization. It finds strong task performance and potential assistance for human-led causal workflows, while emphasizing unpredictable errors and limits on trusting LLMs alone.

Implications: LLMs may automate or assist steps of causal reasoning by transitioning between covariance-based and logic-based causal inference.The proposed workflow includes formulating questions, iterating, and verifying premises and implications.
Evaluation and limitations: The study probes whether strong causal-task performance reflects capability rather than benchmark memorization.The paper discusses memorization tests and evaluates novel datasets created after the models’ training cutoff date.
Causal discovery: 97% accuracy on the Tübingen pairwise causal discovery benchmark exceeded the previous best of 83%.LLM-based methods used variable names as input to build causal DAGs from statistical evidence.
Token causality: 92% accuracy on counterfactual queries was 20 points higher than previously reported accuracy.The benchmark covered basic physics, logic, and common sense.
Token causality: Over 86% accuracy was achieved in identifying necessary and sufficient causes across established and novel 15-vignette benchmarks.The novel benchmark was designed to avoid memorization concerns.
Evaluation and limitations: LLMs achieve high average accuracies but still make simple, unpredictable mistakes, and their robustness depends substantially on prompting.The paper states that more research is needed before outputs can be trusted in critical causal applications.

2 Background and Preliminaries

The paper organizes causal reasoning along methodological and problem-type distinctions, contrasting covariance- and logic-based approaches with type and token causality. It evaluates LLM behavior through prompt-response probing, benchmark tests, memorization checks, and redaction or perturbation tests.

Causal approaches: Covariance-based causality uses statistical data analysis, whereas logic-based causality uses logical reasoning and domain knowledge.Covariance-based applications include drug-efficacy evaluation and policy effects; logic-based applications include legal liability and fault diagnosis.
Causal targets: Type causality concerns relationships between variables and their average effects, while token causality concerns specific events and their causes.Token-causality answers need not generalize across situations.
Causal discovery: Causal discovery searches for graphs consistent with observed covariance, but observational data alone generally cannot identify the correct graph.Multiple graph structures can belong to the same Markov equivalence class.
Evaluation: The study probes LLMs through textual prompts and responses while addressing construct-validity concerns about question structure and memorization.It combines standard benchmarks with memorization tests, novel datasets, and redaction or perturbation tests.

3 Generating causal graphs using LLMs

LLMs construct causal graphs from variable metadata and natural-language context rather than observed data, often matching or exceeding covariance-based methods. Their performance generalizes beyond popular benchmarks, but depends on prompting and remains vulnerable to inconsistent or ambiguous reasoning.

Motivation: Causal discovery cannot generally recover the correct graph from observational data alone, motivating metadata-based reasoning with variable names and domain context.LLMs use information typically supplied by human domain experts when constructing causal graphs.
Robustness and generalization: 98.5% accuracy on the novel dataset showed that GPT-4’s causal-direction capability generalized to variable pairs outside popular datasets.The dataset was created from cause-effect pairs published after the models’ September 2021 training cutoff, using the original single prompt.
Neuropathic pain dataset: Prompt choice substantially affected neuropathic-pain edge-direction accuracy, with GPT-4 reaching 96% and single-pair prompts pushing other models above 85%.With the two-sided prompt, gpt-3.5-turbo achieved 75%, while single prompts increased accuracy for text-davinci-003 and gpt-3.5-turbo to more than 85%.
Failure modes: LLMs sometimes produce incoherent or self-inconsistent causal explanations, while ambiguity in variable names can also cause benchmark mismatches.GPT-4 can identify some self-consistency errors, and additional context can resolve some ambiguous cases.
Full graph generation: GPT-4 achieved an F1 score of 0.73 on full neuropathic-pain graph generation, comparable to the best-reported F1 of 0.68.Full graph generation requires distinguishing causal directions from the option of no edge.

4 LLMs for Token Causality and Causal Judgments

LLMs generate causal arguments about specific events by using natural-language context to answer counterfactual, necessity, sufficiency, and normality questions. They perform strongly across these tasks, but their reasoning remains vulnerable to ambiguity and unpredictable failures.

Motivation: Formal token-causality models struggle to represent background context such as necessity, sufficiency, normality, and responsibility.Token causality requires human-like interpretation of contextual elements that are difficult to formalize in structural causal models.
Counterfactual reasoning: 92.44% accuracy placed GPT-4 within six percentage points of average human performance on the CRASS counterfactual reasoning task.GPT-3.5-turbo reached 87.95 accuracy, while GPT-4 exceeded the previous best accuracy by 20 percentage points.
Counterfactual reasoning: 88.6% accuracy on a post-cutoff synthetic dataset exceeded GPT-3’s 71.3% benchmark, indicating generalization to new counterfactual scenarios.The dataset was uploaded to GitHub in 2022, after the models’ September 2021 training cutoff.
Implications: LLMs can select relevant causal events and adapt causal abstractions to a scenario, unlike structural causal models that require variables to be specified in advance.This flexible causal framing supports natural-language analysis but does not eliminate unpredictable failures.
Necessary and sufficient causes: GPT-4 was accurate for most necessity and sufficiency vignette types without receiving explicit definitions, whereas GPT-3.5-turbo performed near random guess.The models were evaluated on 15 scenarios; GPT-4 nonetheless showed unpredictable failures, including mishandling sufficiency in a short-circuit vignette.

5 A New Frontier for Causality

The paper presents LLMs as potential assistants for causal analysis, combining domain knowledge with flexible natural-language interaction across causal tasks. It also identifies research, reliability, and ethical boundaries that constrain this potential.

LLMs can provide domain knowledge that was previously available mainly through human experts, both explicitly and through background context.
LLMs as a fluid conversational interface: Natural-language interaction lets LLMs work alongside existing causal tools and potentially bridge covariance-based and logic-based reasoning.
Systematizing token causality and attribution: LLMs may help systematize token-causality concepts such as necessity, sufficiency, and normality in practical scenarios.
Augmenting human expertise with LLMs: LLMs may reduce human burden by assisting with causal graph construction, formal assumptions, and robustness checks while preserving human verification.
Applying LLMs to novel scenarios: On datasets created after training cutoffs, similarly high accuracies suggest that dataset-specific memorization is not required for strong causal-task performance.
LLMs are not expected to construct causal assumptions or graphs for concepts beyond their training corpus, while manipulated causal explanations could undermine fairness and trust.
The work is a proof of concept whose effective human-AI interaction requires further advances, and whose causal outputs raise unresolved ethical and research questions.

6 Conclusion

The conclusion characterizes LLMs as imperfect but useful sources of domain knowledge for causal analysis. Their demonstrated capabilities may reduce expert burden while opening substantial questions about reliability and causal reasoning.

LLMs can outperform state-of-the-art causal algorithms in graph discovery and counterfactual inference while operating solely on natural-language input.
LLMs can systematize concepts such as necessity and sufficiency of causes, but their text-based domain-knowledge mimicry is neither explainable nor predictable.
The paper proposes that LLMs could reduce human-expertise burdens in causal graph construction, effect inference, and attribution.
All experiments use gpt-3.5-turbo-0613 and gpt-4-0613 model versions.

A.1 Example prompts for the Tubingen benchmark

Table 14 presents two prompt-template designs for orienting pairwise causal edges in the Tübingen benchmark.

The first template asks two questions for each variable pair, while the Single prompt template asks one question to orient each pairwise edge.

A.2 Example LLM outputs for the Neuropathic pain dataset

The examples illustrate both correct and incorrect LLM outputs for neuropathic-pain causal questions. The responses interpret medical terms, compare directional hypotheses, and select an answer.

Correct outputs explain the meanings of spinal conditions and radiculopathy before evaluating the proposed causal directions.
A second correct example compares both directions and selects DLS T5-T6 causes Left T6 Radiculopathy as more likely.
The appendix also includes an example labeled as an incorrect LLM output.
One correct example selects DLS T5-T6 as the cause of Right T6 Radiculopathy based on nerve-root compression or irritation.

A.3 Novel datasets created after LLMs training cutoff date

The evaluation uses novel datasets, including Tubingen pairs, Alzheimer’s graphs, and counterfactual scenarios, to assess LLM performance on data created after training. These datasets are documented in Tables 15–17.

Novel Tubingen pairs were constructed from recently published books.Table 15 presents a sample of 10 pairs from this dataset.
The novel Alzheimer’s dataset evaluates graph discovery accuracy for GPT-3.5-turbo and GPT-4 against data-based algorithms.Compared algorithms include DAG-GNN and NOTEARS.
The novel counterfactual dataset evaluates GPT-4 against models previously assessed by Li et al. (2023).Table 17 reports the corresponding accuracy results.

B.1 Details on CRASS counterfactual evaluation

The CRASS counterfactual evaluation scores answer selection and uses prompted reasoning over scenario changes and possible outcomes. Examples show both correct answers and failures to resolve ambiguity or preserve contextual details.

Evaluation scoring: 1 indicates a correct answer option, 0 an incorrect option, and 0.33 a skipped question when no valid option is output.The 0.33 score represents the probability of a correct answer under random chance.
Prompt format: The prompt asks the LLM to choose among answer options and return the selection within answer tags.The examples include counterfactual questions about touching fire and catching a water balloon.
Example responses: GPT-4 identified uncertainty between getting wet and remaining dry when the water balloon’s bursting could not be determined.Its response was “Uncertain between B and D.”
Example responses: GPT-4 selected that the man would not pass out, reasoning that other nervousness symptoms were more likely than passing out.The final answer was C.
Failure mode: GPT-4 incorrectly assumed that walking on a bed continued outdoors toward the same destination, despite the intended indoor context.The example illustrates a contextual failure in counterfactual reasoning.

C Evaluation: Necessary and Sufficient Causes

The evaluation examines necessary and sufficient token causes across several established scenario classes, using LLM-generated principles and prompts. It also introduces chemically themed vignettes to test performance beyond memorized examples.

Scenario classes: The evaluation covers seven scenario classes, including symmetric overdetermination, switches, and early and late preemption.The listed classes also include double preemption, bogus preemption, and short circuits.
Prompt development: Complicated definitions of token causality motivate using the LLM to suggest natural-language prompts.The paper notes that expressing actual causality in natural language is non-trivial.
Prompt development: A meta-prompt with example vignettes led the LLM to propose minimal rewriting, which the authors rephrased as the minimal change principle.The principle is used in the necessary-cause prompt.
Sufficient-cause prompting: The examples illustrate minimal-change reasoning for sufficient causes, including jointly firing actors, the first three shooters, and the final droplet causing overflow.The paper also describes adapting this procedure to prompts for sufficient causes.
Reasoning principle: The minimal rewriting principle evaluates which changes to an initial scenario are necessary to obtain candidate outcomes.The selected output is the one requiring the least change.
Novel evaluation data: The paper constructs a new chemistry-lab dataset by perturbing an available dataset’s details and tests whether the LLM can recreate it.The vignettes involve reagents, mixtures, and crystals, with paired necessary- and sufficient-cause questions.

C.4.1 Correct response on an Early Preemption vignette

In an early-preemption vignette, both models correctly judged that Alice was not necessary for the window to remain intact. GPT-4 additionally judged Alice’s action sufficient, whereas GPT-3.5 did not.

Necessary cause: Both models produced the correct answer for the necessary-cause question.
Sufficient cause: GPT-4 alone correctly identified Alice’s action as sufficient for the window remaining intact.The paper contrasts this with GPT-3.5’s incorrect sufficiency judgment.
Sufficient cause: The sufficiency reasoning invokes multiple sufficient causes, including Alice catching the ball and the solid brick wall stopping it.Alice’s action is therefore one sufficient cause rather than the sole determinant of the outcome.

C.4.2 Unpredictable failure mode: An example

GPT-4 usually applies the correct counterfactual reasoning but can unpredictably assess necessity without sufficiency. In the antidote vignette, Carol survives without Alice’s intervention, so Alice is not a sufficient cause for survival.

GPT-4 exhibits an unpredictable failure by assessing necessity without sufficiency in the short-circuit vignette.The model applies correct reasoning for most prompts but fails on this specific prompt.
Alice is not a sufficient cause for Carol’s survival because Carol would survive without the antidote intervention.The counterfactual scenario removes Alice’s action while preserving Carol’s survival.
Responsibility measures how much an event contributes to an outcome relative to other causal events, without requiring the event to be a person’s action.The paper contrasts this graded notion with binary necessary-or-sufficient causation.

D.0.1 Inferring responsibility: Motivating Example

The motivating examples distinguish graded responsibility from binary causal judgments and test whether LLMs can update responsibility when alternative causal pathways arise. The examples reveal both appropriate downgrading under overdetermination and a failure on double prevention.

Responsibility definition: The paper defines responsibility as a relative ranking of how events contribute to the same outcome, rather than an absolute value.The LLM is prompted to rank causes and to apply the definition equally to actions and inanimate events.
Overdetermination: When Susan’s alternative fall would also spill the beer, the LLM reduces the responsibility of Mike’s table bump.Susan’s fall would have produced the same outcome, making both events contributing causal factors.
Double prevention: In the double-prevention scenario, the LLM judges Peter more responsible than Mike because Peter interfered with Jack’s attempt to catch the bottle.The answer prioritizes Peter’s immediate impact over Mike’s role in initiating the sequence.
Limitation: The authors identify this judgment as a failure to capture double-prevention nuance by human standards.The failure concerns ChatGPT-4’s interpretation of responsibility in the extended beer-spilling scene.

E.1 Memorization Tests

The memorization tests compare whether LLMs reproduce content from row-based and unstructured datasets. They find clear memorization for Tübingen, possible partial memorization for neuropathic pain, and no apparent memorization for Arctic Sea Ice or CRASS.

Test design: The tests use row completion for tabular datasets and random 500-character continuation for JSON-formatted datasets.The prompts explicitly ask the model to complete sampled data as a memory test.
Results: Tübingen appears in GPT-3.5 and GPT-4 training data and is at least partially memorized by both models.The memorization result is reported for the row-based dataset.
Results: The neuropathic pain dataset seems partially memorized by GPT-3.5 Turbo, while GPT-4 results are unclear.The evidence does not resolve GPT-4’s memorization status for this dataset.
Results: Arctic Sea Ice and CRASS show no signs of memorization beyond tokens predictable from language or formatting.Only a small number of recovered tokens were observed for these datasets.

F.1 Example of LLM-assisted Causal Analysis Code Generation

The case studies illustrate LLM assistance across causal-analysis setup, code generation, perturbation testing, and control selection. GPT-4 generates a workflow spanning synthetic data, DAG learning, effect estimation, sensitivity analysis, and causal controls, while perturbation results show that an earlier answer strongly influences a later one.

Causal-analysis code generation: GPT-4 generates a causal-analysis workflow using synthetic data, causal-learn for DAG learning, DoWhy for modeling, and EconML for treatment-effect estimation.The example also includes sensitivity analysis to validate results.
Causal-analysis code generation: The generated workflow estimates the treatment effect of X1 on Y in a graph with X1 → X2 → Y and X1 → Y.The example uses three variables and acknowledges that noise and method assumptions can change estimated effects.
Perturbation experiment: Perturbing GPT-3.5’s first answer shows that the initial answer strongly influences the final answer, even when intermediate arithmetic is correct.The experiment intervenes on the first response to examine its effect on the later completion.
Causal controls: The LLM proposes negative controls such as hair color, blood type, handedness, taste preferences, birth month, and unrelated medical procedures.These controls are intended to help assess matching quality and unmeasured confounding.