PaperlayerView exact revision on arXiv

Source-linked AI summary

Causal Reasoning and Large Language Models: Opening a New Frontier for Causality

Emre Kıcıman, Robert Ness, Amit Sharma, Chenhao Tan

arXiv:2305.00050v3cs.AIcs.CLcs.CYcs.HCcs.LGstat.ME

TL;DR

Whether LLMs can perform causal reasoning rather than mimic memorized answers remains debated, especially for high-stakes domains. This study benchmarks LLMs across causal tasks and finds that they often outperform existing causal algorithms, while offering potentially useful support for human causal analysis.

  • Problem

    It remains unclear whether LLMs perform causal reasoning or merely reproduce unreliable memorized responses, an important question for their use in critical domains.

    Source: §1 · p. 2

  • Method

    The paper probes LLMs across causal tasks using textual metadata and tests robustness, including generalization to cause-effect pairs published after model training cutoffs.

    Source: §1 · p. 2, §2.4 · p. 7, §3.1.1 · p. 12

  • Results

    Across tasks, LLM-based methods outperform existing causal algorithms; for Tübingen causal-direction discovery, they achieve 97% accuracy versus 83% previously.

    Source: §1 · pp. 3–4, §6 · p. 33

  • Takeaways & Limitations

    LLMs may reduce the burden of human expertise in causal analysis and help bridge covariance-based and logic-based causal reasoning.

    Source: §1 · p. 2, §6 · p. 34

    The limits and mechanisms of LLM causal capabilities remain poorly understood, and their causal inductive biases are unresolved.

    Source: §5.3 · p. 33

Abstract

from arXiv · show

The causal capabilities of large language models (LLMs) are a matter of significant debate, with critical implications for the use of LLMs in societally impactful domains such as medicine, science, law, and policy. We conduct a "behavorial" study of LLMs to benchmark their capability in generating causal arguments. Across a wide range of tasks, we find that LLMs can generate text corresponding to correct causal arguments with high probability, surpassing the best-performing existing methods. Algorithms based on GPT-3.5 and 4 outperform existing algorithms on a pairwise causal discovery task (97%, 13 points gain), counterfactual reasoning task (92%, 20 points gain) and event causality (86% accuracy in determining necessary and sufficient causes in vignettes). We perform robustness checks across tasks and show that the capabilities cannot be explained by dataset memorization alone, especially since LLMs generalize to novel datasets that were created after the training cutoff date. That said, LLMs exhibit unpredictable failure modes, and we discuss the kinds of errors that may be improved and what are the fundamental limits of LLM-based answers. Overall, by operating on the text metadata, LLMs bring capabilities so far understood to be restricted to humans, such as using collected knowledge to generate causal graphs or identifying background causal context from natural language. As a result, LLMs may be used by human domain experts to save effort in setting up a causal analysis, one of the biggest impediments to the widespread adoption of causal methods. Given that LLMs ignore the actual data, our results also point to a fruitful research direction of developing algorithms that combine LLMs with existing causal techniques. Code and datasets are available at https://github.com/py-why/pywhy-llm.

1 Introduction

The paper evaluates whether LLMs can generate valid causal arguments across causal discovery and token-causality tasks, finding strong average performance alongside unpredictable failures. It argues that LLMs can augment causal analysis by supplying domain knowledge and natural-language reasoning, but should not yet be trusted alone in critical applications.

Motivation

  • The study addresses whether LLMs generate valid causal arguments, how reliable those arguments are, and which causal-reasoning workflows they can support or automate.The debate is motivated by LLMs’ plausible causal explanations and counterfactuals as well as their observed basic logical and causal errors.Source: §1 · p. 2

Contribution

  • LLMs’ capabilities are complementary to existing causal methods because they capture human domain knowledge and may assist or automate steps throughout causal-reasoning workflows.The authors emphasize that, until LLM capacities and mechanisms are better characterized, relying on them alone for critical decisions is not prudent.Source: §1 · pp. 2–4

Causal DAG generation

  • 97% accuracy on the Tübingen pairwise causal benchmark, versus 83% for the previous best algorithm, while LLMs achieve similar accuracy to recent deep-learning methods on full-graph discovery.LLM errors differ from those of data-based discovery algorithms, suggesting that combining both approaches could improve accuracy; performance remains high on datasets released after the training cutoff.Source: §1 · p. 3

Token causality

  • 92% accuracy on counterfactual queries, 20 points above previously reported performance, and over 86% accuracy identifying necessary and sufficient causes across established and novel vignette benchmarks.Performance falls to 70% on normality judgments, where deciding an event’s cause depends heavily on human factors and background context.Source: §1 · p. 4

Implications for causality research

  • LLMs may augment causal analysis by providing general and domain-specific causal knowledge, formalizing scenarios, and helping generate, critique, validate, or refute causal premises under human supervision.These capabilities could support bootstrapping causal DAGs and integrating logical and covariance-based approaches that traditionally rely on human experts.Source: §1 · p. 4

Limitations

  • High average accuracy does not eliminate simple, unpredictable mistakes: results depend substantially on prompting, and further work is needed to establish when LLM outputs can be trusted.The paper probes prompt robustness and the possible contribution of dataset memorization, while noting that these issues remain unresolved.Source: §1 · p. 4

2 Background and Preliminaries

The paper frames causality across complementary methodological, causal-setting, and task dimensions, then positions LLMs as text-based tools whose causal behavior requires systematic probing beyond benchmark accuracy.

2 Background and Preliminaries

  • Together, these preliminaries motivate unified pipelines that connect language-based and data-based causal analyses while recognizing that causal tasks and LLM capabilities require separate evaluation.LLMs’ text input-output interface makes behavioral probing the primary way to study their causal capabilities.Source: §2 · p. 5, §2.2 · p. 6, §2.4 · p. 7

2.1 Many Kinds of Causality

  • Causality spans covariance- and logic-based methods, type- and token-level questions, and diverse domains with distinct conceptualizations and tools.Covariance-based approaches infer causal relationship strengths from data, whereas logic-based approaches use reasoning and domain knowledge; type causality concerns variables and populations, while token causality concerns specific events.Source: §2.1 · p. 5

2.2 Different Causal Tasks and their Connection to Kinds of Causality

  • Causal tasks include graph specification, effect inference, attribution, and judgment, and solving one task does not imply solving the others.These tasks can combine covariance-based and logical approaches across type and token settings, with attribution determining causes and judgment additionally considering morality, normality, and intent.Source: §2.2 · p. 6

2.3 LLMs and Causality

  • LLMs are transformer-based next-word predictors shaped by human feedback, and this paper extends prior work by testing graph generation, counterfactual reasoning, and knowledge transfer across causal methods.With an appropriate prompt, graph-edge retrieval F1 increased from 0.21 to 0.68 in the revisited case study.Source: §2.3 · pp. 6–7

2.4 Probing LLM behaviors

  • Because correct answers may reflect question structure or memorization rather than causal capability, the study combines benchmark evaluation with memorization, novel-dataset, redaction, and perturbation tests.Memorization is assessed by asking the LLM to regenerate dataset rows or file snippets, while redaction tests examine whether changing prompt words changes the answer.Source: §2.4 · p. 7

3 Generating causal graphs using LLMs

This section examines how LLMs generate causal graphs from variable metadata and natural-language context rather than observed data, across medical, atmospheric, and post-training-cutoff datasets. It also considers challenges in full graph generation and how memorization and wording sensitivity affect interpretation.

3 Generating causal graphs using LLMs

  • Knowledge-based causal graph generation uses variable names and natural-language context to supply domain knowledge that covariance-based algorithms cannot identify from observational data alone.This approach formalizes metadata-based reasoning previously performed by human domain experts and is evaluated across ordinary and specialized domains.Source: §3 · p. 8

3.1.1 Tübingen cause-effect pairs dataset

  • 97% weighted accuracy makes GPT-4 substantially outperform the best-known covariance-based causal discovery method on Tübingen cause-effect pairs.GPT-3.5-turbo reaches 86.9% with a causal-reasoning system message, while a step-by-step prompt reaches 92.1%.Source: §3.1.1 · pp. 9–10, Table 2 · p. 10
  • 98.5% accuracy on 67 post-September-2021 cause-effect pairs shows GPT-4 generalizes beyond potentially memorized popular datasets.Using the exact same single prompt, GPT-3.5-turbo obtains 80.3% accuracy on the novel dataset.Source: §3.1.1 · p. 12

3.1.2 Neuropathic pain dataset

  • 96% accuracy on 475 neuropathic-pain edges shows GPT-4 can infer causal directions requiring specialized medical knowledge.With a single prompt, text-davinci-003 and gpt-3.5-turbo both exceed 85%, whereas the two-sided prompt gives gpt-3.5-turbo 75%.Source: §3.1.2 · pp. 12–13, Table 4 · p. 13
  • LLM outputs can be unpredictably incoherent or self-inconsistent, but GPT-4 can critique earlier answers and identify logical inconsistencies.The paper also reports cases where LLM disagreement with ground truth may expose questionable causal directions, such as obesity causing radiculopathy.Source: §3.1.1 · pp. 10, 12, §3.1.2 · pp. 14–15, Table 5 · p. 14

3.1.3 Other properties of pair-wise relationships

  • Prompts can be extended beyond edge direction to ask about confounding, mediation, moderation, monotonicity, heterogeneity, time lags, stability, and spillovers.These relationship properties are presented as straightforward extensions of pairwise edge-existence prompts.Source: §3.1.3 · p. 15

3.2 Full graph generation

  • Full graph generation is harder than pairwise direction inference because each variable pair may have an edge in either direction or no edge, and direct causes must be separated from indirect causes.Whether an edge such as A →C is valid also depends on which intermediate variables are included in the input set.Source: §3.2 · p. 15

3.2.1 Neuropathic pain dataset

  • Adding a “C: No causal relationship exists” option to the prompt raised gpt-3.5-turbo’s F1 to 0.68, over three times 0.21 with a simple prompt and double the 0.33 random baseline.Using the same prompt, GPT-4 obtained F1=0.73, comparable to the Neuropathic dataset’s F1=0.68.Source: §3.2.1 · p. 16

3.2.2 Arctic sea ice dataset

  • 0.22 NHD: gpt-4 produced the lowest distance on the Arctic sea-ice graph, recovering 46 edges with precision 0.63 and F1=0.57 while achieving the lowest NHD ratio.Its NHD was one-third lower than other algorithms, and it correctly recovered 29 edges; gpt-3.5-turbo achieved NHD 0.33 and NHD ratio 0.43.Source: §3.2.2 · p. 17, Table 7 · p. 17
  • 0.14 NHD: gpt-4 outperformed NOTEARS (0.22) and DAG-GNN (0.37) on an expert-validated Alzheimer’s graph created after the training cutoff, with the lowest NHD ratio (0.28).The result supports generalization to novel datasets, while the authors note that novel datasets may still contain pre-cutoff concepts.Source: §3.2.2 · p. 17

3.3 Probing LLM behavior further

  • 61% of remaining cells and 25% of rows without error were recalled by GPT-4 in the Tübingen memorization test, while GPT-3.5 recalled 58% of cells and 19% of rows.The Tübingen dataset was likely in training, but the gap between memorization and accuracy leaves the contribution of memorization unresolved.Source: §3.3 · p. 18
  • The neuropathic dataset was at least partially memorized by GPT-3.5 Turbo but not GPT-4, whereas the Arctic sea-ice dataset was not memorized by either model.Redaction tests found that causal instructions, answer-format tags, and some minor words affected accuracy, suggesting both concept-sensitive processing and grammatical sensitivity.Source: §3.3 · pp. 18–19

4 LLMs for Token Causality and Causal Judgments

This section examines how LLMs generate causal judgments from natural-language scenarios, focusing on counterfactuals, token causality, necessity, sufficiency, normality, and responsibility. It also considers their generalization, limitations, and the potential value of combining them with methods that better guide causal argument construction.

4 LLMs for Token Causality and Causal Judgments

  • Token causality addresses attribution and responsibility, but formal causal models struggle to represent causal frames, normality, intentions, action bias, and other human judgment factors.LLMs are proposed as a way to recover such background context directly from natural-language event descriptions.Source: §4 · p. 19

4.1 Building block of token causality: Counterfactual reasoning

  • 92.44% accuracy makes GPT-4 20 percentage points better than the previous best on CRASS, within six points of average human accuracy.GPT-3.5-turbo reaches 87.95%, indicating a substantial improvement over GPT-3-era models.Source: Table 9 · p. 21, §4.1 · p. 21
  • 88.6% accuracy on a post-cutoff counterfactual dataset exceeds GPT-3’s 71.3%, indicating that GPT-4 generalizes counterfactual arguments to new scenarios.The dataset was uploaded in 2022, after the models’ September 2021 training cutoff.Source: §4.1 · p. 22

4.2.1 Motivating Example: Necessary and sufficient causes

  • In a beer-spilling example, the LLM identifies the open bottle as a relevant causal event and explains robust sufficiency without being given an eligibility list.This adaptive causal framing contrasts with structural causal models, which require variables to be specified before inference.Source: §4.2.1 · p. 23

4.2.2 Evaluating LLMs on inferring necessary and sufficient causes

  • GPT-4 is accurate across most standard necessity-and-sufficiency vignette types, whereas GPT-3.5-turbo performs near random; both models nevertheless show unpredictable failures.GPT-4 can apply the principles without explicit definitions, but it fails on some cases such as short-circuit sufficiency by relying only on counterfactual dependence.Source: §4.2.2 · pp. 24–25, Table 11 · p. 25
  • 92% necessity versus 78% sufficiency accuracy on unseen lab vignettes shows GPT-4 generalizes token-causality judgments, with sufficiency remaining more difficult.The lab-vignettes dataset adapts standard scenarios to laboratory settings; necessity requires flipping one cause, whereas sufficiency requires identifying and altering relevant other variables.Source: §4.2.2 · p. 25
  • LLMs can infer necessary and sufficient causes from natural-language scenarios, but their outputs lack robustness because failures remain unpredictable.On a novel lab-vignettes dataset, GPT-4’s average accuracy stayed comparable to standard vignettes; necessary-cause inference reached 93% versus 78% for sufficient-cause inference.Source: §4.2.2 · p. 25, Table 12 · p. 26

4.2.3 Assessing responsibility

  • LLM-generated causal arguments support responsibility judgments by incorporating intention, moral obligation, and epistemic state that formal structural causal models struggle to capture.The study of these capabilities is presented in Supplementary D.Source: §4.2.3 · p. 26

4.3 Evaluating LLMs ability to infer normality

  • 69.2% accuracy for GPT-3.5-turbo and 71.1% for GPT-4 in inferring normality shows moderate performance comparable to the core causal-judgment benchmark.The benchmark uses reworded practical vignettes varying necessity, sufficiency, normality, omission, and harmful outcomes; GPT-4 sometimes failed where GPT-3.5-turbo succeeded, despite often giving more persuasive rationales.Source: §4.3 · pp. 26–28

4.4 Discussion

  • LLMs perform well at generating causal arguments involving necessity, sufficiency, normality, and responsibility, but lower causal-judgment performance motivates systems that guide their assembly.Formal approaches struggle to represent these background common-sense elements, while direct and chain-of-thought answering performs relatively worse on the causal judgment benchmark.Source: §4.4 · p. 28

5 A New Frontier for Causality

The paper presents LLMs as a new frontier for causality, providing domain knowledge and flexible interaction across causal tasks while complementing, rather than replacing, formal causal methods. It outlines practitioner benefits, ethical risks, and research directions for integrating LLMs with causal tools and human expertise.

What is new with causality and LLMs?

  • LLMs provide domain knowledge and flexible natural-language interaction for causal analysis, potentially broadening access to causal tools and enabling new causal capabilities.They can extract causal primitives such as necessity and sufficiency and support formal token-causality systems.Source: What is new with causality and LLMs? · pp. 29–30, §5 · p. 29

What is not changing with causality and LLMs?

  • LLM-driven causal applications must remain tightly integrated with formal tools because rigorous, documented, and verifiable analyses remain necessary, especially for high-risk decisions.LLMs can frame assumptions and generate code, but statistical causal tools remain necessary for estimating causal quantities.Source: What is not changing with causality… · p. 30, §5.1 · p. 31

5.1 Implications for Causality Practitioners

  • LLMs can augment causal practitioners by proposing or critiquing graphs, supporting robustness checks, and reducing the expertise burden of initiating causal analyses.They can help translate between logical, covariance-based, and counterfactual reasoning modes, while human expertise remains central.Source: §5.1 · pp. 30–31
  • LLMs perform similarly on known and post-cutoff datasets, but their causal generalization remains bounded by training knowledge and may require retrieval augmentation for wholly unfamiliar scenarios.They are not expected to construct causal assumptions or graphs for concepts entirely absent from their training corpus.Source: §5.1 · p. 30

5.2 Ethical Implications and Societal Impacts

  • LLM-generated causal explanations can conceal subjective assumptions and biases, risking misleading claims about fairness, neutrality, and causal validity in high-stakes applications.The paper warns against creating a veneer of causal infallibility because learned causal arguments can be manipulated to reflect problematic viewpoints.Source: §5.2 · pp. 31–32

5.3 New research questions on causality and LLMs

  • Future research should combine LLMs with causal discovery, effect inference, robustness testing, token-causality methods, and human collaboration while investigating their causal mechanisms and unpredictable failures.The paper highlights LLM-based priors, partial-graph reasoning, causal primitives for token judgments, and the limits of interpolation versus learned world models.Source: §5.3 · pp. 32–33

6 Conclusion · A Causal graph generation: Additional details

The paper concludes that LLMs can mimic human domain knowledge to support causal analysis, often outperforming state-of-the-art algorithms despite unpredictable failures. It also identifies practical and research opportunities for integrating LLMs into causal workflows and unifying covariance- and logic-based analysis.

6 Conclusion

  • LLMs can outperform state-of-the-art causal algorithms in graph discovery and counterfactual inference while systematizing necessity and sufficiency from natural language alone.Their errors remain infrequent on average but are neither explainable nor predictable.Source: §6 · p. 33
  • LLMs provide value by mimicking human domain knowledge acquired from vast amounts of human-generated text.This capability reflects a complex training process rather than an interpretable or predictable mechanism.Source: §6 · p. 33
  • LLMs can reduce the burden of human expertise in building causal graphs, inferring effects, and attributing causes.The conclusion presents these as practical applications for causal analysis.Source: §6 · p. 34
  • A flexible natural-language interface could bridge covariance-based and logic-based causal analysis for real-world causal questions.The proposed role is to let analyses seamlessly combine both branches of causal analysis.Source: §6 · p. 34
  • The results motivate further research questions at the intersection of LLMs and causality.The conclusion states that the findings open more questions than they answer.Source: §6 · p. 34

A Causal graph generation: Additional details

A.1 Example prompts for the Tubingen benchmark

The Tübingen benchmark uses two prompt-template designs that differ in how many questions they ask for each variable pair.

A.1 Example prompts for the Tubingen benchmark

  • Table 14 presents templates that either ask two questions per variable pair or use a single question to orient each pairwise edge.Source: Table 14 · p. 40

A.2 Example LLM outputs for the Neuropathic pain dataset

The section illustrates one correct and one incorrect GPT-3.5 causal-discovery output for neuropathic pain, showing both explanatory reasoning and a failure involving anatomical interpretation.

Correct output

  • A correct gpt-3.5-turbo output selects B: DLS T5-T6 causes Left T6 Radiculopathy.It defines both conditions and argues that spinal degeneration can cause nerve-root irritation or compression, making the reverse direction less likely.Source: §A.2 · pp. 40–41
  • The correct example begins by clarifying the terms before comparing the two candidate causal directions.This explanation distinguishes Left T6 Radiculopathy as nerve-root damage or irritation from DLS T5-T6 as degenerative spinal changes.Source: §A.2 · pp. 40–41

Incorrect output

  • An incorrect output misdefines DLS T5-T6 as lumbar spondylosis, interprets the symptom relationship incorrectly, and nevertheless selects B.It describes radiculopathy as nerve-root compression or irritation and concludes that Right T6 Radiculopathy causes symptoms in the DLS T5-T6 region.Source: §A.2 · p. 41

A.3 Novel datasets created after LLMs training cutoff date

The paper evaluates LLM causal reasoning on three novel datasets created after the models’ training cutoff: Tubingen pairs, Alzheimer’s graph discovery, and counterfactual reasoning. These experiments compare LLMs with data-based discovery algorithms or previously evaluated models where specified.

Novel Tubingen dataset

  • Table 15 reports novel Tubingen pairs constructed from recently published books.The dataset tests causal reasoning on material created after the LLMs’ training cutoff date.Source: §A.3 · p. 41

Novel Alzheimer’s dataset

  • Table 16 compares gpt-3.5-turbo and gpt-4 graph discovery accuracy with DAG-GNN and NOTEARS.The comparison uses a novel Alzheimer’s dataset and contrasts LLMs with data-based discovery algorithms.Source: §A.3 · p. 41

Novel Counterfactual dataset

B Evaluation: LLMs and counterfactual reasoning … B.1.2 Example responses from gpt-4

The counterfactual evaluation scores option-based answers against benchmark labels while illustrating GPT-4’s reasoning process through prompts and example responses. These examples show both plausible uncertainty and failures from incorrect assumptions about unstated context.

B.1 Details on CRASS counterfactual evaluation

  • The evaluation assigns 1 for a correct option and 0 for an incorrect option, while non-option outputs receive a random-chance score of 0.33.Source: §B.1 · p. 41

B.1.1 Example prompt

  • The benchmark prompt presents a counterfactual question with four answer choices and requires the model to return its selection within <Answer>A/B/C/D</Answer> tags.Source: §B.1.1 · p. 43
  • The example prompt asks what would happen if a woman touched a fire, with “She would have been burned” as one candidate answer.Source: §B.1.1 · p. 43

B.1.2 Example responses from gpt-4

  • For a nervousness counterfactual, GPT-4 judged passing out possible but not guaranteed and selected C, which the authors present among responses that do not match the benchmark answer.Source: §B.1.2 · pp. 43–44
  • GPT-4 sometimes identifies multiple plausible outcomes and responds “Uncertain between B and D” rather than selecting a single benchmark option.Source: §B.1.2 · p. 43
  • GPT-4 can misinterpret implicit context, incorrectly assuming that a man continues walking outdoors on a bed toward the same destination.Source: §B.1.2 · p. 44
  • In the bed-walking example, GPT-4 reasons that the unusual and inefficient movement would slow the man and potentially make him late.Source: §B.1.2 · p. 44

C Evaluation: Necessary and Sufficient Causes … C.4 Example responses from LLMs

The evaluation examines necessary and sufficient token causality across structured vignette classes, using LLM-generated prompts and a newly perturbed chemistry-lab benchmark designed to reduce memorization. The prompting approach derives “minimal change” and “multiple sufficient causes” principles from demonstrations and applies them to causal judgments.

C.1 Classes of token causality vignette scenarios

  • Token-causality vignettes cover symmetric overdetermination, switches, late and early preemption, double preemption, and bogus preemption.These scenarios vary whether multiple processes produce the same outcome, whether one process preempts another, and whether an interrupted process was causally active.Source: §C.1 · pp. 44–45

C.2 Prompt generation for necessary and sufficient causes

  • Because token-causality definitions are difficult to express naturally, the study uses the LLM itself to help generate task prompts.The method is based on demonstrations pairing vignette inputs with correct necessary or sufficient causes, followed by paraphrasing the generated instruction.Source: §C.2 · p. 45
  • The necessary-cause prompt adopts a “minimal change” principle, derived from LLM analysis of three example vignettes and their correct outputs.The principle selects the candidate requiring the least alteration to the initial scenario to obtain the outcome.Source: §C.2 · pp. 45–46
  • The demonstrations illustrate minimal-change reasoning by selecting both simultaneous bullets, the first three shooters, or only the final water droplet, depending on the outcome.The examples show that the selected cause corresponds to the smallest required modification of the input scenario.Source: §C.2 · pp. 45–46
  • For sufficient-cause identification, the prompting procedure is repeated, and an LLM-generated “multiple sufficient causes” principle is incorporated into the prompt.This extends the demonstration-based prompt-generation method from necessary to sufficient causes.Source: §C.2 · pp. 45–46

C.3 “Lab-vignettes”: A novel token causality benchmark

  • The “lab-vignettes” benchmark perturbs an existing dataset into chemistry-lab scenarios involving reagents, mixtures, and crystals to test necessary and sufficient causes on novel inputs.The study applies a memorization test to verify that the newly created dataset cannot be recreated by the LLM; Table 18 provides example vignettes with separate necessary- and sufficient-cause questions.Source: §C.3 · p. 46, Table 18 · p. 46

C.4.1 Correct response on an Early Preemption vignette

On an early-preemption vignette, both GPT-3.5-Turbo and GPT-4 correctly identify that Alice is not a necessary cause, while only GPT-4 correctly identifies her action as sufficient.

Necessary cause

  • Both models correctly answer that Alice is not a necessary cause of the window remaining intact.If Alice had not caught the ball, it would have hit the solid brick wall and not the window.Source: §C.4.1 · p. 47

Sufficiency

  • Only GPT-4 correctly answers that Alice’s action is a sufficient cause of the window remaining intact.GPT-4 recognizes Alice’s action as one sufficient cause among multiple possible causes, whereas GPT-3.5-Turbo answers no.Source: §C.4.1 · p. 47

C.4.2 Unpredictable failure mode: An example · D Assessing Responsibility

GPT-4 can apply causal reasoning correctly in most prompts yet exhibit unpredictable errors, while responsibility is introduced as a graded measure of causal contribution rather than a binary cause judgment. The paper scopes responsibility to causal contribution, distinguishing it from intention, epistemic state, and moral obligation.

C.4.2 Unpredictable failure mode: An example

  • GPT-4 incorrectly assessed necessity instead of sufficiency in a short-circuit vignette, despite applying correct reasoning for most prompts.This illustrates an unpredictable failure mode rather than a general inability to reason causally.Source: §C.4.2 · p. 47
  • In the counterfactual, Carol survives without Alice’s antidote, showing that Alice’s action was not a sufficient cause of Carol’s survival.The model’s reasoning correctly concludes that survival would still occur when the intervention is removed.Source: §C.4.2 · p. 48

D Assessing Responsibility

  • Responsibility extends binary causal judgments by ranking events according to their relative degree of contribution to the same outcome.Natural-language responsibility can distinguish events such as smoking and dry leaves even when both causally contribute to a forest fire.Source: D Assessing Responsibility · p. 48
  • The SCM definition assigns responsibility as 1/(N+1), where N is the minimum number of other causal events that must be absent for C to become necessary for E.This formulation is associated with Halpern (2016); Chockler & Halpern (2004).Source: D Assessing Responsibility · p. 48
  • Mapping SCM responsibility into ordinary language is difficult because everyday responsibility combines causal contribution with intention, epistemic state, and moral obligation.The case study therefore limits responsibility to causal contribution.Source: D Assessing Responsibility · p. 48
  • Under the paper’s definition, responsibility does not depend on whether the causal event is a person’s action.For two events producing the same outcome, one can be more, less, or equally responsible than the other.Source: D Assessing Responsibility · p. 48

D.0.1 Inferring responsibility: Motivating Example · E Probing LLM Behaviors Further

The study defines responsibility as relative causal contribution, independent of whether the cause is an action, while probing whether LLMs handle overdetermination and double prevention. The LLM reduced Mike’s responsibility under overdetermination but failed to match human judgments in the double-prevention case.

D.0.1 Inferring responsibility: Motivating Example

  • Responsibility is defined as an event’s relative causal contribution to an outcome, independent of whether the cause is a person’s action.The prompt also aims to limit intention, morality, and epistemic-state considerations, treating inanimate and human causes alike.Source: §D.0.1 · pp. 48–49
  • In the basic beer-spilling example, the LLM judged Mike’s table bump highly responsible because it was the immediate and direct cause.The prompt applied the relative-contribution definition to Mike’s bumping of the table.Source: §D.0.1 · p. 49
  • Under overdetermination, the LLM recognized that Susan’s later fall eliminated counterfactual dependence on Mike’s bump.It explained that Susan’s fall would have determined the spill regardless of Mike’s initial bump.Source: §D.0.1 · p. 49
  • With Susan providing an alternative cause, the LLM reduced Mike’s responsibility because both events contributed to the spill.This matches the expected downgrade when two causes produce the same outcome.Source: §D.0.1 · p. 49
  • In the double-prevention scenario, the LLM judged Peter more responsible than Mike because Peter directly interfered with Jack’s attempt to catch the bottle.Although Mike’s bump initiated the sequence, the LLM prioritized Peter’s immediate impact on the outcome.Source: §D.0.1 · p. 50
  • This double-prevention judgment was identified as a ChatGPT4 failure to capture the human causal nuance that Mike, rather than Peter, caused the spill.The scenario involves Mike causing the bottle to fall, Jack attempting prevention, and Peter preventing Jack from succeeding.Source: §D.0.1 · pp. 49–50

E.1 Memorization Tests · E.2 Perturbation Test Example · F Supplementary Discussion

The supplementary analyses test whether benchmark performance reflects dataset memorization and illustrate how token perturbations can diagnose and alter LLM reasoning errors. They find partial memorization for some datasets but show that perturbing an influential answer token can change erroneous outputs toward correctness.

E.1 Memorization Tests

  • The row-based memorization test prompts LLMs to complete sampled dataset rows while explicitly testing memory.This procedure is applied to row-based datasets including Tübingen and CRASS.Source: §E.1 · p. 50
  • For unstructured JSON datasets, the test samples 500 characters and measures how many subsequent characters the LLM reproduces.The method targets files that are not naturally split into rows or lines.Source: §E.1 · p. 50, Table 19 · p. 51
  • The memorization tests find that Tübingen is present in GPT-3.5 and GPT-4 training data and has been at least partially memorized.The neuropathic pain dataset appears partially memorized by GPT-3.5 Turbo, while GPT-4 results are unclear.Source: §E.1 · pp. 50–51, Table 19 · p. 51
  • Arctic Sea Ice and CRASS show no evidence of memorization beyond tokens predictable from language or formatting.The tests use recovered-token counts for unstructured datasets and cell- or row-based completion for Tübingen.Source: §E.1 · p. 51, Table 19 · p. 51

E.2 Perturbation Test Example

  • A perturbation experiment illustrates GPT-3.5 producing an erroneous arithmetic answer despite correctly solving prompted subparts.The experiment tests whether the model follows an initial-answer pattern rather than the preceding arithmetic expression.Source: §E.2 · p. 51
  • Perturbing the first answer often changes the final result, while larger perturbations make the final answer more likely to snap to the correct result.Redacting and perturbing tokens one by one identifies which prompt tokens drive the error; systematic perturbation then tests their influence.Source: §E.2 · p. 51

F.1 Example of LLM-assisted Causal Analysis Code Generation · F.2 Example of LLM-assisted Identification of Negative Controls

The case studies illustrate GPT-assisted causal workflows, from generating code for synthetic-data analysis to proposing negative and positive controls for observational vaccine studies. They also show how such suggestions can support sensitivity checks while remaining dependent on methodological assumptions and expected patterns.

F.1 Example of LLM-assisted Causal Analysis Code Generation

  • GPT-4 generated a step-by-step workflow for synthetic-data causal analysis using causal-learn, DoWhy, and EconML, including DAG learning, treatment-effect estimation, and sensitivity analysis.The example uses X1 -> X2 -> Y and X1 -> Y, while noting that estimated and actual effects may differ because of noise and method assumptions.Source: §F.1 · pp. 52, 54
  • The generated workflow begins by installing packages, importing libraries, and creating synthetic treatment, mediator, and outcome variables before learning the causal graph.The implementation includes numpy, pandas, causal-learn, DoWhy, EconML, and linear regression, with Y defined from X1, X2, and noise.Source: §F.1 · p. 52
  • Perturbing the initial response showed that the first answer strongly influenced the final answer in the illustrated GPT-3.5 interaction.Figure 5 describes erroneous intermediate and final answers across sessions, while Table 20 summarizes the perturbation result.Source: Figure 5 · p. 52, Table 20 · p. 53

F.2 Example of LLM-assisted Identification of Negative Controls

  • GPT-4 identified negative controls as variables or outcomes expected to have zero treatment effects, helping assess matching quality and unmeasured confounding.Examples include hair color, blood type, and handedness, for which vaccinated and unvaccinated groups should show no difference.Source: §F.2 · pp. 54–55
  • GPT-4 proposed zero or limited vaccine effects in groups such as people with pre-existing immunity, immune deficiencies, or inadequate vaccine responses.The response cautions that no direct effect in these groups does not preclude indirect protection from widespread vaccination and herd immunity.Source: §F.2 · p. 55
  • The case study also described positive controls with known but non-zero effects, including age, chronic-condition, pregnancy, smoking, and dosing subgroups.Comparing these expected differences with the overall population can help evaluate whether the analysis captures heterogeneous treatment effects and confounding.Source: §F.2 · p. 56
  • For longitudinal data, GPT-4 proposed time-bound controls: no effect before vaccination or shortly afterward, stronger effects during peak immunity, and smaller effects during waning immunity.Post-booster comparisons were also suggested, with expected positive effects among boosted individuals; matching these temporal patterns can help assess time-varying confounding.Source: §F.2 · pp. 56–57

Related papers

Loading 2305.00050v3…