Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluation

Evaluation approach

• The evaluation covers four ChatGPT versions and uses ECI, causal discovery, and causal explanation generation to test different aspects of causal ability.ECI tests causal relations between events in sentences; causal discovery uses multiple-choice and binary-classification formats, while explanation generation tests production of explanations.Source: §1 · pp. 1–2

Key findings

• ChatGPT is not a good causal reasoner, but is good at causal explanation generation.The evaluation also identifies serious causal hallucination: ChatGPT tends to assume causal relationships even when they may not exist.Source: §1 · p. 2
• Causal hallucinations may reflect natural-language reporting biases and model upgrades such as RLHF, and ICL and CoT prompts can exacerbate them.The proposed explanations are qualified as possible causes.Source: §1 · p. 2
• Causal-reasoning performance is sensitive to prompt wording, and open-ended generation prompts cannot improve it.The paper also reports that close-ended prompts perform better than open-ended prompts.Source: §1 · p. 2
• ChatGPT identifies explicit causality better than implicit causality, and performs worse as event density and lexical distance increase.The distance and event-count trends are reported for events in sentences.Source: §1 · p. 2

2.1 Causal Reasoning in NLP

• Causal-reasoning methods have used external knowledge, causal-specific pre-training, and data augmentation, but depend heavily on annotated training data and perform poorly when it is scarce.The paper evaluates ChatGPT as an approach that does not require training data.Source: §2.1 · p. 2

2.2 Evaluation of ChatGPT’s Capabilities

• Earlier ChatGPT evaluations used small-scale datasets and simple formats that the authors say overestimated causal-reasoning ability.Some used only COPA multiple-choice questions, while another tested causal direction without asking whether causality existed.Source: §2.2 · p. 2
• The paper evaluates ChatGPT across four task forms and five widely used causal-reasoning datasets.Source: §2.2 · p. 2

3.1.1 Event Causality Identification

• Event causality identification is evaluated on EventStoryLine, Causal-TimeBank, and MAVEN-ERE using accuracy, precision, recall, and F1.The evaluation uses the top 20 EventStoryLine topics and MAVEN-ERE’s development set because its test set was unavailable.Source: §3.1.1 · p. 3

3.1.2 Causal Discovery

• Evaluation uses COPA and e-CARE, causal-reasoning datasets containing 1,000 and 21,324 multiple-choice questions, respectively; COPA primarily focuses on everyday-life scenarios, whereas e-CARE covers a wide range of domains.Accuracy is the evaluation metric.Source: §3.1.2 · p. 3

3.1.3 Causal Explanation Generation

• Causal explanation generation is evaluated on e-CARE’s human-annotated explanations using average-BLEU, ROUGE-L, and human judgments of 100 generations per ChatGPT version.Human evaluation labels whether a generated explanation explains its corresponding causal fact.Source: §3.1.3 · p. 3

3.2 Experiment Setting

• ChatGPT is evaluated with instruction prompts in zero-shot settings; four model versions are tested, with temperature set to zero.For GPT-4, 1,000 instances are sampled from each dataset.Source: §3.2 · p. 3

3.3 Baselines

• Baselines are fine-tuned on full training datasets and include PLM classifiers, ECI-specific methods, and generation models.The generation baselines include GRU-based Seq2Seq, GPT2, fine-tuned LLaMA 7B, and FLAN-T5 11B.Source: §3.3 · p. 3

4.1 Event Causality Identification

• On ECI, even GPT-4 is comprehensively outperformed by fine-tuned small PLMs, indicating weak performance on complex causal reasoning.The ECI results are reported across three datasets.Source: §4.1 · p. 3, Table 1 · p. 4
• On ECI, high recall but low precision means ChatGPT falsely labels many non-causal pairs as causal, especially on CTB, which contains more non-causal pairs.The authors suggest that natural-language reporting bias toward describing causal relationships may contribute to this pattern.Source: §4.1 · p. 4
• On ECI, fine-tuned small PLMs identify non-causal pairs better than causal ones, which the authors link to the datasets’ larger number of negative examples.They attribute this pattern to the models learning the training-data distribution.Source: §4.1 · p. 4

4.2 Causal Discovery

• In CD, ChatGPT performs well in multiple choice but poorly in binary classification, so multiple-choice-only evaluations can overestimate its causal reasoning ability.The multiple-choice format lets ChatGPT focus on the more obvious option and ignore the harder one.Source: §4.2 · p. 4, Table 2 · p. 4
• In CD, non-causal pairs are easier for ChatGPT to identify than in ECI because COPA and e-CARE pairs are manually generated, structurally simple, and weakly correlated with input events.On e-CARE, causal-pair accuracy is slightly lower than in ECI, which the authors attribute to broader commonsense and specialized knowledge demands.Source: §4.2 · pp. 4–5
• ChatGPT’s upgrades from text-davinci-003 through GPT-4 increasingly favor classifying event pairs as causal, whether or not that is correct.The authors suggest RLHF’s alignment tax may contribute to this tendency.Source: §4.2 · p. 5

4.3 Causal Explanation Generation

• Human evaluation finds ChatGPT’s causal explanations close to human-generated explanations in accuracy, supporting its strength as a causal explainer.ChatGPT also receives a higher ROUGE-L score than human generation, associated with more complete and detailed explanations.Source: §4.3 · p. 5, Table 3 · p. 5
• ChatGPT’s explanations are more comprehensive and detailed than those from LLaMA and FLAN-T5, while similar ROUGE-L scores do not prevent the fine-tuned models from performing worse in human evaluation.The authors note that fine-tuned models may repeat event wording, whereas ChatGPT can explain causal pairs from different perspectives or in distinct syntactic forms.Source: §4.3 · p. 5

5.1 In-Context Learning

• ICL can improve accuracy on causal pairs while reducing it on non-causal pairs; “4 pos + 48 neg” raises Full accuracy by trading away Pos accuracy.Because ESC has more non-causal pairs, Neg accuracy has greater influence on Full accuracy.Source: §5.1 · p. 6

5.2 Chain-of-Thought Prompting

• CoT zero-shot does not effectively improve ECI, while CoT demonstrations improve causal-pair accuracy but reduce non-causal-pair accuracy.The authors attribute the zero-shot result to low-quality generated reasoning chains and suggest lower-quality chains for non-causal pairs may worsen the imbalance.Source: §5.2 · p. 6, Table 6 · p. 7

5.3 Ways of Expressing Causality in Prompts

• Causal-wording choices matter: counterfactual prompts misclassify almost all non-causal pairs, while “one-step” improves their accuracy at the expense of causal-pair accuracy.Different causal cue words produce significantly different performance; the authors suggest their learned distributions may differ even when the words mean the same to humans.Source: §5.3 · p. 7, Table 7 · p. 7

5.4 Lexical Distance between Events

• As event distance increases, ChatGPT is more inclined to predict non-causality and its F1 scores generally decrease, indicating difficulty with long-distance causal relations.The gpt-4 result for interval [25,30) is an outlier; only 35 of its 1,000 test instances fell in that interval.Source: §5.4 · p. 7, Figure 3 · p. 8

5.5 Density of Events

• As event density increases, most ChatGPT versions more often predict non-causality, and F1 scores decrease as the event context becomes more complex.The authors link the added complexity to greater difficulty capturing correlations between events.Source: §5.5 · p. 8, Figure 4 · p. 8

5.6 Types of Causal Relationship

• ChatGPT performs better on explicit than implicit causality; explicit cases have causal cue words, while implicit cases require contextual and commonsense reasoning.The figure reports accuracy for the two relationship types.Source: §5.6 · p. 8, Figure 5 · p. 8

5.7 Prompts in the Form of Open-Ended Generation

• Open-ended prompts decrease ChatGPT’s performance because they require event extraction as well as causal identification, and the authors report that ChatGPT is not good at event extraction.The evaluation uses a relaxed P, R, and F1 rule: a predicted pair is correct when at least one token overlaps with each labeled event.Source: §5.7 · p. 9, Table 8 · p. 9

• ChatGPT is not a good causal reasoner, but is good at causal explanation generation.Source: §6 · p. 9
• ChatGPT has serious causal hallucination, which the authors say may relate to causal reporting biases and model-upgrading processes.Source: §6 · p. 9
• ICL and CoT can further exacerbate ChatGPT’s causal hallucination.Source: §6 · p. 9
• ChatGPT is sensitive to how prompts express causal concepts, and open-ended causal reasoning prompts are not suitable for it.Source: §6 · p. 9
• For sentence events, ChatGPT performs better on explicit causality, lower event density, and smaller event distances.Source: §6 · p. 9

• ChatGPT’s capabilities are constantly updated, so current test results may change as the system evolves.Source: Limitations · p. 10
• Unclear implementation details make it difficult to explain performance differences among ChatGPT versions or isolate effects of data and training techniques.Source: Limitations · p. 10
• Different prompts may outperform the reported results, but the authors believe prompting alone cannot fundamentally solve ChatGPT’s causal-reasoning issues.Source: Limitations · p. 10

• With Anthropic’s open-source RLHF dataset, the authors compare zero-shot FLAN-T5-Large performance with and without RLHF because ChatGPT’s RLHF details are unavailable.Source: A Effect of RLHF on Causal Reasoning · p. 12
• RLHF also exacerbates causal hallucination in FLAN-T5-Large, according to the authors’ test.Source: A Effect of RLHF on Causal Reasoning · p. 12
• Only 10.29% of “Why” questions in the examined RLHF data include “not,” which the authors interpret as a tendency to guide models about causality rather than non-causality.Source: A Effect of RLHF on Causal Reasoning · p. 12
• In the gold responses, “yes” appears approximately twice as often as “no,” which could increase the likelihood of positive labels.Source: A Effect of RLHF on Causal Reasoning · p. 12
• The authors suggest both RLHF-data patterns could potentially exacerbate ChatGPT’s causal hallucination and encourage assuming causal relations even when they do not exist.Source: A Effect of RLHF on Causal Reasoning · p. 12

B Effect of the Label Distribution in ICL Demonstrations

• For k=4, the label-distribution analysis groups ESC instances by how many causal demonstrations appear among their top four similar examples.The five groups contain 0, 1, 2, 3, or 4 causal demonstrations; the section defines “Proportion+” and the x-pos/y-neg labels.Source: B Effect of the Label Distribution in… · p. 12, Table 10 · p. 12
• With mixed-label demonstrations, Neg accuracy rises as causal examples increase while Pos accuracy changes less; single-class sets achieve higher Pos accuracy.The analysis uses subsets with 0–4 causal examples among the top four similar demonstrations.Source: Table 10 · p. 12, B Effect of the Label Distribution in… · p. 12

C Effect of the Order of ICL Demonstrations

• Causal-first demonstrations incline ChatGPT more toward causal classifications than non-causal-first, and every tested few-shot order is more causal-leaning than zero-shot.The authors suggest earlier demonstrations have stronger impact.Source: C Effect of the Order of ICL… · pp. 12–13, Table 12 · p. 13

D Error Analysis for the Causal Reasoning of ChatGPT

• Manual annotation of 100 ESC errors per model identifies four recurring types: fabricated conditions, misunderstood event relations, wrong event references, and incorrect commonsense.Fabricated conditions may be absent from or incorrect relative to the input sentence.Source: D Error Analysis for the Causal… · p. 13
• ChatGPT’s unreliable commonsense, event understanding, and logical reasoning can compound errors, while pretrained causal associations may not fit the input context.The authors present these as explanations for the observed errors.Source: D Error Analysis for the Causal… · p. 13
• Compared with text-davinci-003, gpt-3.5-turbo and gpt-4 more clearly tend to fabricate causal conditions, while gpt-4 introduces fewer incorrect commonsense claims.The authors suggest dialog data may encourage longer, more divergent responses; they attribute gpt-4’s lower incorrect-commonsense errors to better knowledge grasp.Source: D Error Analysis for the Causal… · p. 13

E Details of CoT Experiments

• The section presents both the few-shot CoT demonstrations and ChatGPT’s generated reasoning chains as examples.Source: E Details of CoT Experiments · p. 13, Figure 7 · p. 14, Figure 8 · p. 15
• Figure 7 provides examples of demonstrations used in the few-shot CoT experiments.Source: E Details of CoT Experiments · p. 13, Figure 7 · p. 14
• Figure 8 provides examples of reasoning chains generated by ChatGPT.Source: E Details of CoT Experiments · p. 13, Figure 8 · p. 15