Source-linked AI summary
CounterBench: Evaluating and Improving Counterfactual Reasoning in Large Language Models
Yuefei Chen, Vivek K. Singh, Jing Ma, Ruixiang Tang
TL;DR
LLMs often struggle to reason counterfactually, and existing evaluation lacks a dedicated benchmark for this task. The paper introduces CounterBench and CoIn, a reasoning approach based on iterative inference and backtracking. Most models perform near-randomly, while CoIn reaches 89.9% accuracy on CounterBench and improves on alternative strategies with Gemini-1.5-flash.
Problem
LLMs lack a dedicated benchmark for counterfactual tasks and often struggle to produce logically consistent, contextually appropriate counterfactuals.
Source: §1 · p. 2Method
The paper builds CounterBench with over 1,200 varied counterfactual questions and proposes CoIn, which guides models through abduction, action, iterative prediction, and backtracking validation.
Source: §1 · p. 2Results
Takeaways & Limitations
The reported results indicate that CoIn improves counterfactual reasoning performance across LLMs.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Counterfactual reasoning is widely recognized as one of the most challenging and intricate aspects of causality in artificial intelligence. In this paper, we evaluate the performance of large language models (LLMs) in counterfactual reasoning. In contrast to previous studies that primarily focus on commonsense causal reasoning, where LLMs often rely on prior knowledge for inference, we specifically assess their ability to perform counterfactual inference using a set of formal rules. To support this evaluation, we introduce a new benchmark dataset, CounterBench, comprising 1K counterfactual reasoning questions. The dataset is designed with varying levels of difficulty, diverse causal graph structures, distinct types of counterfactual questions, and multiple nonsensical name variants. Our experiments demonstrate that counterfactual reasoning poses a significant challenge for LLMs, with most models performing at levels comparable to random guessing. To enhance LLM's counterfactual reasoning ability, we propose a novel reasoning paradigm, CoIn, which guides LLMs through iterative reasoning and backtracking to systematically explore counterfactual solutions. Experimental results show that our method significantly improves LLM performance on counterfactual reasoning tasks and consistently enhances performance across different LLMs.Our dataset is available at https://huggingface.co/datasets/CounterBench/CounterBench.
1 Introduction
The introduction identifies a lack of dedicated evaluation for counterfactual reasoning and persistent difficulties with logically consistent, multi-step inference. It presents CounterBench and CoIn, reporting strong results for CoIn where existing models and strategies often struggle.
1 Introduction
- CoIn reaches 89.9% accuracy on CounterBench, nearly 20% better than alternative strategies with Gemini-1.5-flash.The comparison is across Standard, CausalCoT, and Solver strategies.Source: Figure 1 · p. 1, §1 · p. 2
- The paper addresses the absence of a dedicated counterfactual benchmark and models’ difficulty producing logically consistent, contextually appropriate counterfactuals.It frames these as two central challenges for evaluating and improving LLM counterfactual reasoning.Source: §1 · p. 2
- Most models, including GPT-4o and Deepseek-V3, show limited counterfactual inference ability, with accuracy around 50% for many models.The introduction characterizes this level as equivalent to random guessing and reports only marginal gains from state-of-the-art inference strategies for most models.Source: §1 · p. 2
- CoIn guides models through abduction, action, iterative prediction, and backtracking validation to explore counterfactual paths.Its search process assesses inference steps and can revert to more promising points to support logical consistency in long-chain dependencies.Source: §1 · p. 2
- CounterBench contains over 1,200 long-chain counterfactual questions spanning difficulty levels, causal graph structures, question types, and nonsensical name variants.Source: §1 · p. 2
2 CounterBench
CounterBench provides a formal-rule benchmark of counterfactual reasoning, built from queries derived from deterministic structural causal models. Its 1,200 questions span five query types and five difficulty levels, with balanced yes/no answers and nonsensical variable names.
2 CounterBench
- The dataset section introduces a benchmark for measuring LLMs’ ability to handle complex causal reasoning and describes its query-generation methodology.Figure 2 situates the dataset alongside model benchmarking, error analysis, and the CoIn reasoning framework.Source: §2 · p. 2, Figure 2 · p. 3
2.1 Dataset Structure
- CounterBench represents each query-answer pair as a binary response derived from a deterministic structural causal model with exogenous variables, endogenous variables, and structural equations.Intervening on variables changes the model to determine counterfactual outcomes.Source: §2.1 · p. 3
- The five query types cover basic, joint, and nested counterfactuals, plus conditional queries with observed conditions and backdoor queries involving confounding paths.Basic queries ask about one hypothetical intervention; joint queries set multiple variables simultaneously, while nested queries capture sequential dependencies.Source: §2.1 · pp. 3–4
2.2 Query Generation and Quality Assessment
- The 1,200 questions are split across five counterfactual types and five difficulty levels, with balanced yes/no answers and nonsensical names that discourage reliance on memorized knowledge.Each difficulty level has 240 questions and an even yes/no split; query names are artificially generated.Source: §2.2 · p. 5
3 Benchmarking LLMs on CounterBench
The benchmark evaluates standard prompting and CausalCoT across LLMs using inference accuracy. Most models perform near random without specific instructions, and CausalCoT yields little improvement, while inference errors dominate its failures.
3 Benchmarking LLMs on CounterBench
- The evaluation tested ten named LLMs with standard prompting and CausalCoT, classifying responses as yes, no, or incomprehensible and using inference accuracy as its primary metric.Inference temperature was set to 0.Source: §3 · p. 5
3.1 Experimental Results
- 68.0% accuracy was Gemini-1.5-flash’s highest baseline result, while GPT-4o’s 75.8% was the best CausalCoT result; most models showed minimal or no CausalCoT improvement.Without specific instructions, most models performed no better than random guessing.Source: §3.1 · p. 5
- 86% of CausalCoT errors occurred in inference, even when causal graphs were well constructed.The error analysis distinguishes failures in causal relationships, inference, and conclusion derivation.Source: §3.1 · pp. 5–6, Figure 3 · p. 6
4 Proposed Reasoning Strategy
CoIn organizes counterfactual reasoning into five phases, from extracting facts and inferring the factual world to applying interventions, predicting consequences, and validating the reasoning chain.
4.1 Extraction
- During Extraction, the process gathers explicitly stated scenario information, constructs a causal graph by identifying relationships between events, and collects variable values from background information and questions without unsupported inferences.Source: §4.1 · p. 6
4.2 Abduction
- Abduction infers constraints on exogenous noise variables or parent assignments that make observations consistent with structural equations, then stores them as factual-world knowledge.Those inferred values remain fixed during intervention and forward inference.Source: §4.2 · p. 6, §4 · p. 6
4.3 Intervention Action
- Intervention Action applies the query’s hypothetical changes by replacing intervened variables’ equations with constants in the causal rules.The updated rules are incorporated into the knowledge base.Source: §4.3 · pp. 6–7, §4 · p. 6
4.4 Forward Inference
- Forward Inference predicts unobserved variables by evaluating nodes whose parents are known, progressing from interventions and inferred noise toward the target variable.When a node cannot yet be calculated, the framework selects another suitable node and continues.Source: §4 · p. 6, §4.4 · p. 7
4.5 Back-tracking Validation
- Back-tracking Validation re-evaluates each non-noise variable’s equation against its predicted value and flags mismatches for possible reevaluation.This final phase checks the logical consistency of the reasoning chain.Source: §4 · p. 6, §4.5 · p. 7
5 Experiments
The experiments compare CoIn with standard and CausalCoT strategies on CounterBench, then test its generalization on CLADDER. CoIn improves accuracy across models and outperforms both baselines on the CLADDER counterfactual subset.
5.1 Experiment Setup
- The experiments use the same LLMs as Section 3 and compare standard solver strategies with CausalCoT; standard solving integrates LLMs with SCM tools, with inference temperature set to zero.CausalTool's counterfactual procedure was re-implemented from its paper because its source code was unavailable.Source: §5.1 · p. 7, §5.2 · p. 7
5.2 Main Result
- GPT-4o accuracy rises from 75.8% to 89.4% with CoIn, and GPT-4o mini reaches 79.9%; the model comparisons are reported in Table 3.CoIn also performs better across all five GPT-4o mini query types, particularly on basic questions compared with more complex variants.Source: §5.2 · p. 7, Table 3 · p. 7, Table 4 · p. 8
- Inference errors decrease with CoIn, while accuracy decreases as the number of variants increases for both CoIn and CausalCoT.The paper reports the error analysis in Appendix D and the variant-count analysis in the following section.Source: §5.2 · p. 8
5.3 Validating Generalization Ability
- On CLADDER's counterfactual subset, CoIn scores 78.98% accuracy, versus 64.77% for CausalCoT and 64.20% for Standard.Performance is reported as stable across commonsense and anti-commonsense reasoning paradigms.Source: §5.3 · p. 8, Figure 4 · p. 8
6 Related Work
Related work describes counterfactual reasoning's role across several fields and the challenges of applying it to LLMs. Existing approaches use commonsense knowledge or formal causal graphs and external tools.
Counterfactual Reasoning
- Counterfactual reasoning examines how outcomes change when variables differ from their historical states, with applications including policy analysis, medicine, interpretability, and fairness.The passage distinguishes deterministic counterfactuals from probabilistic ones that incorporate uncertainty.Source: §6 · p. 8
LLMs in Counterfactual Learning
- LLM counterfactual methods commonly use either commonsense scenarios or formal causal graphs with external computation; graph-based approaches often offload key calculations outside the LLM.The related work notes that complex variable relationships in high-dimensional text remain challenging for robust causal inference.Source: §6 · pp. 8–9
7 Conclusion
The paper concludes that CounterBench evaluates LLMs across five counterfactual problem types and that CoIn improves their counterfactual reasoning. Its prompt design uses few-shot demonstrations and step-by-step instructions to guide systematic path exploration.
7 Conclusion
- CounterBench is a dataset for evaluating LLMs across five counterfactual problem types, and most LLMs perform near-randomly while state-of-the-art methods show minimal improvement.The conclusion presents CoIn as an iterative reasoning and backtracking paradigm that significantly enhances counterfactual reasoning capabilities.Source: §7 · p. 9
A. Method Prompt Design
- The prompt combines multiple few-shot examples with step-by-step instructions to guide LLMs through CoIn's structured reasoning process.The design aims to support systematic solution-path exploration rather than intuitive responses and reduce erroneous intermediate steps.Source: A. Method Prompt Design · p. 11
B. Human Evaluation
Human evaluators achieved high accuracy on a random sample of CounterBench questions, which they reported required careful causal reasoning. Their judgments also showed strong agreement.
B. Human Evaluation
- Two causal-inference Ph.D. students averaged 97.75% accuracy on 200 randomly selected questions, taking four hours to answer them.Their feedback described the questions as requiring consideration of multiple causal factors and explicit counterfactual reasoning steps.Source: B. Human Evaluation · p. 11
- The two annotators reached approximately 95% inter-annotator agreement after practice examples and calibration discussions.They were given detailed reasoning guidelines before full annotation began.Source: B. Human Evaluation · p. 11
C. More Experiment Details
CoIn consistently outperforms Solver across task categories and model architectures, while performance varies by task type and model. Newer models show higher average accuracy than their predecessors.
C. More Experiment Details
- CoIn consistently outperforms Solver across all task categories and model architectures on CounterBench.The improvement varies by category and is especially pronounced for newer architectures.Source: C. More Experiment Details · pp. 11–12
- 100.0%: GPT-4o’s accuracy on basic reasoning tasks under CoIn.Source: C. More Experiment Details · p. 11
- Basic tasks yield the highest CoIn accuracy but show the most variation across architectures, while joint-task performance is relatively stable across models.Source: C. More Experiment Details · p. 11
- 89.8% for GPT-4o and 89.4% for Claude-3 (Sonnet): the average accuracies reported for these newer models.The paper reports that model recency correlates with performance across task types, with improvement magnitude varying by category.Source: C. More Experiment Details · pp. 11–12
D. Error Analysis comparison between Our Method and CausalCoT
The error analysis reports fewer inference-related errors under the proposed method, while the examples contrast commonsense with anti-commonsense reasoning. CoIn uses structured path exploration, and its forward-inference procedure evaluates target events with iterative reasoning and backtracking.
D. Error Analysis comparison between Our Method and CausalCoT
- Inference-related errors fell from 86% to 46%, while relationship-related errors rose from 12% to 50% in the sampled comparison.The analysis randomly sampled 50 instances and reports that the overall number of errors also decreased.Source: D. Error Analysis comparison between… · p. 12
E. CLADDER Dataset Example
- Anti-commonsense examples test whether models follow explicitly stated causal relations rather than intuitive prior knowledge.The paired CLADDER examples cover both commonsense and anti-commonsense reasoning.Source: E. CLADDER Dataset Example · p. 12
F. Answer of our methods
- CoIn decomposes counterfactual reasoning into information collection and systematic exploration of inference paths, unlike the minimally guided standard prompt.The paper attributes the illustrated correct answer to explicit variable mapping and structured processing of causal dependencies.Source: F. Answer of our methods · p. 12
G. Forward Inference Algorithm
- The forward-inference procedure applies information gathered during extraction, abduction, and action to evaluate the target event.It iteratively infers event values and uses evaluation and backtracking to optimize reasoning paths.Source: G. Forward Inference Algorithm · p. 12
H. Benchmark Result Analysis: CausalCoT
Under CausalCoT, Babbage-002 achieved 8.0% accuracy, with most responses classified as incomprehensible. The analysis categorizes errors and includes an example with inconsistent variable assignments.
H. Benchmark Result Analysis: CausalCoT
- 8.0%: Babbage-002’s accuracy under CausalCoT; 82.4% of its responses were classified as incomprehensible.The paper says this helps explain why its accuracy was below random guessing.Source: H. Benchmark Result Analysis: CausalCoT · p. 13
- The error analysis distinguishes blank responses, repetitions of the question or supplied information, and type mismatches in expected yes/no answers.Source: H. Benchmark Result Analysis: CausalCoT · p. 13
- The worked response assigns conflicting variable mappings and values while attempting to derive the counterfactual outcome.It first maps X to kraz and V2 to pexu, then gives a different mapping in its answer and repeats X = 0.Source: H. Benchmark Result Analysis: CausalCoT · p. 14