PaperlayerView exact revision on arXiv

Source-linked AI summary

CausalFlip: A Benchmark for LLM Causal Judgment Beyond Semantic Matching

Yuzhe Wang, Yaochen Zhu, Jundong Li

arXiv:2602.20094v2cs.AI

TL;DR

Traditional reasoning benchmarks can yield high scores through semantic patterns rather than causal reasoning, leaving a gap in evaluating LLM causal judgment. CausalFlip tests this with paired questions across causal structures and noisy prefixes, while the paper examines training with explicit and internalized reasoning. Intermediate-step supervision improves accuracy, and implicit causal reasoning is more robust than explicit CoT under noisy prefixes.

  • Problem

    Traditional reasoning benchmarks may not reliably reveal whether LLMs reason from causal structure, since semantic correlations can support high scores.

    Source: §1 · p. 1

  • Method

    CausalFlip pairs semantically similar questions with opposite labels across confounder, chain, and collider structures, and evaluates answer-only, explicit-CoT, and implicit-reasoning training.

    Source: §1 · p. 2, §5.2 · p. 6

  • Results

    Intermediate causal reasoning supervision improves accuracy over pretraining and no-CoT fine-tuning, while implicit reasoning is more robust than explicit-CoT under noisy prefixes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings support using CausalFlip to assess whether LLM causal judgments track causal structure rather than semantic patterns.

    Source: §7 · p. 8

    The formulation is scoped to binary Yes/No causal judgments associated with an underlying causal graph and target label.

    Source: §2.3 · p. 4

Abstract

from arXiv · show

As large language models (LLMs) witness increasing deployment in complex, high-stakes decision-making scenarios, it becomes imperative to ground their reasoning in causality rather than spurious correlations. However, strong performance on traditional reasoning benchmarks does not guarantee true causal reasoning ability of LLMs, as high accuracy may still arise from memorizing semantic patterns instead of analyzing the underlying true causal structures. To bridge this critical gap, we propose a new causal reasoning benchmark, CausalFlip, designed to encourage the development of new LLM paradigm or training algorithms that ground LLM reasoning in causality rather than semantic correlation. CausalFlip consists of causal judgment questions built over event triples that could form different confounder, chain, and collider relations. Based on this, for each event triple, we construct pairs of semantically similar questions that reuse the same events but yield opposite causal answers, where models that rely heavily on semantic matching are systematically driven toward incorrect predictions. To further probe models' reliance on semantic patterns, we introduce a noisy-prefix evaluation that prepends causally irrelevant text before intermediate causal reasoning steps without altering the underlying causal relations or the logic of the reasoning process. We evaluate LLMs under multiple training paradigms, including answer-only training, explicit Chain-of-Thought (CoT) supervision, and a proposed internalized causal reasoning approach that aims to mitigate explicit reliance on correlation in the reasoning process. Our results show that explicit CoT can still be misled by spurious semantic correlations, where internalizing reasoning steps yields substantially improved causal grounding, suggesting that it is promising to better elicit the latent causal reasoning capabilities of base LLMs.

1 Introduction

CausalFlip targets a gap between strong benchmark performance and causal reasoning by testing whether LLM judgments follow causal structure rather than semantic cues. The paper introduces paired-question evaluation, internalized causal reasoning, and noisy-prefix testing, and reports that internalized reasoning is more robust to noisy prefixes than explicit CoT.

1 Introduction

  • Traditional reasoning benchmarks may reward semantic correlations, leaving a gap in evaluating whether LLMs can reason from causal structure.The introduction frames this gap as hindering development of LLM paradigms or training algorithms grounded in causality.Source: §1 · p. 1
  • Explicit CoT can improve accuracy and inspectability on traditional benchmarks, but it has higher latency and token usage and may still rely on semantic patterns.The introduction notes that whether internalization encourages causal reasoning in model weights remains unclear.Source: §1 · p. 1
  • CausalFlip covers confounder, chain, and collider structures using semantically similar question pairs with opposite causal labels.The paired questions reuse the same events, making semantic matching a potential source of incorrect judgments.Source: §1 · p. 2, Figure 2 · p. 3
  • Alternative phrasings and a pairwise train-test split further penalize semantic shortcuts by holding out each training question’s similar counterpart with the opposite label.The four categories are Base–Default, Base–Alternative, Opposite–Default, and Opposite–Alternative.Source: §1 · p. 2
  • Implicit causal reasoning progressively masks supervised intermediate reasoning tokens to encourage models to encode reasoning internally, while noisy prefixes test reliance on semantic patterns.The prefixes are causally irrelevant and do not change the CoT content or underlying causal structure.Source: §1 · p. 2
  • With a fixed noisy prefix, explicit-CoT degrades more, while implicit causal reasoning remains robust and achieves higher accuracy than explicit-CoT.The prefix leaves the causal structure unchanged.Source: §1 · p. 2

2 Preliminaries

The preliminaries formalize autoregressive token prediction and Chain-of-Thought (CoT) as conditional generation, then cast the task as binary causal-judgment question answering linked to causal graphs.

2.1 LLM Basics

  • Autoregressive LLMs factor sequence probabilities into next-token conditionals and generate left-to-right from prefixes; maximum-likelihood training rewards predictive context regularities, including spurious downstream-label correlations.The next-token objective may therefore reward semantic correlations even when they are spurious.Source: §2.1 · p. 3

2.2 Chain-of-Thought Reasoning

  • CoT generates intermediate reasoning tokens s before answer y, which is conditioned on x and s, but its steps remain conditional rather than grounded in true causal structure.At inference, the model generates a reasoning-step rollout and then predicts the answer conditioned on it.Source: §2.2 · pp. 3–4

2.3 Problem Formulation

  • Each causal-judgment question is linked to a causal graph G and correct binary label a ∈ {Yes, No}, with answers predicted by maximizing pθ(a|q).Source: §2.3 · p. 4

3 Benchmark Design

CausalFlip tests whether models judge causal structure rather than match question semantics, using semantically similar question pairs whose labels flip with the underlying causal graph. It balances causal structures and question templates to reduce shortcuts based on wording or category frequencies.

3 Benchmark Design

  • CausalFlip builds binary causal-judgment questions over event triples in confounder, chain, and collider sub-datasets, with Base and Opposite structures yielding flipped labels.Its design aims to penalize predictions based on spurious semantic correlations.Source: §3 · p. 4, §3.2 · p. 4

3.1 Semantically Similar, Label-Flipped Pairs with Pairwise Train–Test Split

  • Each pair reuses the same event triples and similar semantics but asks about different causal relations, so its two questions have opposite labels.The pairwise train–test split puts one question in training and its counterpart in testing, making semantic matching alone insufficient for the test question.Source: §3.1 · p. 4

3.2 Causal Structures and Induced Questions

  • Each dataset balances Base and Opposite causal structures, preventing question type from having a fixed label across structures.Base follows the dataset’s canonical graph; Opposite uses an alternative graph that flips the labels for the same question types.Source: §3.2 · p. 4

3.2.1 Confounder Dataset.

  • In the confounder Base graph, Z causes both X and Y, while the paired direct-effect question has no directed causal path from X to Y.The Opposite graph instead has X cause Y, and the two question types’ labels flip.Source: §3.2.1 · p. 4

3.2.2 Chain Dataset.

  • In the chain Base graph, X affects Z through Y, while the Opposite chain structure has no causal path from X to Y or from Y to Z.The collider Base graph instead has X and Y as independent parents of Z, with no directed path between X and Y.Source: §3.2.2 · pp. 4–5, §3.2.3 · p. 5

3.3 Question Templates

  • Default templates phrase questions as direct binary questions, while Alternative templates use declarative statements judged Yes or No.The two template families are balanced across Base and Opposite structures, producing matched BD, BA, OD, and OA categories.Source: §3.3 · p. 5

4 Implicit Causal Reasoning

The paper introduces implicit causal reasoning to reduce explicit CoT’s reliance on spurious semantic correlations. Its training strategy progressively masks early causal-reasoning tokens from supervision, shifting learning toward the remaining steps and answer.

4 Implicit Causal Reasoning

  • Implicit causal reasoning is introduced to reduce explicit CoT’s reliance on spurious semantic correlations in causal questions.CausalFlip is designed to penalize models that rely on semantics for prediction.Source: §4 · p. 5, §4.1 · p. 5

4.1 Motivation and Overview

  • The proposed progressive masking strategy removes causal-reasoning steps from the training loss to encourage generations grounded in causal structure.It is presented as a preliminary exploration of grounding LLM reasoning in causality.Source: §4.1 · p. 5

4.2 Progressive Causal Reasoning Steps Mask

  • Training samples pair a question and binary answer with intermediate reasoning steps derived from the corresponding causal graph.Those steps state key structural conditions that lead to the answer.Source: §4.2 · p. 5

4.2.1 Explicit Causal Reasoning Steps Supervision.

  • Explicit causal-reasoning supervision applies the autoregressive next-token objective to all intermediate reasoning-step tokens and the final answer.The steps are represented as a sequence of tokens, with the cited passage identifying their token-wise likelihood formulation.Source: §4.2.1 · p. 5

4.2.2 Progressive Causal Reasoning Steps Removal.

  • At training step t, progressive removal masks the first r(t) reasoning tokens from supervision and computes loss only on the remaining tokens.The mask marks each token as either excluded from loss or given standard supervision; the schedule controls how many tokens are masked over training.Source: §4.2.2 · pp. 5–6

5 Empirical Study

The empirical study compares four training strategies on CausalFlip and tests whether implicit causal reasoning is more robust than explicit-CoT to semantically distracting prefixes. Intermediate causal reasoning supervision improves clean-input accuracy, while implicit reasoning retains higher accuracy under noisy prefixes.

5.1 Research Questions

  • The study asks how the strategies compare on CausalFlip and whether implicit reasoning is more robust than explicit-CoT when semantic distractions are added.The noisy-prefix accuracy change is used as an indicator of sensitivity to semantic patterns versus causal structure.Source: §5.1 · p. 6

5.2 Baseline and Training Strategies

  • Implicit causal reasoning progressively masks intermediate reasoning-token supervision while continuing to supervise the final decision; explicit-CoT supervises the full reasoning and answer.The four-way comparison also includes the untuned base model and answer-only no-CoT fine-tuning.Source: §5.2 · p. 6

5.3 Training and Evaluation Setups

  • All strategies use the same fixed train/test split; evaluation presents question-only prompts and scores the final binary answer against the ground truth.Accuracy is reported on clean inputs and in the noisy-prefix setting.Source: §5.3 · p. 6, §5.3.2 · p. 6, §5.3.3 · p. 7

5.3.4 Noisy-Prefix Evaluation

  • The noisy prefix is causally irrelevant and leaves the underlying causal structure unchanged, allowing accuracy degradation to probe sensitivity to semantic distraction.The evaluation compares explicit-CoT and implicit causal reasoning under the same prefix condition.Source: §5.3.4 · p. 7

5.4 Performance on the CausalFlip (RQ1)

  • Confounder accuracy: explicit-CoT/implicit 0.892/0.900; Chain: 0.690/0.757; Collider: 0.856/0.849.These scores exceed the naive-pretraining and no-CoT results reported for each dataset.Source: §5.4 · p. 7, Table 1 · p. 7, Table 2 · p. 7
  • Naive pretraining and no-CoT performed poorly across all three datasets, whereas training with intermediate causal reasoning steps substantially improved accuracy.No-CoT was near chance on Confounder (0.529/0.524 for naive/no-CoT); on Chain, no-CoT reached 0.639 versus 0.612 for naive pretraining.Source: §5.4 · p. 7

5.5 Performance under Noisy Prefix (RQ2)

  • Under noisy prefixes, implicit causal reasoning outscored explicit-CoT on Confounder (0.769 > 0.699), Chain (0.692 > 0.547), and Collider (0.768 > 0.710).Average accuracy drops were about 0.161 for explicit-CoT and 0.092 for implicit causal reasoning.Source: §5.5 · p. 7

6 Related Work

Prior work studies LLM causal inference and offers a range of causal benchmarks, but benchmark accuracy can be inflated by spurious semantic correlations. CausalFlip complements this work with opposite-label, semantically similar pairs and a pairwise split; Figure 3 reports greater noisy-prefix robustness for implicit reasoning than explicit-CoT.

Causal Inference in LLMs

  • Figure 3 compares explicit-CoT and implicit causal reasoning across three CausalFlip sub-datasets with clean inputs and noisy prefixes; implicit reasoning degrades less under the prefix.The caption attributes this pattern to reduced reliance on spurious semantic correlations.Source: Figure 3 · p. 8
  • CausalFlip complements prior LLM causal-inference studies with a training strategy aimed at reducing semantic reliance and a benchmark that penalizes semantic matching through opposite-label pairs and pairwise splitting.Questions are built over event triples and evaluate causal judgment under different structures.Source: §6 · p. 8

Causal Benchmarks for LLMs

  • Existing causal and general reasoning benchmarks can yield high accuracy through label-correlated spurious patterns, so accuracy alone may not establish reasoning from causal structure.CausalFlip uses semantically similar, label-flipped pairs to make shortcut matching insufficient for correct predictions.Source: §6 · p. 8

7 Conclusion

The paper introduces CausalFlip to evaluate LLM causal judgment beyond reliance on label-correlated semantic patterns. Its experiments find that intermediate-step supervision improves accuracy and that progressively masking those steps can reduce reliance on explicit semantic correlations.

7 Conclusion

  • CausalFlip evaluates confounder, chain, and collider judgments; experiments find that intermediate-step supervision improves accuracy and progressive masking reduces reliance on possible spurious semantic correlations.The implicit strategy masks intermediate causal-reasoning tokens from supervision.Source: §7 · p. 8

8 Ethics and Fairness

The paper describes dataset review and intended use as part of its ethics and fairness discussion.

8 Ethics and Fairness

  • CausalFlip contains no user data or human-subject records, and manual review was used to mitigate sensitive, offensive, or stereotyped content.The authors describe the benchmark as intended for research evaluation and encourage responsible downstream use.Source: §8 · p. 8

A.1 Reducing Data Skewness

CausalFlip can retain event-distribution shortcuts despite semantically similar, opposite-answer pairs. The authors identify and reduce skew in event frequencies and semantic neighborhoods.

  • Event-frequency imbalances can associate particular events with Base or Opposite labels, creating shortcuts that bypass causal structure.Source: §A.1 · pp. 9–10
  • The skewness audit counts event frequencies across labels and categories, then uses stella-400M cosine neighbors to identify generic or near-duplicate examples.Source: §A.1 · p. 10
  • Iterative replacement of the top five cases balances overrepresented events and reduces highly repeated neighbor patterns.Source: §A.1 · p. 10

A.2 Hyperparameters

All fine-tuned strategies share LoRA settings and a fixed three-epoch training setup, with a fixed random seed for fair comparisons.

  • All fine-tuned strategies use LoRA with r=4, α=8, dropout=0.05, targeting q_proj and v_proj.Source: §A.2 · p. 10
  • Training runs for 3 epochs at learning rate 1 × 10−4, batch size 4, and maximum sequence length 256.Source: §A.2 · p. 10
  • The setup uses paged AdamW 8-bit, bf16 training, gradient checkpointing, and a fixed random seed for fair comparisons.Source: §A.2 · p. 10

A.3 Benchmark Generalizability

On IBM-PowerMoE-3B, standard fine-tuning underperforms pretraining across all three CausalFlip subsets. Representation comparisons show implicit causal reasoning resembles explicit-CoT more than no-CoT.

A.3 Benchmark Generalizability

  • IBM-PowerMoE-3B accuracy falls from 51.9% to 50.8% on Confounder, 50.8% to 50.0% on Chain, and 53.2% to 52.8% on Collider after standard fine-tuning versus pretraining.Source: §A.3 · p. 10
  • Valid outputs were 1000/1000 pretrained on each dataset and 999/1000, 993/1000, and 995/1000 after fine-tuning on Confounder, Chain, and Collider, respectively.The authors report that the accuracy result is not driven by invalid output.Source: §A.3 · p. 10

A.4 Interpretability in Internalization

  • Implicit causal reasoning representations are more similar to explicit-CoT than no-CoT across Confounder, Chain, and Collider.Cosine similarity, implicit-CoT vs. no-CoT: Confounder 0.8928 vs. 0.1174, Chain 0.7263 vs. 0.1364, Collider 0.8146 vs. 0.0905; linear CKA, implicit-CoT vs. no-CoT: Confounder 0.9890 vs. 0.4757, Chain 0.9038 vs. 0.5376, Collider 0.9431 vs. 0.4429.Source: §A.4 · p. 10, Table 5 · p. 10

Related papers

Loading 2602.20094v2…