PaperlayerView exact revision on arXiv

Source-linked AI summary

Can LLMs Explain Themselves Counterfactually?

Zahra Dehghanighobadi, Asja Fischer, Muhammad Bilal Zafar

arXiv:2502.18156v2cs.CLcs.AI

TL;DR

Because LLM outputs are difficult to explain, this paper studies whether models can generate counterfactual explanations of their own predictions. It evaluates SCE generation and model predictions on the resulting inputs, finding that those predictions often miss the intended target and are influenced by context. The authors conclude that LLMs remain far from reliable in counterfactual self-explanation.

  • Problem

    LLMs’ complex behavior makes their outputs difficult to explain, while self-generated counterfactual explanations offer a form of contrastive explanation.

    Source: §1 · p. 1

  • Method

    The study tests whether LLMs generate SCEs and whether their predictions on those SCEs align with the intended target across multiple models, datasets, and tasks.

    Source: §1 · p. 1

  • Results

    Models often predict a label other than the SCE’s intended target, and their predictions are strongly influenced by the original prediction and SCE instruction in context.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    The findings indicate that LLMs remain far from being able to explain their own predictions counterfactually.

    Source: §8 · p. 9

    Evaluation relied primarily on automated metrics, with human annotation limited to one task.

    Source: §9 · p. 10

Abstract

from arXiv · show

Explanations are an important tool for gaining insights into the behavior of ML models, calibrating user trust and ensuring regulatory compliance. Past few years have seen a flurry of post-hoc methods for generating model explanations, many of which involve computing model gradients or solving specially designed optimization problems. However, owing to the remarkable reasoning abilities of Large Language Model (LLMs), self-explanation, that is, prompting the model to explain its outputs has recently emerged as a new paradigm. In this work, we study a specific type of self-explanations, self-generated counterfactual explanations (SCEs). We design tests for measuring the efficacy of LLMs in generating SCEs. Analysis over various LLM families, model sizes, temperature settings, and datasets reveals that LLMs sometimes struggle to generate SCEs. Even when they do, their prediction often does not agree with their own counterfactual reasoning.

1 Introduction

LLMs’ growing capabilities coexist with difficulty explaining their outputs, motivating the study of self-generated counterfactual explanations (SCEs). The paper tests whether LLMs can generate SCEs, whether their predictions align with them, and how context affects that alignment.

1 Introduction

  • LLMs’ complex behavior makes their outputs difficult to explain despite their strong performance across a range of tasks.The cited work attributes their capabilities to large datasets, model sizes, and alignment with human preferences.Source: §1 · p. 1
  • SCEs are self-generated counterfactual explanations, a form of contrastive explanation that prior work argues better aligns with human expectations and regulatory needs.A counterfactual modifies input x to lead to a different output than the original prediction.Source: §1 · p. 1
  • The study evaluates LLM efficacy in generating SCEs by asking whether they can generate SCEs at all, whether those counterfactuals faithfully reflect model reasoning, and whether generation avoids large-scale input changes.Source: §1 · p. 1
  • The evaluation procedure obtains an initial prediction, asks the model for an SCE, and then queries its prediction on that SCE.The study finds that the original prediction and SCE instruction in the chat history strongly influence the later prediction.Source: §1 · pp. 1–2
  • In many cases, models’ predictions on SCEs do not yield the requested target label, exposing a mismatch between counterfactual reasoning and model predictions.The authors conclude that LLMs remain far from reliable in counterfactual self-explanation.Source: §1 · p. 2

2 Related Work

Related work spans established explainability methods, counterfactual explanations, and LLM self-explanation. This paper’s focus is model-generated counterfactuals for explaining the LLM itself.

Explainability in ML

Counterfactual explanations in ML

  • Generating valid and plausible counterfactual explanations remains challenging, with prior work also emphasizing on-manifold counterfactuals for plausibility and robustness.Delaney et al. (2023) report that human-generated counterfactuals make larger, more meaningful modifications, while computational methods prioritize minimal edits.Source: §2 · p. 2

Self-explanation (SEs) by LLMs

  • LLM self-explanations such as chain-of-thought and feature attributions may fail to reflect actual decisions, while this paper uses chain-of-thought for evaluation rather than as an explanation.Madsen et al. (2024) also evaluate SCEs; this work systematically studies generation success, varied generation strategies, input distance, and hidden-state differences between valid and invalid SCEs.Source: §2 · pp. 2–3

LLMs for explanations

  • This study explains the LLM itself, unlike work using LLMs to explain other models; some prior approaches provide gold labels without assessing the model’s independent decision or understanding.Jacovi and Goldberg (2020) argue that faithfulness evaluation should not rely on human-provided gold labels.Source: §2 · p. 3

3 Generating and evaluating SCEs

The paper prompts LLMs to generate counterfactual inputs and evaluates whether those inputs produce the intended outputs, alongside generation success and input distance.

3.1 Generating counterfactuals

  • The generation procedure obtains a prediction on the original input, asks the model itself for a counterfactual targeting a different output, and then predicts on that counterfactual.The study avoids the common optimization-based approach; it tests unconstrained, rationale-based, and chain-of-thought prompting strategies.Source: §3.1 · p. 3
  • For the final prediction on an SCE, the model can retain the original prediction and generation instruction in context or receive the SCE without them.These conditions are termed prediction with context and prediction without context.Source: §3.1 · p. 3

3.2 Evaluating CEs

  • The evaluation measures generation percentage, the share of attempts producing an SCE, and counterfactual validity, whether the SCE produces its intended target label.Generation failures include stop-word outputs or counterfactuals much shorter than the original input.Source: §3.2 · p. 4
  • Edit distance measures changes between the original input and SCE; it is reported only for valid SCEs and separately for predictions made with prior steps in context.The distance is normalized by the length of the longer string and multiplied by 100; the in-context measure is denoted EDC.Source: §3.2 · p. 4

4 Experimental setup

The experimental setup draws on datasets from decision-making, sentiment classification, mathematics, and natural language inference. The listed datasets cover decision-making, sentiment, and math tasks.

4.1 Datasets

  • The experiments span four domains: decision-making, sentiment classification, mathematics, and natural language inference.The listed datasets include DiscrimEval and FolkTexts for decision-making, Twitter financial news and SST2 for sentiment, and GSM8K for math.Source: §4.1 · p. 4
  • DiscrimEval contains 70 hypothetical decision scenarios with binary decisions, where yes is always desirable; FolkTexts predicts whether an individual’s yearly income exceeds $50K.DiscrimEval scenarios vary gender, race, and age, which this study fixes to female, white, and age 20.Source: §4.1 · p. 4

6. Multi-Genre Natural Language Inference (MGNLI)

MGNLI uses premise–hypothesis sentence pairs classified as entailment, neutral, or contradiction. The experiments cover several model sizes and types, two temperature settings, and a bounded sample of dataset examples.

6. Multi-Genre Natural Language Inference (MGNLI)

  • MGNLI instances pair a premise with a hypothesis, and the task is to classify their relationship as entailment, neutral, or contradiction.Source: §6 · p. 4

4.2 Models, infrastructure, and parameters

  • The evaluated models include small and medium instruction-tuned models from Gemma, Llama, and Mistral, plus the DeepSeek-R1-Distill-Qwen-32B reasoning model.Source: §4.2 · p. 4
  • Experiments used T = 0 and T = 0.5; unconstrained and rationale-based prompting at T = 0.5 had five trials, while CoT at T = 0.5 had three.Source: §4.2 · p. 5
  • The study sampled the first 250 examples per dataset, or per class for classification datasets, due to inference cost.Source: §4.2 · p. 5

5 Results

The results show that most models generate SCEs, but whether those explanations yield the target label varies substantially across tasks, models, and prompting conditions. Validity, edit distance, and length patterns also depend on context and generation settings.

RQ1: Ability of LLMs to generate SCEs

  • Most models generate SCEs in the vast majority of cases, except GEMs on DISCRIMEVAL and FOLKTEXTS; CoT and T = 0.5 improve generation for GEMs and most models, respectively.Source: RQ1: Ability of LLMs to generate SCEs · p. 5

RQ2: Do SCEs yield the target label?

RQ3: Changes required to generate SCEs

6 Characterization of Failure Cases

The failure-case analyses compare model-assessed validity with human-assessed correctness and test whether readability or embedding representations distinguish SCE outcomes. Validity and correctness align in some conditions, but the measures do not consistently track one another.

  • The human annotation evaluates whether each SCE actually evaluates to its target label, while readability and embedding analyses provide complementary automatic measures.Source: §6 · p. 7
  • On 50 randomly selected GSM8K samples per model, validity correlates significantly with human-rated correctness only without context by Spearman correlation (ρ = 0.76, p = 0.05); with context, neither correlation is significant.Source: §6 · p. 7
  • Readability shows no significant relationship with correctness or validity, with or without context.Source: §6 · pp. 7–8
  • Embedding drift has no correlation with correctness; for validity, it has no effect with context, while without context Pearson correlation is significant (r = 0.76, p = 0.05) but Spearman correlation is not.Source: §6 · p. 8
  • The study uses cosine distance between problem and SCE embeddings to measure drift, and applies k-means with k = 2 to probe whether valid and invalid SCE representations separate.Source: §6 · p. 8

7 Why do models struggle with SCEs?

The authors consider why LLMs struggle to produce valid SCEs and examine whether standard model-quality measures track this ability. They point to next-token prediction and sensitivity to context as possible factors, while finding no consistent alignment with model rankings.

  • The authors argue that LLMs’ next-token-prediction training differs from human counterfactual reasoning, and propose contrastive training examples pairing correct and incorrect SCEs as a possible improvement.Source: §7 · pp. 8–9
  • The authors suggest that context can influence apparent reasoning: validity improves with original predictions and counterfactual instructions present, especially on GSM8K.Source: §7 · p. 9
  • Model size, few-shot perplexity, and leaderboard rank show no strong or consistent correlation with SCE validity; weaker-ranked models sometimes match or exceed stronger-ranked ones.Source: §7 · p. 9
  • The authors conclude that leaderboard rank alone does not reflect a model’s counterfactual reasoning ability.Source: §7 · p. 9

8 Conclusion and future work

The authors conclude that LLMs struggle to produce valid self-generated counterfactual explanations, and their predictions often fail to match the intended target. They also report that context affects predictions and limit their experiments to relatively simple tasks.

8 Conclusion and future work

  • LLMs consistently struggle to generate valid SCEs, and predictions on them often miss the target label the explanation was crafted to achieve.The authors argue that LLMs remain far from explaining their own predictions counterfactually.Source: §8 · p. 9
  • The model’s prediction on an SCE is significantly influenced by whether the original prediction and SCE-generation instructions remain in context.The authors suggest exploring counterfactual explanation capabilities as part of LLM training.Source: §8 · p. 9
  • Experiments cover relatively simple classification and integer-answer mathematics tasks; broader language-generation tasks would require substantial human annotation to assess validity.The authors identify scaling evaluation to such tasks as an important direction for future work.Source: §8 · p. 9

9 Limitations

This section outlines evaluation and deployment limitations, including privacy and strategic-gaming risks, reliance on automated metrics with limited human annotation, and supplementary prompting results at T = 0 after invalid or incomplete generations were excluded.

9 Limitations

  • Even faithful, comprehensive explanations may expose sensitive information or create security risks, although using publicly available models and datasets mitigates these concerns.The authors note explanations could also be exploited for adversarial attacks on the model itself.Source: §9 · pp. 9–10
  • Savvy users could exploit counterfactual explanations to unfairly maximize positive outcomes, so detecting and limiting this behavior is important before real-world deployment (Tsirtsis and Gomez Rodriguez, 2020).The authors identify this as a desideratum for integrating LLM-generated counterfactual explanations into decision-making systems.Source: §9 · p. 10
  • The study relied primarily on automated SCE-quality metrics and small-scale human annotation for only one task, leaving comprehensive evaluation of explanation plausibility for future work.The authors suggest human feedback could help improve performance, for example through direct preference optimization (Rafailov et al., 2024).Source: §9 · p. 10

C Postprocessing model outputs

  • Postprocessing removes rationale-based cases when the model cannot identify rationales, because those SCEs do not meet the generation instruction.It also normalizes answers, filters invalid answers and empirically defined short generations, and removes answer tags from SCEs when they bias predictions.Source: C Postprocessing model outputs · pp. 18–19

D Additional results for various prompting strategies

  • At T = 0, mean accuracy was 68% with CoT and 66% with unconstrained and rationale-based prompting, but the difference was not statistically significant.CoT had slightly lower variance, but a Wilcoxon signed-rank test found no consistent accuracy improvement across datasets and models.Source: D Additional results for various… · p. 20

E Statistical Analysis of Results

The section describes how confidence intervals were computed and interpreted when comparing experimental conditions.

E Statistical Analysis of Results

  • The authors compute 95% confidence intervals using the standard error of the mean to assess differences between conditions.Non-overlapping intervals indicate differences beyond what is expected from random variation and usually correspond roughly to p < 0.05.Source: E Statistical Analysis of Results · p. 20
  • The interval calculation uses the mean, standard deviation, and sample count; 1.96 corresponds to a 95% confidence level under a normal distribution.Source: E Statistical Analysis of Results · p. 20

F Correlation between validity and popular performance metrics

The analyses find no clear, consistent relationship between SCE validity and popular model-performance measures, with model-size trends varying by dataset. A GSM8K annotation study used two independent annotators to assess whether SCEs yielded the correct target label.

F Correlation between validity and popular performance metrics

G Annotation Protocol

  • Two annotators independently judged 350 GSM8K SCEs generated under CoT prompting at T = 0, resolving roughly 5% disagreements through discussion.Consensus labels were used to compute correlations between validity and correctness.Source: G Annotation Protocol · pp. 21–22

H Clustering of SCE Representations: Methodology and Results

The clustering analysis finds that normalized cosine distance gives the highest average separation score, while separation varies across datasets. Permutation tests detect more significant context-condition differences than confidence-interval overlap alone, and bootstrap intervals avoid a normality assumption.

H Clustering of SCE Representations: Methodology and Results

  • The analysis clustered representations from the first generated token, last generated token, or last input token using raw or normalized Euclidean or cosine distance.Normalization standardized last-layer hidden-state vectors to zero mean and unit variance.Source: H Clustering of SCE Representations:… · pp. 23–24
  • The average separation score summarizes the absolute differences between valid and invalid cases in clusters 0 and 1 across evaluated model–dataset pairs.Source: H Clustering of SCE Representations:… · pp. 24–25
  • 178.9 was the highest average separation score for normalized cosine distance, versus 176.5 for raw Cosine, 175.7 for normalized Euclidean, and 175.2 for raw Euclidean.The scores differed little overall; normalized cosine distance was adopted as the primary metric.Source: H Clustering of SCE Representations:… · p. 25
  • GSM8K showed consistently low separation, whereas TWITTER and SST2 had higher separation between valid and invalid SCE cases.Larger ∆ values indicate stronger separation.Source: H Clustering of SCE Representations:… · p. 25

I Statistical Significance via Permutation Testing

  • Permutation testing detected more statistically significant context-condition differences in validity than confidence-interval overlap alone, with effect magnitudes varying across datasets and prompting strategies.The tests compared with-context and without-context conditions under unconstrained and rationale-based prompting at T = 0.Source: I Statistical Significance via… · p. 26

J Bootstrap Confidence Intervals

  • The study used 10,000 bootstrap resamples to estimate 95% confidence intervals for normalized differences in counterfactual lengths without assuming normality.The percentile intervals can accommodate asymmetry.Source: J Bootstrap Confidence Intervals · p. 26

Related papers

Loading 2502.18156v2…