Source-linked AI summary
An Empirical Study of Counterfactual Self-Explanations in LLMs
Giannis Kalyvas, Giorgos Filandrianos, Orfeas Menis Mastromichalakis, Vassilis Lyberatos, Giorgos Stamou
TL;DR
LLM self-explanations may sound convincing without faithfully reflecting model behavior, so this paper tests them through prediction-flipping counterfactual edits. Across two tasks and ten models, scale most strongly predicts faithfulness, while rationale guidance improves minimality and human alignment without consistently improving faithfulness.
Problem
The paper asks when fluent LLM self-explanations can be treated as reliable evidence about the factors determining the model’s predictions.
Method
The study evaluates counterfactual self-explanations across sentiment analysis and natural language inference using ten LLaMA-3 and Qwen-2.5 models and measures faithfulness, minimality, and rationale alignment.
Results
Model scale is the strongest determinant of explanation quality: larger models more consistently flip their own predictions and target evidence aligned with human rationales, whereas rationale guidance improves minimality and human alignment without consistently improving faithfulness.
Takeaways & Limitations
Counterfactual self-explanations can provide useful behavioral evidence, but their reliability depends strongly on model capacity and prompting strategy.
Takeaways & Limitations
The analysis is restricted to binary classification, limiting generalization to multiple-choice and open-ended generative settings, and its metrics may not capture human understandability or usefulness.
Abstract
from arXiv · showhide
Large language models can easily generate explanations for their own outputs, but such self-explanations are not necessarily faithful to the model's behavior. We study this issue through counterfactual self-explanations, where a model minimally edits an input so that its own prediction changes. Across sentiment analysis and natural language inference, we evaluate ten instruction-tuned models from the LLaMA-3 and Qwen-2.5 families, measuring faithfulness, minimality, and alignment with human-annotated rationales. Our results show that model scale is the strongest determinant of explanation quality: larger models are substantially more likely to generate counterfactuals that flip their own predictions and target decision-relevant evidence. In contrast, the rationale-guided condition produces edit-minimal counterfactuals that are also more human-aligned. However, it does not consistently improve faithfulness. Overall, counterfactual self-explanations can provide useful behavioral evidence about model decisions, but their reliability depends strongly on model capacity and should be empirically validated rather than assumed.
1 Introduction
The paper asks when LLM self-explanations can be trusted as evidence about their own predictions. It studies counterfactual edits across tasks, models, metrics, and prompting strategies to assess faithfulness, minimality, and human alignment.
- Approach: Counterfactual self-explanations ask the same model to minimally edit an input so its own prediction changes.Successful prediction flips provide behavioral evidence, while overlap with human rationales indicates alignment with human reasoning.
- Research Questions: The study examines whether counterfactual self-explanations are faithful, minimal, and aligned with decision-relevant evidence.It focuses on whether edits flip the model’s prediction, modify human-supported evidence, and respond to prompting choices.
- Study Scope: The evaluation spans sentiment analysis and natural language inference using ten instruction-tuned open-weight models from the LLaMA-3 and Qwen-2.5 families.The study measures faithfulness by prediction flips, minimality by closeness to the original input, and human alignment by rationale overlap.
- Evaluation: The paper introduces Evidence-Supported Modification Precision (ESMP) to measure whether edits target human-annotated evidence rather than arbitrary tokens.ESMP complements faithfulness and minimality by evaluating alignment with decision-relevant rationales.
2 Methodology
The methodology generates and reclassifies counterfactuals with the same LLM, then evaluates prediction flips, input closeness, semantic preservation, and alignment with annotated evidence. It compares tasks, model families, and prompting strategies while explicitly treating minimality as empirical edit closeness rather than global optimality.
- Tasks and Data: The study evaluates movie-review sentiment and e-SNLI natural language inference using ERASER test sets and human rationale annotations.The two tasks provide different forms of language understanding for assessing counterfactual explanations.
- Models: Ten instruction-tuned LLaMA-3 and Qwen-2.5 models ranging from 1B to 70B+ are compared within architectural families to isolate scale effects.The families provide broad size ranges, consistent architectures, and public instruction-tuned variants.
- Counterfactual Generation: The pipeline uses the same LLM to label the original input, generate a minimally edited prediction-flipping counterfactual, and reclassify it.These are three sequential stages with role-specific prompts.
- Prompting: Three prompting variants test how guidance affects counterfactuals: a baseline targeting the opposite label, Chat-History, and Rationale-Guided generation.The baseline directly requests a minimally edited counterfactual without additional structure or guidance.
- Metrics: Faithfulness is measured by the flip rate, while minimality is assessed through normalized edit distance, closeness, and MPNet-based semantic similarity.Semantic similarity captures meaning preservation beyond lexical overlap, and flip rate records successful label changes.
- Evidence Alignment: ESMP uses minimal-edit alignment to classify edits inside annotated evidence as true positives and unsupported edits as false positives, reporting precision.Precision is chosen because minimal counterfactuals should concentrate edits on human-supported evidence rather than broadly cover it.
3 Results and Discussion
Results show that model scale most strongly determines counterfactual self-explanation faithfulness and human-aligned targeting, while prompting introduces a faithfulness–minimality trade-off. Generated counterfactuals generally remain close to their inputs, but task structure affects how these properties co-vary.
- Faithfulness: Model size strongly correlates with flip rate (ρ = 0.87), making larger models more likely to produce behaviorally faithful self-explanations.A regression controlling for dataset, model family, and prompting strategy associates each doubling in model size with a 12-percentage-point increase in faithfulness.
- Minimality: Counterfactuals remain close to original inputs across settings, indicating generally localized edits rather than broad rewrites.Closeness and semantic similarity are reported across datasets, model families, model sizes, and prompting strategies.
- Human alignment: Model size strongly correlates with ESMP (ρ = 0.78), indicating that larger models make more targeted edits overlapping human-annotated rationales.Its associations with closeness (ρ = 0.32) and semantic similarity (ρ = 0.16) are much weaker.
- Prompting: Chat-History achieves the best faithfulness score in 12 of 20 settings, whereas Rationale-Guided prompting lowers faithfulness while improving ESMP and closeness.The rationale-guided prompt explicitly steers models toward evidence-like tokens, which can make explanations appear more human-aligned and minimal without better reflecting model behavior.
- Task structure: Task structure changes the relationship among metrics: Movies permits lexical substitutions that improve flip rate, ESMP, and semantic similarity, while e-SNLI requires larger semantic edits.In e-SNLI, larger models improve flip rate and ESMP, but semantic similarity does not increase similarly.
4 Conclusions
The study finds that counterfactual self-explanations offer behavioral evidence about model decisions, but their reliability depends on model scale and prompting. They should therefore be empirically validated rather than treated as direct access to internal reasoning.
- 4 Conclusions: Larger models produce more faithful and human-aligned self-explanations, while rationale-guided prompting can improve minimality and alignment without improving faithfulness.The conclusion characterizes counterfactuals as useful behavioral evidence, not direct access to internal reasoning.
Limitations
The study is limited to binary classification and evaluates explanations with metrics that may not fully capture human understandability or usefulness.
- Limitations: The binary-classification setting limits generalization to multiple-choice and open-ended generative tasks, while faithfulness and minimality may not capture human understandability or usefulness.These metrics are informative but do not necessarily reflect explainability’s usual end goal.
A Model Identifiers
The experiments use locally executed models whose exact identifiers are listed in Table 1.
- A Model Identifiers: Table 1 lists the exact model identifiers used in the experiments.All models were run locally on a dedicated GPU infrastructure rather than through external APIs.
B Prompts
The prompts implement counterfactual self-explanations by first classifying an input, then minimally editing it toward the opposite label, and finally reclassifying the edit.
- B Prompts: Rationale-Guided generation first identifies words causing the original label, then changes only a minimal set of those words.The sentiment example identifies “dull, predictable, failed, engage” before generating a positive counterfactual.
- B Prompts: In the sentiment example, changing negative descriptors produces a paragraph classified as positive.The generated edit changes “dull, predictable, and failed” to positive language, and Stage 3 returns “Positive.”
- B Prompts: In the entailment example, changing “animal” to “cat” yields a paragraph classified as not entailing the target statement.The original paragraph is classified as entailing, while the edited paragraph receives “No.”
C Detailed Analysis
Aggregate and full-result analyses compare faithfulness, semantic similarity, closeness, and ESMP across datasets, model families, sizes, and prompting strategies.
- C Detailed Analysis: Model size has a strong positive association with faithfulness (ρ = 0.87) and ESMP (ρ = 0.78), but weaker associations with closeness (ρ = 0.32) and semantic similarity (ρ = 0.16).The pattern indicates that scaling relates more strongly to faithful and evidence-aligned explanations than to textual preservation.
- C Detailed Analysis: Faithfulness results are evaluated by whether generated self-explanations change the model’s original prediction across datasets, families, and prompting conditions.Table 2 reports the full faithfulness comparison, with higher values indicating more frequent prediction changes.
- C Detailed Analysis: Semantic similarity measures preservation between the original input and generated self-explanation, while closeness measures how near the edit remains to the original.The full results separately report semantic similarity and closeness across settings.
- C Detailed Analysis: Chat-History achieves the highest faithfulness in 12 out of 20 settings, whereas Rationale-Guided achieves the highest ESMP and closeness in 15 out of 20 settings each.Rationale-Guided therefore tends to produce more human-like and minimal edits, but not always the most faithful explanations.
- C Detailed Analysis: ESMP measures alignment between model-edited evidence and human-annotated rationales, adding a human-alignment dimension to the evaluation.The ESMP results are reported across datasets, model families, and prompting conditions.
D Aggregate Statistical Analysis
Aggregate analyses describe how model size and prompting strategy relate to counterfactual explanation metrics across reused datasets, model families, and settings.
- D Aggregate Statistical Analysis: The aggregate analyses provide descriptive evidence over metric values rather than instance-level causal estimates because datasets, model families, and prompting settings are reused across conditions.Interpret the reported associations as aggregate trends, not causal effects for individual examples.
- D Aggregate Statistical Analysis: Each doubling in model size is associated with an 11.9-percentage-point increase in faithfulness and a 3.2-point increase in ESMP.The effects on closeness and semantic similarity are much smaller.
- D Aggregate Statistical Analysis: Each doubling in model size approximately doubles the odds of producing a faithful self-explanation (odds ratio = 2.00, 95% CI [1.68, 2.40], p < 0.001).A complementary fractional-logit model supports the same qualitative conclusion.
- D Aggregate Statistical Analysis: The largest model is substantially more faithful than the smallest across all dataset–family combinations, with ESMP increasing in the same direction.Table 7 averages values over prompting strategies for smallest-to-largest comparisons.
- D Aggregate Statistical Analysis: Chat-History has the highest faithfulness in 12 out of 20 settings, while Rationale-Guided has the highest ESMP and closeness in 15 out of 20 settings each.These aggregate comparisons support a distinction between faithfulness and human-like, minimal editing.