Are self-explanations from Large Language Models faithful?

Counterfactual explanation

• The authors propose measuring interpretability-faithfulness with inference-API-only self-consistency checks, addressing inaccessible ground truth, free-form explanations, and impractical confidence-based metrics.They limit explanations to those verifiable by self-consistency checks and carefully prompt explanation and classification generation.Source: Counterfactual explanation · p. 2
• Self-consistency re-evaluates predictions after an explanation-guided change: a counterfactual is faithful if the edited input produces the requested opposite prediction.The explanation and check must occur in separate sessions.Source: Counterfactual explanation · pp. 1–2
• The study evaluates counterfactual, feature-attribution, and redaction explanations across sentiment, multi-choice, and two-paragraph classification datasets, using Llama2, Falcon, and Mistral models.The models include Llama2 70B and 7B, Falcon 40B and 7B, and Mistral 7B; datasets include IMDB, bAbI, MCTest, and RTE.Source: Counterfactual explanation · p. 2
• Faithfulness varies by model, explanation, and task: for Llama2 70B, counterfactuals work only on IMDB, while feature attribution works only on RTE and bAbI.The findings are robust to prompt variations.Source: Counterfactual explanation · p. 2

1 Introduction

• Convincing but wrong self-explanations can create unsupported confidence, and the authors conclude they cannot generally be trusted.The concern arises because LLMs can hallucinate and self-explanations may not reflect the model’s reasoning.Source: Counterfactual explanation · pp. 1–2

2 Self-explanations

• Self-explanations are new because the predictive LLM itself generates explanations, whereas earlier interpretability methods used a separate model or algorithm (Madsen et al., 2022b).This development creates new challenges and opportunities (Singh et al., 2024).Source: §2 · p. 2
• Earlier counterfactuals could come from models largely independent of the classifier, including systems fine-tuned on human explanations, so they may not represent classifier behavior.The authors note that humans lack insight into the classifier’s behavior, weakening the basis for trusting such counterfactuals.Source: §2 · pp. 2–3
• The authors use the same language model for classification and counterfactual generation, enabling explanation generation to access or simulate classification behavior through shared weights.This is linked to self-model capabilities; Kadavath et al. (2022) found poor self-modeling capabilities in LLMs using a self-consistency check.Source: §2 · p. 3
• The study focuses on faithfulness evaluation, since self-model capabilities may be helpful but are not necessary if the LLM reproduces human behavior consistently in classification and explanation.Under that condition, explanations could be self-consistent and faithful, so the authors limit their scope rather than re-exploring self-model capabilities.Source: §2 · p. 3

3.1 Self-consistency check

• A counterfactual is faithful if reevaluation of its modified input yields the intended opposite label, using a new chat session to avoid history-driven answers.The authors call this reevaluation a self-consistency check.Source: §3.1 · p. 3

3.2 Negative results and robustness

• A failed check is inconclusive about whether faithful explanations are possible, but one non-faithful pattern among six prompts is considered problematic for robustness.The six patterns assess whether explanations remain faithful under natural prompt variations and minor sampling differences relevant to end users.Source: §3.2 · p. 3

4 Prompt-patterns for self-explanation and self-consistency checks

• Prompts are illustrated with sentiment classification but are intended to generalize to multi-choice Q&A and multi-paragraph classification, using chat-tuned framing or proper framing for instruction-tuned models.The authors note that the prompts convey a one-time input-output relation and cannot prove that a model understands user intent; they prioritize user intent when evaluating potential deployment impact.Source: §4 · pp. 3–4

4.1 Counterfactual explanation

• Counterfactual prompts request an edit producing the opposite sentiment, with explicit and implicit target-sentiment variants; extra explanations are excluded because they cannot be validated.The implicit variant replaces “is {opposite sentiment}” with “becomes the opposite of what it currently is”; any unsolicited explanation is removed.Source: §4.1 · p. 4

4.2 Feature attribution explanation

• Feature-attribution prompts elicit important words, which are then replaced with “[REDACTED]”; faithfulness is assessed by whether the model can still classify the modified input.This check assumes the model understands masking during classification; the explanation is unfaithful if classification remains possible.Source: §4.2 · p. 4

4.3 Redaction explanation

• Redaction prompts ask the model to replace important content itself, optionally using “[REMOVED]” instead of “[REDACTED]”; faithfulness means the redacted paragraph cannot be classified.The authors suggest built-in copying mechanisms may make this easier for LLMs; unconstrained whole-paragraph redaction would be faithful but not useful to humans, and was not observed in practice.Source: §4.3 · pp. 4–5

4.4 Classification

• Classification compares the original paragraph with the explanation paragraph, or the modified paragraph for feature attribution, and supports “[REDACTED]” tokens and an “unknown” prediction when information is missing.The prompt is designed to avoid ungrammatical inputs from deletion and resulting out-of-distribution issues, while remaining robust to redaction tokens.Source: §4.4 · p. 5, Figure 5 · p. 5
• The same classification template is used before and after every explanation, retaining redaction and unknown-class support even for counterfactuals to ensure consistency and comparability.Those features are unnecessary for counterfactuals but remain in the prompt across cases.Source: §4.4 · p. 5

5 Results

• Experiments span IMDB sentiment classification, bAbI-1 and MCTest multi-choice QA, and RTE entailment, with explanations modifying the main paragraph rather than questions, choices, or hypotheses.The authors state that this task variation demonstrates generalization to paragraph-based multi-choice and two-paragraph tasks.Source: §5 · p. 5

5.1 Classification

• For Llama2-70B, classification accuracy is not affected much by persona or redaction-instruction; under default prompt settings, performance depends strongly on model type and size, and Falcon 40B performs slightly worse than Falcon 7B.Source: Figure 6 · p. 6, §5.1 · p. 6, Figure 7 · p. 6

5.2 Faithfulness

• Faithfulness is the ratio of faithful observations, measured only on correctly classified, meaningful outputs; incorrect predictions are discarded.This avoids treating an explanation as faithful when the input may not have changed despite a classification error.Source: §5.2 · p. 6, §5.1 · p. 6

5.2.2 Feature attribution

• Llama2-70B is faithful only on RTE and bAbI-1; Llama2-7B and Falcon-7B perform very poorly in faithfulness, while Mistral (7B) drastically outperforms every other model at identifying important words for sentiment classification.The authors note that bAbI-1 is easy because, from a human perspective, only two words—a location and an individual—are relevant to answering its question.Source: §5.2.2 · p. 7

5.2.3 Redaction

• Redaction faithfulness resembles feature attribution, including near-identical prompt-variation results for Llama2-70B, supporting that model-generated masking does not create out-of-distribution inputs.Falcon 40B performs better on IMDB and MCTest but worse than Falcon 7B on bAbI-1.Source: §5.2.3 · p. 7

5.2.1 Counterfactual

• Counterfactual faithfulness is robust to persona and target phrasing except on multi-choice tasks, while Llama2 and Mistral are consistently faithful only on IMDB and Falcon performs poorly.Implicit targets allow multiple correct answers on bAbI-1 and MCTest, unlike explicit targets.Source: §5.2.1 · p. 6, Figure 8 · p. 7

5.2 Faithfulness

• Faithfulness generally increases with model size except for Falcon, and depends heavily on model type, task, and explanation type.The reported size exception is particularly evident for Falcon in the counterfactual case.Source: Figure 9 · p. 7

6 Related work

• Earlier feature-attribution checks removed allegedly important input information and measured prediction changes, often using external explanations and logits rather than discrete classifications (Samek et al., 2017; Hooker et al., 2019; Madsen et al., 2022a).Logit differences before and after removal offered a more nuanced faithfulness measure.Source: §6 · pp. 7–8
• Huang et al. (2023) found ChatGPT confidence and word-importance scores uncalibrated, and that it resisted sentiment classification when information was missing, motivating better LLM self-explanation metrics.Their study used a 100-observation SST subset and asked ChatGPT for confidence and word-importance scores.Source: §6 · p. 8
• This paper avoids confidence and importance scores by using discrete classification with an “unknown” option, addressing issues identified by Huang et al. (2023).Source: §6 · p. 8
• Lanham et al. (2023) tested mathematical Chain-of-Thought explanations by inserting false facts and checking whether predictions followed, but such edits may be out-of-distribution or mistaken for typos.They, like this paper, found faithfulness to be model- and task-dependent.Source: §6 · p. 8

6.1 Non-faithfulness works

• Self-consistency checks have also probed LLM self-modeling, mathematical-question judgments, and meta-linguistic capabilities (Kadavath et al., 2022; Li et al., 2023; Hu and Levy, 2023).Source: §6.1 · p. 8
• Counterfactual self-explanations can be convincing to humans without correlating with simulatability, while faithfulness avoids the challenge of separating explanation use from world knowledge (Chen et al., 2023; Agarwal et al., 2024).Source: §6.1 · p. 8
• Rationalizations from humans or models trained on human explanations are studied mainly for plausibility, with some simulatability and faithfulness evaluations; Parcalabescu and Frank (2023) argue the latter measure consistency, not faithfulness.Rationalizations should not be treated as explanations of the model (Jacovi and Goldberg, 2020).Source: §6.1 · p. 8

7 Conclusion

• Self-explanation faithfulness is highly model- and dataset-dependent, and the paper measures it for counterfactual, feature-attribution, and redaction explanations.The authors note that this conclusion is similar to previous work (Lanham et al., 2023; Madsen et al., 2022a; Bastings et al., 2022).Source: §7 · p. 8
• Because faithfulness depends on the task, LLM self-explanations cannot generally be trusted, increasing the risk that people mistakenly believe models can explain themselves (Chen et al., 2023).The authors also caution against trusting free-form explanations when self-consistency checks cannot evaluate their faithfulness.Source: §7 · p. 8

7.1 Future work

• Future work developing instruction-tuned LLMs should evaluate self-explanation faithfulness with this paper’s methodology to help reduce LLM-related risk.The authors argue this evaluation is a necessary direction.Source: §7.1 · p. 9
• Fine-tuning may improve self-explanations, since Kadavath et al. (2022) improved self-modeling, but their model was unavailable for evaluation.The authors suggest self-modeling may also improve self-explanation capabilities, but could not test this possibility.Source: §7.1 · p. 9
• The authors propose treating self-explanation faithfulness as an out-of-domain evaluation problem, such as testing whether counterfactual-faithfulness tuning also improves feature attribution and redaction.This could provide evidence about general explanation capabilities, including free-form explanations that are harder to evaluate with self-consistency checks.Source: §7.1 · p. 9

Absolute faithfulness

• Discrete token-sequence outputs prevented graded faithfulness measurement, although single-observation absolute scores were averaged for a nuanced model-level view.The authors agree with the view that faithfulness should be measured by degree rather than as a binary property.Source: Absolute faithfulness · p. 9

Assuming the prompt is interpreted correctly

• Prompt interpretation cannot be completely verified despite natural-sounding prompts, and comprehension failures may explain unfaithful explanations or inaccurate classifications.At least one dataset showed high classification accuracy or explanation-faithfulness, which the authors say indicates sufficient prompt comprehension but inability to perform the requested task.Source: Assuming the prompt is interpreted… · p. 9

No in-context learning

• The study used zero-shot explanations because in-context learning would require known true explanations that humans cannot provide.Source: No in-context learning · p. 9

Poor classification performance creates challenging comparison

• Incorrect classification observations were discarded, making comparisons harder and potentially introducing class imbalance.The models often had low classification accuracy, but the study focused on interpretability-faithfulness rather than accuracy.Source: Poor classification performance… · p. 9

Impossible to show lack of capability

• Unfaithful explanations for specific prompt templates do not prove models are fundamentally incapable of explaining themselves.Because explanations can be convincing despite being unfaithful, the authors warn users not to trust them and raise an AI safety concern, while noting the evidence is not definitive.Source: Impossible to show lack of capability · p. 9

We provide interpretability-faithfulness results on 3

• The study covers only counterfactuals, importance measures, and redaction, with importance and redaction providing similar explanations.The authors say these are currently the only explanation types measurable with self-consistency checks and hope future work finds new checks.Source: We provide… · pp. 9–10

7.2 Not measuring on ChatGPT, GPT-4

• ChatGPT, GPT-4, and other popular closed models were excluded because frequent updates prevent reproducible results.The authors also cite unrestricted indemnity clauses and personal financial-liability risk, concluding that a truly unbiased analysis could not be provided under those conditions.Source: §7.2 · p. 10

Ethics Statement

• The authors warn that methodological mistakes could cause unsupported confidence or skepticism, but argue the paper’s message discourages assuming LLM explanations are faithful.They describe unsupported confidence as potentially dangerous, while saying additional skepticism is unlikely to cause ethical issues and the paper itself is unlikely to create unsupported confidence.Source: Ethics Statement · p. 10
• The authors selected models whose licenses do not restrict their commentary, which they say supports confidence that their views are unbiased.They contrast this choice with ChatGPT and GPT4, whose license indemnity clause could make them financially liable for damages.Source: Ethics Statement · p. 10

Oana-Maria Camburu, Tim Rocktäschel, Thomas

• The bibliography includes work on natural-language explanations and counterfactual simulatability, including e-SNLI and studies asking whether models explain themselves.Source: Oana-Maria Camburu, Tim Rocktäschel,… · p. 10, Shiyuan Huang, Siddarth Mamidanna,… · p. 11

Shiyuan Huang, Siddarth Mamidanna, Shreedhar

• It also lists research on defining and evaluating faithfulness, measuring chain-of-thought faithfulness, and recursively masking important tokens.Source: Shiyuan Huang, Siddarth Mamidanna,… · p. 11, Tamera Lanham, Anna Chen, Ansh… · p. 11, Andreas Madsen, Nicholas Meade,… · p. 12

Andreas Madsen, Nicholas Meade, Vaibhav Adlakha,

• Other entries cover post-hoc interpretability, generator-validator consistency, and self-explanations generated by large language models.Source: Tamera Lanham, Anna Chen, Ansh… · p. 11, Shiyuan Huang, Siddarth Mamidanna,… · p. 11, Andreas Madsen, Nicholas Meade,… · p. 12

• The public datasets were selected to provide diversity in task, input, and target representation, including sentiment classification and multiple-choice question answering.Source: A Datasets · p. 13
• Prompts were developed using training splits, while all reported results use test splits; the authors had no reason to suspect the split affected results.Source: A Datasets · p. 13

B Models

• The analysis covers Llama 2, Falcon, and Mistral, selected from publicly available models without an indemnity clause.The models are listed in Table 2.Source: B Models · p. 13
• Although the methods could apply to any instruction-tuned generative model, including API-only models, the study excludes fine-tuned derivatives to avoid extra computing costs and because they were unlikely to add valuable insights.Source: B Models · p. 13

C Experimental setup

• Generation used TGI 1.1.0 with HuggingFace chat parameters, seed 0, and a single inference sample.The authors cite response stability and inference expense as reasons for using one sample.Source: C Experimental setup · p. 13, Table 3 · p. 14
• Llama2 and Mistral used no system prompt, whereas Falcon used its default system prompt.Source: C Experimental setup · p. 13
• The classification prompt was reused across each explanation variation.Source: C Experimental setup · p. 14

D Compute

• The compute section refers readers to Table 5 for hardware specifications and states that the electricity is 99% hydroelectric.Table 5 reports only allocated resources.Source: D Compute · p. 14, Table 5 · p. 14
• Inference time is reported as TGI time, not wall-time; 50 prompts run in parallel, so dividing by 50 is a decent wall-time approximation.Previously computed prompts are computed only once.Source: D Compute · p. 14, Table 4 · p. 15

E Prompt examples

• The prompt examples use outputs from Llama2-70B chat-tuned, with simple-heuristic extraction manually verified for every observation.Extracted content includes predicted labels and explanations.Source: E Prompt examples · p. 16

E.1.1 Counterfactual

• Counterfactual sentiment examples are marked faithful across objective, human, and you personas and explicit and implicit targets.Each example has a correct initial prediction and a true label of negative.Source: Figure 10 · p. 16, Figure 11 · p. 17, Figure 12 · p. 17

E.1.2 Feature attribution

• Feature-attribution sentiment examples are marked not faithful across objective, human, and you personas with both “redacted” and “removed” instructions.The examples show a correct initial prediction and a true label of negative.Source: Figure 16 · p. 19, Figure 17 · p. 20, Figure 18 · p. 20

E.1.3 Redacted

• Redaction sentiment examples are marked not faithful across objective, human, and you personas with both “redacted” and “removed” instructions.The examples show a correct initial prediction and a true label of negative.Source: Figure 22 · p. 22, Figure 23 · p. 23, Figure 24 · p. 23

E.2.1 Counterfactual

• The objective-persona, explicit-target counterfactual example for the question “Where is Mary?” is marked not faithful.The true label is “office,” and the initial prediction was correct.Source: Figure 28 · p. 25
• Counterfactual examples were not faithful for both explicit-target configurations and implicit-target “objective” and “you” configurations, but were faithful for implicit-target “human.”The explicit-target examples use both human and you personas; the implicit-target examples vary across objective, human, and you personas.Source: Figure 29 · p. 26, Figure 30 · p. 26, Figure 32 · p. 27

E.2.2 Feature attribution

• Feature-attribution examples were faithful across objective, human, and you personas with both “redacted” and “removed” redaction instructions.Figures 34–39 report faithful evaluations for all six shown configurations.Source: Figure 34 · p. 28, Figure 35 · p. 29, Figure 36 · p. 29

E.2.3 Redacted

• Redaction examples using the “redacted” instruction were not faithful for objective, human, and you personas.Each example’s initial prediction was correct and its true label was “office.”Source: Figure 40 · p. 31, Figure 41 · p. 32, Figure 42 · p. 32
• Mary-location redaction examples were judged not faithful with objective, human, and you personas, although the prompt requested removing answer-critical words.Source: Figure 43 · p. 33, Figure 44 · p. 33, Figure 45 · p. 34

E.3.1 Counterfactual

• For Hotmail entailment, explicit counterfactuals were faithful only with the human persona, while implicit counterfactuals were not faithful with any of the three personas.Source: Figure 46 · p. 34, Figure 47 · p. 35, Figure 48 · p. 35

E.3.2 Feature attribution

• For Hotmail entailment, feature-attribution redactions were judged faithful with all three personas using [REDACTED] and with objective and human personas using [REMOVED].Source: Figure 52 · p. 37, Figure 53 · p. 38, Figure 54 · p. 38
• The feature-attribution consistency check evaluated the correct “yes” prediction as faithful.The example asks for important words and tests the prediction after words are marked [REMOVED].Source: Figure 57 · p. 40, §E.3.2 · p. 40

E.3.3 Redacted

• Across six redaction configurations, the correct “yes” prediction was evaluated as faithful.The examples vary persona instructions (objective, human, or you) and use either “redacted” or “removed” markers.Source: Figure 58 · p. 40, Figure 59 · p. 41, Figure 60 · p. 41