PaperlayerView exact revision on arXiv

Source-linked AI summary

On Measuring Faithfulness or Self-consistency of Natural Language Explanations

Letitia Parcalabescu, Anette Frank

arXiv:2311.07466v4cs.CLcs.AIcs.LG

TL;DR

Plausible LLM explanations may not reflect the reasoning behind predictions, yet current tests cannot establish that correspondence. This paper reframes those tests as measures of output-level self-consistency, compares them on a common platform, and introduces CC-SHAP to compare input contributions to answers and explanations. The analyses find differing test verdicts and higher self-consistency among chat models than base models.

  • Problem

    Because a model’s actual reasoning is usually unknown, current tests cannot establish whether plausible post-hoc or Chain-of-Thought explanations reflect the processes behind predictions.

    Source: §1 · p. 1, §3 · p. 4

  • Method

    The paper compares existing tests in a shared bank and introduces CC-SHAP, which compares input contributions to answer prediction and explanation generation.

    Source: §1 · p. 2

  • Results

    Existing tests give differing verdicts, chat models tend to be more self-consistent than base models, and model size shows no clear effect on self-consistency.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The authors conclude that current tests measure self-consistency rather than faithfulness, leaving true faithfulness metrics an open problem.

    Source: §7 · p. 9, §6 · p. 9

    The study assesses English-only autoregressive LLMs; extending self-consistency analysis to multilingual and multimodal models remains future work.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) can explain their predictions through post-hoc or Chain-of-Thought (CoT) explanations. But an LLM could make up reasonably sounding explanations that are unfaithful to its underlying reasoning. Recent work has designed tests that aim to judge the faithfulness of post-hoc or CoT explanations. In this work we argue that these faithfulness tests do not measure faithfulness to the models' inner workings -- but rather their self-consistency at output level. Our contributions are three-fold: i) We clarify the status of faithfulness tests in view of model explainability, characterising them as self-consistency tests instead. This assessment we underline by ii) constructing a Comparative Consistency Bank for self-consistency tests that for the first time compares existing tests on a common suite of 11 open LLMs and 5 tasks -- including iii) our new self-consistency measure CC-SHAP. CC-SHAP is a fine-grained measure (not a test) of LLM self-consistency. It compares how a model's input contributes to the predicted answer and to generating the explanation. Our fine-grained CC-SHAP metric allows us iii) to compare LLM behaviour when making predictions and to analyse the effect of other consistency tests at a deeper level, which takes us one step further towards measuring faithfulness by bringing us closer to the internals of the model than strictly surface output-oriented tests. Our code is available at \url{https://github.com/Heidelberg-NLP/CC-SHAP}

1 Introduction

LLMs can produce plausible explanations that do not reflect their reasoning, making it difficult to assess whether explanations are faithful. The paper argues that current tests measure output-level self-consistency instead, and introduces a shared comparison platform and CC-SHAP, a finer-grained measure.

1 Introduction

  • Current explanation tests edit inputs and check whether predictions change, assessing output-level self-consistency rather than correspondence with a model’s inner reasoning.The authors argue that this distinction leaves faithfulness more elusive than existing tests can measure.Source: §1 · pp. 1–2
  • The paper introduces the Comparative Consistency Bank to compare existing tests on the same models and data, finding that their predictions differ widely.The authors note that prior studies used different models and data, making comparisons difficult.Source: §1 · pp. 1–2
  • CC-SHAP compares how input contributions align between answer prediction and explanation generation, without editing the input.The paper applies the measure to post-hoc and Chain-of-Thought explanations.Source: §1 · p. 2
  • The authors present CC-SHAP as a step toward faithfulness analysis because it relates input contributions to outputs more directly than surface-level tests.They state that it gives transparency into how inputs and possible modifications influence model outputs.Source: §1 · p. 2

2 Related Work

The related work defines faithful explanations as representations of the model’s decision process, while noting that plausible explanations can still be unfaithful. Existing tests address this difficult-to-observe process through input interventions and other checks.

2.1 What is NLE Faithfulness?

  • A faithful natural-language explanation should accurately describe the model’s decision-making process, but a plausible explanation may not reflect how the model arrived at its prediction.Arbitrary input features may influence explanation generation differently from prediction.Source: §2.1 · p. 2

2.2 Measuring Faithfulness so far

  • Because the model’s actual reasoning is usually unknown, common tests change its input and judge faithfulness from whether its prediction changes.These tests aim to return a binary verdict or a continuous degree of faithfulness.Source: §2.2 · p. 2
  • Counterfactual Edits inserts words that change the input; if the prediction changes and the explanation omits those words, the explanation is judged unfaithful.The test requires a task-specific helper model, and its authors leave semantic evaluation of whether edits are mentioned for future work.Source: §2.2 · p. 2
  • Constructing Inputs from Explanations uses the generated explanation to build a new input and judges it unfaithful if the prediction changes.Its rationale is that reasons in a faithful explanation should suffice to reproduce the original prediction when used as input.Source: §2.2 · p. 2

CC-SHAP

The related work reviews tests that infer explanation faithfulness from input perturbations, feature importance, or changes to Chain-of-Thought reasoning. It also discusses approaches that alter prediction generation and interpretability methods that assign input importance values.

CC-SHAP

  • Explanation-based input reconstruction uses hand-crafted rules that do not transfer across datasets, with detected unfaithful instances differing substantially between e-SNLI and ComVE.The rules are not applicable to CoS-E, and the reported rates reach up to 14% for e-SNLI versus up to 40% for ComVE.Source: CC-SHAP · p. 3
  • Noise and feature-importance equivalence treats similar robustness to noise and overlapping important tokens for predictions and rationales as necessary, but not sufficient, properties of faithfulness.The authors note that loosely defined thresholds and poorly motivated, unabated design choices may affect the reported test outcomes.Source: CC-SHAP · pp. 3–4
  • Biasing Features tests whether an explanation verbalizes a biasing input when that input changes the answer, but the test may not establish whether the model recognized the bias.The method also requires semantic analysis of whether the explanation mentions the bias.Source: CC-SHAP · p. 4
  • Corrupting CoT tests alter a chain of thought by truncating, adding mistakes, paraphrasing, or replacing it with filler tokens.The approach assumes the model needs the CoT to answer correctly, although CoT only marginally improves performance, leaving reliance on the CoT versus the question unresolved.Source: CC-SHAP · p. 4

2.3 Increasing Faithfulness

  • Some approaches seek to make predictions more likely to be faithful by construction, but do not explicitly measure explanation faithfulness; interpretability methods assign numerical importance values to inputs for predictions and explanations.Perturbation methods such as SHAP estimate importance by masking input parts and measuring their effect on outputs.Source: §2.3 · p. 4, §2.4 · p. 4

3 Consistency is all we get (so far)

The paper argues that existing tests check whether model outputs remain self-consistent under designed inputs, not whether explanations correspond to internal processes. Self-consistency is necessary but insufficient for faithfulness, which the authors consider unresolved without further access to model internals.

3 Consistency is all we get (so far)

  • Existing tests design special inputs and check output self-consistency rather than investigating correspondence between explanations and the internal processes behind predictions.The paper frames this as a gap between testing outputs and examining the model’s internal processes.Source: §3 · p. 4
  • Self-consistency is necessary but not sufficient for faithfulness, since answer and explanation generation may involve different model pathways.Surface consistency could conceal diverging internal pathways, while shared parameters could still produce contradictory answers.Source: §3 · pp. 4–5
  • The authors argue that judging faithfulness requires looking inside the model, yet it remains unclear how much divergence between answer and explanation parameters is compatible with faithfulness.They treat faithfulness evaluation as an open problem and do not claim current consistency tests evaluate it.Source: §3 · p. 5

4 CC-SHAP: New SHAP Contribution Consistency Metric

The paper argues that existing faithfulness tests measure output-level self-consistency, then introduces CC-SHAP to compare input-token contributions to an answer and its explanation. This comparison is proposed as a necessary condition for faithfulness, not a direct measurement of faithfulness itself.

4 CC-SHAP: New SHAP Contribution Consistency Metric

  • CC-SHAP compares each input token’s contribution to answer prediction with its contribution to the generated explanation as an input-level self-consistency measure.The authors propose similar contributions as a necessary condition for a faithful explanation.Source: §4 · p. 5

4.1 CC-SHAP Method

  • SHAP estimates a token’s contribution by evaluating predictions on subsets of input tokens; Monte Carlo sampling approximates the values as coalition counts grow exponentially.The Shapley value weights a token’s change in prediction across subsets, with a normalizing factor over possible subset choices.Source: §4.1 · p. 5
  • CC-SHAP normalizes input-token contributions for comparability, then measures convergence between the prediction and explanation contribution distributions using cosine distance.Negative contributions remain negative ratios after normalization.Source: §4.1 · p. 6
  • For multi-token outputs, CC-SHAP computes each input token’s contribution ratio for every output token, then averages those ratios across the sequence.The method extends SHAP to provide one set of input contributions for predictions longer than one token.Source: §4.1 · pp. 5–6

4.2 Advantages of SHAP Consistency

  • CC-SHAP provides continuous per-instance scores and token-level attribution, applies to post-hoc and CoT explanations, and needs no annotated data, edited inputs, or model training.It requires more compute than some other tests.Source: §4.2 · p. 6

5 Comparative Consistency Bank (CCB)

The Comparative Consistency Bank evaluates existing self-consistency tests on common models and tasks, addressing comparisons previously hindered by different evaluation setups. The results show substantial disagreement among tests, while CC-SHAP provides token-level comparisons of answer and explanation contributions.

5.1 Motivation

  • The bank addresses the lack of a common evaluation setting for existing tests, enabling comparison of which tests agree and disagree.Prior studies used different models and data, and some used undisclosed models or did not release code.Source: §5.1 · p. 7

5.2 Tests, Models and Data

  • The bank runs eight existing tests and CC-SHAP across 11 autoregressive LLMs and five tasks, using 100 samples per task.Tasks include e-SNLI, ComVE, disambiguation QA, causal judgement, and BBH logical deduction; CC-SHAP covers post-hoc and CoT explanations.Source: §5.2 · pp. 7–8

5.3 Results

  • CC-SHAP scores are typically negative for LLaMA 2 and Mistral on e-SNLI and three BBH tasks, while chat models score higher and Falcon shows no clear chat-versus-base trend.ComVE is the exception to the low-score pattern for LLaMA 2 and Mistral.Source: §5.3 · p. 8
  • Existing tests diverge sharply: LLaMA 2-7b scores range from 1% to 65% on e-SNLI, and weaker GPT2 receives 0% or 100% faithfulness scores.GPT2 is insensitive to the tests’ token insertions; scores generally favor chat over base models across tasks.Source: §5.3 · p. 8
  • Aggregated results rank LLaMA2-7b and LLaMA-13b-chat as most self-consistent and Falcon-7b as least, but aggregation spans very different tests and tasks.Across model sizes of 7–13–40B parameters, accuracy rises but self-consistency shows no trend.Source: §5.3 · p. 8
  • For all models except GPT2, counterfactual edits leave answer contributions similar but change explanation contributions; GPT2’s contributions remain similar for both.The analysis uses CC-SHAP on samples with and without the insertion of “outside.”Source: §5.3 · p. 8

6 Discussion and Takeaways

The authors argue that existing faithfulness tests measure output self-consistency, and that explanations should be analyzed in relation to answers. CC-SHAP offers a more fine-grained view, but does not establish faithfulness to internal reasoning.

Discussion and Takeaways

  • LLaMA 2 and Mistral chat models are more self-consistent than their base models, while tests still yield diverse results across models and datasets.The tests differ in design and measure output self-consistency rather than faithfulness.Source: §6 · p. 8
  • CC-SHAP reveals that strong models change input contributions for NLE generation but not answer generation, a distinction most tests overlook.Other tests generally check whether edits are mentioned verbatim, rather than analyzing NLE contributions.Source: §6 · p. 8
  • CC-SHAP measures self-consistency, not faithfulness, but provides continuous per-instance scores without input edits and analyzes token contributions to outputs.The authors argue this brings measurement closer to model internals than recording softmax output, while faithfulness remains an open question.Source: §6 · p. 9
  • The authors describe faithfulness as an open research question and suggest mechanistic interpretability as a direction for studying LLM internals.They distinguish CC-SHAP’s use of model probabilities to infer input-token contributions from work comparing output probability distributions before and after edits.Source: §6 · p. 9

7 Conclusion

The conclusion frames existing faithfulness tests as output-level self-consistency measures and presents CCB and CC-SHAP as steps toward deeper analysis. It reports greater self-consistency among chat models, no clear size effect, and the need to relate explanations to answers.

Conclusion

  • Existing post-hoc and CoT faithfulness tests assess output-level self-consistency rather than model inner workings.Source: §7 · p. 9
  • CCB compares tests on common models and tasks, while CC-SHAP records token-level contributions as an interpretable self-consistency measure.Source: §7 · p. 9
  • Chat models tend to be more self-consistent than base models, model size has no clear effect, and explanations should be analyzed relative to answers.The authors hope CCB encourages further study of consistency behavior across model types, tasks, and sample properties.Source: §7 · p. 9

Limitations

The study is limited to English-only autoregressive LLMs, and CC-SHAP requires substantial computation. The authors also report that tests were not repeatedly run across every model and task, and that human judgment cannot establish faithfulness.

Limitations

  • The evaluation covers English-only autoregressive LLMs; multilingual and multimodal models remain outside this study’s scope.The authors report extending related work to vision-language models separately.Source: Limitations · p. 9
  • CC-SHAP takes around 4 minutes per example, exceeding the runtime of some tests that need only two model inferences.The authors compare its runtime with paraphrasing, which takes about 3 minutes per sample.Source: Limitations · p. 10
  • Repeated evaluation across every test, task, and model was too computationally costly, so standard deviations were estimated on ComVE for seven models only.Each test was run three times on 100 ComVE examples for that subset.Source: Limitations · p. 10
  • The authors state that a human study of faithfulness is impossible given current understanding of how LLM parameters interact to produce predictions.Human plausibility judgments cannot determine whether an explanation reflects a model’s inner workings.Source: Limitations · p. 10

Ethics Statement

The paper uses publicly available datasets and models, which may carry their biases and imperfections.

Ethics Statement

  • Using publicly available datasets and models means the work may carry their biases and imperfections.Source: Ethics Statement · p. 10

A.1 Definition of Faithfulness

The paper defines faithfulness as an explanation accurately representing the model’s true reasoning, but argues that existing tests assess output-level self-consistency instead. It presents this distinction as a reason to compare tests on common data and models while moving toward analysis of input contributions.

A.1 Definition of Faithfulness

  • Existing tests inspect output behavior, so the authors characterize them as self-consistency tests rather than measures of faithfulness to internal reasoning.They argue that plausible surface consistency cannot rule out explanations disconnected from the model’s underlying processes.Source: §A.1 · p. 14
  • Without ground truth for faithfulness, the authors compare tests’ predictions on shared models and data, reporting wide disagreement among tests.They define self-consistency as output correspondences for prior tests or input-contribution correspondences for CC-SHAP.Source: §A.1 · pp. 14–15

A.2 Overview of Data and Models of Current & Prior Work

A.3 SHAP values for long explanations: Technical Detail

  • The paper flags rare long-explanation cases where tokens with little input contribution may become large after normalization and disrupt aggregation.It implemented a check for these cases.Source: §A.3 · p. 15

A.4 Prompts

  • For LLaMA 2, the prompts prepend a system prompt and use [INST] tags, while Falcon prompts use “User:” and “Assistant:”.Source: §A.4 · p. 15

A.5 Results on Causal Judgement and Logical Deduction (BBH)

Tests give substantially different judgments of models’ self-consistency, and chat models generally score higher than base counterparts, except Falcon. In the reported logical-deduction example, scores for LLaMA-7b range from 2% to 68%.

A.5 Results on Causal Judgement and Logical Deduction (BBH)

  • Chat models are generally more self-consistent than their base counterparts, except for Falcon.Source: §A.5 · p. 15
  • Table 4 reports accuracy and faithfulness/self-consistency test results for post-hoc and CoT explanations on 100-sample causal-judgement and logical-deduction-five-objects data, with random-accuracy baselines of 50% and 20%, respectively.Source: Table 4 · p. 16
  • Different tests give very different judgments of models’ self-consistency, consistent with their differing principles for how predictions may change.Source: §A.5 · p. 16

A.6 Aggregated Results

Aggregated test scores rank some models as more self-consistent, but the ranking warrants caution and CC-SHAP and other tests can show different trends. Across additional analyses, CC-SHAP is more stable across runs, correlations with other tests vary, and model size does not track self-consistency in the tested range.

A.6 Aggregated Results

  • LLaMA2-7b, LLaMA2-13b-chat, and Mistral-7b-chat rank as most self-consistent, while Falcon-7b ranks lowest; the aggregate spans inherently different tests and should be interpreted cautiously.CC-SHAP trends favor chat over base models, whereas non-CC-SHAP tests give lower and inconsistent trends for those pairs; Mistral trends agree, and Falcon shows no CC-SHAP differences.Source: §A.6 · pp. 17–18

A.7 Standard Deviation of Self-Consistency Tests and Accuracy

  • Other tests show considerable run-to-run standard deviation, while CC-SHAP shows low variation because input contributions remain almost equal across different generations.The estimate uses three runs on 100 ComVE examples for a subset of seven models.Source: §A.7 · p. 18

A.8 Correlation between CC-SHAP and other Tests

  • CC-SHAP correlates substantially positively with Counterfactual Edits across tasks, but its agreement with other tests varies by task and test.It aligns with most tests on ComVE except Paraphrasing and Constructing Input from Explanation, and with all tests on causal judgement.Source: §A.8 · p. 18

A.9 Relationship between Size, Accuracy and Self-Consistency

  • For tested model sizes of 7–13–40B parameters, accuracy trendlines generally rise with size, while self-consistency trendlines are mixed.CC-SHAP scores are very close between CoT and post-hoc settings.Source: §A.9 · pp. 18–19, Figure 4 · p. 20

B.1 Post-hoc Tests

  • The post-hoc examples compare CC-SHAP with Counterfactual Editing and Constructing Input from Explanation on a ComVE lobster instance.Source: §B.1 · p. 21

B.2 CoT Tests

  • The CoT examples compare CC-SHAP with Biasing Feature and Early Answering on the same ComVE lobster instance.Source: §B.2 · p. 21

B.3 Combining CC-SHAP with other Tests

This section examines CC-SHAP alongside counterfactual edits and two CoT consistency tests, focusing on how changes affect answer and explanation contributions. It reports that explanation contributions can shift more than prediction contributions, while most models’ answer attributions remain stable under counterfactual edits.

B.3 Combining CC-SHAP with other Tests

  • GPT2 shows low sensitivity to counterfactual edits in both answer and explanation contributions, unlike the other models.For post-hoc, the passage describes GPT2’s answer contributions as similar before and after insertion; the CoT result also reports low sensitivity in both contributions.Source: §B.3 · p. 21

CoT

  • In the CoT setting, all models except GPT2 show little sensitivity to edits in answer contributions but stark changes in explanation-generation contributions.The reported explanation effect is even stronger than in the post-hoc case.Source: §B.3 · p. 21

B.3 Combining CC-SHAP with other Tests

  • Because most other tests ignore explanation contributions apart from checking verbatim mention, the authors argue explanations should be considered more directly.The passage notes an exception for tests that construct input from the explanation.Source: §B.3 · p. 21
  • For post-hoc explanations, all models except GPT2 show similar answer contributions before and after counterfactual edits, but different explanation contributions.Table 15 illustrates the post-hoc comparison; the broader pattern is reported for Tables 15–19.Source: §B.3 · p. 21, Table 15 · p. 31

CoT

  • Adding “outside” changes prediction contributions little but affects explanation contributions more in the Mistral-7b-chat example.The example combines CC-SHAP with the Counterfactual Edit test, which adds a word or phrase to try to change a prediction and checks whether the new explanation mentions it.Source: Table 22 · p. 38, CoT · p. 38
  • CC-SHAP compares input contributions to a prediction and its explanation, yielding a continuous self-consistency value from −1 to 1, with higher values indicating greater consistency.The Counterfactual Edit test changes the input with a word or phrase and checks whether the explanation mentions the change that produced a different prediction.Source: CoT · p. 38
  • The Biasing Feature test deems a model unfaithful if adding a suggestive phrase (“I think the answer is...”) changes its CoT prediction.The test compares a prediction with CoT against a prediction on the same sample after adding a biasing feature.Source: Table 25 · p. 41
  • The Early Answering test deems a model unfaithful to its CoT if deleting most of the reasoning does not change its prediction.It compares a CoT prediction with one made after the CoT is corrupted by early answering.Source: Table 26 · p. 42

Related papers

Loading 2311.07466v4…