Source-linked AI summary
The Probabilities Also Matter: A More Faithful Metric for Faithfulness of Free-Text Explanations in Large Language Models
Noah Y. Siegel, Oana-Maria Camburu, Nicolas Heess, Maria Perez-Ortiz
TL;DR
The paper addresses whether plausible free-text explanations faithfully capture the factors behind LLM predictions, an important question for AI oversight. It introduces CEF, which measures continuous prediction impact alongside explanation mention patterns, and applies it to CT as CCT. Across Llama2 models and three NLP tasks, CCT detects faithfulness trends that the existing CT metric misses, while results vary across datasets and remain bounded by the intervention and few-shot explanation settings.
Problem
It is unclear whether plausible LLM-generated free-text explanations truly capture the factors responsible for model predictions, limiting their evidential value for oversight.
Method
The paper introduces CEF, which relates continuous intervention impact to explanation mention importance, and instantiates it on CT as CCT.
Results
Across Llama2 models and three datasets, CCT captures faithfulness trends that the existing metric used in CT misses.
Takeaways & Limitations
Model explanations are more likely to mention inserted words when those words have greater prediction impact, suggesting faithfulness that increases with model size on these tasks.
Takeaways & Limitations
The evaluation inherits CT limitations, including single-word adjective and adverb interventions, possible semantic incoherence, untested synonyms, and no check of semantic usage.
Abstract
from arXiv · showhide
In order to oversee advanced AI systems, it is important to understand their underlying decision-making process. When prompted, large language models (LLMs) can provide natural language explanations or reasoning traces that sound plausible and receive high ratings from human annotators. However, it is unclear to what extent these explanations are faithful, i.e., truly capture the factors responsible for the model's predictions. In this work, we introduce Correlational Explanatory Faithfulness (CEF), a metric that can be used in faithfulness tests based on input interventions. Previous metrics used in such tests take into account only binary changes in the predictions. Our metric accounts for the total shift in the model's predicted label distribution, more accurately reflecting the explanations' faithfulness. We then introduce the Correlational Counterfactual Test (CCT) by instantiating CEF on the Counterfactual Test (CT) from Atanasova et al. (2023). We evaluate the faithfulness of free-text explanations generated by few-shot-prompted LLMs from the Llama2 family on three NLP tasks. We find that our metric measures aspects of faithfulness which the CT misses.
1 Introduction
The paper argues that faithful free-text explanations must distinguish impactful from non-impactful factors, then introduces metrics designed to capture this distinction and reports that they reveal faithfulness trends missed by prior evaluation.
- Faithful explanations could support oversight by exposing model behaviors such as racial or gender bias, deception, or power-seeking.
- Informatively faithful explanations should mention significant factors more often than insignificant ones, rather than merely mentioning significant factors.
- Correlational Explanatory Faithfulness (CEF) captures both input-feature impact and the difference in mention frequency between impactful and non-impactful factors.
- The authors introduce the Correlational Counterfactual Test (CCT) by applying CEF to Atanasova et al. (2023)'s Counterfactual Test.
- Experiments with Llama2 models on three datasets show that CCT captures faithfulness trends missed by the existing metric used in CT.
2 Related Work
Related work distinguishes explanatory faithfulness, which concerns whether an explanation reflects a model's decision-making process, from causal faithfulness, which concerns whether the model's prediction depends on its reasoning trace.
- Natural language explanations are studied because they can communicate justifications more expressively than restrictive highlight-based explanations.
- Explanatory faithfulness asks whether an explanation reflects the model's decision-making process and is often assessed through input interventions.
- Causal faithfulness additionally asks whether the model's prediction causally depends on the generated reasoning trace.
- The literature also distinguishes explainability from interpretability or transparency, with this paper primarily focusing on explainability.
- 2.2 The Counterfactual Test: The Counterfactual Test inserts text into an input and checks whether the changed prediction-related text appears in the new explanation.
3 Methods
The methods identify two weaknesses in the Counterfactual Test and introduce CEF and its CCT instantiation to measure whether explanations mention interventions in proportion to their prediction impact.
- The CT can be trivially gamed by repeating the entire input, whereas faithful explanations should mention impactful features and omit non-impactful ones.
- CEF applies to tests with an intervention, a scalar prediction-impact measure I, and an explanation-mention measure M.It quantifies faithfulness as the Pearson correlation between prediction impact and mention importance, with positive values indicating higher mention importance for more impactful interventions.
- CCT instantiates CEF on CT by measuring intervention impact with total variation distance between the model's predicted distributions before and after intervention.TVD is symmetric and gives less weight to shifts between very small probabilities than KL divergence.
- For binary mention measures, CCT is maximized when explanations mention interventions whose TVD exceeds a distribution-dependent threshold.M indicates whether the explanation mentions the intervention, and |M| counts examples with such mentions.
- CCT's maximum correlation requires mentioning impactful interventions while avoiding non-impactful ones, so it cannot be achieved without a signal about impactfulness.
4 Experiments
The experiments evaluate few-shot Llama-2 explanations across three natural-language classification datasets, comparing the Counterfactual Test with CCT under different prompting and intervention settings.
- The study compares CT and CCT for LLM-generated natural-language explanations on three natural-language classification tasks.
- The datasets are e-SNLI for entailment, contradiction, or neutrality; ComVE for common-sense violations; and ECQA with five multiple-choice options.
- The evaluation uses Llama-2 7B, 13B, and 70B foundation models in a few-shot setting with 20 randomly selected labeled training examples.
- Models generate explanations either after predictions in predict-then-explain or before predictions in explain-then-predict prompting.
- The intervention procedure inserts random adjectives before nouns or adverbs before verbs at randomly selected positions using WordNet candidates, with an instruction-tuned model filtering unnatural sentences.
- For each model, prompting strategy, and dataset, researchers compare whether explanations mention interventions according to their TVD-measured impact after rerunning the model on counterfactual inputs.Greedy sampling reduces textual variation, while predicted-class token probabilities are retained for TVD computation.
5 Results
The results show that CCT captures faithfulness patterns that CT misses, especially the difference between mentioning impactful factors and mentioning factors indiscriminately. Faithfulness varies across datasets and models, with Llama2-70B strongest on e-SNLI and ComVE.
- Dataset comparison: e-SNLI shows an upward relationship between intervention impact and explanation mentions, whereas ECQA shows a relatively flat relationship.The ECQA pattern may reflect verbose explanations that repeat large portions of their inputs.
- Measurement: Figure 1 compares explanation mention frequency with intervention impact measured by TVD, where higher TVD indicates greater prediction impact.Rows represent datasets and columns represent models in the PE setting.
- Experimental results: Table 2 reports accuracy before intervention, CT, and CCT across datasets, models, and predict-then-explain versus explain-then-predict prompt orders.The table also includes a random CCT-faithfulness baseline based on independence between explanation mentions and prediction impact.
- Dataset comparison: ECQA explanations frequently mention impactful inserted factors but provide little signal about impact because they also frequently mention any inserted factors.This explains why low CT unfaithfulness can be misleading for ECQA.
- Overall results: CCT is more informative than CT about qualitative faithfulness patterns, distinguishing stronger relevance information for e-SNLI than ECQA.The comparison reflects whether explanations mention impactful inserted factors more often than non-impactful ones.
6 Summary and outlook
The paper introduces Counterfactual Explanatory Faithfulness and the Correlational Counterfactual Test to measure whether explanations informatively track the importance of mentioned factors. Across these tasks, inserted words are more likely to appear in explanations when they have greater prediction impact, with faithfulness increasing with model size but varying substantially by dataset.
- Contribution: The Correlational Counterfactual Test measures how informative explanations are about the importance of the factors they mention.It is presented as Counterfactual Explanatory Faithfulness instantiated through the CCT.
- Findings: Inserted words are more likely to appear in explanations when they have greater impact on model predictions, suggesting faithfulness that increases with model size.The authors also report significant variation between datasets.
- Outlook: Future work could apply CCT to instruction-tuned models and explanations generated with strategies such as question decomposition.These directions are proposed for extending the evaluation.
Limitations
The evaluation inherits limitations from the original Counterfactual Test and depends on the few-shot explanation examples used for each task. Its interventions cover only single adjectives or adverbs, may produce incoherent text, omit synonym testing, and do not assess semantic usage of mentions.
- Evaluation scope: The CCT inherits the original CT’s scope limits: interventions insert only single adjectives or adverbs, so other parts of speech are not evaluated.Random interventions can also produce semantically incoherent text despite LLM filtering.
- Evaluation scope: The evaluation does not test synonyms, which could cause some explanations to be labeled inaccurately.This is identified as an inherited limitation of the intervention-based setup.
- Metric scope: The metrics do not consider the semantic usage of word mentions and therefore would not penalize illogical explanations with the correct word-inclusion pattern.The limitation concerns semantic interpretation rather than mention frequency alone.
- Generalization: Few-shot results may depend as much on the example explanations and annotation instructions as on the model, limiting extrapolation to other tasks.ECQA explanations are reported to be significantly longer than CoS-E explanations despite sharing the CQA base dataset.
A "Causal" vs. "Explanatory" Faithfulness
The paper distinguishes explanatory faithfulness from causal faithfulness: an explanation may reflect useful information about predictive factors without being causally required for the prediction. Causal dependence alone may also fail to ensure that reasoning text provides useful information.
- Causal reasoning limits: CoT may yield relatively minor accuracy gains outside math-focused domains, and stronger models may solve more tasks without verbalized CoT.These observations complicate reliance on causal reasoning traces as explanations.
- Conceptual distinction: Causal faithfulness asks whether the model’s prediction causally depends on the generated reasoning trace, whereas explanatory faithfulness asks whether the explanation reflects the model’s decision-making process.The distinction separates causal dependence from the informativeness of textual content.
- Why the distinction matters: Causal dependence does not guarantee useful explanations because models may use language differently from humans, including through semantic drift or steganography.The text therefore treats usefulness of reasoning content as a separate criterion.
B “Explainability” vs. “Transparency/Interpretability”
The paper distinguishes explainability, which uses auxiliary models to explain black-box systems, from transparency, which constrains model form so its inner mechanics can be observed. Because terminology varies, the authors use “explainability” and “transparency” for this distinction and note that both can in principle be faithful.
- “Interpretability” and “explainability” are used inconsistently, sometimes interchangeably and sometimes as distinct concepts.
- The paper calls systems that expose their inner mechanics “transparent” and auxiliary explanations of black-box models “explainable.”
- The authors use “explainability” and “transparency” because these labels are least ambiguous for the distinction.
- Both transparent and explainable systems can be faithful if their explanations accurately represent the model’s reasoning process.
C “Faithfulness” or “Self-Consistency”?
The paper discusses whether intervention-based tests measure faithfulness or only self-consistency. It retains the faithfulness label because prediction changes after interventions reveal model dependence on the intervened input, while acknowledging that mechanistic tests are not automatically robust.
- Some researchers argue that metrics including the Counterfactual Test measure “self-consistency” because they do not account for mechanistic inner workings.
- The authors retain the term faithfulness because an intervention-induced output-distribution change reveals that the model depends on the intervened input in context.
- The proposed test measures whether explanations reflect the model dependency revealed by input interventions.
- Mechanistic tests are not guaranteed to be robust, and the authors suggest robust faithfulness measurement may require combining mechanistic and black-box tests.
- Figure 2 applies the intervention-impactfulness versus explanation-mention analysis in the Explain-then-Predict setting.
E Example Interventions
The paper presents randomly selected intervention examples across three datasets, showing original and modified inputs, predictions, highlighted inserted adjectives or adverbs, and explanation mentions.
- Examples show original and modified problems, resulting predictions, inserted interventions, and any mentions of the intervention in explanations.The displayed examples use Llama2 70B with predict-then-explain prompting and cover e-SNLI, ECQA, and ComVE.
F CCT Variants
The paper evaluates alternative CCT configurations by replacing TVD with Jensen-Shannon divergence or Pearson correlation with Spearman correlation. These variants preserve the main qualitative ordering across datasets and generally favor the largest model.
- F CCT Variants: TVD is chosen because it downweights shifts among very small probabilities, while Pearson uses cardinality rather than only rank information.The approach can also be computed with other distance and correlation choices.
- F CCT Variants: CCT (Jensen-Shannon) replaces TVD with Jensen-Shannon divergence, while CCT (Spearman) replaces Pearson correlation with Spearman’s rank correlation.
- F CCT Variants: The variants show the highest values for e-SNLI, lower values for ECQA and comVE, and generally slightly higher faithfulness for the largest model.The exception is CCT (Spearman) in the Explain-then-Predict setting, where the 13B model has the highest value.
- F CCT Variants: The evaluation prompts Llama2 models with few-shot demonstrations containing labels and explanations, using either predict-then-explain or explain-then-predict ordering.
- F CCT Variants: The intervention examples include dataset-specific text formats and generated explanations for e-SNLI, ECQA, and ComVE.