PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanations

Shiyuan Huang, Siddarth Mamidanna, Shreedhar Jangam, Yilun Zhou, Leilani H. Gilpin

arXiv:2310.11207v1cs.CLcs.LG

TL;DR

The paper asks whether LLM-generated explanations faithfully support predictions and how they compare with traditional methods. It evaluates ChatGPT feature-attribution explanations for sentiment analysis, finding faithfulness parity but substantial disagreement. The findings also point to limits in applying existing interpretability evaluations to these models.

  • Problem

    The paper asks whether LLM-generated explanations faithfully support predictions, how they compare with traditional explanations, and whether negative evaluations reflect model or evaluation limitations.

    Source: §I · p. 1

  • Method

    Using ChatGPT on sentiment analysis, the study elicits feature-attribution explanations in two prediction/explanation orders and compares them with occlusion and LIME.

    Source: §I · p. 1

  • Results

    ChatGPT self-explanations perform on par with traditional methods on faithfulness evaluations but differ greatly on agreement metrics.

    Source: §I · p. 2

  • Takeaways & Limitations

    The authors argue that interpretability pipelines may be ill-suited to LLMs because predictions are not sufficiently fine-grained for meaningful explanation evaluation.

    Source: §I · p. 2

    The study is scoped to ChatGPT, feature-attribution explanations, and sentiment analysis.

    Source: §I · p. 1

Abstract

from arXiv · show

Large language models (LLMs) such as ChatGPT have demonstrated superior performance on a variety of natural language processing (NLP) tasks including sentiment analysis, mathematical reasoning and summarization. Furthermore, since these models are instruction-tuned on human conversations to produce "helpful" responses, they can and often will produce explanations along with the response, which we call self-explanations. For example, when analyzing the sentiment of a movie review, the model may output not only the positivity of the sentiment, but also an explanation (e.g., by listing the sentiment-laden words such as "fantastic" and "memorable" in the review). How good are these automatically generated self-explanations? In this paper, we investigate this question on the task of sentiment analysis and for feature attribution explanation, one of the most commonly studied settings in the interpretability literature (for pre-ChatGPT models). Specifically, we study different ways to elicit the self-explanations, evaluate their faithfulness on a set of evaluation metrics, and compare them to traditional explanation methods such as occlusion or LIME saliency maps. Through an extensive set of experiments, we find that ChatGPT's self-explanations perform on par with traditional ones, but are quite different from them according to various agreement metrics, meanwhile being much cheaper to produce (as they are generated along with the prediction). In addition, we identified several interesting characteristics of them, which prompt us to rethink many current model interpretability practices in the era of ChatGPT(-like) LLMs.

I. INTRODUCTION

The paper asks whether LLM self-explanations faithfully support predictions and how they compare with traditional interpretability methods. It studies ChatGPT feature-attribution explanations for sentiment analysis and finds faithfulness parity alongside substantial disagreement.

I. INTRODUCTION

  • The study examines whether self-explanations faithfully support predictions, how they compare with traditional explanations, and whether negative evaluations reflect model or evaluation limits.It focuses on ChatGPT, feature attribution, and sentiment analysis, choices the authors describe as popular in their respective areas.Source: §I · p. 1
  • Explanations are elicited in explain-then-predict and predict-and-explain settings, using either full word-importance lists or top-k words, and compared with occlusion and LIME.The comparisons use faithfulness metrics and disagreement measurements.Source: §I · p. 2
  • ChatGPT self-explanations perform on par with traditional methods on faithfulness evaluations but differ greatly on agreement metrics.The authors argue that interpretability pipelines may be ill-suited to LLMs because their predictions are not sufficiently fine-grained for meaningful explanation evaluation.Source: §I · p. 2

II. RELATED WORK

  • Because ChatGPT is closed-source and provides no input gradients, gradient-based attribution methods cannot be calculated for it.The paper places its work in a relatively new area and describes itself, to the authors’ knowledge, as the first rigorous study of LLM-generated feature-attribution explanations.Source: A. Feature Attribution Explanations · p. 2, §II · p. 2

A. Feature Attribution Explanations

  • Feature importance is based on responses to inputs with features removed; the study uses mainstream word deletion and leaves alternatives such as mask infilling for future work.Occlusion measures prediction change after removing a word; LIME and SHAP remove multiple features to capture interactions.Source: A. Feature Attribution Explanations · p. 2

B. Evaluations of Feature Attribution Explanations

This section describes the evaluation dimensions and related approaches used to study feature-attribution explanations. It also situates the work among LLM-generated explanations and outlines its systematic analysis of sentiment-domain self-explanations.

B. Evaluations of Feature Attribution Explanations

  • Faithfulness metrics test whether modifying features identified as important changes predictions, while disagreement metrics measure correlation between self-explanations and traditional techniques.The study uses comprehensiveness, sufficiency, decision-flip measures, deletion rank correlation, and disagreement metrics.Source: B. Evaluations of Feature Attribution… · p. 2
  • Dataset-retraining evaluations that test alignment with expected reasoning are left for future work because finetuning is costly and API access is limited.These evaluations address the absence of ground-truth model reasoning by training models to use specified reasoning for high accuracy.Source: B. Evaluations of Feature Attribution… · p. 2

C. LLM-Generated Self-Explanations

  • Camburu et al. [7] studied both prediction-then-explanation and explanation-then-prediction, while finding a slight accuracy drop when models were trained to generate explanations.Prior work also considered chain-of-thought explanations and self-generated counterfactual explanations, which were reported as often misleading.Source: C. LLM-Generated Self-Explanations · p. 3

III. METHODOLOGY

  • The paper’s contribution is a suite of systematic analyses of LLM-generated self-explanations in sentiment analysis.The methodology section says it introduces prompting strategy, traditional comparison methods, and evaluation metrics.Source: §III · p. 3

A. Auto-regressive LLMs

The section models autoregressive generation as producing tokens from prior context until a stop token, using deterministic greedy decoding in the experiments. Conversational context is built by concatenating system, user, and assistant messages.

A. Auto-regressive LLMs

  • Autoregressive generation produces the next token from prior context until a stop token is generated.The experiments use deterministic greedy decoding with temperature = 0.Source: A. Auto-regressive LLMs · p. 3
  • The generation function maps context words x_c to generated words x_g.The context sequence concatenates past messages, including system, user, and assistant turns.Source: A. Auto-regressive LLMs · p. 3
  • The example context appends the assistant turn marker after the user asks whether x=1 has other solutions.The supplied E-P and P-E prompt tables are identified as examples of those setups.Source: A. Auto-regressive LLMs · p. 3, Table I · p. 3, Table II · p. 4

B. Prompting

The prompting section elicits sentiment predictions and feature-attribution explanations in explain-then-predict and predict-and-explain orders, using full attribution lists or top-k words. The prompts also address partial inputs and output formatting, while top-k prompts offer a more natural alternative to detailed scores.

B. Prompting

  • The study uses both explain-then-predict and predict-and-explain, asking for either full word-attribution scores or only the top-k words.The prompt designs request explanation and prediction together, in a defined order, and guide the model to answer even when words have been removed.Source: B. Prompting · p. 4
  • Prompts use Python-format output for consistent parsing, and the model’s decimal confidence value serves as its prediction because model weights are unavailable.The authors report that models sometimes refuse partial inputs without guidance, so prompts ask for their best guess.Source: B. Prompting · pp. 4–5
  • Top-k prompts aim for more natural outputs and set k dynamically as min(1, ⌊L/5⌋), where L is sentence length in words.The authors raise concern that unnatural Python-format guidance may make attribution values less faithful to the model’s true reasoning.Source: B. Prompting · p. 4
  • The in-the-wild example presents ChatGPT’s explanation as a few highlighted phrases accompanying its response.The paper also illustrates an E-P top-k prompt and example query.Source: Table III · p. 4, Table IV · p. 5

C. Traditional Interpretability Methods

Traditional perturbation methods estimate word importance from prediction changes after removing words, while the paper evaluates explanation faithfulness using prediction-based metrics and separately measures disagreement. Top-k explanations receive adapted versions of selected metrics.

C. Traditional Interpretability Methods

  • Occlusion assigns a word importance from the prediction change when that word is removed, while LIME samples random word subsets and fits a linear regression to attribute prediction changes.LIME addresses potentially non-additive effects of removing multiple words.Source: C. Traditional Interpretability Methods · p. 5

D. Evaluation Metrics

  • The evaluation uses comprehensiveness, sufficiency, DFMIT, DFFrac, and RankDel, based on model predictions for the original input and inputs with selected words removed.Tied saliency values receive random tie-breaking when determining word-removal order.Source: D. Evaluation Metrics · pp. 5–6
  • For top-k explanations, comprehensiveness, sufficiency, and DFMIT are adapted using inputs with the top-k words removed or retained.The top-k format has no ranking among selected words or among the remaining words.Source: D. Evaluation Metrics · p. 6
  • The study also evaluates agreement or disagreement between explanations, separately from faithfulness scores for individual explanations.The agreement evaluation follows definitions from Krishna et al. [23].Source: D. Evaluation Metrics · p. 6

E. A Caveat on Model Sameness

Because prompts shape the effective model, the E-P and P-E prompts are treated as different models, so their self-explanations are not comparable. The setup also visualizes example explanations on SST.

E. A Caveat on Model Sameness

  • Changing the system prompt effectively changes the model, so E-P and P-E self-explanations are not comparable; prediction evaluation must use the original prompt.For occlusion comparisons, the authors also use that original prompt for the input with a word removed.Source: E. A Caveat on Model Sameness · p. 6
  • Top-k explanations require a different prompt, so the authors treat them as external explainers for the target E-P or P-E model.The prompt needed to elicit top-k output effectively results in a different model.Source: E. A Caveat on Model Sameness · p. 6

A. Experiment Setup and Explanation Visualization

B. Model Accuracy

On this sentiment-analysis setup, E-P is less accurate than P-E, and both explanation-generating models trail the explanation-free model. Accuracy drops further in the top-k settings.

B. Model Accuracy

  • 85% vs 88%: E-P accuracy is lower than P-E, which the authors attribute to attribution scores inadequately representing sentiment reasoning.Source: B. Model Accuracy · p. 7
  • 92%: the explanation-free model is more accurate than both explanation-generating models.Source: B. Model Accuracy · p. 7

Example output: (<int classification>, <float confidence>)

C. Faithfulness Evaluation Results

Faithfulness metrics show no clear overall winner among self-explanations, occlusion, and LIME. The top-k explanations likewise do not emerge as distinctively better, while self-explanations offer a lower-cost alternative to LIME.

  • Across faithfulness metrics, no method clearly wins: occlusion leads on DFMIT, while LIME performs evenly with SELFEXP on the remaining metrics.Source: C. Faithfulness Evaluation Results · p. 7
  • RankDel values are close to zero, and prediction insensitivity to single-word removal makes the metric prone to stochasticity from tie-breaking.Source: C. Faithfulness Evaluation Results · p. 7
  • Occlusion is best on none of the remaining metrics, which the authors relate to its inability to capture nonlinear feature-removal effects considered by LIME.Source: C. Faithfulness Evaluation Results · p. 7
  • SELFEXP could be a low-cost substitute for LIME because it is generated with the prediction, whereas LIME takes about 20 minutes per explanation in the authors’ setup.They adaptively average 183 perturbations per explanation.Source: C. Faithfulness Evaluation Results · p. 7
  • Top-k explanations are not distinctly better on compatible metrics, so the results warrant caution when interpreting top-k explanations in the wild.Source: C. Faithfulness Evaluation Results · p. 8

D. Agreement Evaluation Results

Explanations can disagree substantially even when their faithfulness scores are similar. The authors discuss limits of the evaluation and the explanations, including coarse attribution levels and an untested understandability trade-off.

D. Agreement Evaluation Results

  • Explanation pairs show high disagreement even though faithfulness scores are close, unlike the larger method-performance differences reported for traditional classifier explanations.Source: D. Agreement Evaluation Results · p. 8
  • The authors suggest either that the evaluations lack power to distinguish explanation quality or that current methods miss a potentially better explanation.They leave investigation of the latter possibility to future work.Source: D. Agreement Evaluation Results · pp. 8–9

1) Level of Saliency Values:

  • For sentences averaging 18.2 words, attribution values have only 2.72 unique levels for E-P and 5.69 for P-E; the authors attribute the rounded, coarse levels to human-like behavior.Source: 1) Level of Saliency Values · p. 9
  • The proposed trade-off—vaguer explanations that may be easier for people to understand—remains a hypothesis requiring formal human-subject evaluation.Source: 1) Level of Saliency Values · p. 9
  • Few-shot prompting produced less-rounded scores when given fine-grained examples, but still yielded few levels and sometimes repeated example values for the same word.The authors therefore based experiments on the more stable zero-shot responses.Source: 1) Level of Saliency Values · p. 9

2) Prediction Value Roundedness and Insensitivity:

ChatGPT’s prediction and attribution values are often coarse, and removing words often leaves predictions unchanged. This behavior produces zero occlusion saliency for many words and affects faithfulness evaluations, especially DFMIT and RankDel.

2) Prediction Value Roundedness and Insensitivity:

  • Few-shot exemplars leave attribution values on a few saliency levels, though they are no longer well-rounded, and the model memorizes a supplied word-score pair.Source: Table X · p. 10
  • ChatGPT prediction values are often well-rounded and drawn from a few distinct values; the study defines predictions as textual confidence scores because token probabilities are unavailable.Source: 2) Prediction Value Roundedness and… · pp. 9–10
  • Removing a few words often does not change predictions, which the authors attribute to inferring missing words and alignment with human tendencies to ignore errors.Source: 2) Prediction Value Roundedness and… · p. 10
  • Occlusion saliency is exactly 0 for 82.6% of words in E-P and 54.5% in P-E.Source: 2) Prediction Value Roundedness and… · p. 10
  • DFMIT and RankDel are most severely affected, with all explanations scoring very low on these metrics compared with finetuned RoBERTa; unchanged predictions also lower comprehensiveness and raise sufficiency.The reported RoBERTa comparison is ∼25% for DFMIT and ∼0.5% for RankDel on average.Source: 2) Prediction Value Roundedness and… · pp. 10–11

V. CONCLUSION AND FUTURE WORK

The paper assesses ChatGPT-generated word-attribution explanations against traditional methods on faithfulness and agreement metrics. It finds no distinct faithfulness advantage but substantial differences in agreement, while coarse prediction and attribution values may limit evaluation’s ability to distinguish explanations.

V. CONCLUSION AND FUTURE WORK

  • The study evaluates ChatGPT on SST, eliciting full word-attribution scores and top-k explanations in explain-then-predict and predict-and-explain settings, then comparing them with occlusion and LIME.Source: §V · p. 11
  • No explanation has a distinct advantage in faithfulness evaluations, yet the explanations differ greatly on agreement metrics.Source: §V · p. 11
  • ChatGPT’s prediction and attribution values are highly well-rounded, which may leave faithfulness metrics unable to distinguish good explanations from bad ones.Source: §V · p. 11
  • The authors suggest improving elicitation and rethinking evaluation, including carefully designed human studies, because current practices may not suit ChatGPT’s reasoning style.Source: §V · p. 11
  • Future work could compare other LLMs, study counterfactual or concept-based explanations, and test whether explanations can be manipulated or hide fairness issues.Source: §V · p. 11
Loading 2310.11207v1…