Source-linked AI summary
From Plausible to Actionable: A Position on LLM Self-Explanations
Elize Herrewijnen, Benedetta Muscato, Gizem Gezici, Fosca Giannotti
TL;DR
LLM self-explanations appear promising for XAI, but whether they reflect models’ underlying reasoning remains unresolved. This position paper identifies limitations in standard plausibility and faithfulness evaluations, argues that self-explanations can be plausible yet questionably faithful, and emphasizes their actionability for supporting informed decisions.
Problem
Whether natural-language self-explanations faithfully reflect LLMs’ underlying decision processes remains unresolved despite their apparent plausibility and accessibility.
Method
The paper examines limitations of standard XAI evaluations and develops practical guidance and a research agenda centered on plausibility, faithfulness, and actionability.
Results
Self-explanations can be highly plausible but questionably faithful, while their value can lie in supporting informed decision-making and appropriate action.
Takeaways & Limitations
Self-explanations should be treated as communicative interfaces and decision-support tools rather than literal accounts of models’ internal reasoning.
Takeaways & Limitations
Plausibility and faithfulness evaluations are constrained by stakeholder expertise, LLM nondeterminism, prompt sensitivity, and unreliable perturbation assumptions.
Abstract
from arXiv · showhide
Large Language Models (LLMs) can generate natural language explanations that rationalize their own decisions, a phenomenon commonly referred to as self-explanations. Such explanations have emerged as a promising direction for explainable artificial intelligence (XAI), particularly for interpreting LLM behavior. However, while self-explanations often appear plausible, whether they faithfully reflect a model's underlying reasoning process remains an open question. In this opinion paper, we argue that self-explanations can be highly plausible, questionably faithful, and yet highly actionable. From a traditional XAI perspective, we identify the limitations of standard evaluation protocols for LLM-generated self-explanations and propose practical guidelines for assessing their plausibility and faithfulness.Moreover, we argue that evaluation should extend beyond these criteria to actionability, highlighting applications of LLM rationalization capabilities that support informed decision-making and appropriate action across diverse stakeholders.
1 Introduction
LLM self-explanations offer accessible natural-language accounts of model outputs, but their apparent alignment with human reasoning does not establish that they reflect the model’s underlying decision process.
- Traditional post-hoc XAI methods are accessible but difficult to scale to LLMs, while mechanistic interpretability is complex for non-experts.Post-hoc methods can be computationally costly for models with hundreds of billions of parameters, whereas mechanistic approaches require advanced expertise.
- LLM self-explanations generate natural-language explanations or intermediate reasoning for their outputs without relying on external explanation methods or internal-mechanism analysis.
- Self-explanations appear transparent and aligned with human reasoning, making them attractive for high-stakes applications such as clinical decision support.Their accessibility can support trust among clinicians and patients, but apparent transparency does not guarantee alignment with the model’s actual decision process.
- The paper argues that self-explanations can be highly plausible yet questionably faithful to the model’s underlying reasoning.
- The authors propose evaluating plausibility and faithfulness while shifting attention toward actionability for informed decisions and concrete actions.They frame self-explanations as potential support for stakeholders with diverse goals, expertise, and technical backgrounds.
2 Self-Explanations: Highly Plausible, Questionably Faithful
Self-explanations often seem plausible because they align with human expectations and coherent language, but their faithfulness is uncertain and standard evaluation protocols miss important LLM-specific conditions.
- 2.1 Plausibility: Sycophancy, coherent helpful responses, and accessible natural language can make self-explanations align with stakeholder expectations and appear interpretable.
- 2.1 Plausibility: Self-explanations are often judged plausible through human ratings or similarity to human explanations, but these evaluations can overlook expertise and explanatory variation.The paper recommends assessing explanations with stakeholders whose expertise matches the task and allowing multiple valid explanations for ambiguous or subjective decisions.
- 2.2 Faithfulness: Self-explanations are questionably faithful because models lack access to their internal output-generating processes, explanations may influence decisions, and rationales may be misleading.The paper identifies sycophancy, hallucinations, and biases as sources of coherent explanations that may not correspond to decision causes.
- 2.2 Faithfulness: Perturbation-based faithfulness methods rely on assumptions that do not hold reliably for LLM self-explanations.The relevant assumptions concern model equivalence, prediction stability, and linearity of input contributions.
- 2.2 Faithfulness: Faithfulness evaluation must account for nondeterminism and prompt sensitivity because identical or similar inputs can produce different outputs and explanations.Single-run outputs are therefore insufficient as definitive evidence of faithfulness.
- 2.2 Faithfulness: Input perturbations can be unreliable because LLMs may rely on internal knowledge, label bias, or unwarranted behavior rather than the altered task input.Masking task words may leave predictions unchanged, while unnatural or synthetic inputs can trigger refusal or other behavior.
- 2.3 Implications of High Plausibility but Questionable Faithfulness: Highly plausible but unfaithful explanations can persuade users to accept incorrect predictions and increase automation bias, especially in high-stakes domains.
3 Towards Actionable Self-Explanations
Because self-explanations may not faithfully reflect model decision-making, the paper argues that their value should be assessed through actionability: helping diverse stakeholders understand outputs, deliberate, and act. It presents self-explanations as rationalizing interfaces that support—not replace—human judgment.
- Actionability evaluates whether self-explanations help diverse stakeholders make informed decisions and take effective actions despite uncertain faithfulness.The paper frames self-explanations as outputs of an LLM’s rationalization capability: generating justifications for decisions.
- Making XAI accessible to non-expert stakeholders: Self-explanations can translate complex faithful explanations into accessible natural-language rationales tailored to stakeholders’ expertise and information needs.They are presented as communicative interfaces rather than standalone explanations, and may convey uncertainty or potential hallucinations.
- LLMs in support of human decision-making: In high-stakes domains, self-explanations should present arguments, assumptions, uncertainties, and alternatives for critical evaluation rather than final decisions.Safety protocols can constrain models to generate supporting or challenging evidence while leaving the final decision to a human expert.
- LLMs as advocates to facilitate deliberation: Different LLMs can provide complementary arguments and counterarguments, expanding considerations available for human deliberation while people remain final decision-makers.The paper calls this paradigm “LLMs as advocates” and links variation to richer deliberation.
4 Conclusion
The paper concludes that existing XAI frameworks are limited for evaluating LLM self-explanations’ plausibility and faithfulness. It therefore positions actionability—not literal faithfulness—as the primary objective for supporting informed decisions across diverse stakeholders.
- Existing XAI evaluation frameworks have limitations for measuring the plausibility and faithfulness of LLM-generated self-explanations.
- Self-explanations should be treated as communicative interfaces, decision-support tools, and facilitators of deliberation rather than literal accounts of internal reasoning.
- Actionability emphasizes enabling stakeholders with diverse goals, expertise, and backgrounds to make informed decisions and take appropriate actions.