Source-linked AI summary
Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Models
Chirag Agarwal, Sree Harsha Tanneru, Himabindu Lakkaraju
TL;DR
LLM self-explanations are often plausible and user-friendly, but their faithfulness to the models’ actual reasoning remains poorly understood. This review characterizes the plausibility–faithfulness dichotomy, examines its implications for high-stakes use, and calls for methods and evaluations that improve faithfulness without losing usability.
Problem
The central gap is that plausible, coherent self-explanations may not accurately reflect LLM reasoning, creating concerns for high-stakes decisions.
Method
The paper reviews self-explanation techniques, defines plausibility and faithfulness, analyzes their application-dependent requirements, and surveys approaches for evaluating faithfulness.
Results
The review concludes that self-explanations lack faithfulness guarantees and that no universally agreed-upon faithfulness metrics currently exist.
Takeaways & Limitations
Faithfulness is critical in high-stakes applications, so the community should develop methods, metrics, and benchmarks that support transparent deployment.
Takeaways & Limitations
LLM plausibility can mimic logical reasoning without inherent understanding of truth or factual accuracy.
Abstract
from arXiv · showhide
Large Language Models (LLMs) are deployed as powerful tools for several natural language processing (NLP) applications. Recent works show that modern LLMs can generate self-explanations (SEs), which elicit their intermediate reasoning steps for explaining their behavior. Self-explanations have seen widespread adoption owing to their conversational and plausible nature. However, there is little to no understanding of their faithfulness. In this work, we discuss the dichotomy between faithfulness and plausibility in SEs generated by LLMs. We argue that while LLMs are adept at generating plausible explanations -- seemingly logical and coherent to human users -- these explanations do not necessarily align with the reasoning processes of the LLMs, raising concerns about their faithfulness. We highlight that the current trend towards increasing the plausibility of explanations, primarily driven by the demand for user-friendly interfaces, may come at the cost of diminishing their faithfulness. We assert that the faithfulness of explanations is critical in LLMs employed for high-stakes decision-making. Moreover, we emphasize the need for a systematic characterization of faithfulness-plausibility requirements of different real-world applications and ensure explanations meet those needs. While there are several approaches to improving plausibility, improving faithfulness is an open challenge. We call upon the community to develop novel methods to enhance the faithfulness of self explanations thereby enabling transparent deployment of LLMs in diverse high-stakes settings.
1. Introduction
LLMs can generate coherent, human-understandable self-explanations, but these explanations may not faithfully reflect the models’ actual reasoning. This review examines the plausibility–faithfulness gap and its importance for high-stakes applications.
- LLMs generate coherent and contextually adaptable self-explanations for applications including healthcare, law, and finance.
- Because plausible explanations may diverge from actual model reasoning, the gap raises reliability and trustworthiness concerns in high-stakes scenarios.
- Plausibility concerns how convincing an explanation appears to humans, whereas faithfulness concerns whether it accurately reflects how the model reached its decision.
- It calls for evaluation metrics, benchmarks, and methods that can assess and improve both properties of self-explanations.
- The review focuses on high-stakes applications and examines when explanations need to be plausible, faithful, or both.
2. Self-Explanations
Self-explanations are model-generated accounts of decision-making that can take the form of reasoning traces, influential-token highlights, or counterfactuals. These formats aim to make LLM behavior more transparent and understandable.
- Self-explanations elicit model reasoning in human-understandable language and include chain-of-thought, token importance, and counterfactual formats.
- Chain-of-thought: Chain-of-thought explanations provide intermediate steps leading to a final response, making the decision process easier to inspect.
- Token importance: Token-importance explanations identify input words or phrases that significantly influence the model’s decision.
- Counterfactual explanations: Counterfactual explanations describe how changing an input could alter the model’s response.
- Together, these forms are presented as tools for increasing transparency, user confidence, and understanding of LLM information processing.
3. Plausibility
LLMs can produce explanations that are contextually relevant and convincing because they mimic logical reasoning patterns and adapt to prompts. However, plausibility does not guarantee factual correctness or faithfulness to the model’s prediction process.
- LLMs generate contextually relevant and convincing explanations that are perceived as plausible by human practitioners and stakeholders.
- Plausibility: Plausibility is defined as coherence with human reasoning and understanding.
- Plausibility: Different LLMs may produce distinct explanations for the same prompt that users nevertheless perceive as equally valid.
- Chain-of-thought reasoning: Chain-of-thought reasoning has been associated with improved performance on complex tasks, but its impact varies across reasoning skills.
- Mechanisms of plausibility: Extensive training, RLHF, and prompt adaptation help LLMs imitate logical structures and produce coherent, context-tailored explanations.
- Limitations: Plausible reasoning may lack factual grounding, fail to entail the model’s prediction, and cause users to over-rely on incorrect explanations.
4. Faithfulness
Faithfulness asks whether a self-explanation accurately represents an LLM’s underlying reasoning, but evaluating this is difficult because ground-truth explanations are unavailable and models are opaque. Proposed approaches use input perturbations or explanation interventions to identify unfaithfulness, yet no agreed metric or guarantee currently exists.
- Faithfulness means that an explanation accurately represents the reasoning of the underlying model.
- Evaluating self-explanation faithfulness is difficult because ground-truth explanations are unavailable and LLM scale and proprietary design impede gradient-based assessment.
- Turpin et al. (2023) test faithfulness by perturbing features identified as important or unimportant and measuring whether predictions change as expected.On BBH, reordering options so the correct answer is first caused altered explanations and incorrect predictions, indicating unfaithfulness.
- Lanham et al. (2023) estimate post-hoc reasoning by truncating explanations or adding mistakes, then measuring answer consistency across intervention levels.Faithfulness is quantified through the area over curves relating explanation fraction or mistake fraction to answer consistency.
- Self-explanations currently lack faithfulness guarantees, universally agreed metrics, and consensus on what faithfulness should mean.The paper argues that plausible yet unfaithful explanations can foster misplaced trustworthiness in LLMs.
5. Plausibility or Faithfulness - Which one do we need ?
LLMs are trained and evaluated in ways that favor plausible, human-convincing explanations, even when those explanations may not reflect model reasoning. The paper contrasts the risks and benefits of plausibility and faithfulness, arguing that the appropriate balance depends on users’ goals and application stakes.
- What led to explanations being more plausible than faithful?: Training and evaluation practices favor plausible explanations because human-like text generation and RLHF reward coherence to human evaluators.The paper notes that this optimization may conflict with producing faithful self-explanations.
- 5.2. What if explanations are plausible but not faithful ?: Plausible but unfaithful explanations can encourage misplaced trust, causing high-stakes users to accept inaccurate predictions and risk misdiagnosis or inappropriate treatment.The medical example contrasts plausible reliance on WBC count and serotonin levels with faithful reliance on spurious visit-timing features.
- 5.2. What if explanations are plausible but not faithful ?: Plausible explanations can also conceal unsafe heuristics: a refusal explanation may cite harmful words while contextual prompts reveal that the model still generates harmful content.
- 5.4. Choosing Faithfulness vs. Plausibility is Use-Case Driven: High-stakes applications such as clinical decision support require faithful explanations, whereas learning interactions often benefit more from explanations that align with human reasoning.
- 5.3. What if explanations are faithful but not plausible ?: Faithful but implausible explanations may be technically accurate yet too complex for users, reducing usability and acceptance in interactive settings.For a factorial question, describing the model’s internal computation may help less than the human-readable calculation 5! = 5×4×3×2×1 = 120.
- 5.4. Choosing Faithfulness vs. Plausibility is Use-Case Driven: The value of plausibility or faithfulness is determined by users’ end goals, so developers and stakeholders should prioritize these properties according to the application.
6. Call for Community
The paper calls for explanations that remain faithful to LLM reasoning while matching the differing needs of high-stakes, interactive, educational, and recreational applications. It proposes coordinated research on evaluation, generation, interpretability, and domain-specific deployment.
- Faithfulness is foundational for trustworthy LLM deployment, because plausible explanations can obscure the models’ reasoning.
- Explanation requirements vary by context: high-stakes domains prioritize faithfulness, whereas conversational, educational, creative, and recreational settings benefit more from plausibility and interactivity.
- The community should develop reliable faithfulness metrics, standardized benchmarks, human-in-the-loop assessments, and methods that transparently expose LLM decision-making.
- Proposed strategies include fine-tuning on aligned domain data, in-context examples of faithful explanations, and mechanistic interpretability of model components.
- LLMs for High-Stakes Domains: High-stakes healthcare, legal, and financial applications require highly faithful explanations that experts can use to understand model decisions before deployment.
- The paper calls for collaboration to resolve the plausibility-faithfulness dichotomy and enhance user trust in LLM explanations.
7. Conclusion
The conclusion frames faithful and plausible self-explanations as a difficult but necessary balance for reliable real-world LLM use. It calls for explanations that are both accurate and accessible, aligned with clarity, trust, and accountability.
- Reliable LLM deployment requires explanations that are coherent to humans while accurately reflecting models’ decision-making processes.