PaperlayerView exact revision on arXiv

Source-linked AI summary

LExT: Towards Evaluating Trustworthiness of Natural Language Explanations

Krithi Shailya, Shreya Rajpal, Gokul S Krishnan, Balaraman Ravindran

arXiv:2504.06227v1cs.CL

TL;DR

Evaluating LLM explanations is difficult because standard text metrics do not capture all relevant explanation qualities. LExT combines plausibility and faithfulness into a trustworthiness score; its six-model evaluation compares overall scores and examines trade-offs.

  • Problem

    BLEU, ROUGE, and related text metrics overlook explanation qualities such as factual accuracy, consistency, and faithfulness.

    Source: §1 · p. 2

  • Method

    LExT evaluates explanations through plausibility and faithfulness and combines those dimensions in a trustworthiness score.

    Source: §1 · p. 2

  • Results

    Across six models, the evaluation computes plausibility, faithfulness, and overall LExT scores, and compares trade-offs between the dimensions.

    Source: §1 · p. 3, §5 · p. 12

  • Takeaways & Limitations

    The framework provides a structured way to compare model explanations across plausibility and faithfulness.

    Source: §1 · p. 2, §6 · p. 15

    Future work includes addressing bias and fairness in explanations and developing comprehensive ground-truth datasets for factual verification.

    Source: §6 · p. 15

Abstract

from arXiv · show

As Large Language Models (LLMs) become increasingly integrated into high-stakes domains, there have been several approaches proposed toward generating natural language explanations. These explanations are crucial for enhancing the interpretability of a model, especially in sensitive domains like healthcare, where transparency and reliability are key. In light of such explanations being generated by LLMs and its known concerns, there is a growing need for robust evaluation frameworks to assess model-generated explanations. Natural Language Generation metrics like BLEU and ROUGE capture syntactic and semantic accuracies but overlook other crucial aspects such as factual accuracy, consistency, and faithfulness. To address this gap, we propose a general framework for quantifying trustworthiness of natural language explanations, balancing Plausibility and Faithfulness, to derive a comprehensive Language Explanation Trustworthiness Score (LExT) (The code and set up to reproduce our experiments are publicly available at https://github.com/cerai-iitm/LExT). Applying our domain-agnostic framework to the healthcare domain using public medical datasets, we evaluate six models, including domain-specific and general-purpose models. Our findings demonstrate significant differences in their ability to generate trustworthy explanations. On comparing these explanations, we make interesting observations such as inconsistencies in Faithfulness demonstrated by general-purpose models and their tendency to outperform domain-specific fine-tuned models. This work further highlights the importance of using a tailored evaluation framework to assess natural language explanations in sensitive fields, providing a foundation for improving the trustworthiness and transparency of language models in healthcare and beyond.

1 Introduction

The introduction argues that explanations need evaluation beyond text similarity, especially in high-stakes settings. It presents LExT as a framework combining plausibility and faithfulness, then applies it to compare medical-model explanations.

1 Introduction

  • Generic language-generation metrics can miss coherence, factual accuracy, and reasoning fidelity, making convincing but unreliable explanations a concern.The introduction notes that BLEU, ROUGE, and METEOR emphasize semantic word similarity, while BERTScore does not address consistency or robustness.Source: §1 · p. 2
  • LExT evaluates explanation trustworthiness through plausibility and faithfulness, combining perceived validity with correspondence to the model’s actual reasoning.Plausibility includes alignment with expert knowledge, semantic accuracy, and contextual relevance; faithfulness concerns the model’s reasoning and task adherence.Source: §1 · p. 2
  • The framework introduces Context Relevancy, QAG score, Counterfactual Stability, and Contextual Faithfulness as metrics within an integrated evaluation.The metrics are described as inspired by prior research but redefined and quantified for this framework.Source: §1 · p. 3
  • Experiments apply the framework to two medical datasets and evaluate six models, including medically fine-tuned and general-purpose LLMs.The introduction describes the work as a comparative analysis of explanatory capabilities.Source: §1 · p. 3
  • The paper compares models’ plausibility-faithfulness trade-offs to identify areas for improvement.The introduction says this comparative analysis is visualized, but gives no specific winner or score.Source: §1 · p. 3

2 Related Work

Prior work identifies limits in text-similarity and faithfulness metrics, and studies trade-offs between plausible and faithful explanations. The paper positions LExT as a quantitative framework integrating both dimensions.

2 Related Work

  • BLEU, ROUGE, and embedding similarity do not provide a comprehensive explanation assessment, including consistency and robustness to counterfactual or paraphrased prompts.These limitations motivate broader evaluation of explanation reliability.Source: §2 · p. 3
  • Chen et al. [8] assess whether explanations let people predict outputs on modified inputs, while OpenXAI benchmarks post-hoc explanations across stability and fairness [3].The cited OpenXAI work reports trade-offs between reliability and variability.Source: §2 · p. 3
  • Prior work describes plausibility-faithfulness tensions: Chain-of-Thought explanations may favor plausibility over faithful reasoning [36], while Agarwal et al. [4] call for balancing both dimensions.The cited discussion says current studies largely conceptualize this challenge without quantitative assessment frameworks.Source: §2 · p. 4
  • Madsen et al. [26] introduce LLM self-explanation faithfulness metrics, but the related-work discussion says these are not integrated into a unified trustworthiness paradigm.The paper frames its approach as quantifying both faithfulness and plausibility.Source: §2 · p. 4
  • The paper proposes a framework intended to support transparent, reliable AI in diverse real-world applications, particularly high-stakes domains.The discussion names high error costs as a motivation for that focus.Source: §2 · p. 4

3 Proposed Methodology

The framework evaluates model explanations through Plausibility and Faithfulness, aggregating metrics calculated per prompt and averaged across the dataset into a model-level LExT score. Plausibility covers correctness and consistency, with specialized measures for semantic alignment, context relevance, and response stability.

3 Proposed Methodology

  • The framework combines Plausibility and Faithfulness metrics, averaging per-prompt scores across the dataset to produce an overall LExT score for each model.Source: §3 · p. 4

3.1 Plausibility

  • Plausibility assesses whether explanations appear reasonable from a human perspective and incorporates metrics for correctness and consistency.Source: §3.1 · p. 4

3.1.1 Correctness.

  • Correctness combines semantic similarity with medical NER overlap, while Context Relevancy provides a fallback when NER overlap fails.The fallback uses a larger model to generate a question from the predicted explanation and compares it with the ground-truth question.Source: §3.1.1 · p. 6

3.1.2 Consistency.

  • Consistency measures response stability across repeated or paraphrased prompts; examples show similarity falling from 0.866533 to 0.32523147 by Iteration 5.Paraphrased prompts can also yield generic or less specific explanations rather than retaining ground-truth alignment.Source: §3.1.2 · p. 7

Faithfulness

The faithfulness evaluation combines Question Answer Generation, Counterfactual Stability, and Contextual Faithfulness to probe whether explanations reflect a model’s reasoning across different circumstances. LExT then combines plausibility and faithfulness while accounting for their overall magnitude and disagreement.

3.2 Faithfulness

  • Question Answer Generation tests whether the model can answer questions derived from its predicted explanation using that explanation as context.A larger LLM generates questions, which are passed to the target model for a binary answerability judgment.Source: §3.2 · p. 8

3.2.2 Counterfactual Stability.

  • Counterfactual Stability tests adaptation by rephrasing an explanation to imply the opposite label and scoring whether the model flips its response.A matching flip scores +1, no meaningful response scores 0, and retaining the original label scores −1; scores are min-max normalized.Source: §3.2.2 · pp. 8–9
  • Contextual Faithfulness removes key information in complete and sequential redaction phases to assess how explanations depend on context.The model is asked to identify essential keywords; failure on complete redaction yields a score of 0, while sequential redaction measures responses to each removed keyword.Source: §3.2.2 · p. 9

3.3 Evaluation Framework

  • LExT combines plausibility and faithfulness, rewarding strong performance while penalizing large disagreements and downweighting small disagreements when both scores are low.The framework gives the dimensions equal weight and scales the disagreement penalty to avoid overreacting to noise from low-performing models.Source: §3.3 · p. 10
  • The framework treats plausibility as alignment with human expectations and faithfulness as reflection of the model’s actual reasoning.It averages each dimension’s components and integrates both dimensions into a comprehensive explanation-quality measure.Source: §3.3 · pp. 10–11

4 Implementation

The evaluation applies the framework to QPain and PubMedQA and compares three healthcare fine-tuned models with three general-purpose models. The datasets provide medical questions and expert-level explanations for assessing generated explanations.

4.1 Datasets

  • QPain and PubMedQA provide complementary medical evaluation settings, spanning pain-management scenarios and biomedical research questions.QPain includes patient attributes and expert-generated explanations for treatment recommendations and pain assessment; PubMedQA supplies long answers used as ground explanations for ground-truth-dependent metrics.Source: §4.1 · p. 11

4.2 Models

  • The six evaluated models comprise three healthcare fine-tuned systems and three general-purpose models, enabling comparison of specialized and general LLMs.The study includes Biomistral, MMed-Llama3, and Meditron alongside Llama3, Gemma, and Phi3.5.Source: §4.2 · p. 11
  • Llama 70B was used for specific faithfulness-evaluation tasks that require more advanced models.The study focuses on smaller models for lower computational requirements, but notes that flipped-explanation tasks need larger, more stable models.Source: §4.2 · pp. 11–12

5 Results and Discussion

Across QPain and PubMedQA, general-purpose models Llama 3 and Gemma outperform some medically fine-tuned models on plausibility and faithfulness, with performance varying by dataset. The results also show why explanation evaluation must consider multiple dimensions and how they trade off.

5.2 R2: How do these metrics reveal differences in model performance across specific domains or tasks?

  • Llama 3 and Gemma outperform MMed-Llama3 and Meditron in plausibility and faithfulness, while Gemma leads on PubMedQA and Llama 3 shows high trustworthiness on QPain.The reported overall LExT scores combine plausibility and faithfulness; the prose also notes that the fine-tuned models benchmark better on accuracy.Source: Table 6 · p. 12, §5.2 · pp. 13–14

5 Results and Discussion

  • The discussion evaluates six models across correctness, consistency, QAG score, counterfactual stability, and contextual faithfulness, then computes plausibility, faithfulness, and LExT scores.Tables 6 and 7 present overall scores and individual metric analysis across QPain and PubMedQA.Source: §5 · p. 12

5.1 R1: Why is it necessary to evaluate not just the answers but also how the models generate them?

  • Correctness and consistency alone miss differences in robustness and contextual faithfulness, with Gemma strong on correctness but performing poorly on QPain.The metric analysis also reports MMed-Llama excelling on PubMedQA in QAG, Gemma and MMed-Llama handling input variations effectively, and Llama performing strongly in contextual faithfulness.Source: §5.1 · pp. 12–13, Table 7 · p. 13

5.3 R3: Why are faithfulness and plausibility related, and what does their relationship reveal?

  • The faithfulness–plausibility comparison places Llama and Gemma high on both dimensions, while MMed-Llama3 and Biomistral are plausible but less faithful.Phi 3.5 and Meditron are reported as low on both dimensions.Source: Figure 6 · p. 14, §5.3 · p. 14

5.4 R4: How generalizable are these metrics across datasets and domains?

  • The authors describe the metrics as dataset-agnostic and say the framework can be adapted to other high-stakes areas such as legal or financial question-answering.The framework aggregates the metrics to provide a unified assessment across datasets and domains.Source: §5.4 · pp. 14–15

6 Conclusion and Future Scope

The paper presents LExT as a structured framework for evaluating and comparing model explanations across dimensions and their trade-offs. Its future scope includes addressing explanation bias and fairness, building ground-truth datasets for factual verification, and optimizing smaller models.

6 Conclusion and Future Scope

  • LExT integrates explanation metrics to compare model strengths and weaknesses and examine trade-offs between performance and interpretability.The conclusion frames faithfulness, plausibility, and trustworthiness as key evaluation metrics for high-stakes domains.Source: §6 · p. 15
  • Future work includes addressing explanation bias and fairness across diverse contexts and developing comprehensive ground-truth datasets for factual verification.The paper also identifies optimizing smaller language models as a future direction to reduce dependence on large models for complex tasks.Source: §6 · p. 15

A.1 Dataset Modification and Augmentation

The QPain dataset was augmented with demographic variations to support evaluation of language models across social contexts. Patient vignettes, questions, and explanations were personalized to match those variations.

A.1 Dataset Modification and Augmentation

  • Names were selected across six racial and ethnic demographics, alongside gender-specific attributes and pronouns.The listed groups are Black, White, Asian, Hispanic, Indian/South Asian, and Middle Eastern.Source: §A.1 · p. 22
  • Vignettes were personalized by replacing demographic and grammatical placeholders, and questions and explanations were updated to use matching names.Source: §A.1 · p. 22
  • The expanded dataset varies race, gender, and associated pronouns to assess model performance, consistency, fairness, and contextual understanding in medical QA.Source: §A.1 · p. 22

A.2 Derivation of the LExT Score

The LExT derivation chooses a disagreement penalty scaling tied to the harmonic mean of plausibility and faithfulness. The authors say this scaling avoids disproportionate penalties for low-performing models.

A.2 Derivation of the LExT Score

  • The penalty function g is scaled in relation to the harmonic mean of plausibility P and faithfulness F.The passages describe substituting a choice of g to obtain the proposed metric form, but do not provide a readable expression for that form.Source: §A.2 · p. 22
  • The authors state that this scaling makes the disagreement penalty proportional to the overall magnitude of P and F, avoiding disproportionate penalties for low-performing models.Source: §A.2 · p. 22

B Experimental Analysis

This section describes prompts and procedures for evaluating accuracy and explanation properties on QPain and PubMedQA. It also tests response variation under paraphrased inputs.

B Experimental Analysis

  • The experimental analysis uses prompts based on each model’s documented preferred format to elicit responses for metric evaluation.Source: B Experimental Analysis · p. 22

B.1.1 QPain Dataset.

  • For QPain, models act as medical advisors, recommend a low or high dosage, and provide an explanation; extracted answers and explanations are compared with ground explanations.Source: §B.1.1 · p. 23

B.1.2 PubMedQA Dataset.

  • For PubMedQA, models answer medical-research questions with Yes or No and provide an explanation, given a context and question.Source: §B.1.2 · p. 23

B.2 Contextual Relevancy

  • Context Relevancy is the cosine similarity between the original question and a question generated from the model’s explanation, measured with a BERT-based model.Source: §B.2 · p. 23

B.3 Paraphrasing

  • Slightly paraphrased inputs can lead to widely different model responses.The experiment paraphrases the context three times before asking the model to generate an answer.Source: §B.3 · p. 24

B.4 QAG Score

  • The QAG score tests whether an explanation can answer questions generated from that explanation.Source: §B.4 · p. 24

Part 1: Important Words and Context Modification.

  • The contextual-faithfulness experiment extracts five context keywords, redacts them, and tests model predictions on the modified contexts.The section describes this as a two-part experiment to assess behavior under modified contexts and quantify faithfulness to contextual reasoning.Source: §B.5 · p. 24, Part 1: Important Words and Context… · pp. 24–25

Part 2: Faithfulness and Accuracy Evaluation.

The section tests contextual faithfulness by redacting important context words and checking predictions, and tests counterfactual stability by feeding an opposite-label explanation back to the model. An example reports a contextual-faithfulness score of 0.6.

Part 2: Faithfulness and Accuracy Evaluation.

  • The procedure also iteratively restores redacted words and records the resulting predictions to evaluate consistency and faithfulness to the context.Source: Part 2: Faithfulness and Accuracy… · p. 25
  • The evaluation extracts five prediction-relevant keywords, tests the fully redacted context, then restores keywords one at a time to assess their necessity.Responses are classified as Yes, No, Unknown, or Random.Source: Part 2: Faithfulness and Accuracy… · p. 25

Keyword Extraction.

  • The keyword-extraction prompt requests five words from the context that led to the predicted label and without which the label could not be predicted.Source: Keyword Extraction · p. 26

Label Prediction Analysis.

B.6 Counter Factual Stability

  • Counterfactual stability checks whether a model’s label flips after its explanation is rephrased to suggest the opposite label and returned with the original question.Source: §B.6 · p. 26
  • The counterfactual procedure rephrases the explanation to suggest the opposite label, then compares the model’s new label with its original label.Source: §B.6 · p. 26

B.7 Example

Contextual Faithfulness Response (Score: 0.6)

Trustworthiness Metrics

The reported trustworthiness metrics are Plausibility 0.69 and Faithfulness 0.53.

Trustworthiness Metrics

Related papers

Loading 2504.06227v1…