A Positive Case for Faithfulness: LLM Self-Explanations Help Predict Model Behavior

Harry Mayne; Justin Singh Kang; Dewi Gould; Kannan Ramchandran; Adam Mahdi; Noah Y. Siegel

Source-linked AI summary

A Positive Case for Faithfulness: LLM Self-Explanations Help Predict Model Behavior

Harry Mayne, Justin Singh Kang, Dewi Gould, Kannan Ramchandran, Adam Mahdi, Noah Y. Siegel

arXiv:2602.02639v1Feb 2, 2026cs.AIcs.LG

TL;DR

Faithfulness evaluations for LLM explanations remain limited because existing methods often rely on adversarial vulnerabilities or reasoning errors. The paper introduces NSG, which measures whether explanations improve prediction of model behavior on natural counterfactuals, and finds that self-explanations provide predictive information and outperform external explanations. The approach is constrained by counterfactual quality, tabular main results, classification settings, and non-state-of-the-art predictors.

Problem
Existing faithfulness metrics rely on adversarial vulnerabilities or reasoning errors, leaving a limited evaluation signal for frontier LLMs.
Method
NSG measures the predictive information in explanations by comparing observer accuracy with and without explanations on natural counterfactual inputs.
Results
Self-explanations encode valuable predictive information for evaluated models and consistently outperform explanations from external models, even when those models are stronger.
Takeaways & Limitations
Self-explanations make a positive case for faithfulness by revealing information about models’ decision-making criteria that external observers may not access.
Takeaways & Limitations
NSG depends on counterfactual quality; the main results use tabular data and classification tasks, while free-text generation remains open and the predictors are not state-of-the-art.

Abstract

from arXiv · show

LLM self-explanations are often presented as a promising tool for AI oversight, yet their faithfulness to the model's true reasoning process is poorly understood. Existing faithfulness metrics have critical limitations, typically relying on identifying unfaithfulness via adversarial prompting or detecting reasoning errors. These methods overlook the predictive value of explanations. We introduce Normalized Simulatability Gain (NSG), a general and scalable metric based on the idea that a faithful explanation should allow an observer to learn a model's decision-making criteria, and thus better predict its behavior on related inputs. We evaluate 18 frontier proprietary and open-weight models, e.g., Gemini 3, GPT-5.2, and Claude 4.5, on 7,000 counterfactuals from popular datasets covering health, business, and ethics. We find self-explanations substantially improve prediction of model behavior (11-37% NSG). Self-explanations also provide more predictive information than explanations generated by external models, even when those models are stronger. This implies an advantage from self-knowledge that external explanation methods cannot replicate. Our approach also reveals that, across models, 5-15% of self-explanations are egregiously misleading. Despite their imperfections, we show a positive case for self-explanations: they encode information that helps predict model behavior.

1. Introduction

The paper introduces NSG to evaluate faithfulness by testing whether explanations help observers predict model behavior on natural counterfactuals. Across frontier models, self-explanations provide predictive information and may benefit from privileged self-knowledge.

Motivation: Prior faithfulness metrics often depend on adversarial vulnerabilities or reasoning errors, creating a vanishing signal as model capabilities improve.The paper frames this as an evaluation gap for frontier LLMs.
Method: NSG measures whether an explanation helps an observer predict a model’s behavior on related counterfactual inputs, testing what explanations reveal rather than only exposing failures.The framework compares predictor accuracy with and without access to the reference model’s explanation.
Method: Natural-data counterfactuals are used instead of synthetic perturbations to capture more meaningful changes while evaluating complex model logic.The authors identify counterfactual quality as central to NSG’s validity.
Results: Self-explanations consistently outperform explanations from external models, including stronger external models, suggesting privileged access to self-knowledge.This comparison supports the paper’s positive case for self-explanation faithfulness.

2. A test of faithfulness

The paper operationalizes faithfulness as an explanation’s ability to improve prediction of a model’s behavior on related counterfactual inputs. NSG compares prediction with and without explanations using plausible local counterfactuals.

2.1. Characterizing faithfulness: The framework uses counterfactual inputs similar to the original but differing in selected features, testing whether explanations predict behavior across a local region.
2.2. Measuring faithfulness: Predictor accuracy is compared when the predictor receives the reference answer and counterfactual alone versus the same information plus the explanation.
2.1. Characterizing faithfulness: NSG measures how much explanations improve an observer’s prediction of the reference model on related counterfactuals.An NSG of 1 means explanations enable perfect counterfactual prediction, while 0 means they provide no predictive benefit.
2.2. Measuring faithfulness: NSG interpretation is sensitive to predictor choice, counterfactual selection, and dataset size because variance increases when the denominator is small.
2.3. Defining the counterfactual region of interest: Counterfactuals are drawn from real tabular data using semantic closeness and Hamming distance, producing multivariate, plausible, and local perturbations.

3. Experimental setup

The experiments cover seven tabular classification datasets, 18 reference models, and an ensemble of five predictor models. Counterfactual pairs are sampled at Hamming distance at most 2, yielding 7,000 evaluation pairs.

3. Experimental setup: The study evaluates seven tabular datasets spanning health, business, and ethics, converting each into a binary classification task.
3. Experimental setup: 7,000 question–counterfactual pairs are constructed by sampling 1,000 examples per dataset with Hamming distance at most 2.
3. Experimental setup: Counterfactual design balances multivariate changes with relevance by restricting selected examples to Hamming distance at most 2.
3. Experimental setup: The evaluation includes 18 reference models from popular proprietary and open-weight LLM families, with one output generated per unique input.
3. Experimental setup: Results are averaged across five predictor models to reduce dependence on any single predictor.

4. Results

Self-explanations improve prediction of model behavior, but faithfulness varies with model scale, dataset, consistency, and feature type, with some explanations egregiously misleading.

4.1. A positive case for faithfulness: 3.8–10.8% absolute simulatability gain and 11.0–36.5% NSG show that self-explanations improve prediction across all reference models.For the best-performing models, explanations fix roughly a third of incorrect predictions.
4.1. A positive case for faithfulness: Faithfulness scales monotonically with parameter count for Qwen 3 and trends upward for Gemma 3, but shows no clear trend among proprietary models.The authors suggest scaling relationships break down beyond a modest capability threshold.
4.3. Characterizing unfaithfulness: Small Qwen-3-0.6B and gemma-3-1b-it models have about 15% egregiously unfaithful explanations, versus about 7% for frontier models.Egregious unfaithfulness occurs when an explanation causes all predictors to make the incorrect prediction.
4.3. Characterizing unfaithfulness: Unfaithfulness is dataset-dependent: NSG is 6.0% on Moral Machines and 42.8% on Pima Diabetes, with some model–dataset pairs statistically indistinguishable from zero.These positive NSG results are average-case findings.
4.3. Characterizing unfaithfulness: Repeated-rollout analysis finds measured NSG of 35.6% for Qwen3-32B and 34.9% for gemma3-27b-it, while inconsistency explains only part of the gap.The consistency upper bound for Qwen3-32B is 77.8%; the supplied passage truncates the corresponding Gemma upper bound.
4.3. Characterizing unfaithfulness: Education level and Occupation increase income-prediction unfaithfulness, while Race, Sex, and Age are neutral; Radiation therapy has RR = 1.83 for breast-cancer recurrence.The analysis reports RR = 1.60 for Education level and RR = 1.29 for Occupation in Figure 7.

5. Do models have privileged self-knowledge?

The paper tests whether self-explanations contain information unavailable to external explainers and finds a consistent self-explanation advantage across model families.

5. Do models have privileged self-knowledge?: The cross-explanation design swaps self-explanations for explanations from different model families while restricting predictors outside both families.This isolates the effect of privileged access under the paper’s comparison design.
5. Do models have privileged self-knowledge?: Self-explanations consistently encode more predictive information than cross-model explanations, even when the external explainer is stronger.The comparison is averaged within model families using the top three models from each family.
5. Do models have privileged self-knowledge?: The positive advantage also appears for Gemma 3, the only non-reasoning family tested, providing evidence for an introspection effect beyond reasoning-trace access.The authors also identify privileged access to a reasoning trace as a possible source of advantage.

6. Related work

Prior faithfulness evaluations mainly detect adversarial vulnerabilities or reasoning errors, while this work builds on counterfactual simulatability to measure predictive information relative to a no-explanation baseline.

6. Related work: Existing faithfulness tests use hidden cues or reasoning errors, but their signals weaken as models become less vulnerable to those failure modes.The paper characterizes this as a vanishing signal problem.
6. Related work: Counterfactual simulatability evaluates whether an observer can predict model behavior from an explanation.The paper extends prior LLM simulatability work by measuring accuracy relative to a baseline without explanation access.

7. Discussion

The paper argues that imperfect self-explanations still provide useful predictive information, while identifying open questions about malicious obfuscation, training, and safety guarantees.

7. Discussion: Self-explanations should not be discarded: despite widespread unfaithfulness, they encode valuable predictive information about model behavior.The paper presents this as a positive case for faithfulness.
7. Discussion: NSG addresses a limited toolkit for testing explanation faithfulness, but its relationship to malicious settings where models may obfuscate reasoning remains unanswered.The safety implications are therefore established for benign evaluation settings, not malicious ones.
7. Discussion: Model scale matters up to a modest capability threshold and privileged self-knowledge improves faithfulness, while other drivers remain an open question.The paper suggests finetuning and using NSG as a training incentive as possible future directions.
7. Discussion: NSG depends on counterfactual quality, remains average-case, does not yet extend to free-text generation, and may be confounded by evaluation awareness.The paper also notes that predictor models are not state-of-the-art and worst-case safety guarantees may require more than NSG.

8. Conclusion

NSG addresses the limitation that existing faithfulness metrics do not scale to frontier LLMs by measuring predictive information in self-explanations.

NSG addresses the limitation that existing faithfulness metrics do not scale to frontier LLMs.

Impact statement

Self-explanations encode decision-making information that improves prediction of model behavior, while some explanations are misleading and their faithfulness varies across models, datasets, and features.

Self-explanations produce statistically significant positive simulatability gains across all reference models, improving predictors’ ability to anticipate model behavior.Table 3 reports predictor accuracy with and without explanations, absolute gains, and NSG averaged across five predictor models.
Faithfulness varies substantially by dataset and model, including significantly negative NSG for some models and datasets such as Moral Machines.Gemma3-12b-it reaches NSG = −30.4% with 95% bootstrap CIs [−44.9, −17.4]% on Moral Machines.
Predictor ensembles preserve model rankings strongly, with Kendall’s W reaching 0.950 for four-model leave-one-out ensembles and declining to 0.861 for two-model ensembles.Most rank changes occur among models whose differences are statistically insignificant.
Self-explanations consistently provide more predictive information than complete chain-of-thought traces for the evaluated Qwen 3 models.The comparison used three predictor models and found higher NSG for user-facing explanations than raw reasoning traces.
Unfaithfulness is concentrated in some technically complex features, whereas sensitive attributes in the Income dataset show approximately neutral unfaithfulness rates.Radiation therapy has RR = 1.83 and Education level has RR = 1.60, while Race, Sex, and Age have RR ≈1.0.
Feature importance alone does not explain unfaithfulness: overall correlation with answer-change risk is weak, and the relationship differs by domain.Employee Attrition and Income show stronger alignment between important and difficult-to-explain features, while medical and other domains can diverge.

A.9. Cross-model explanation ablations

The ablation compares self-explanations with cross-model explanations while holding the original answer fixed. Self-explanations provide more predictive information across model families and reasoning strengths, though the advantage varies with counterfactual distance and dataset.

Experimental design: The cross-model comparison holds the explained answer fixed and excludes predictors from the relevant model families.This isolates explanation quality from differences in the underlying answer and family overlap.
Self-explanation uplift: Self-explanations yield statistically significantly higher NSG than cross-model explanations across all five model families.The uplift is +4% for Qwen 3, +4.4% for GPT-5, and +0.8% for Gemma 3.
Reasoning strength: Self-explanation advantage persists across reasoning strengths and does not systematically increase with reasoning strength.The result suggests privileged access to internal reasoning contributes beyond access to explicit reasoning traces.
Representative example: In an Attrition example, all five predictors were correct with self-explanations but incorrect with cross-explanations.Both reference models gave the same original answer, but their explanations emphasized different factors.
Metric: Self-explanation uplift is defined as the difference between same-model and cross-model NSG.A positive uplift indicates that self-explanations encode more predictive information than cross-model explanations.
Counterfactual distance: NSG decreases monotonically as counterfactual Hamming distance increases in the Qwen-3-32B evaluation.The analysis samples counterfactuals at distances up to 5 across six datasets, excluding Moral Machines.

B.2. Relationship to prior metrics

The paper contrasts NSG with prior counterfactual-based faithfulness tests. Its approach supports multivariate, coherent counterfactuals sampled from natural data, addressing limitations of single-feature or randomly edited alternatives.

Prior metrics: Biased-prompt tests vary a single cue and increasingly face resistance from frontier models.Their limited complexity and variety constrain the forms of unfaithfulness they can test.
Prior metrics: Random feature perturbation can produce incoherent counterfactuals and alter unimportant words.These methods generally edit one randomly selected word at a time.
Prior metrics: Concept-space perturbation changes one feature at a time and can generate implausible examples when concepts are correlated.Single-feature changes miss complex interaction logic.
Proposed approach: The approach supports multivariate counterfactuals, natural-data coherence, and balanced ground-truth labels.These design choices are presented as improvements over prior counterfactual-generation approaches.
Evaluation data: The evaluation combines six tabular classification datasets with the Moral Machines dataset and samples 1,000 records from each.The paper provides dataset-specific prompts and feature counts separately.

C.1. Model performance on datasets

The dataset benchmark shows uneven reference-model performance across tasks. Bank Marketing is easiest for many models, Attrition is hardest for most, and model scale or extra reasoning does not reliably predict accuracy.

Evaluation scope: Moral Machines is excluded from the reference-accuracy analysis because it lacks objective ground-truth answers.The exclusion applies to this accuracy analysis rather than to the entire paper.
Dataset performance: Bank Marketing has the strongest performance, with many models exceeding 80% accuracy, while Attrition is most challenging for most models.The accompanying table reports reference-model accuracy by dataset.
Scaling: Model scale does not reliably predict dataset accuracy, with non-monotonic scaling in the Gemma and Qwen families.Within Gemma, the 27B model performs worse than two smaller counterparts on average.
Scaling: Reasoning effort has minimal impact on accuracy on these datasets.The result is reported alongside the observed non-monotonic family-level scaling.

D. Error case studies and taxonomy

The error taxonomy identifies spurious attribution, critical omission, uncertainty, and evaluation awareness as distinct issues in explanation faithfulness. Case studies show that explanations can misidentify influential features or omit features that change predictions.

Spurious attribution: Spurious attribution occurs when an explanation assigns importance to a feature that does not govern the model’s behavior.A Moral Machines example claims an inaction principle but violates it on a related counterfactual.
Critical omission: Critical omission occurs when an explanation leaves out a feature that demonstrably changes the model’s prediction.In an income task, Claude Sonnet 4.5 omits race and marital status despite changing its prediction when they vary.
Uncertainty: The current NSG protocol does not assess explanations expressing uncertainty about the model’s choice.Addressing this case would require resampling answers or comparing output distributions.
Evaluation awareness: Evaluation awareness appeared in six examples among more than 145,000 explanations, with at least one judge scoring above 3.Four cases involved Gemini-3-Pro recognizing that the setting resembled a known dataset.
Case-study interpretation: The case studies illustrate that misleading explanations can preserve an answer while offering reasoning that would mislead a predictor.The Moral Machines examples contrast stated principles with behavior under gender-swapped counterfactuals.