PaperlayerView exact revision on arXiv

Source-linked AI summary

Faithfulness Without Anchors: A Position Paper on the Structural Misalignments of Explainable AI Faithfulness Metrics for Feature Importance

Marco Zullich, Emily Schiller, Ivan Gentile, David Dembinsky, Adriano Lucieri, Kilian Göller, Steffen Seitz

arXiv:2610.12493v1cs.AIcs.LGstat.ML

TL;DR

Faithfulness metrics for feature importance are intended to assess how explanations align with the model, but existing approaches have important limitations. This position paper analyzes those shortcomings and proposes research directions; it concludes that current approaches have five major flaws that limit faithfulness assessment.

  • Problem

    Existing feature-importance faithfulness metrics can be unreliable, relying on perturbations affected by out-of-distribution data and arbitrary baselines, or conflating faithfulness with other properties.

    Source: §3.2 · p. 3, §3.4 · p. 4

  • Method

    The paper analyzes recent studies of faithfulness-metric unreliability and organizes its criticism around five shortcomings and research directions for evaluation.

    Source: §1 · p. 1, §6 · p. 7

  • Results

    The paper identifies five shortcomings: classification focus, conflation with other XAI facets, arbitrary baselines, perturbation-induced out-of-distribution data, and misaligned faithfulness metrics.

    Source: §6 · p. 7

  • Takeaways & Limitations

    The authors argue that faithfulness research should move beyond established but unreliable metrics toward approaches grounded in faithfulness goals, model uncertainty, and expected feature-importance properties.

    Source: §1 · p. 1

    ROAR evaluates explanations using a retrained model that may differ internally from the original model.

    Source: §3.2 · p. 3

Abstract

from arXiv · show

Explainable Artificial Intelligence (XAI) tools have often been presented as methods to increase trust in machine learning models by making aspects of their algorithms more human-understandable. However, evidence accumulated over the last decade points to an overall failure of this objective, instead promoting overreliance on incorrect predictions, which can subsequently lead to algorithmic aversion. Rather, XAI appears to function primarily as a debugging tool, useful for obtaining anecdotal insights into, for example, features learned by the model. One of the main issues underlying this limitation is that XAI methods often oversimplify the inner workings of the model, resulting in misalignment between the explanation and the model itself - a behavior referred to as unfaithfulness or infidelity. Faithfulness can be interpreted as the degree of approximation between an explanation and the model. Computing faithfulness in a rigorous manner would thus make it possible to quantify one of the various elements of trustworthiness of an explanation. However, existing metrics present significant limitations, making this quantification either infeasible or computationally expensive. In this position paper, we outline five major shortcomings of current faithfulness metrics and propose four possible avenues for research to bolster their reliability.

1 Introduction

XAI aims to make black-box models more understandable, but evidence suggests it can fall short and foster overreliance on incorrect predictions. This paper examines why current feature-importance faithfulness metrics are unreliable and points toward research directions for improving them.

1 Introduction

  • XAI can foster overreliance on incorrect predictions, while infidelity between explanations and models is identified as a source of these shortcomings.The cited discussion notes possible serious consequences in safety-critical applications and potential algorithmic aversion.Source: §1 · p. 1
  • Current feature-importance faithfulness metrics can be unreliable because of out-of-distribution data, arbitrary baselines, retraining, or conflation with other XAI properties.The resulting uncertainty concerns whether feature importance explanations are unfaithful or whether the metrics themselves are dubious.Source: §1 · p. 1
  • The paper analyzes recent research on metric unreliability and argues that XAI should prioritize more trustworthy faithfulness evaluation.It proposes moving beyond pixel flipping and ROAR [Hooker et al., 2019] toward approaches considering faithfulness goals, model uncertainty estimation, and expected feature-importance properties.Source: §1 · pp. 1–2

2 Background

The background defines feature importance and distinguishes faithfulness from other ways of evaluating explanations. It describes perturbation-based evaluation and correlation- and area-based metrics.

2.1 Feature Importance

  • Feature-importance explanations assign each feature a score indicating its importance in the model’s predictions; explanations may be global or local.Score magnitude indicates importance, while sign indicates whether a feature increases or decreases the prediction.Source: §2.1 · p. 2

2.2 Evaluation of Feature Importance

  • Explanation evaluation can address content, presentation, and user perspectives; content includes faithfulness, robustness, and compactness, while plausibility concerns alignment with human prior knowledge.The passage attributes this categorization to Nauta et al. [2023] and the faithfulness grouping to Dembinsky et al. [2026].Source: §2.2 · p. 2

Faithfulness in Feature Importance

  • Faithfulness evaluation commonly perturbs features and tracks model output; guided MoRF or LeRF orders features by importance, while correlation and curve-area measures summarize the resulting behavior.For classification, the example plots confidence against perturbation iteration; Figure 1 illustrates expected MoRF and LeRF trends, though curves can be noisy.Source: Faithfulness in Feature Importance · pp. 2–3, Figure 1 · p. 3

3 Issues with existing faithfulness metrics

The section argues that feature-importance faithfulness metrics can diverge from explanation–model alignment because of task assumptions, perturbation choices, and scoring rules that reward unsuitable behaviors.

3.1 Faithfulness computation is almost exclusively operated on classification

  • Most faithfulness metrics target classification, while regression lacks a settled target, comparable normalization, and a clear monotonic-degradation criterion.The paper contrasts changes in predicted values with changes in ground-truth error; focusing on prediction error can conflate model accuracy with explanation quality.Source: §3.1 · p. 3

3.2 Some faithfulness metrics conflate fidelity with other properties of explanations

  • Some metrics conflate explanation–model alignment with plausibility or ground-truth performance rather than assessing the model’s own predictions.The paper notes that ROAR [Hooker et al., 2019] also evaluates a retrained model that may differ internally from the original.Source: §3.2 · p. 3

3.3 Perturbation as a stand-in for feature removal requires arbitrary baselines

  • Perturbation baselines are arbitrary because no single approach reproduces feature removal or defines missing information across data types.Image methods include fixed imputation, blurring, noise, and inpainting; text and tabular data use different strategies.Source: §3.3 · pp. 3–4

3.4 Many perturbation-based approaches cause OoD issues with models

  • Perturbations can trigger erratic, overconfident OoD responses that conflict with metrics’ monotonic-decrease assumption; augmentation and inpainting remain limited, image-focused workarounds.The paper says augmentation only works to a reasonable perturbation level, while these approaches may not transfer readily beyond image data.Source: §3.4 · p. 4

3.5 The Formal Divergence of Correlation and Area-Based Metrics from Explanatory Faithfulness

  • AOPC can give normalized score 1 to a curve where only the highest-importance feature changes prediction, contrary to the expected proportionality.The paper says AOPC would be valid only when the explanation is binarized.Source: Figure 3 · p. 5, §3.5 · p. 5
  • Correlation scores can reward monotonic curves even when feature importance and prediction drops lack proportionality, so they need not reflect faithful behavior.Pearson’s correlation requires uniform confidence drops across iterations; Spearman’s correlation considers monotonicity but not convexity.Source: Figure 2 · p. 4, §3.5 · p. 5
  • For non-predicted classes, removing important features may increase confidence, making current approaches infeasible for measuring faithfulness.The paper links this behavior to models’ catch-all categories for OoD examples [Miró-Nicolau et al., 2025].Source: §3.5 · p. 5

4 Our proposal for the direction of FI faithfulness research

The paper proposes four directions for improving feature-importance faithfulness evaluation: account for model calibration and out-of-distribution robustness, cover non-predicted classes and regression, and align metric design with its purpose.

Proposal I

  • Faithfulness should be evaluated on well-calibrated models robust to out-of-distribution data, because perturbations can expose overconfident predictions on unfamiliar inputs.Figure 4 illustrates a catch-all class that remains highly confident as an image is replaced with black patches, unlike the 1/K behavior expected of a well-calibrated model.Source: §4 · p. 5, Figure 4 · p. 6

Proposal II

  • Classification faithfulness metrics should generalize to non-predicted classes, which may be needed for contrastive explanations.Until such methods exist, the authors recommend limiting explanations to the predicted category.Source: §4 · p. 5

Proposal III

  • Regression needs dedicated faithfulness metrics because classification-based measures do not trivially transfer and may rely on incompatible assumptions.Regression metrics must assess whether feature importance predicts both the direction and magnitude of changes in predicted values; calibration is also less established for regression models.Source: §4 · pp. 5–6

Proposal IV

  • Faithfulness metrics should be designed around explicit expectations for faithful explanations rather than current correlation- and area-based approaches.The authors call for formalizing an optimal perturbation-curve shape, while noting that perturbation choice and gradient starvation may complicate this.Source: §4 · p. 6
  • Perturbation curves should track the importance of removed features, not merely the iteration number, so prediction changes can be related to feature importance.Many approaches use iteration number, which does not directly indicate the importance of removed features.Source: §4 · p. 6
  • If an optimal perturbation-curve shape is established, faithfulness evaluation could draw on expected-curve comparisons such as expected calibration error.Figure 5 illustrates ECE as differences between binned average accuracy and the diagonal representing perfect calibration.Source: §4 · p. 6, Figure 5 · p. 6
  • Faithfulness metrics should be relative, since absolute output-logit values can differ between models and make scores difficult to interpret or compare.The paper uses IROF [Rieger and Hansen, 2020] as an example of an absolute metric.Source: §4 · p. 6

5 Contextualizing Faithfulness Assessment

Prior work has addressed aspects of faithfulness evaluation, including perturbation-related out-of-distribution issues, metric inconsistency, and governance concerns. This paper positions its proposals as a technical contribution focused on feature-importance evaluation.

Prior approaches

  • Pixel flipping, ROAR [Hooker et al., 2019], ROAD [Rong et al., 2022], and F-fidelity [Zheng et al., 2024] address perturbation-related out-of-distribution issues through retraining, blurring, or augmentative finetuning.Miró-Nicolau et al. [2025] report that common approaches may misalign with faithfulness goals on simple tasks with known important features.Source: §5 · p. 7
  • Chan et al. [2022] find inconsistencies among faithfulness metrics on NLP tasks and favor correlation-based metrics, whose issues this paper discusses.The paper also notes that Miró-Nicolau et al. [2025] offer a critical analysis without concrete future proposals.Source: §5 · p. 7
  • Seth and Sankarapu [2025] criticize metric fragmentation and manipulability from a governance perspective, without addressing technical details.Haufe et al. [2026] discuss fragile evaluation metrics and data-generating distributions but do not propose integration into existing methods.Source: §5 · p. 7

Positioning

  • The paper frames its proposals as technical responses to prior calls for more formal XAI evaluation, including defining desired perturbation curves and requiring stronger model calibration.It restricts its claims to feature importance, where substantial research is available.Source: §5 · p. 7

6 Discussion and conclusion

The discussion defines faithfulness as alignment between a feature-importance explanation and its model, then identifies five shortcomings in current metrics and proposes four research directions.

6 Discussion and conclusion

  • Faithfulness is alignment between an FI explanation and the model’s internals; metrics typically perturb features and summarize output changes with a curve correlation or area.Features are incrementally removed or revealed, and model outputs are tracked against the fraction of features removed.Source: §6 · p. 7
  • The paper identifies five shortcomings: classification focus, conflation with other evaluation facets, arbitrary baselines, out-of-distribution perturbations, and metric misalignment.Source: §6 · p. 7
  • It proposes considering model calibration, evaluating explanations for predicted and non-predicted classes, expanding regression research, and aligning metrics with faithfulness’s actual goal.Source: §6 · p. 7
  • The paper excludes other popular XAI tasks, lacks issue-summarizing experiments apart from Figure 4, and offers research recommendations without concrete solutions in some cases.The authors specifically identify Proposals II and III as recommendations without concrete approaches.Source: §6 · p. 7
  • The proposed directions do not resolve the arbitrary perturbation-baseline problem, though the paper aims to summarize shortcomings and guide future research.Source: §6 · pp. 7–8

Related papers

Loading 2610.12493v1…