Source-linked AI summary
Assessing Alignment and Stability of Feature Importance Explanations via Weight of Evidence
Eddie Conti, Claudio Daka, Álvaro Parafita, Antonio L. Alfeo, Axel Brando, Mario G. C. A. Cimino
TL;DR
Feature importance scores alone provide limited evidence about an explanation’s reasoning and alignment. The paper embeds stochastic FIM explanations in a WoE-based hypothesis-testing framework, using domain knowledge, ground truth, or mean attribution as references. It theoretically links lower attribution variability to higher WoE and empirically applies the framework to LIME and SHAP across reference settings.
Problem
FIM evaluation lacks a measure of how well explanations align with a reference hypothesis, while explanation-quality criteria are often implicit or inconsistent.
Method
The framework converts repeated stochastic FIM explanations into feature-relevance hypotheses and uses WoE to quantify support for a reference hypothesis over an alternative.
Results
Lower attribution variance produces higher WoE with mean attribution as reference, while experiments assess alignment, faithfulness, and stability using LIME and SHAP across complementary settings.
Takeaways & Limitations
WoE provides a complementary, contrastive way to evaluate FIM alignment with domain knowledge, ground truth, model mechanisms, or repeated-run behavior.
Takeaways & Limitations
The approach depends on design choices such as priors, exact hypothesis matching, hypothesis-generation strategies, and repeated FIM executions with associated computational cost.
Abstract
from arXiv · showhide
Feature importance Methods (FIMs) are widely used in Explainable AI to interpret model predictions, yet attribution scores alone often provide limited insight into the underlying reasoning process. In this work, we introduce a novel perspective by embedding FIMs within a hypothesis-testing framework based on Weight of Evidence (WoE). We quantify how strongly the observed evidence supports any given hypothesis on feature importance. The reference hypothesis can stem from domain knowledge, ground truth, or be derived from the FIM itself. This formulation enables a principled evaluation of FIMs, capturing both their alignment with prior knowledge and their variability. We further provide theoretical results linking WoE to attribution variance. Empirical results shows the applicability and flexibility of our strategy analyzing LIME and SHAP explanations in settings with different reference hypotheses. Overall, our framework offers a complementary tool for assessing FIMs through a contrastive, evidence-based lens.
1 Introduction
The paper motivates evaluating feature importance methods through explicit, testable hypotheses because explanation quality lacks consistent criteria. Its framework adapts Weight of Evidence to assess alignment and stability across reference hypotheses.
- XAI helps expose how complex black-box models process information and generate predictions, especially in high-stakes domains requiring trust and accountability.
- Existing evaluation studies often rely on implicit assumptions because interpretability and explainability lack universally accepted criteria.
- The framework treats explanations from domain knowledge, model mechanisms, or stochastic FIMs as testable hypotheses evaluated with WoE.
- Depending on the reference hypothesis, WoE tests local–global alignment, faithfulness and ground-truth alignment, or stochastic FIM stability.
- A theoretical result links low FIM variability to +∞ WoE when the reference hypothesis is generated from mean attribution.
2 Related Work
Prior XAI work uses various metrics and has applied WoE mainly to generate or structure explanations. This paper instead uses explanations as testable feature-relevance hypotheses and WoE as their evaluation criterion.
- XAI has established metrics for different FIM properties, but lacks a measure of alignment with a reference hypothesis.
- FIM denotes a feature importance method, while an attribution vector is the vector of feature importances it produces.
- Earlier WoE applications evaluate progressively refined output hypotheses or support explanation generation in goal recognition.
- In contrast, this paper treats FIM- or domain-derived explanations as testable hypotheses about which features are responsible for a model decision.
3 Weight of Evidence
The paper defines WoE as contrastive evidence strength and applies it to hypotheses extracted from repeated stochastic FIM runs. Reference hypotheses determine whether WoE measures stability or alignment.
- WoE measures the strength of evidence supporting hypothesis h over alternative h′ using log-odds.
- Positive WoE indicates that evidence supports h over h′, while the framework commonly uses the complementary explanation as the alternative.
- Repeated FIM executions on one instance produce attribution vectors that serve as empirical evidence about explanatory behavior.
- Each attribution vector is converted into a binary hypothesis identifying relevant and non-relevant features, with sparsity focusing attention on the relevant set.
- A mean-attribution reference makes WoE measure explainer stability, whereas domain or ground-truth references make it measure alignment while accounting for intrinsic variability.
- Top-k selection retains the highest-ranked k features, while adaptive selection retains enough ranked features to explain at least fraction τ of attribution mass.
4 Connecting WoE with Attribution Variability
The theoretical analysis shows that low attribution variability stabilizes selected features and increases WoE when the reference is derived from mean attribution. The argument combines concentration of selection outcomes with binomial posterior behavior.
- When the mean attribution generates the reference hypothesis, WoE diverges to +∞ as attribution variability approaches zero.
- Concentrated attribution vectors make selected feature sets and their cardinalities stable, so nearly every run reproduces the mean-based hypothesis.
- For adaptive selection, low variability stabilizes k∗ around the value computed from mean attributions; top-k selection has fixed cardinality by construction.
- Low variability also makes higher-mean relevant features outrank lower-mean non-relevant features with high probability.
- The number of runs reproducing the reference follows a binomial distribution, whose upper-tail probability increases with its success probability.
- As the probability of reproducing the reference approaches 1, the posterior probability approaches 1 and WoE against the complementary hypothesis diverges positively.
5 Experimental Setup
The experiments evaluate LIME and SHAP with WoE under domain-knowledge, ground-truth, and stability-oriented reference hypotheses. Results examine alignment, variability, and sensitivity across datasets, models, noise levels, and repeated runs.
- Experimental design: The study tests LIME and SHAP against domain knowledge, synthetic ground truth, and mean-attribution references to assess alignment and internal consistency.The reference hypothesis changes what WoE measures, including domain alignment, faithfulness, and stability.
- Domain knowledge: 27 of 50 LIME instances had +∞ WoE and 23 had finite positive WoE against Titanic survival knowledge, whereas SHAP produced 23 +∞ and 27 −∞ cases.The reference features were sex and pclass; SHAP’s negative divergences indicate selections beyond those features.
- Domain knowledge: 54% of PCA variance was explained, separating LIME’s finite and divergent cases and more strongly separating SHAP’s +∞ and −∞ cases.The separation suggests that alternative variables such as age influence explanations in specific feature-space regions.
- Synthetic ground truth: As λ increased to at least 0.70, WoE_G_T and WoE_M_o_d_e_l converged because redundant features became too noisy for the model to rely on.At lower noise, LIME aligned more with the model than with ground truth; reducing LIME’s kernel width at λ = 0.65 progressively aligned model WoE with ground-truth WoE.
- No prior knowledge: Across Diabetes, Heart Disease, Mobile, and Churn, LIME and SHAP WoE distributions were comparable, while SHAP more consistently produced the same explanation.WoE was generally positive, but decreased on more complex datasets; for Mobile, LIME had 0 positive and 5 negative divergent values.
6 Limitations
The framework’s flexibility depends on several design choices that the authors do not claim are optimal. Exact hypothesis matching can be restrictive, and repeated explanation runs make computational cost dependent on the feature-importance method.
- The framework adopts intuitive design choices without claiming they are optimal, including a uniform prior when no prior knowledge is available.
- Exact equality between feature sets may be too restrictive; softer partial-match or distance-based alternatives could provide smoother similarity.
- Repeatedly running the explanatory method makes the procedure dependent on the feature-importance method’s computational cost.The authors report that stable WoE estimates can nevertheless be obtained with just a few runs.
7 Conclusion and Future Work
The paper embeds feature-importance methods in a Weight of Evidence hypothesis-testing framework whose reference hypothesis determines what is measured. It connects FIM stability to WoE theoretically and reports experimental insights on discrepancies, alignment, and internal consistency, while proposing broader evaluations and softer design choices for future work.
- Conclusion: The reference hypothesis determines whether WoE measures domain-knowledge alignment, faithfulness to intrinsic mechanisms, or internal FIM stability.
- Conclusion: The framework establishes a formal connection between FIM stability and WoE, while experiments detect local-global discrepancy and Data-Model-FIM disalignment.
- Conclusion: WoE measures the internal consistency of feature-importance methods through their variability and stability across repeated runs.
- Future Work: Future work will explore softer partial-match posteriors, alternative priors, broader FIMs, model types, and data modalities.The authors also identify faithfulness as a promising direction for examining disalignment between models and underlying data-generating processes.