Source-linked AI summary
A Formal Methodological Framework for Auditing Robustness and Fidelity in Explainable AI: From Application to Trust Certification
Rosa Elysabeth Ralinirina, Jean Christian Ralaivao, Niaiko Michaël Ralaivao, Alain Josué Ratovondrahona, Thomas Mahatody
TL;DR
Post-hoc explanations can vary under small perturbations, raising questions about their reliability and faithfulness. The paper audits them by measuring robustness and fidelity and combining both into a Trust Score, then finds that highly accurate or overfitted models can still yield degenerate or uninformative explanations. These findings motivate evaluating explanation quality alongside predictive performance.
Problem
Small input perturbations can substantially change SHAP and LIME attributions, while no built-in mechanism establishes whether highlighted features drive the model’s prediction.
Method
The protocol measures robustness under input noise and fidelity through feature ablation, combines them into a Trust Score, and evaluates classifiers and explainers on a Malagasy food-security dataset.
Results
Near-perfect AUC can coexist with degenerate explanations, while overfitting can flatten fidelity and reduce its ability to distinguish explanation quality.
Takeaways & Limitations
Explanation auditing can reveal instability and vacuity that predictive metrics alone do not identify, especially when explanations inform sensitive decisions.
Takeaways & Limitations
The dataset contains only 253 instances, including 3 Acceptable examples, limiting per-class metric reliability and preventing standard stratified 5-fold cross-validation from using all folds.
Abstract
from arXiv · showhide
SHAP and LIME are now standard tools for interpreting black-box predictions, yet their outputs can vary substantially when the input is perturbed by small amounts of noise--a problem we observed firsthand in our previous work on food security in Madagascar (Ralinirina et al., 2025). This variability raises the question of whether such explanations can be trusted at all. We address it by constructing an auditing protocol that measures two properties of any post-hoc explainer: robustness (how stable the explanation is under input perturbation) and fidelity (whether the features deemed important actually drive the model's prediction). These two quantities are combined into a single Trust Score. We run the protocol on a multi-sectoral dataset from Madagascar (83 features, 253 records, 4 malnutrition classes) using three classifiers and two explainers, plus their regularized counterparts. The results are sobering: models with AUC above 0.99 can produce numerically degenerate or flatly uninformative explanations, and fidelity scores lose discriminative power when the model is overfitted. These findings suggest that auditing XAI outputs is not optional but necessary, particularly when they inform decisions in sensitive domains.
1 Introduction
Post-hoc explanations for black-box predictions can change substantially under small input perturbations, while their highlighted features may not reflect the model’s actual reasoning. The paper therefore proposes auditing explanations through robustness, fidelity, and a combined Trust Score.
- Motivation: Small input perturbations can substantially change SHAP and LIME attributions, undermining their reliability for food-security predictions.This problem was observed in Madagascar and includes shifts in important-feature rankings between explainers.
- Motivation: Prior studies report that LIME and SHAP can be fooled, local explanations can be unstable, and visual saliency inspection can be insufficient.The cited literature includes theoretical, adversarial, survey, healthcare, and NLP evidence.
- Contribution: The proposed protocol measures explanation robustness under noise and fidelity to the model, then combines them into a Trust Score.The protocol is validated with three classifiers, two explainers, and regularized variants on the Malagasy food-security dataset.
- Contribution: The paper argues that explanation auditing is necessary because near-perfect predictive performance does not guarantee stable or informative explanations.The introduction frames auditing as a methodological requirement for explanations used in sensitive decision contexts.
2 Related Work
Although SHAP and LIME are widely used model-agnostic explainers, research evaluating explanation quality remains less developed than explanation generation. This framework addresses that gap by evaluating robustness and fidelity side by side and reporting a directly comparable Trust Score.
- Existing explainers: LIME approximates a black-box predictor locally with a linear model, whereas SHAP assigns Shapley values; both can produce fragile explanations.Prior work also demonstrated adversarial attacks that induce arbitrary explanations.
- Evaluation gap: Explanation evaluation has received less attention than explanation generation, motivating quantitative assessment beyond visual inspection.Related work includes causability and domain-specific evaluations in healthcare and agriculture.
- Evaluation gap: Existing domain-specific studies evaluate explainers in isolation rather than through a unified protocol that compares them on equal footing.The framework targets this comparison gap without claiming novelty for the individual metrics.
- Framework distinction: The framework computes Jensen-Shannon-divergence robustness and feature-ablation fidelity on the same data and model, producing a comparable score across explainer–model pairs.Its stated contribution is pairing established metrics within one pipeline and Trust Score.
3 Methodological Framework Architecture
The framework evaluates each explanation through independent robustness and fidelity measurements before combining them into a Trust Score. Robustness tests sensitivity to perturbed inputs, while fidelity tests whether highly ranked features affect model predictions.
- Architecture: For each test instance, the auditor applies model f and explainer E to compute robustness R(x) and fidelity F(x), then forms T(x) = αR(x) + βF(x) with α + β = 1.The pipeline treats the two dimensions independently before aggregation.
- Robustness: Robustness compares normalized explanations for x and perturbed inputs x + δ, using Gaussian noise and divergence across N trials.A score near 1 indicates little change under perturbation, while a score near 0 indicates substantial change.
- Robustness: An ϵ = 10^-10 smoothing term prevents undefined Jensen-Shannon divergence when TreeSHAP outputs zero importance for every feature.The fix prevents division by zero while having negligible numerical effect.
- Fidelity: Fidelity masks the explainer’s top-k features by replacing them with feature means and measures the resulting prediction change across ablation levels K.Overall fidelity is the mean across the selected ablation levels.
- Fidelity: Fidelity near 1 indicates that ablating highlighted features shifts predictions, whereas fidelity near 0 indicates that highlighted features are not used by the model.Extreme model confidence can make ablation effects similar across levels and render fidelity uninformative.
- Trust Score: The default Trust Score assigns equal weights, α = β = 0.5, while allowing domain-specific weighting of stability or faithfulness.The score simplifies comparison but its interpretation depends on the chosen weights.
4.1 Data Description
The dataset combines official Malagasy agricultural, public-health, development, climate, socioeconomic, nutrition, and demographic information across 23 regions and 13 years. After preprocessing, it contains 83 features and 253 instances assigned to four malnutrition classes, with severe imbalance in the Acceptable class.
- Data sources: The dataset aggregates official reports from Madagascar’s agriculture, public-health, and national-development institutions across 23 regions from 2010–2023.It integrates multiple sectoral sources rather than a single-domain dataset.
- Feature domains: Climate variables include rainfall, cyclonic events, and temperature anomalies.
- Feature domains: Agricultural variables include rice production, soil quality, and pest incidence.
- Feature domains: Socioeconomic variables include market prices, household income, and transport quality.
- Feature domains: Nutrition variables include IPC malnutrition phases, child stunting rates, and vaccination coverage, while demographics include population, literacy, and school enrollment.
- Preprocessed dataset: 83 features and 253 instances remain after preprocessing, with four ordinal malnutrition classes including only 3 Acceptable instances.This imbalance limits per-class AUC reliability and prevents stratified cross-validation folds from containing all four classes.
4.2 Experimental Setup
The experiment used three classifiers, regularized variants, and SHAP/LIME explanations on a stratified train/test split. Cross-validation produced very high macro-average AUC values, although extreme class imbalance limited the reliability of per-class estimates.
- The dataset was divided using a stratified 82/18 split, with 208 training and 45 test instances.
- The classifiers were Random Forest, XGBoost, and a three-hidden-layer Neural Network.
- 0.998 ± 0.002 macro-average AUC was obtained for RF across the three valid folds, compared with 0.999 ± 0.002 for XGB and 0.994 ± 0.001 for NN.Only three of five stratified folds were valid because the Acceptable class contained three instances.
- Three regularized variants were also trained to assess their impact on explanation quality.The variants were RFreg, NNreg, and a regularized XGBoost configuration.
- The audit applied SHAP and LIME to 30 randomly sampled test instances using σ = 0.1, N = 10 perturbations, and five ablation levels.Ablation levels were K = {3, 5, 10, 41, 82}, spanning small removals to nearly all 83 features.
4.3 Results
The results show that high predictive performance does not guarantee stable or informative explanations. Robustness varies across model–explainer pairs, while overconfident XGBoost models produce nearly flat fidelity and lower Trust Scores.
- Robustness: SHAP was more robust than LIME on tree models: RF scored 0.902 vs. 0.700, and XGB scored 0.978 vs. 0.658.The XGBoost+SHAP score of 0.978 required ϵ-smoothing because TreeSHAP sometimes returned zero-valued importance vectors.
- Robustness: Regularization improved SHAP robustness on RF from 0.902 to 0.923 and on the Neural Network from 0.694 to 0.748, while XGBoost changed from 0.978 to 0.958.LIME robustness remained between 0.66 and 0.76 across models.
- Fidelity: XGBoost fidelity stayed approximately 0.620 from F3 through F82 for both SHAP and LIME, so ablation level did not discriminate feature importance.The consistent moderate prediction shift reflects overconfidence rather than uniformly high fidelity.
- Fidelity: NN+LIME fidelity fell from 0.984 at F3 to 0.734 at F82, whereas XGBoost and RF showed flatter and downward fidelity patterns, respectively.The regularized Neural Network showed an even stronger pattern, with fidelity from 0.977 at F3 to 0.802 at F82.
- Trust Scores: RFreg+SHAP achieved the highest Trust Score at 0.871, while XGBoost+LIME had the lowest at 0.640.Regularization improved Trust Scores across all six model–explainer pairs, with the largest gains for XGBoost.
- Feature Importance: Rice production, rainfall anomalies, and market prices consistently ranked among the top three features across models and explainers.Some institutional indicators had high SHAP values but low robustness, suggesting possible model artifacts.
5 Discussion
The discussion shows that explanation trustworthiness depends on more than predictive AUC: overfitting can make fidelity uninformative, while explainer choice affects robustness. The Trust Score also entails a weighting trade-off and should be interpreted cautiously in severely overfitted settings.
- AUC above 0.99 does not guarantee trustworthy explanations because XGBoost can produce zero-valued SHAP importance vectors.The numerical smoothing fix prevents undefined JSD calculations but does not make the explanations informative.
- ≈0.620 fidelity at every XGBoost ablation level and for both explainers shows that extreme confidence can erase fidelity’s discriminative power.Regularization increases XGBreg+SHAP fidelity variation only modestly, from F3 = 0.715 to F82 = 0.712.
- Before auditing, severe overfitting should be checked because it reduces fidelity’s—and somewhat robustness’s—discriminative power.The authors recommend interpreting the Trust Score cautiously under this condition.
- Robustness favors SHAP over LIME on tree-based models: RF scores 0.902 versus 0.700, and XGB scores 0.978 versus 0.658.The discussion attributes this pattern to TreeSHAP’s exact computation path versus LIME’s sampling variability.
- The Trust Score makes stability–faithfulness priorities explicit, but compressing both dimensions into one number inevitably loses nuance.The weighting choice depends on context: stability may dominate field use, whereas fidelity may take priority in regulation.
- The framework’s evidence base is constrained by a 253-instance dataset with only 3 Acceptable examples and by domain-independent Gaussian perturbations.Additional boundaries include O(d · n) ablation cost, feature attribution scope, judgment-dependent weights, and limited transferability from highly separable Malagasy data.
6 Conclusion
The framework combines robustness and fidelity into a Trust Score and exposes cases where near-perfect AUC coexists with degenerate explanations or flat fidelity. Future work should extend the approach to richer data structures, domain-specific perturbations, and real-time decision support.
- Near-perfect AUC can coexist with zero-valued XGBoost+SHAP importance vectors, while XGBoost fidelity remains ≈0.620 across ablation levels.The case study used 253 instances, 83 features, and 4 malnutrition classes.
- Further work should address temporal and spatial structures, domain-specific perturbations, and real-time decision support.