PaperlayerView exact revision on arXiv

Source-linked AI summary

Reliable Post hoc Explanations: Modeling Uncertainty in Explainability

Dylan Slack, Sophie Hilgard, Sameer Singh, Himabindu Lakkaraju

arXiv:2008.05030v4cs.LGstat.ML

TL;DR

Existing local explanations can be unstable, inconsistent, difficult to assess, computationally inefficient, and dependent on poorly guided hyperparameters. The paper introduces Bayesian versions of LIME and KernelSHAP that quantify uncertainty, with experiments reporting greater stability, calibrated uncertainty, and faster perturbation sampling, while retaining limitations of local linear approximations.

  • Problem

    Existing local explanation methods lack reliable quality assessment, can be unstable and inconsistent, and require costly queries and poorly guided perturbation settings.

  • Method

    The paper develops a Bayesian framework instantiated as BayesLIME and BayesSHAP, producing feature-importance credible intervals and uncertainty-guided perturbation estimates and sampling.

  • Results

    53% more stability than LIME/SHAP on average was achieved, while focused sampling sped explanation generation by up to a factor of 2 and credible intervals were well calibrated.

  • Takeaways & Limitations

    Uncertainty estimates provide concrete reliability information and support explanations meeting user-specified confidence levels with more stable and efficient generation.

  • Takeaways & Limitations

    Highly non-linear local decision surfaces may not be accurately captured because the framework relies on local linear approximations.

Abstract

from arXiv · show

As black box explanations are increasingly being employed to establish model credibility in high-stakes settings, it is important to ensure that these explanations are accurate and reliable. However, prior work demonstrates that explanations generated by state-of-the-art techniques are inconsistent, unstable, and provide very little insight into their correctness and reliability. In addition, these methods are also computationally inefficient, and require significant hyper-parameter tuning. In this paper, we address the aforementioned challenges by developing a novel Bayesian framework for generating local explanations along with their associated uncertainty. We instantiate this framework to obtain Bayesian versions of LIME and KernelSHAP which output credible intervals for the feature importances, capturing the associated uncertainty. The resulting explanations not only enable us to make concrete inferences about their quality (e.g., there is a 95% chance that the feature importance lies within the given range), but are also highly consistent and stable. We carry out a detailed theoretical analysis that leverages the aforementioned uncertainty to estimate how many perturbations to sample, and how to sample for faster convergence. This work makes the first attempt at addressing several critical issues with popular explanation methods in one shot, thereby generating consistent, stable, and reliable explanations with guarantees in a computationally efficient manner. Experimental evaluation with multiple real world datasets and user studies demonstrate that the efficacy of the proposed framework.

1 Introduction

Popular local explanation methods can be unstable, inconsistent, difficult to assess, computationally inefficient, and sensitive to hyperparameters. The paper proposes Bayesian local explanations that quantify uncertainty, improve reliability, and guide perturbation sampling.

  • Local explanations may change substantially under small perturbations or across repeated runs with identical settings.
  • Common quality metrics, hyperparameter choices, and large black-box query requirements limit the reliability and efficiency of existing methods.The paper specifically identifies fidelity metrics, perturbation counts, and computational cost as unresolved issues.
  • The proposed Bayesian framework augments LIME and KernelSHAP with feature-importance estimates and credible intervals representing uncertainty.The framework is instantiated as BayesLIME and BayesSHAP.
  • Uncertainty estimates support concrete quality inferences, user-specified confidence levels, and more consistent and stable explanations.
  • 53% more stability than LIME/SHAP on average was observed across the evaluated datasets, while focused uncertainty sampling accelerated generation by up to a factor of 2.The evaluation included COMPAS, German Credit, ImageNet, and MNIST, plus a 31-person user study.

2 Notation & Background

The paper frames LIME and KernelSHAP as model-agnostic methods that approximate a black-box classifier locally with a linear model. Their main distinction is the proximity function used to weight perturbations.

  • The notation defines f as a black-box classifier, φ as feature importances, Z as sampled perturbations, and Y as their corresponding predictions.
  • LIME and KernelSHAP learn a local linear model to approximate classifier behavior near an instance x.
  • The shared objective has a closed-form solution for the local weighted approximation.
  • LIME chooses proximity heuristically using cosine or l2 distance, whereas KernelSHAP uses game-theoretic principles with guaranteed properties.

3 Our Framework: Bayesian Local Explanations

The framework models local black-box behavior Bayesianly, producing feature-importance and error uncertainty alongside point estimates. It also uses uncertainty to estimate required perturbations and target sampling for faster convergence.

  • 3.1 Constructing Bayesian Local Explanations: A Bayesian weighted local linear model represents each perturbed prediction as φ^Tz plus Gaussian error whose variance depends on proximity to x.Nearby perturbations receive more accurate modeling, while farther perturbations allow more error.
  • 3.1 Constructing Bayesian Local Explanations: The framework captures both uncertainty in feature importances and uncertainty in the error describing local decision-surface mismatch.
  • 3.1 Constructing Bayesian Local Explanations: Using the same proximity function, the posterior mean feature importances equal those produced by LIME or KernelSHAP.
  • 3.1 Constructing Bayesian Local Explanations: Credible intervals are estimated around posterior mean feature importances by sampling from their posterior distribution.
  • 3.1 Constructing Bayesian Local Explanations: The posterior error distribution supports a constant-time density calculation at zero as a proxy for how well the explanation captures the local decision surface.
  • 3.1 Constructing Bayesian Local Explanations: As N →∞, estimated feature importances converge to the true scores, their uncertainty converges to 0, and error uncertainty converges to local-model bias.
  • 3.1 Constructing Bayesian Local Explanations: BayesLIME uses a distance-based proximity function, while BayesSHAP uses the KernelSHAP proximity and interprets uncertainty as reliability of variable contributions.
  • 3.2 Estimating the Number of Perturbations: Perturbations-to-go estimates additional samples needed for a target credible-interval width and confidence level, using weighted squared errors and the inverse normal CDF.

4 Experiments

Experiments evaluate uncertainty calibration, perturbation estimates, focused sampling, stability, and user-facing informativeness across structured and image datasets. The Bayesian methods produce well-calibrated uncertainty estimates, reliable perturbation-to-go estimates, faster convergence, and generally greater stability than LIME and SHAP.

  • Uncertainty estimates: Both BayesLIME and BayesSHAP are well calibrated: their 95% credible intervals include true feature importances at rates close to the 95.0 target.True values are estimated using 10,000 perturbations, while intervals use 100 perturbations.
  • Setup: Experiments span COMPAS, German Credit, MNIST, and ImageNet using black-box models and standard LIME and KernelSHAP baselines.The datasets cover criminal justice, credit scoring, handwritten digits, and ImageNet images.
  • Perturbations-to-go: 200-5,000 perturbations are estimated for lower certainty, increasing to 200-20,000 for higher certainty across instances.Observed credible-interval widths closely match desired widths, supporting the reliability of perturbations-to-go estimates.
  • Focused sampling: Focused sampling converges faster than random sampling when evaluated by error uncertainty over model queries on 100 ImageNet French bulldog images.The comparison reports mean and standard error.
  • Stability: BayesLIME and BayesSHAP are 53% more stable on average than LIME and SHAP, with the exception of BayesSHAP on German Credit.Wilcoxon signed-rank tests are significant in the other cases (ρ < 1e-2), but not for BayesSHAP on German Credit (ρ > 0.05).
  • User study: A 31-subject MNIST user study tests whether lower-uncertainty BayesLIME explanations are more meaningful to humans.Participants guess masked digits after the most important features selected by BayesLIME or LIME are masked.

5 Related Work

Related work covers interpretable models, post hoc explanations, vulnerabilities of perturbation-based methods, formal reasoning, and Bayesian approaches. The paper positions its contribution around uncertainty modeling for local explanations, which prior Bayesian approaches did not focus on.

  • Interpretability Methods: Interpretability methods include additive models, decision lists and sets, and instance-based explanations, while post hoc methods explain flexible black-box models.LIME and SHAP are highlighted as popular post hoc methods because of their broad applicability.
  • Vulnerabilities of Post hoc Explanations: Post hoc explanations can be sensitive to small input changes, manipulable, unfaithful to black boxes, variable across runs, and dependent on hyperparameters.These concerns motivate assessing reliability and stability in perturbation-based explanation methods.
  • Logical and Formal Reasoning: Formal reasoning methods identify feature subsets sufficient for predictions but are not model agnostic and do not define local feature importances.They also do not address improving locally weighted explanations such as LIME and SHAP.
  • Bayesian Methods in Explainable ML: Prior Bayesian explainability work includes global non-parametric surrogates and informative-prior studies, but neither focuses on uncertainty in local explanations.This paper addresses that local-explanation uncertainty focus.

6 Conclusion

The paper presents a Bayesian framework that augments local explanations with credible intervals and uses uncertainty to guide sampling and hyperparameter estimation. It reports improved reliability, consistency, and stability while retaining limitations inherited from local linear approximations and perturbation procedures.

  • 6 Conclusion: BayesLIME and BayesSHAP provide feature-importance estimates with credible intervals that quantify explanation uncertainty.These intervals support quality inference and user-specified uncertainty levels.
  • 6 Conclusion: The framework uses uncertainty measures to estimate critical hyperparameters, including the number of perturbations, and introduces focused sampling for faster convergence.
  • 6 Conclusion: Highly nonlinear local decision surfaces may not be accurately captured because the framework relies on local linear approximations.
  • 6 Conclusion: Using LIME or SHAP perturbation procedures can leave BayesLIME and BayesSHAP vulnerable to the attacks identified by Slack et al.

A Derivations

The derivations obtain posterior distributions for feature-importance parameters, noise variance, and posterior predictions in the Bayesian explanation model. They express the feature-importance posterior through a closed-form mean and variance and derive a Student’s t posterior predictive distribution.

  • A Derivations: The model derivation writes the joint posterior and groups its terms according to the feature-importance vector φ.
  • A Derivations: The conditional posterior of φ has mean φ̂ and variance σ2(ZT diag(Πx(Z))Z + I)^−1.
  • A Derivations: The conditional posterior for σ2 is identified as a scaled inverse-χ2 distribution using the definition of s2.
  • A Derivations: The posterior predictive distribution for a prediction at z is a Student’s t distribution centered at φ̂Tz with scale determined by zT Vφz and s2.

B Proof of Theorems

The proofs analyze how feature-importance estimates, uncertainty, and prediction error behave as the number of perturbations increases. Under stated sampling and large-sample assumptions, uncertainty over feature importance vanishes, while residual uncertainty converges to local model error.

  • B Proof of Theorems: Under Bernoulli(0.5) perturbation sampling and near-zero priors, the proofs analyze the posterior and its large-sample behavior.
  • B Proof of Theorems: The proof assumes sufficiently large perturbation counts and replaces the weighted design product in Vφ with its expected value.
  • Convergence of Var(φ): As N →∞, Vφ approaches the null matrix, so uncertainty over feature importance also approaches zero.
  • Consistency of φ̂: As N →∞, the feature-importance estimate converges in probability to the true value φ*.
  • Convergence of Var(ϵ): As N →∞, predictive error variance converges to s2, which reduces to local model error or squared bias of the local model.
  • Extended experimental results: A hyperparameter grid search on MNIST finds that explanation uncertainty is robust to choices of n0 and σ2_0.

C.2 PTG Estimate Results

The appendix evaluates perturbation-count estimation and provides additional examples of the framework’s behavior. PTG estimates vary with the desired credible-interval width, while ImageNet complexity requires limiting the uncertainty range because perturbation needs become very high.

  • C.2 PTG Estimate Results: PTG estimates the number of additional perturbations needed to reach a desired level of feature-importance certainty.
  • C.2 PTG Estimate Results: The estimated number of perturbations is highly variable depending on the desired credible-interval width W.
  • C.2 PTG Estimate Results: For ImageNet’s French bulldog class, the uncertainty range is limited because the data complexity makes the required number of perturbations very high.
  • C.3 User Study: The user study asks participants to identify MNIST digits after masking features selected as important by BayesLIME or LIME.
  • Additional results: A toy nonlinear decision surface illustrates that LIME produces high-variance explanations, whereas BayesLIME captures explanation uncertainty in a single explanation.

C.4 Focused Sampling Results

Focused sampling improves BayesLIME’s convergence efficiency while retaining convergence to the ground-truth explanation. The uncertainty-based procedure also yields meaningful credible intervals and efficient closed-form inference.

  • Wall Clock Time of Focused Sampling: Focused sampling is more time efficient than random sampling for BayesLIME.The comparison uses wall-clock time to convergence over 100 ImageNet images.
  • Bias of Focused Sampling: 30−40% improved query efficiency: focused sampling reaches L1 distance 0.1 at 300 queries, versus upwards of 450 for random sampling.At approximately 1000 queries, the L1 distance is around 0.06, making the explanations practically identical to ground truth.
  • Bias of Focused Sampling: Focused sampling converges to the ground-truth explanation faster than random sampling, indicating minimal bias in the procedure.The comparison evaluates convergence using the L1 distance between explanations and ground truth.
  • Benchmarking: 20.3 seconds and 21.1 seconds: BayesLIME and BayesSHAP explain one ImageNet VGG16 instance faster than Guo et al.’s method at 139.2 seconds.The benchmark is conducted under the same conditions and attributes the efficiency to the closed-form solution.
  • Explaining a Ground Truth Function: Tighter credible intervals correspond to BayesLIME explanations closer to the ground-truth feature importances.The experiment measures L1 distance against maximum credible-interval width.

E Compute Used

All experiments were run using one NVIDIA 2080TI GPU and one NVIDIA Titan RTX GPU.

  • All experiments used a single NVIDIA 2080TI GPU and a single NVIDIA Titan RTX GPU.

F Dataset licenses

The appendix reports licensing information for German Credit, COMPAS, MNIST, and ImageNet.

  • German Credit is in the public domain, while COMPAS uses the MIT license.
  • MNIST uses the Creative Commons Attribution-Share Alike 3.0 license, while ImageNet does not hold copyright of images.
Loading 2008.05030v4…