Source-linked AI summary
On the Robustness of Interpretability Methods
David Alvarez-Melis, Tommi S. Jaakkola
TL;DR
Point-wise interpretability explanations may change substantially under small input perturbations, even when model predictions remain nearly unchanged. The paper formalizes local robustness, introduces local Lipschitz-based metrics, evaluates popular methods, and discusses enforcement strategies; experiments find that most methods are not robust, with perturbation-based methods more unstable than gradient-based ones.
Problem
Point-wise explanations can be brittle, so the paper asks whether popular interpretability methods remain stable under small input perturbations.
Method
The paper formalizes local explanation robustness and evaluates it using neighborhood-based Lipschitz metrics across interpretability methods and model settings.
Results
Most current methods are not robust; model-agnostic perturbation-based methods are more prone to instability than gradient-based counterparts.
Takeaways & Limitations
Robustness is a key desideratum when explanations are intended to reflect stable relevant aspects of inputs or support local predictive use.
Takeaways & Limitations
The paper focuses on small perturbations with minimal or no effect on the underlying model’s predictions.
Abstract
from arXiv · showhide
We argue that robustness of explanations---i.e., that similar inputs should give rise to similar explanations---is a key desideratum for interpretability. We introduce metrics to quantify robustness and demonstrate that current methods do not perform well according to these metrics. Finally, we propose ways that robustness can be enforced on existing interpretability approaches.
1. Introduction
The paper argues that point-wise interpretability explanations can be brittle, motivating robustness to local input perturbations as a key property. It formalizes this goal and investigates whether popular gradient- and perturbation-based methods satisfy it.
- 1. Introduction: Current interpretability methods include saliency approaches based on gradients or output decomposition and perturbation approaches based on local model queries.These methods generally explain individual input/prediction pairs and typically require little or no model modification.
- 1. Introduction: Point-wise explanations can be brittle and may create a false sense of understanding.The paper motivates examining model behavior in neighborhoods rather than relying only on individual inputs.
- 1. Introduction: Robust explanations should remain similar for similar inputs under small local perturbations.This supports validity around a point and, for predictive explanations, approximate use of the simplified model within a small neighborhood.
- 1. Introduction: The paper investigates robustness by formalizing the notion, evaluating popular gradient and perturbation-based methods, and discussing ways to enforce it.The study proceeds from formalization to experiments and then to robustness-enforcement approaches.
2. Robustness
The paper defines robustness locally: explanations should change proportionally little for nearby inputs, with a weaker finite-sample alternative for discrete settings. It uses local Lipschitz estimates to quantify this behavior, while noting computational and interpretive constraints.
- 2. Robustness: Similar inputs should not produce substantially different explanations when perturbations leave predictions nearly unchanged.The paper identifies this lack of robustness as a central failure of many current interpretability methods.
- 2. Robustness: LIME and SHAP are stable for a linear SVM but vary considerably for neighboring inputs of a neural-network classifier.The neural-network explanations can also be inconsistent with each other, illustrating the instability under study.
- 2. Robustness: Local Lipschitz continuity measures explanation changes relative to input changes within a neighborhood, rather than across the entire input space.The local criterion is intended to assess stability for neighboring inputs, where global explanation uniformity is not expected.
- 2. Robustness: Local robustness constants depend on the anchor point and are estimated by optimizing the explanation difference over nearby inputs.For each point of interest, the estimate searches for the largest relative explanation change within a radius-ϵ neighborhood.
- 2. Robustness: For discrete inputs or overly restrictive continuous perturbations, robustness can instead be estimated over finite-sample neighborhoods.The paper presents this as a weaker empirical stability notion based on examples in the test data.
- 2. Robustness: The local Lipschitz estimates are computed on 100 test points across various UCI classification datasets.The estimates provide an objective alternative to subjective visual inspection of attribution changes.
3. Experiments
The experiments evaluate explanation robustness across tabular and image-classification settings using local Lipschitz criteria and controlled perturbations. Across datasets, small input changes can produce substantially different explanations despite nearly unchanged predictions.
- Experimental setup: The study compares LIME, SHAP, SALIENCY, GRADIENT*INPUT, INTegrated GRADients, ϵ-Layerwise Relevance Propagation, and OCCLUSION sensitivity across tabular and image-model settings.Attribution arrays are treated as vector-valued explanations, and robustness is estimated with local or discrete Lipschitz criteria.
- Benchmark Classification and Regression Datasets: Black-box experiments train random forest models, explain 200 sampled test points, and compute local robustness for each point.COMPAS instead uses a discrete, sample-based criterion because its categorical variables make continuous perturbations less meaningful.
- Benchmark Classification and Regression Datasets: Extremely close BOSTON inputs can yield considerably different explanations when one input is the adversarially chosen maximizer of the Lipschitz estimate.Figure 3 compares explanations for an example and its worst-case local deviation.
- Explaining Digit Predictions: On MNIST, Gaussian-noise perturbations barely change predicted-class probabilities but can change explanations considerably, sometimes dramatically for LIME and OCCLUSION.Dataset-level robustness is evaluated over multiple test points, with local Lipschitz estimates computed on 100 points.
- Explaining Digit Predictions: All MNIST methods are significantly affected by minor perturbations, with LIME especially sensitive because its sparse superpixel explanations respond strongly to small input changes.Worst-case perturbations and corresponding explanations are shown for a particular input.
- Explaining Image Classification: For a ResNet natural-image example, a nearly indistinguishable perturbation preserves the bull mastiff prediction and probability while producing remarkably different SALIENCY explanations.The probabilities are 0.7308 and 0.7307 for the original and perturbed images, respectively.
4. Discussion
The experiments find that popular interpretability frameworks are generally not robust to small input modifications, with model-agnostic perturbation methods more unstable than gradient-based methods. The discussion frames robustness as dependent on the explanation goal and points to robust-by-construction and adversarial-training-inspired approaches.
- Discussion: For the most part, current interpretability frameworks are not robust to small input modifications, and model-agnostic perturbation methods are more prone to instability than gradient-based methods.This conclusion summarizes the paper’s experimental findings across its evaluated settings.
- Discussion: The study focuses on perturbations with minimal or no effect on model predictions but significant effects on explanations.This defines the scope of the reported robustness concern.
- Discussion: Whether explanations should include input artifacts depends on the interpretability goal: debugging may favor broad influence, whereas understanding the predictor and phenomenon may favor greater robustness.The discussion contrasts tracing outputs to inputs with understanding the modeled phenomenon.
- Discussion: The paper identifies robust-by-construction interpretable networks and adversarial-training techniques as possible ways to enforce explanation robustness.A generalized criterion can train interpretable neural networks, while adversarial-training methods may inspire interpretability approaches.