Source-linked AI summary
Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods
Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, Himabindu Lakkaraju
TL;DR
Post hoc explanation methods such as LIME and SHAP may not reliably reveal discriminatory behavior because they rely on input perturbations. The paper introduces scaffolding that preserves a biased classifier’s predictions on real-world inputs while controlling its behavior on perturbed inputs, and shows that such classifiers can fool these explanations across multiple datasets.
Problem
Black-box models require interpretable explanations for experts to diagnose errors and biases, but the reliability of perturbation-based methods such as LIME and SHAP is vulnerable to adversarial manipulation.
Method
The framework detects perturbed or out-of-distribution inputs and scaffolds a biased classifier to behave differently on them while preserving behavior on the input distribution.
Results
Across COMPAS, Communities and Crime, and German loan lending, highly discriminatory classifiers hid sensitive attributes from LIME and SHAP explanations, with LIME more vulnerable than SHAP.
Takeaways & Limitations
Existing perturbation-based post hoc explanations are not sufficiently robust for ascertaining discriminatory behavior in sensitive applications.
Takeaways & Limitations
The framework assumes the adversary has a biased classifier and sampled real-world inputs but no access to the underlying input distribution.
Abstract
from arXiv · showhide
As machine learning black boxes are increasingly being deployed in domains such as healthcare and criminal justice, there is growing emphasis on building tools and techniques for explaining these black boxes in an interpretable manner. Such explanations are being leveraged by domain experts to diagnose systematic errors and underlying biases of black boxes. In this paper, we demonstrate that post hoc explanations techniques that rely on input perturbations, such as LIME and SHAP, are not reliable. Specifically, we propose a novel scaffolding technique that effectively hides the biases of any given classifier by allowing an adversarial entity to craft an arbitrary desired explanation. Our approach can be used to scaffold any biased classifier in such a way that its predictions on the input data distribution still remain biased, but the post hoc explanations of the scaffolded classifier look innocuous. Using extensive evaluation with multiple real-world datasets (including COMPAS), we demonstrate how extremely biased (racist) classifiers crafted by our framework can easily fool popular explanation techniques such as LIME and SHAP into generating innocuous explanations which do not reflect the underlying biases.
INTRODUCTION
The paper argues that post hoc explanations help experts inspect black-box behavior and bias, but perturbation-based methods can be adversarially manipulated to hide discriminatory classifiers.
- Black-box models are difficult for domain experts to understand, motivating interpretable tools for diagnosing errors and potential biases.
- LIME and SHAP explain individual predictions by estimating feature contributions from perturbed instances.
- The proposed framework exploits perturbation-based explanations to construct scaffolds whose explanations can be arbitrarily controlled.
- Scaffolded classifiers can remain discriminatory while LIME and SHAP explanations appear innocuous.
- Across COMPAS, Communities and Crime, and German loan lending, explanations did not flag sensitive attributes such as race as important.
BUILDING ADVERSARIAL CLASSIFIERS TO FOOL EXPLANATION TECHNIQUES
The paper presents a framework for constructing adversarial classifiers that fool post hoc explanation techniques relying on input perturbations.
- The framework section first reviews LIME and SHAP before presenting the method for constructing adversarial classifiers.
Background: LIME and SHAP
LIME and SHAP locally approximate black-box predictions with interpretable models, balancing fidelity near an instance against explanation complexity.
- Complex models are treated as black boxes, so simpler interpretable models can approximate their behavior for human understanding.
- LIME and SHAP are model-agnostic local methods that estimate feature attributions for individual predictions.
- The dataset consists of N feature vectors with corresponding class labels, and the classifier maps each input to a class label.
- Their objective is to approximate the black box accurately near x while keeping the explanation model simple and interpretable.
- LIME chooses complexity and proximity heuristically, whereas Kernel SHAP grounds them in game-theoretic principles.
Proposed Framework
The framework detects whether inputs resemble the real-world distribution or perturbations, then preserves biased predictions on real inputs while altering behavior on perturbed ones.
- Proposed Framework: The adversary supplies a biased classifier and sampled input data, without access to the underlying real-world distribution.
- Proposed Framework: The scaffolded classifier behaves like the biased classifier on real-world inputs while concealing its biases from LIME and SHAP.
- Proposed Framework: LIME and SHAP probe local neighborhoods containing synthetic perturbations that may fall outside the input distribution.
- Proposed Framework: PCA showed that LIME-style perturbations were distributed significantly differently from input data, making perturbation detection feasible.
- Proposed Framework: The adversary exploits this distinction by making the classifier biased on in-distribution inputs and innocuous on perturbed inputs.
- Proposed Framework: An OOD classifier is trained by labeling original instances false and perturbed instances true, then fitting a classifier on their union.
EXPERIMENTAL RESULTS
The evaluation tests adversarial classifiers across recidivism, violent-crime, and credit-scoring datasets, using LIME and SHAP under specified perturbation and classifier constructions. Figures compare feature-ranking behavior for biased and adversarial classifiers.
- Datasets: The evaluation covers COMPAS, Communities and Crime, and German credit datasets spanning recidivism, violent crime, and credit scoring.The datasets include sensitive attributes of race, percentage of Caucasian population, and gender, respectively.
- Experimental setup: The framework uses 90% of each dataset for training and 10% for testing.
- Classifier construction: The biased classifier predicts purely from race for COMPAS and Communities and Crime, and from gender for German Credit.
- Perturbations and OOD classifier: LIME perturbations add N(0, 1) noise to every feature, while SHAP masks randomly selected features using a 10-cluster kmeans background distribution.
- Adversarial classifier construction: The unbiased classifier ψ uses one or two synthetic features with zero correlation to sensitive attributes, including XOR when two features are used.
- Evaluation measures: Figures 2–4 compare top-three feature rankings for the biased classifier and adversarial classifiers across COMPAS, Communities and Crime, and German credit.German credit uses Loan Rate % Income as an existing predictive feature and Gender as the sensitive feature.
Effectiveness of Adversarial Classifiers
The adversarial classifiers substantially redirect LIME and SHAP feature rankings away from sensitive features while retaining strong agreement with the original classifier in the evaluated settings. LIME is fooled across all three datasets, whereas SHAP is less successful when two uncorrelated features are used.
- LIME: 100% of held-out COMPAS and Communities and Crime points shift LIME importance from the sensitive feature to one uncorrelated feature, versus 91% for German Loan % Income.
- LIME: LIME is effectively fooled by the adversarial classifier on all three datasets.
- SHAP: 100% of Communities and Crime, 84% of COMPAS, and 85% of German points shift SHAP importance from the sensitive feature when one uncorrelated feature is used.
- SHAP: With two uncorrelated features, SHAP removes the bias feature from first place in 67% of COMPAS and 71% of Communities and Crime instances.The passage attributes this lower success to SHAP’s local accuracy property, which distributes attributions across several features when no single feature is identifiable.
- Fidelity: For LIME, adversarial and original classifier predictions agree on 100% of held-out instances across all datasets; SHAP fidelity is 75% for Communities and Crime and 80% for German.
Effect of Perturbation Detection Accuracy
Attack success depends on the OOD classifier’s ability to distinguish perturbed instances from input-data points: higher detection accuracy makes LIME and SHAP less likely to identify race as most important.
- Effect of Perturbation Detection Accuracy: When OOD detection is inadequate, both LIME and SHAP identify race as the most important feature, so the attack fails.Improved OOD classification enables the scaffolded behavior on perturbed points that hides the bias in explanations.
- Effect of Perturbation Detection Accuracy: Table 2 reports feature coefficients for a single-feature LIME explanation before and after an attack on one COMPAS instance.The supplied table caption identifies the comparison but does not provide the coefficient values.
- Effect of Perturbation Detection Accuracy: LIME becomes highly vulnerable once the OOD classifier’s F1 score exceeds 0.8, whereas SHAP begins responding around F1 score ∼0.45 but hides bias more gradually.SHAP requires a very accurate OOD classifier to completely conceal the bias.
- Effect of Perturbation Detection Accuracy: The attack is effective when perturbed instances and input-data points can be differentiated with reasonable accuracy.Figure 5 measures effectiveness by the percentage of COMPAS data points for which race is the most important feature against OOD-classifier F1 score.
Robustness to Hyperparameters
The attacks remain effective across several test-time choices for LIME and SHAP, including kernel distance, kernel width, and background-cluster count.
- Robustness to Hyperparameters: LIME attacks remain equally effective when the kernel width changes from its default or when l1 distance replaces l2 distance.The reported kernel-width range is truncated in the supplied passage, but the robustness conclusion is explicit.
- Robustness to Hyperparameters: SHAP attacks remain equally effective when the test-time K-means background uses 5 or 15 clusters instead of the default 10.Using an all-zero SHAP background makes the attacks even more pronounced.
- Robustness to Hyperparameters: Changing the SHAP background distribution to all zeroes makes the attacks even more pronounced.This result is reported alongside robustness to alternative K-means cluster counts.
Example Explanations
Example COMPAS explanations show that the attack hides race, the original classifier’s sole important feature, from LIME and substantially reduces its prominence in SHAP.
- Example Explanations: Race is completely hidden from the LIME explanation after the attack, despite being the sole important feature for the original classifier f.The example concerns an instance from the COMPAS dataset.
- Example Explanations: In the SHAP explanation, race is considerably hidden after the attack but is not completely nullified.The passage contrasts this partial suppression with the complete hiding reported for LIME.
RELATED WORK
Related work covers perturbation-based explanation methods, criticisms of post hoc faithfulness, adversarial manipulation of saliency maps, and challenges in detecting bias even with interpretability or transparency.
- Perturbation-based Explanation Methods: Perturbation-based post hoc methods include LIME, SHAP, anchors, and GAM, which produce local feature or rule-based explanations.LIME and SHAP are described as prominent local, model-agnostic approaches.
- Criticism of Post hoc Explanations: Prior critiques argue that post hoc explanations may be unfaithful, sensitive to small input changes, or based on unintuitive perturbation points.These concerns are especially noted for structured data and unchanged classifier predictions.
- Adversarial Explanations: Earlier adversarial-explanation work showed that imperceptible input modifications can arbitrarily change saliency maps in image classification.The cited studies include attacks on saliency maps by Dombrowski et al. and Heo et al.
- Interpretability and Bias Detection: Interpretability has been proposed for evaluating bias, but post hoc explanations cannot definitively establish unfairness, and correlated variables complicate bias detection even for transparent models.The related work frames interpretability and transparency as insufficient for conclusively detecting or preventing bias.
CONCLUSIONS AND FUTURE WORK
The framework hides discriminatory biases from perturbation-based explanations while preserving biased predictions, and experiments find LIME more vulnerable than SHAP. The findings suggest existing post hoc explanations are insufficient for ascertaining discriminatory behavior in sensitive applications and motivate robust alternatives.
- The framework generates adversarial classifiers that fool LIME and SHAP while preserving biased predictions on the input distribution.It creates scaffolding around biased classifiers and controls behavior on perturbed data points so explanations appear innocuous.
- Extensive experiments on criminal justice and credit-scoring data find that LIME is more vulnerable than SHAP.
- Existing post hoc explanation techniques are not sufficient for ascertaining discriminatory classifier behavior in sensitive applications.
- Future research directions: Future work should test whether other post hoc explanation classes, including gradient-based approaches, are vulnerable to adversarial attacks.
- Future research directions: Future work should develop adversarially robust explanations that withstand attacks like those presented in this work.