PaperlayerView exact revision on arXiv

Source-linked AI summary

Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods

Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, Himabindu Lakkaraju

arXiv:1911.02508v2cs.LGcs.AIstat.ML

TL;DR

Post hoc explanation methods such as LIME and SHAP may not reliably reveal discriminatory behavior because they rely on input perturbations. The paper introduces scaffolding that preserves a biased classifier’s predictions on real-world inputs while controlling its behavior on perturbed inputs, and shows that such classifiers can fool these explanations across multiple datasets.

  • Problem

    Black-box models require interpretable explanations for experts to diagnose errors and biases, but the reliability of perturbation-based methods such as LIME and SHAP is vulnerable to adversarial manipulation.

    Source: INTRODUCTION · p. 1

  • Method

    The framework detects perturbed or out-of-distribution inputs and scaffolds a biased classifier to behave differently on them while preserving behavior on the input distribution.

    Source: Proposed Framework · p. 3

  • Results

    Across COMPAS, Communities and Crime, and German loan lending, highly discriminatory classifiers hid sensitive attributes from LIME and SHAP explanations, with LIME more vulnerable than SHAP.

    Source: INTRODUCTION · p. 2, CONCLUSIONS AND FUTURE WORK · p. 7

  • Takeaways & Limitations

    Existing perturbation-based post hoc explanations are not sufficiently robust for ascertaining discriminatory behavior in sensitive applications.

    Source: INTRODUCTION · p. 2, CONCLUSIONS AND FUTURE WORK · p. 7

    The framework assumes the adversary has a biased classifier and sampled real-world inputs but no access to the underlying input distribution.

    Source: Proposed Framework · p. 3

Abstract

from arXiv · show

As machine learning black boxes are increasingly being deployed in domains such as healthcare and criminal justice, there is growing emphasis on building tools and techniques for explaining these black boxes in an interpretable manner. Such explanations are being leveraged by domain experts to diagnose systematic errors and underlying biases of black boxes. In this paper, we demonstrate that post hoc explanations techniques that rely on input perturbations, such as LIME and SHAP, are not reliable. Specifically, we propose a novel scaffolding technique that effectively hides the biases of any given classifier by allowing an adversarial entity to craft an arbitrary desired explanation. Our approach can be used to scaffold any biased classifier in such a way that its predictions on the input data distribution still remain biased, but the post hoc explanations of the scaffolded classifier look innocuous. Using extensive evaluation with multiple real-world datasets (including COMPAS), we demonstrate how extremely biased (racist) classifiers crafted by our framework can easily fool popular explanation techniques such as LIME and SHAP into generating innocuous explanations which do not reflect the underlying biases.

INTRODUCTION

The paper argues that post hoc explanations help experts inspect black-box behavior and bias, but perturbation-based methods can be adversarially manipulated to hide discriminatory classifiers.

  • Black-box models are difficult for domain experts to understand, motivating interpretable tools for diagnosing errors and potential biases.Source: INTRODUCTION · p. 1
  • LIME and SHAP explain individual predictions by estimating feature contributions from perturbed instances.Source: INTRODUCTION · p. 1
  • The proposed framework exploits perturbation-based explanations to construct scaffolds whose explanations can be arbitrarily controlled.Source: INTRODUCTION · p. 1
  • Scaffolded classifiers can remain discriminatory while LIME and SHAP explanations appear innocuous.Source: INTRODUCTION · p. 2
  • Across COMPAS, Communities and Crime, and German loan lending, explanations did not flag sensitive attributes such as race as important.Source: INTRODUCTION · p. 2

BUILDING ADVERSARIAL CLASSIFIERS TO FOOL EXPLANATION TECHNIQUES

The paper presents a framework for constructing adversarial classifiers that fool post hoc explanation techniques relying on input perturbations.

Background: LIME and SHAP

LIME and SHAP locally approximate black-box predictions with interpretable models, balancing fidelity near an instance against explanation complexity.

Proposed Framework

The framework detects whether inputs resemble the real-world distribution or perturbations, then preserves biased predictions on real inputs while altering behavior on perturbed ones.

Proposed Framework

  • The adversary supplies a biased classifier and sampled input data, without access to the underlying real-world distribution.Source: Proposed Framework · p. 3
  • The scaffolded classifier behaves like the biased classifier on real-world inputs while concealing its biases from LIME and SHAP.Source: Proposed Framework · p. 3
  • LIME and SHAP probe local neighborhoods containing synthetic perturbations that may fall outside the input distribution.Source: Proposed Framework · p. 3
  • PCA showed that LIME-style perturbations were distributed significantly differently from input data, making perturbation detection feasible.Source: Proposed Framework · p. 3
  • The adversary exploits this distinction by making the classifier biased on in-distribution inputs and innocuous on perturbed inputs.Source: Proposed Framework · p. 3
  • An OOD classifier is trained by labeling original instances false and perturbed instances true, then fitting a classifier on their union.Source: Proposed Framework · p. 3

EXPERIMENTAL RESULTS

The evaluation tests adversarial classifiers across recidivism, violent-crime, and credit-scoring datasets, using LIME and SHAP under specified perturbation and classifier constructions. Figures compare feature-ranking behavior for biased and adversarial classifiers.

Datasets

  • The evaluation covers COMPAS, Communities and Crime, and German credit datasets spanning recidivism, violent crime, and credit scoring.The datasets include sensitive attributes of race, percentage of Caucasian population, and gender, respectively.Source: EXPERIMENTAL RESULTS · pp. 3–4

Experimental setup

Classifier construction

  • The biased classifier predicts purely from race for COMPAS and Communities and Crime, and from gender for German Credit.Source: EXPERIMENTAL RESULTS · p. 4

Perturbations and OOD classifier

  • LIME perturbations add N(0, 1) noise to every feature, while SHAP masks randomly selected features using a 10-cluster kmeans background distribution.Source: EXPERIMENTAL RESULTS · p. 4

Adversarial classifier construction

  • The unbiased classifier ψ uses one or two synthetic features with zero correlation to sensitive attributes, including XOR when two features are used.Source: EXPERIMENTAL RESULTS · p. 4

Evaluation measures

  • Figures 2–4 compare top-three feature rankings for the biased classifier and adversarial classifiers across COMPAS, Communities and Crime, and German credit.German credit uses Loan Rate % Income as an existing predictive feature and Gender as the sensitive feature.Source: Figure 2 · p. 4, Figure 3 · p. 5, Figure 4 · p. 5

Effectiveness of Adversarial Classifiers

The adversarial classifiers substantially redirect LIME and SHAP feature rankings away from sensitive features while retaining strong agreement with the original classifier in the evaluated settings. LIME is fooled across all three datasets, whereas SHAP is less successful when two uncorrelated features are used.

LIME

SHAP

  • 100% of Communities and Crime, 84% of COMPAS, and 85% of German points shift SHAP importance from the sensitive feature when one uncorrelated feature is used.Source: Effectiveness of Adversarial Classifiers · p. 5
  • With two uncorrelated features, SHAP removes the bias feature from first place in 67% of COMPAS and 71% of Communities and Crime instances.The passage attributes this lower success to SHAP’s local accuracy property, which distributes attributions across several features when no single feature is identifiable.Source: Effectiveness of Adversarial Classifiers · p. 5

Fidelity

Effect of Perturbation Detection Accuracy

Attack success depends on the OOD classifier’s ability to distinguish perturbed instances from input-data points: higher detection accuracy makes LIME and SHAP less likely to identify race as most important.

Effect of Perturbation Detection Accuracy

  • When OOD detection is inadequate, both LIME and SHAP identify race as the most important feature, so the attack fails.Improved OOD classification enables the scaffolded behavior on perturbed points that hides the bias in explanations.Source: Effect of Perturbation Detection… · p. 5
  • Table 2 reports feature coefficients for a single-feature LIME explanation before and after an attack on one COMPAS instance.The supplied table caption identifies the comparison but does not provide the coefficient values.Source: Table 2 · p. 6
  • LIME becomes highly vulnerable once the OOD classifier’s F1 score exceeds 0.8, whereas SHAP begins responding around F1 score ∼0.45 but hides bias more gradually.SHAP requires a very accurate OOD classifier to completely conceal the bias.Source: Effect of Perturbation Detection… · p. 6
  • The attack is effective when perturbed instances and input-data points can be differentiated with reasonable accuracy.Figure 5 measures effectiveness by the percentage of COMPAS data points for which race is the most important feature against OOD-classifier F1 score.Source: Effect of Perturbation Detection… · p. 6, Figure 5 · p. 6

Robustness to Hyperparameters

The attacks remain effective across several test-time choices for LIME and SHAP, including kernel distance, kernel width, and background-cluster count.

Robustness to Hyperparameters

  • LIME attacks remain equally effective when the kernel width changes from its default or when l1 distance replaces l2 distance.The reported kernel-width range is truncated in the supplied passage, but the robustness conclusion is explicit.Source: Robustness to Hyperparameters · p. 6
  • SHAP attacks remain equally effective when the test-time K-means background uses 5 or 15 clusters instead of the default 10.Using an all-zero SHAP background makes the attacks even more pronounced.Source: Robustness to Hyperparameters · p. 6
  • Changing the SHAP background distribution to all zeroes makes the attacks even more pronounced.This result is reported alongside robustness to alternative K-means cluster counts.Source: Robustness to Hyperparameters · p. 6

Example Explanations

Example COMPAS explanations show that the attack hides race, the original classifier’s sole important feature, from LIME and substantially reduces its prominence in SHAP.

Example Explanations

  • Race is completely hidden from the LIME explanation after the attack, despite being the sole important feature for the original classifier f.The example concerns an instance from the COMPAS dataset.Source: Example Explanations · p. 6
  • In the SHAP explanation, race is considerably hidden after the attack but is not completely nullified.The passage contrasts this partial suppression with the complete hiding reported for LIME.Source: Example Explanations · p. 6

RELATED WORK

Related work covers perturbation-based explanation methods, criticisms of post hoc faithfulness, adversarial manipulation of saliency maps, and challenges in detecting bias even with interpretability or transparency.

Perturbation-based Explanation Methods

  • Perturbation-based post hoc methods include LIME, SHAP, anchors, and GAM, which produce local feature or rule-based explanations.LIME and SHAP are described as prominent local, model-agnostic approaches.Source: RELATED WORK · p. 6

Criticism of Post hoc Explanations

  • Prior critiques argue that post hoc explanations may be unfaithful, sensitive to small input changes, or based on unintuitive perturbation points.These concerns are especially noted for structured data and unchanged classifier predictions.Source: RELATED WORK · pp. 6–7

Adversarial Explanations

  • Earlier adversarial-explanation work showed that imperceptible input modifications can arbitrarily change saliency maps in image classification.The cited studies include attacks on saliency maps by Dombrowski et al. and Heo et al.Source: RELATED WORK · p. 7

Interpretability and Bias Detection

  • Interpretability has been proposed for evaluating bias, but post hoc explanations cannot definitively establish unfairness, and correlated variables complicate bias detection even for transparent models.The related work frames interpretability and transparency as insufficient for conclusively detecting or preventing bias.Source: RELATED WORK · p. 7

CONCLUSIONS AND FUTURE WORK

The framework hides discriminatory biases from perturbation-based explanations while preserving biased predictions, and experiments find LIME more vulnerable than SHAP. The findings suggest existing post hoc explanations are insufficient for ascertaining discriminatory behavior in sensitive applications and motivate robust alternatives.

  • The framework generates adversarial classifiers that fool LIME and SHAP while preserving biased predictions on the input distribution.It creates scaffolding around biased classifiers and controls behavior on perturbed data points so explanations appear innocuous.Source: CONCLUSIONS AND FUTURE WORK · p. 7
  • Extensive experiments on criminal justice and credit-scoring data find that LIME is more vulnerable than SHAP.Source: CONCLUSIONS AND FUTURE WORK · p. 7
  • Existing post hoc explanation techniques are not sufficient for ascertaining discriminatory classifier behavior in sensitive applications.Source: CONCLUSIONS AND FUTURE WORK · p. 7

Future research directions

Related papers

Loading 1911.02508v2…