PaperlayerView exact revision on arXiv

Source-linked AI summary

When Explainability Meets Adversarial Learning: Detecting Adversarial Examples using SHAP Signatures

Gil Fidel, Ron Bitton, Asaf Shabtai

arXiv:1909.03418v1cs.LGcs.AIcs.CRstat.ML

TL;DR

The paper asks whether SHAP explanation patterns can help detect adversarial examples and generalize to attacks unknown during training. It uses penultimate-layer SHAP values as detector features and reports strong detection and cross-attack generalization on CIFAR-10 and MNIST.

  • Problem

    The paper asks whether SHAP explanation patterns can help detect adversarial examples and generalize to attacks unknown during training.

    Source: §I · p. 1, §VI · p. 6

  • Method

    The method uses SHAP importance scores for penultimate-layer neurons as features for a supervised binary adversarial-example detector.

    Source: §IV · p. 3, D. Training the detector · p. 6

  • Results

    The detector generalized well to attacks unseen during training; AUC-ROC was 0.966/0.967 for CIFAR-10/MNIST, respectively.

    Source: RQ1: baseline performance of the… · p. 7, §VII · p. 8

  • Takeaways & Limitations

    Combining SHAP signatures with signatures from other detectors in a continuously updated framework could advance practical defense against adversarial examples.

    Source: §VII · p. 8

    The explanation–feature-robustness connection is only a first step toward understanding and warrants rigorous study.

    Source: §VII · p. 8

Abstract

from arXiv · show

State-of-the-art deep neural networks (DNNs) are highly effective in solving many complex real-world problems. However, these models are vulnerable to adversarial perturbation attacks, and despite the plethora of research in this domain, to this day, adversaries still have the upper hand in the cat and mouse game of adversarial example generation methods vs. detection and prevention methods. In this research, we present a novel detection method that uses Shapley Additive Explanations (SHAP) values computed for the internal layers of a DNN classifier to discriminate between normal and adversarial inputs. We evaluate our method by building an extensive dataset of adversarial examples over the popular CIFAR-10 and MNIST datasets, and training a neural network-based detector to distinguish between normal and adversarial inputs. We evaluate our detector against adversarial examples generated by diverse state-of-the-art attacks and demonstrate its high detection accuracy and strong generalization ability to adversarial inputs generated with different attack methods.

I. INTRODUCTION

DNNs can misclassify inputs after slight, potentially imperceptible perturbations, while adversarial-example detection remains an open problem. The paper uses SHAP-based signatures to detect adversarial inputs and reports high accuracy across attack methods, including methods unseen during training.

I. INTRODUCTION

  • Adversarial perturbations are slight input changes that can cause misclassification, and effective detection remains an open problem amid advanced attacks.For image classifiers, pixel changes may be imperceptible to people while changing the classifier's output.Source: §I · p. 1
  • DNNs are difficult to explain, limiting the ability to justify their decisions in production-grade contexts.The introduction connects this challenge to research on explainable AI.Source: §I · p. 1
  • The method applies SHAP to the penultimate layer to create XAI signatures, which a detector uses to distinguish normal from adversarial inputs.The paper presents this approach as a way to use a proposed connection between explainability and adversarial examples for detection.Source: §I · p. 1
  • Across CIFAR-10 and MNIST, the detector achieves high detection accuracy (AUC ˜97%) and performs well on examples from both trained and untrained attack methods.The evaluation includes white-box and black-box attacks.Source: §I · p. 1
  • The study presents adversarial-example detection as a contribution and a first step toward linking adversarial learning with explainable AI.The authors also report strong effectiveness against diverse attacks.Source: §I · pp. 1–2

II. BACKGROUND

The background defines adversarial examples as inputs altered to induce misclassification under a perturbation constraint. It distinguishes attack goals and access assumptions, and outlines common generation methods and perturbation measures.

A. Adversarial Attacks

  • An adversarial example is formed by adding a perturbation to an input so the classifier returns a label different from its correct class, subject to a maximum perturbation size.The size constraint is intended to keep the altered sample visually indistinguishable from the original.Source: A. Adversarial Attacks · p. 2
  • L0 counts changed input features, L2 measures Euclidean distance, and L∞ measures the largest single-feature difference; each approximates perceptual change differently.Source: A. Adversarial Attacks · p. 2
  • Targeted attacks aim for an attacker-chosen class, whereas non-targeted attacks seek any incorrect class.Source: A. Adversarial Attacks · p. 2
  • FGSM takes one gradient step [19], BIM and PGD take multiple smaller steps, [6], and C&W formulates attack generation as metric-specific optimization.Source: A. Adversarial Attacks · p. 2
  • White-box attackers can access a model's structure and parameters, while black-box attackers observe outputs only; transferability can let attacks on a surrogate fool another model.Source: A. Adversarial Attacks · p. 2

B. Adversarial Defenses

The background groups defenses into improving model robustness and detecting adversarial examples, while describing detectors that inspect input irregularities or internal model behavior. It also presents robust and non-robust features as an account of adversarial examples.

B. Adversarial Defenses

C. Understanding adversarial examples

  • Non-robust features can be highly predictive yet fragile to small perturbations, while robust features are predictive and change less readily [18].The cited account connects non-robust features to adversarial examples and their transferability across models.Source: C. Understanding adversarial examples · p. 2

D. Explainable AI

The paper uses SHAP to interpret DNN decisions and proposes using penultimate-layer importance scores to detect adversarial inputs. It motivates this approach through a robust/non-robust feature hypothesis and exploratory visualizations of SHAP signatures.

D. Explainable AI

  • SHAP estimates how inputs or neurons contribute to model decisions, and DeepExplainer is a variant optimized for explaining DNNs.The paper situates SHAP among methods for explaining supervised models.Source: D. Explainable AI · p. 3

III. RELATED WORK

  • Prior detectors target input or model-behavior irregularities; this work proposes a new approach and evaluates cross-attack generalization across diverse attacks.Table I summarizes prior detectors’ concepts, setups, results, pros, and cons.Source: §III · p. 3, Table I · p. 4

IV. ROBUSTNESS THROUGH EXPLAINABILITY

  • The method hypothesizes adversarial inputs rely more on non-robust features, then uses SHAP importance scores for penultimate-layer neurons as detector features.Those neurons are treated as high-level features of the classifier.Source: §IV · p. 3
  • In an illustrative targeted PGD L2 example, the adversarial automobile signature combines patterns associated with normal automobiles and cats.The authors present the robust/non-robust feature interpretation as speculation requiring further research.Source: §IV · p. 3
  • The CIFAR-10 embedding shows normal samples clustered by class and adversarial examples in a separate cluster, while attack types disperse around adversarial clusters.The authors interpret this separation and dispersion as grounds for expecting useful detector features and cross-attack generalization.Source: §IV · pp. 3–4

V. PROPOSED METHOD

The proposed solution has three phases: build a repository of normal and adversarial examples, generate XAI signatures, and construct a detector. Its notation defines the classifier, layer outputs, inputs, and ground-truth labels.

V. PROPOSED METHOD

  • The proposed method comprises repository creation, XAI-signature generation, and detector construction.Figure 4 is identified as depicting the detector training process.Source: §V · p. 4

A. Notation

  • f(·) denotes the neural-network classifier, and f[i](·) denotes layer i’s output, with layer 0 as input and layer l as final softmax output.These definitions specify the model and its layer outputs used in the method.Source: A. Notation · p. 4
  • x denotes an input vector, while Y(x) denotes its ground-truth label.Source: A. Notation · p. 4

B. Creating a repository of normal and adversarial examples

The repository is built from normal samples and successfully generated adversarial examples, with attack parameters varied to expose the detector to different perturbations.

B. Creating a repository of normal and adversarial examples

  • Normal examples are sampled from the classifier’s training dataset, and varied attack hyperparameters produce diverse perturbations intended to improve detector generalization.The varied parameters include distance metrics, perturbation budget, iteration count, and attack step.Source: B. Creating a repository of normal and… · p. 4
  • The generation procedure randomly selects a sample, attack method, distance metric, preferences, and a target class different from the ground truth.It stores an adversarial example when the targeted classifier assigns the requested target class.Source: B. Creating a repository of normal and… · pp. 4–5
  • The accompanying figures illustrate signatures across classes and original/adversarial examples, plus CIFAR-10 UMAP views of normal and adversarial signatures and different attacks.Source: Figure 1 · p. 5, Figure 2 · p. 5, Figure 3 · p. 5

C. Generating XAI signatures

The method applies SHAP DeepExplainer to penultimate-layer neurons and concatenates the resulting values into a signature vector. Normal signatures model ordinary decisions, while attack signatures model perturbation effects; the repository should be updated as new attacks emerge.

C. Generating XAI signatures

  • For each sample, SHAP DeepExplainer interprets penultimate-layer neurons, producing values across target classes that are concatenated into a flat XAI-signature vector.Its size is the number of target classes times the number of penultimate-layer outputs.Source: C. Generating XAI signatures · p. 5
  • Normal signatures provide a baseline for normal decision-making in activation space, while attack signatures model perturbation effects on decisions.Source: C. Generating XAI signatures · p. 6
  • Production use calls for continuously adding signatures from newly discovered attacks to improve detection of new attack classes.The authors also state that a good detector should generalize from known attacks to unknown ones.Source: C. Generating XAI signatures · p. 6

D. Training the detector

The detector uses SHAP values as features for supervised classification of normal versus adversarial samples. The evaluation asks whether it performs well on known attacks and generalizes to unseen attacks across CIFAR-10 and MNIST models.

D. Training the detector

  • SHAP values serve as features for a supervised binary detector that classifies samples as normal or adversarial.The implementation uses a fully connected feed-forward neural network; at inference, a sample’s SHAP values are fed to the classifier.Source: D. Training the detector · p. 6

VI. EVALUATION

  • The evaluation tests baseline performance on attacks represented in both training and test sets, and generalization to attacks unknown during training.These are the stated RQ1 and RQ2 questions.Source: §VI · p. 6

A. Evaluation setup

  • The evaluation uses CIFAR-10 with ResNet-56 and MNIST with a CNN; their reported test accuracies are 93.39% and 99.25%, respectively.The MNIST architecture is taken from the Keras MNIST example.Source: A. Evaluation setup · p. 6

Adversarial example generation.:

The authors generate adversarial examples using established frameworks, compute SHAP features from the classifiers’ last fully connected layers, and train a neural-network detector on those features.

Adversarial example generation.:

  • Adversarial examples are generated with Foolbox and Adversarial Robustness Toolbox, while SHAP explanations are generated with the SHAP framework.The passage also points to Table II for the normal and adversarial sample counts used for training and testing.Source: Adversarial example generation · p. 6
  • Table II describes the normal and adversarial sample counts used for training and testing on CIFAR-10 and MNIST.The provided text does not give the table’s individual counts.Source: Adversarial example generation · p. 6, Table II · p. 6

SHAP values computation.:

  • SHAP values are computed on each classifier’s last fully connected layer, yielding 640 features per CIFAR-10 sample and 1280 per MNIST sample.The CIFAR-10 layer has 64 neurons and the MNIST layer has 128.Source: SHAP values computation · p. 6

Training the adversarial example detector.:

  • A three-hidden-layer feed-forward neural network is trained on SHAP features, using an 80/20 train-validation split, AdaBound, and early stopping.The hidden layers have 256, 128, and 16 neurons; training runs for at most 500 epochs and stops if validation loss fails to improve for 20 epochs.Source: Training the adversarial example… · p. 7

B. Results and Discussion

The detector performs well on known attacks and achieves similar overall performance when tested on attacks withheld from training. Per-attack results also report high true-positive rates for most attacks at a false-positive rate of 0.05.

RQ1: baseline performance of the detector.:

RQ2: Generalization across different attack types.:

VII. CONCLUSIONS AND FUTURE WORK

The authors report detection across diverse attacks and generalization to attacks absent from training, while identifying training requirements and further evaluation as open work. They also propose a broader signature-based framework and further study of explanation patterns and feature robustness.

VII. CONCLUSIONS AND FUTURE WORK

  • Experiments support detection across state-of-the-art attacks and generalization to attacks not used for training.The authors connect these results to patterns in penultimate-layer SHAP values and the importance of robust versus non-robust features.Source: §VII · p. 8
  • The supervised method requires a big adversarial-example training set; the authors suggest semi-supervised detection as a possible way to streamline training.They state that this approach could use the same features and further improve detector performance.Source: §VII · p. 8
  • The authors envision extending the method into a framework that collects benign and malicious samples, extracts signatures, and classifies or forwards samples for review.The proposed framework could combine SHAP-based signatures with signatures from other state-of-the-art detectors.Source: §VII · p. 8
  • The paper presents its study as a first step toward understanding how model explanations relate to feature robustness.The authors say rigorous study of this connection could benefit detector performance and understanding of adversarial examples.Source: §VII · p. 8
  • Future work includes testing additional datasets and models, evaluating detector transfer across classifiers, and assessing customized attacks adapted to the detector.The authors note that SHAP computation is not differentiable, making such customized attacks nontrivial.Source: §VII · p. 8
Loading 1909.03418v1…