PaperlayerView exact revision on arXiv

Source-linked AI summary

Detecting Adversarial Samples Using Influence Functions and Nearest Neighbors

Gilad Cohen, Guillermo Sapiro, Raja Giryes

arXiv:1909.06872v2cs.LGstat.ML

TL;DR

Adversarial examples threaten DNN classifiers, motivating detectors that can identify attacks without altering a pre-trained network. The paper combines influence functions with embedding-space nearest neighbors to compare influential training samples with nearby samples, then trains a detector on their ranks and distances. NNIF achieves state-of-the-art adversarial detection across six attacks and three datasets, although influence computation is costly and alternative distance metrics may improve results.

  • Problem

    DNNs are vulnerable to imperceptible adversarial perturbations that induce confident misclassification, while sensitive applications require reactive detection methods.

  • Method

    NNIF uses influence functions to find helpful and harmful training samples, compares them with embedding-space k-NN through ranks and distances, and trains a Logistic Regression detector.

  • Results

    NNIF achieves state-of-the-art adversarial detection performance for six attacks—FGSM, JSMA, Deepfool, CW, PGD, and EAD—on CIFAR-10, CIFAR-100, and SVHN.

  • Takeaways & Limitations

    The correlation between nearest neighbors and influential training examples is strong for normal images but breaks for adversarial inputs, providing a detection signal.

  • Takeaways & Limitations

    NNIF requires costly influence calculations over the entire training set, creating a deployment challenge for real-time detection; its analysis mainly uses L2 distance.

Abstract

from arXiv · show

Deep neural networks (DNNs) are notorious for their vulnerability to adversarial attacks, which are small perturbations added to their input images to mislead their prediction. Detection of adversarial examples is, therefore, a fundamental requirement for robust classification frameworks. In this work, we present a method for detecting such adversarial attacks, which is suitable for any pre-trained neural network classifier. We use influence functions to measure the impact of every training sample on the validation set data. From the influence scores, we find the most supportive training samples for any given validation example. A k-nearest neighbor (k-NN) model fitted on the DNN's activation layers is employed to search for the ranking of these supporting training samples. We observe that these samples are highly correlated with the nearest neighbors of the normal inputs, while this correlation is much weaker for adversarial inputs. We train an adversarial detector using the k-NN ranks and distances and show that it successfully distinguishes adversarial examples, getting state-of-the-art results on six attack methods with three datasets. Code is available at https://github.com/giladcohen/NNIF_adv_defense.

1. Introduction

The paper targets reactive detection of adversarial inputs by testing whether training examples that influence a prediction correspond to nearby embedded training neighbors. This correspondence is strong for normal inputs but breaks for adversarial ones, enabling detection with a lightweight classifier.

  • 1. Introduction: The method detects adversarial attacks for any pre-trained neural network by identifying breaks between training-data influence and network decisions.The approach focuses on reactive detection without modifying the DNN.
  • 1. Introduction: Influence functions measure how each training sample affects the loss of a given test sample when that training point is upweighted.This provides a per-training-example measure of influence on the test classification.
  • 1. Introduction: Embedding-space k-NN provides a second measure because prior work links nearest-neighbor resemblance with the network’s decision and output.The neighbors are selected from training examples in the DNN embedding space.
  • 1. Introduction: For normal inputs, the most helpful training samples strongly correlate with embedding-space k-NN, whereas adversarial inputs show weak correspondence.Figure 1 compares 25 nearest neighbors with 25 most helpful samples for a normal image and its adversarial version.
  • 1. Introduction: The detector ranks influential training samples by k-NN distance and uses those ranks and distances as features for Logistic Regression.For an unseen input, influential training examples are selected first, then their embedding-space ranks and L2 distances are computed.
  • 1. Introduction: The evaluation reports an advantage over other leading detection techniques across various attack methods and datasets.The results also support the previously reported resemblance between embedding-space k-NN and DNN decisions.

2. Related work

Related work spans proactive robustness defenses, reactive adversarial detectors, adversarial attack algorithms, and theoretical analyses of DNN vulnerability and robustness. The paper situates its approach among methods using DNN representations, nearest neighbors, density, dimensionality, and distributional distance.

  • 2. Related work: Theoretical work analyzes robustness through robust optimization, input-loss gradients, decision-boundary curvature, and formal robustness guarantees.Madry et al. identify PGD as an optimal first-order adversary within their framework.
  • 2. Related work: The surveyed attacks include fast one-step FGSM, sparse iterative JSMA, non-targeted Deepfool, and targeted CW optimization attacks.JSMA is reported to achieve 97% success while modifying 4.02% of input features on average, although it is iterative and costly.
  • 2. Related work: Adversarial defenses divide into proactive methods that improve robustness and reactive methods that detect attacks without changing the DNN.The surveyed proactive approaches include adversarial training, distillation, gradient masking, feature squeezing, regularization, Parseval networks, and k-NN search.
  • 2. Related work: Reactive detectors distinguish adversarial from natural images using features from DNN layers or learned encoders, including kernel-density and Bayesian-uncertainty features.The related work also includes detectors derived from existing DkNN outputs.
  • 2. Related work: LID detection estimates how the number of data objects expands with distance, using extreme-value modeling of nearest-neighbor distances at DNN layers.The cited procedure applies attacks and Gaussian noise to normal examples before using the resulting data for detection.
  • 2. Related work: DkNN fits k-NN models at every network layer and uses calibration-based nonconformity counts to estimate prediction credibility and robustness.The paper converts DkNN’s empirical p-values into features for a Logistic Regression adversarial detector.
  • 2. Related work: Mahalanobis-based detection models class-conditional activation distributions and uses distances to those distributions as Logistic Regression features.Means and covariances are computed for each class and layer from training activations.

3. Method

NNIF combines influence functions with embedding-space nearest-neighbor statistics to detect adversarial inputs. It selects influential training examples, measures their neighbor ranks and distances, and trains a Logistic Regression detector, but its influence computation is costly.

  • 3. Method: NNIF hypothesizes that adversarial attacks move inputs toward embedding-space regions where harmful training data can induce misclassification.The method tests this hypothesis using k-NN models on DNN activation layers and influence functions.
  • 3. Method: Influence functions identify training samples that help or hinder a DNN’s prediction by measuring their effect on a specific test-image loss.The influence calculation uses the model Hessian, loss, and parameters.
  • 3. Method: For each test example, NNIF selects the top M helpful and harmful training samples, then computes each selected sample’s embedding-space rank and distance.The k-NN model is fitted using embedding vectors for all training examples.
  • 3. Method: The detector uses four NNIF feature groups—helpful and harmful ranks and distances—to train Logistic Regression on normal and attacked examples.The same feature extraction process is repeated after applying an adversarial attack to the test image.
  • 3. Method: Training requires influence calculations over every training-validation pair, with time complexity O(Ntrain · Nval), while test-time detection has complexity O(Ntrain).The authors identify the computation time as a deployment issue, especially for real-time systems.

4. Results

NNIF detects adversarial inputs by testing whether influence-selected training examples retain the nearest-neighbor relationships expected for normal inputs. Across six attacks and three datasets, it outperforms existing detectors on most attack–dataset settings, while its helpful-example distance is the strongest ablation feature.

  • Experimental setup: NNIF combines influence functions with k-NN ranks and L2 distances of helpful and harmful training examples, then trains a logistic-regression adversarial detector.The algorithm generates attacked validation data, fits k-NN models across activation layers, extracts influence-selected neighbor features, and trains the detector on normal and adversarial examples.
  • Experimental setup: The test-time influence approximation uses 10k randomly selected training samples instead of the full 49k, and the authors report that it remains sufficient for state-of-the-art detection.The approximation is described as coarse but is used only for the test set because influence computation over the entire training set is very slow.
  • Detection of adversarial attacks: NNIF surpasses all other detectors for Deepfool, CW, and PGD across CIFAR-10, CIFAR-100, and SVHN, while matching Mahalanobis on FGSM and JSMA and leading EAD on two datasets.For EAD, NNIF achieves new state-of-the-art results on CIFAR-10 and SVHN, but not CIFAR-100.
  • Ablation study: DM↑, the embedding-space L2 distance to the most helpful training examples, is the most influential feature and is nearly as effective alone as all four features.RM↑ does not improve results when added to DM↑, suggesting its information is already inferred from the helpful-example distance; DM↓ is more useful for CW than Deepfool.
  • Ablation study: A simple threshold detector using only DM↑ is possible for all attacks, avoiding direct exposure of a learned logistic-regression classifier’s parameters.The authors motivate this design because an attacker could potentially access the parameters of a learning-based detector.

5. Discussion and conclusions

The paper concludes that influence-based distances and nearest-neighbor rankings enable strong adversarial detection across attacks and datasets, while identifying metric and computational limitations.

  • 5. Discussion and conclusions: SOTA detection performance was achieved for six attacks across CIFAR-10, CIFAR-100, and SVHN, with robustness in a white-box setting.The detector combines the DM↑ distance to helpful training inputs with training-input nearest-neighbor ranking order.
  • 5. Discussion and conclusions: The L2 distance from a test embedding to its most helpful training inputs is a strong adversarial-detection measure.
  • 5. Discussion and conclusions: Future work should examine alternative distance metrics or transformations because the analysis mainly used L2 distance.Mahalanobis distance is suggested as a possible improvement.
  • 5. Discussion and conclusions: Computing influence functions over the entire training set is time-consuming and must be accelerated for real-time deployment.The paper suggests hashing nearest neighbors to influential training examples as one possible solution.

A. Method

NNIF computes influence scores for training samples, selects the most helpful or harmful examples, and uses these ranked outputs for downstream detection.

  • A. Method: Influence functions are evaluated for every training sample to measure its effect on a test sample’s loss, after which influential examples are sorted.The algorithm takes a test sample, training set, and M, the number of top influence samples to collect.
  • A. Method: The influence calculation uses the test-loss gradient, model Hessian inverse, and training-loss gradient to quantify sample impact.The displayed approximation is described using the model Hessian, loss, and parameters.
  • A. Method: The method retains the top M helpful training examples for further processing.The algorithm also initializes and tracks harmful-example indices.

B. Experimental setup

The experiments compare adversarial attacks and detection methods across datasets, using tuned hyperparameters and reporting clean accuracy, attack success, and detector settings.

  • B. Experimental setup: The evaluation includes FGSM, JSMA, Deepfool, CW, CW-Opt, PGD, and EAD attack success rates.Attack success rates are reported for the studied datasets, with higher rates noted for CIFAR-100.
  • B. Experimental setup: Clean DNN accuracies are reported for normal images without attack.
  • B. Experimental setup: Hyperparameters for DkNN, LID, Mahalanobis, and NNIF were selected by nested cross-entropy validation using detection ROC AUC.NNIF tunes M, while the other methods tune their respective neighbor or noise parameters.

C. Detection of adversarial attacks

NNIF outperforms prior detection methods on representative CIFAR-10 attacks and is evaluated across six attacks, three datasets, and embedding-based features.

  • C. Detection of adversarial attacks: NNIF achieves better classification power than previous state-of-the-art methods for Deepfool and CW attacks on CIFAR-10.Figure 3 reports ROC curves for these two attacks.
  • C. Detection of adversarial attacks: AUC comparisons cover FGSM, JSMA, Deepfool, CW, PGD, and EAD attacks on CIFAR-10, CIFAR-100, and SVHN.Table 8 uses only features from the DNN embedding space.

D. Ablation study

The ablation study identifies which influence-function and nearest-neighbor features contribute most to adversarial detection and shows that feature usefulness depends on the attack.

  • D. Ablation study: DM↑ is the most beneficial feature among the four evaluated features across FGSM, JSMA, Deepfool, and CW on CIFAR-10.DM↑ measures the L2 distance from the most helpful training examples in the DNN embedding space.
  • D. Ablation study: 98.27% versus 81.91% and 99.79% versus 97.27% show that RM↑ and DM↑ detect Deepfool better than CW.The comparisons correspond to RM↑ and DM↑ alone, respectively.
  • D. Ablation study: 89.97% versus 82.11% shows that DM↓ detects CW better than Deepfool.Figure 4 reports probability densities for RM↑, DM↑, and DM↓ on CIFAR-10 for both attacks.

E. Generalization to other attacks

The study evaluates whether a detector trained on FGSM transfers to unseen attacks using features from all activation layers. The generalization comparison does not identify a definite winner among the methods.

  • E. Generalization to other attacks: The generalization results do not have a definite winner method.The comparison includes NNIF, DkNN, and Mahalanobis-based methods.
  • E. Generalization to other attacks: Generalization is tested by training the LR classifier on FGSM features and evaluating it on JSMA, Deepfool, CW, PGD, and EAD.All activation layers provide features for this comparison, except DkNN, which uses the DNN embedding space.

F. Attack against NNIF

The white-box evaluation tests attacks designed to evade NNIF across CIFAR-10, SVHN, and CIFAR-100. NNIF remains robust overall, although the effect varies by dataset and feature set.

  • F. Attack against NNIF: NNIF detection accuracy decreases by only 1% on CIFAR-10 under the proposed white-box attack, supporting robustness in that setting.The same attack also decreases DkNN and LID accuracies because they use L2 nearest-neighbor distances in embedding space.
  • F. Attack against NNIF: 20% impairment occurs for NNIF on SVHN under CW-Opt, while DkNN and LID decrease by more than 20%.The attack flips 46% of SVHN labels, compared with 96% for vanilla CW, so the white-box evaluation considers the hardest samples for detection.
  • F. Attack against NNIF: CIFAR-100 shows increased NNIF detection accuracy in the white-box setting, unlike the other datasets.The attack targets the helpful-distance feature, while NNIF also uses other parameters.
  • F. Attack against NNIF: Using only DM↑ on CIFAR-100 reduces detection accuracy from 74% to 65%, confirming that the white-box attack affects distance features.The reported defense-accuracy table covers the DM↑-only variant.
  • F. Attack against NNIF: Overall, NNIF is reported to be robust to white-box attacks, with rank features adding strength because ranks are non-differentiable.This makes rank-based optimization difficult for the attacker.

G. Influence function smoothness

The influence-function calculation raises a smoothness concern because ReLU-based cross-entropy is not continuously differentiable. The authors argue that this does not affect practical backpropagation.

  • G. Influence function smoothness: ReLU activations make the cross-entropy loss non-continuously differentiable, creating a technical concern for calculating the influence function.The authors assume the issue is negligible because discontinuities have measure zero and problematic activation points are not encountered in backpropagation.
Loading 1909.06872v2…