PaperlayerView exact revision on arXiv

Source-linked AI summary

Fooling Neural Network Interpretations via Adversarial Model Manipulation

Juyeon Heo, Sunghwan Joo, Taesup Moon

arXiv:1902.02041v3cs.LGcs.AIcs.CVstat.ML

TL;DR

The paper asks whether interpretation methods can be fooled by fine-tuning model parameters to alter explanations without substantially harming predictive accuracy. It incorporates interpretation results into the fine-tuning penalty and shows that saliency-map interpreters can be fooled, with effects generalizing across validation data and transferring between methods. The authors argue that stability against this manipulation should be checked when developing robust interpreters.

  • Problem

    The paper investigates whether neural network interpretation methods are stable against adversarial model manipulation, a form of fine-tuning that changes explanations without significantly hurting accuracy.

  • Method

    The authors incorporate interpretation results into the fine-tuning objective and evaluate Passive and Active fooling against LRP, Grad-CAM, and SimpleGradient.

  • Results

    LRP, Grad-CAM, and SimpleGradient are vulnerable, with ImageNet accuracy drops around 2% for Top-1 and 1% for Top-5 accuracy; fooled explanations generalize and transfer across methods.

  • Takeaways & Limitations

    Stability against adversarial model manipulation should be treated as an important criterion for developing robust and reliable interpretation methods.

Abstract

from arXiv · show

We ask whether the neural network interpretation methods can be fooled via adversarial model manipulation, which is defined as a model fine-tuning step that aims to radically alter the explanations without hurting the accuracy of the original models, e.g., VGG19, ResNet50, and DenseNet121. By incorporating the interpretation results directly in the penalty term of the objective function for fine-tuning, we show that the state-of-the-art saliency map based interpreters, e.g., LRP, Grad-CAM, and SimpleGrad, can be easily fooled with our model manipulation. We propose two types of fooling, Passive and Active, and demonstrate such foolings generalize well to the entire validation set as well as transfer to other interpretation methods. Our results are validated by both visually showing the fooled explanations and reporting quantitative metrics that measure the deviations from the original explanations. We claim that the stability of neural network interpretation method with respect to our adversarial model manipulation is an important criterion to check for developing robust and reliable neural network interpretation method.

1 Introduction

The paper examines whether interpretation methods remain stable when model parameters are fine-tuned to change explanations without substantially reducing accuracy. It finds representative saliency-map interpreters vulnerable to these manipulations, with fooling generalizing across validation data and methods.

  • Motivation: Adversarial model manipulation fine-tunes model parameters to dramatically alter interpretation results without significantly hurting the original model’s accuracy.This stability notion differs from robustness to input perturbations or attacks.
  • Motivation: Interpretation methods can help debug models and reveal unintended bias, making fooled explanations consequential for applications such as income prediction.The introduction highlights fairness checking as a setting where interpretations may be used to detect race-related bias.
  • Conclusion: The paper calls for interpretation methods to be evaluated for stability against adversarial model manipulation as part of developing robust and reliable interpreters.The authors frame this stability as an important additional criterion for interpretation research.
  • Contributions: LRP, Grad-CAM, and SimpleGradient are vulnerable to the proposed model manipulation, with ImageNet accuracy drops around 2% for Top-1 and 1% for Top-5 accuracy.These representative saliency-map interpreters can be fooled while retaining nearly the original predictive performance.
  • Contributions: The fooled explanations generalize to the entire validation set and transfer across interpretation methods rather than affecting only selected inputs or one interpreter.Figure 1(b) illustrates manipulated VGG19 models with Passive and Active foolings while reporting about 1% Top-5 accuracy differences across ImageNet validation.

2 Related Work

Prior interpretability work includes black-box and gradient/saliency-map methods, alongside sanity checks based on perturbing inputs or models. This paper focuses on whether gradient/saliency-map interpreters can be fooled by fine-tuning model parameters without perturbing inputs.

  • Interpretation methods: Interpretability frameworks are broadly categorized into black-box methods and gradient/saliency-map-based methods.The latter generally access model architectures and parameters directly.
  • Interpretation methods: Gradient/saliency-map methods are typically simpler and less computationally intensive for complex neural networks, motivating the paper’s focus on them.The study examines LRP, Grad-CAM, and SimpleGradient as representative state-of-the-art methods.
  • Sanity checking: Earlier sanity checks mainly test prediction or interpretation stability by perturbing input data or models, often in the context of adversarial attacks.These approaches differ from the paper’s model-level fine-tuning manipulation.
  • Relation to prior work: Unlike methods that attack each input image, this work changes model parameters through fine-tuning and leaves input data unperturbed.The authors link this distinction to fooling that generalizes across the validation data.

3 Adversarial Model Manipulation

The paper formulates adversarial model manipulation by fine-tuning a pretrained network with interpretation-dependent penalties, then defines Passive and Active fooling schemes. These schemes respectively make explanations uninformative or swap explanations between target classes while using training choices intended to preserve classification accuracy.

  • 3.1 Preliminaries and notations: The considered interpreters generate heatmaps that represent the relevance or importance of input data points for a prediction.LRP assigns positive or negative pixel relevance, Grad-CAM combines gradients with class activation maps, and SimpleGrad visualizes prediction-score gradients with respect to input pixels.
  • 3.2 Objective function and penalty terms: The manipulation objective combines ordinary classification loss with an interpretation penalty computed on a possibly smaller fooling dataset.The classification loss uses the training data, while the penalty term uses D_fool and λ controls the trade-off between them.
  • 3.2.1 Passive fooling: Passive fooling makes explanations uninformative through Location, Top-k, and Center-mass penalties.Location targets a predefined input region, Top-k suppresses pixels that originally had the highest heatmap values, and Center-mass maximizes displacement from the original heatmap center.
  • 3.2.1 Passive fooling: Location fooling directs high importance toward a predefined region by using a binary mask over input locations.Mask entries equal 1 where high importance is desired and 0 elsewhere.
  • 3.2.2 Active fooling: Active fooling swaps explanations between two target classes by applying opposing penalties to images containing both class objects.Separate datasets are used for classification loss and interpretation loss, focusing fooling on the target classes while maintaining classification accuracy.

4 Experimental Results

The experiments evaluate Passive and Active fooling across three pretrained ImageNet models, measuring Fooling Success Rate and accuracy while examining visual alterations and transferability. Passive fooling generally succeeds across architectures, whereas Active fooling is strongest for VGG19 and ResNet50; manipulated models retain nearly the original classification accuracy.

  • Experimental setup: The study uses ImageNet training data with VGG19, ResNet50, and DenseNet121, constructing two-class synthetic images for Active fooling.Active fooling uses images containing African Elephant and Firetruck in varying spatial arrangements.
  • Evaluation metric: Fooling Success Rate measures the fraction of validation examples whose test loss falls within a fooling-specific threshold interval.The reported intervals are [0, 0.2], [0, 0.3], [0.1, 1], and [0.5, 2] for Location, Top-k, Center-mass, and Active fooling.
  • Passive fooling: Passive fooling alters explanations toward uninformative image regions, hides critical evidence, and achieves FSRs above 50% in matched cases except Location fooling with LRPT for DenseNet121.Location, Top-k, and Center-mass fooling produce distinct changes in relevance locations, highlighted pixels, and heatmap centers.
  • Active fooling: Active fooling clearly swaps explanations for the two classes in VGG19, nearly does so in ResNet50, and is not clearly successful in DenseNet121.For DenseNet121, nearly zero FSR may partly reflect the conservative threshold, because the visualizations still show meaningful fooling.
  • Transferability: Fooling transfers to other interpretation methods, with transferability depending on fooling type, model architecture, and interpreter.Center-mass fooling with LRPT also alters Grad-CAM, while Top-k fooling and VGG19 show comparatively larger transferability.

5 Discussion and Conclusion

The discussion finds that adversarial model manipulation can fool interpretation methods while preserving model reasoning and resisting simple detection, including for adversarially trained models. Gradient-based explanations help explain fooling transferability, while SmoothGrad remains vulnerable and robustness checks are proposed.

  • Discussion of model reasoning: AOPC curves remain nearly identical when evaluated with the original heatmap, but manipulated heatmaps score below random perturbation, indicating misleading explanations rather than changed model reasoning.The comparison uses DenseNet121 with Grad-CAM under Top-k fooling and averages results over 10K validation images.
  • Discussion of detectability: Gaussian perturbations produce similar Top-1 accuracy and non-radical FSR changes for original and Location-fooled ResNet50 models, so detection or reversal is not simple.The result concerns Location fooling with LRPT on ImageNet validation.
  • Discussion of adversarial training: Adversarially trained ResNet50 can also be manipulated with Location fooling while evaluating accuracy on both ImageNet validation and PGD-attacked validation data.The manipulation uses Grad-CAM while adversarial training is maintained.
  • Intuition and limitations: Because SimpleG, Grad-CAM, and LRP all use gradient-related information, different decision boundaries with similar accuracy can yield radically different interpretations and help explain transferability.The paper notes that asymmetry of transferability requires further analysis.
  • Intuition and limitations: Location fooling also affects SmoothGrad, which integrates SimpleG maps across Gaussian-noise inputs, showing that aggregation does not prevent the demonstrated fooling.The supplied passage identifies SmoothGrad as a robustness-check example but truncates the experimental outcome.
  • Conclusion: The paper proposes robustness to adversarial model manipulation as an additional criterion for evaluating interpretation methods and motivates methods designed to defend against it.This recommendation is framed alongside existing sanity checks.

Appendix A Back-propagation for fooling

The appendix describes a computational graph in which classification and fooling losses are differentiated through interpretation operations during model fine-tuning. Forward and backward paths are distinguished, and Active fooling uses separate datasets for the two losses.

  • Computational graph: The computational graph applies to LRP, Grad-CAM, and other saliency-map methods, with layer inputs, parameters, outputs, and heatmaps represented explicitly.The interpretation recursion is written as h^(ℓ) = R(x^(ℓ), w^(ℓ), h^(ℓ+1)), with R varying by method.
  • Loss gradients: The yellow-box terms represent cross-entropy loss and fooling loss, both contributing gradient paths used to fine-tune model parameters.Training considers all red-arrow paths from the two losses to each layer's parameters.
  • Pass directions: Black arrows denote the forward pass, while red arrows denote the reverse-direction backward pass through the computational graph.For Grad-CAM, ordinary back-propagation used to compute a heatmap is treated as an interpretation-sequence forward pass in the implementation.
  • Active fooling: Active fooling computes the classification and fooling losses using two different datasets.The appendix refers to the dataset choice described in Section 3.2.2.

Appendix B Experiment details

The experiments use a 1,300-image fooling dataset, ImageNet validation for classification evaluation, and specified model-visualization implementation details. A mask-based Location fooling scheme highlights the image frame.

  • Datasets: The fooling dataset contains 1,300 images, with 1,100 for fine-tuning and the remainder for validation.ImageNet validation contains 50,000 images and is used for classification accuracy; FSR uses ImageNet validation for Passive fooling and 200 held-out fooling images for Active fooling.
  • Hyperparameters: Table 4 identifies learning rate lr and regularization strength λ as hyperparameters of the trained models.The table associates λ with the objective in equation (2).
  • Location fooling: For Location fooling, the mask vector m ∈ R^(H×W) selects a frame-like region so the interpretation highlights the image boundary.The passage states that other masks can also be used.

Appendix C Threshold determination process in FSR.

Fooling thresholds are selected by comparing interpretation visualizations with test loss across training iterations. Location fooling is declared successful below a test loss of 0.2, with separate ranges reported for other fooling types.

  • Threshold selection: Thresholds for each fooling method are determined by comparing visualization changes and test loss over varying training iterations.The procedure is applied across the Location, Center-mass, Top-k, and Active fooling visualizations.
  • Location fooling: Location fooling uses a test-loss threshold of 0.2, with boundary-directed highlighted regions observed as training loss decreases.The reported threshold region is [0, 0.2].
  • Center-mass fooling: Center-mass fooling uses the threshold region [0.1, 1].The threshold is reported with the corresponding training visualization.
  • Top-k fooling: Top-k fooling uses the threshold region [0, 0.3].The threshold is reported with the corresponding training visualization.
  • Active fooling: Active fooling uses the threshold region [0.5, 2].The supplied figure reports this range alongside visualization and test loss during training.

Appendix D Top-5 accuracy for c1 and c2 class.

Appendix D evaluates whether Active fooling’s accuracy drop is concentrated in the fooled classes. The reported results indicate that accuracies for c1 and c2 remain close to baseline, so the drop is distributed across classes.

  • The slight Active-fooling accuracy drop is not caused by the fooled classes alone, but by the entire classes.
  • Table 5 evaluates ImageNet validation accuracy for c1 and c2 after Active fooling with c1 and c2.Each class has 50 validation images.
  • The accuracies of c1 and c2 remain quite similar to baseline after fooling.The passage qualifies this comparison by noting the validation-set size.

Appendix E Visualizations of Passive foolings and Active fooling

Appendix E provides additional qualitative visualizations of both Passive and Active fooling. Passive figures repeat the content of Figure 2, while Active figures additionally show interpretations for c2.

  • Passive fooling: Figures 11–14 present additional qualitative results for Passive fooling.Their content is the same as Figure 2.
  • Active fooling: Figures 15–17 present additional qualitative results for Active fooling.
  • Active fooling: The Active-fooling visualizations include interpretations for c2, which Figure 3 omits.
Loading 1902.02041v3…