PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpretation of Neural Networks is Fragile

Amirata Ghorbani, Abubakar Abid, James Zou

arXiv:1710.10547v2stat.MLcs.LG

TL;DR

Reliable explanations matter for trusting neural-network predictions, but the paper asks whether interpretations themselves remain trustworthy. It defines and systematically investigates interpretation fragility, finding that imperceptibly perturbed inputs can preserve labels while substantially changing interpretations, with Hessian analysis offering insight into why.

  • Problem

    Reliable interpretations are important for trust and transparency, yet the paper examines how much interpretation of a prediction can be trusted.

  • Method

    The paper defines interpretation fragility and develops efficient, visually imperceptible perturbation attacks, evaluation metrics, and Hessian-based analysis across feature- and sample-importance methods.

  • Results

    Interpretations can differ substantially for similar inputs with the same predicted label; random and targeted perturbations affect feature-importance methods, while influence functions are also fragile.

  • Takeaways & Limitations

    Interpretation robustness is an important challenge because users in biomedical and financial settings may be as interested in explanations as in predictions.

  • Takeaways & Limitations

    The study focuses on standard image benchmarks and popular interpretation tools, leaving fragility in biomedical, economic, and other settings as an open research agenda.

Abstract

from arXiv · show

In order for machine learning to be deployed and trusted in many applications, it is crucial to be able to reliably explain why the machine learning algorithm makes certain predictions. For example, if an algorithm classifies a given pathology image to be a malignant tumor, then the doctor may need to know which parts of the image led the algorithm to this classification. How to interpret black-box predictors is thus an important and active area of research. A fundamental question is: how much can we trust the interpretation itself? In this paper, we show that interpretation of deep learning predictions is extremely fragile in the following sense: two perceptively indistinguishable inputs with the same predicted label can be assigned very different interpretations. We systematically characterize the fragility of several widely-used feature-importance interpretation methods (saliency maps, relevance propagation, and DeepLIFT) on ImageNet and CIFAR-10. Our experiments show that even small random perturbation can change the feature importance and new systematic perturbations can lead to dramatically different interpretations without changing the label. We extend these results to show that interpretations based on exemplars (e.g. influence functions) are similarly fragile. Our analysis of the geometry of the Hessian matrix gives insight on why fragility could be a fundamental challenge to the current interpretation approaches.

1 Introduction

Reliable interpretations are needed for trust and transparency, but the paper shows that visually indistinguishable inputs with unchanged predictions can receive substantially different interpretations. Complex decision boundaries help explain why small perturbations can alter feature importance.

  • Motivation: Interpretations help users understand which input features led to consequential predictions in applications such as medicine and finance.Such explanations are presented as important for establishing trust and transparency between users and algorithms.
  • Fragility: Robust explanations are necessary because visually indistinguishable, same-label inputs may receive very different interpretations.A clinician could otherwise be directed toward different image regions despite essentially unchanged predictions.
  • Fragility: The paper defines interpretation fragility as changing an interpretation substantially through a perceptively indistinguishable perturbation while preserving the prediction label.The study examines feature-importance methods and influence functions under targeted perturbations.
  • Illustration: Figure 1 shows that center attacks with ϵ = 8 leave predicted labels unchanged while shifting saliency maps toward features humans would not consider salient.The example covers simple gradients, DeepLIFT, and integrated gradients.
  • Security implications: Label-preserving interpretation attacks create a security concern because adversaries may redirect attention away from relevant features or toward chosen features.This risk is especially consequential when interpretations guide medical interventions.
  • Mechanism: Complex deep-network decision boundaries can move inputs into regions with different loss contours, changing gradients and feature-importance scores.The same intuition applies to exemplar-based analyses when the most influential training point changes after perturbation.

2 Related Works and Preliminaries

The paper situates interpretation attacks alongside prediction attacks and introduces feature- and sample-importance explanations. It describes the methods and metrics used to compare interpretations before and after perturbation.

  • Related work: Earlier adversarial-attack work changed neural-network predictions using visually indistinguishable inputs, whereas this paper targets interpretations while preserving predictions.The discussion references ℓ2, ℓ0, and ℓ∞ measures for quantifying perturbation size.
  • Interpretation methods: Post-hoc interpretability explains predictions using test-example features or the contributions of training examples without exposing black-box mechanisms.These approaches aim to provide human users insight into model behavior.
  • Feature importance: Feature-importance methods assign normalized absolute scores to input dimensions that affect the predicted class score.The paper studies simple gradients, integrated gradients, and DeepLIFT.
  • Feature importance: Integrated gradients sums gradients across scaled inputs relative to a reference point, while DeepLIFT propagates relevance based on changes from that reference.Integrated gradients addresses the saturation drawback of simple gradients; DeepLIFT uses the Rescale rule here.
  • Sample importance: Influence functions explain a test prediction through training examples whose up-weighting or down-weighting most affects the test loss.The influence calculation uses gradients and the empirical Hessian over training examples.
  • Evaluation: Interpretation similarity is measured with Spearman rank correlation and top-k intersection of the most important features or training examples.These metrics capture both ordering similarity and overlap among the highest-ranked items.

3 Methods: Generating Perturbations

The authors generate small, label-preserving perturbations that maximize dissimilarity between original and perturbed interpretations. They use iterative attacks for feature importance and a single-step gradient-sign attack for influence functions.

  • Problem formulation: The perturbation objective is to change the interpretation of a fixed test input while preserving its predicted label under an L∞ norm constraint.The interpretation may be a vector of feature scores or scores for training examples.
  • Baselines: Random sign perturbation independently changes each pixel by ±ϵ as a baseline for feature- and sample-importance attacks.Its purpose is to compare unguided changes with adversarial perturbations.
  • Feature-importance attacks: Iterative feature-importance attacks maximize a differentiable dissimilarity function between original and perturbed interpretations.The top-k attack suppresses initially important features, mass-center attack displaces saliency spatially, and targeted attack changes semantic focus.
  • Influence attacks: Influence-function attacks linearize the influence equation and apply an optimal single-step perturbation under an L∞ constraint.The chosen attack decreases the influence of the three most influential training images, while affecting other influences as well.
  • Influence attacks: For influence functions, the calculation uses final-layer parameters and back-propagates the resulting input perturbation through the network.This makes the influence calculation feasible but produces the perturbation at the final layer before back-propagation.
  • Feature-importance attacks: The algorithm repeatedly follows the signed gradient of interpretation dissimilarity, clips inputs to ||x_p−x_t||∞≤ϵ, and returns the label-preserving perturbation with greatest dissimilarity.The center-of-mass objective uses the spatial distribution of feature-importance mass.

4 Experiments & Results

Experiments on ImageNet, CIFAR-10, and a flower-classification influence-function setup show that small or adversarial perturbations substantially change interpretations without changing predictions. Targeted attacks can produce especially perceptible changes, while integrated gradients is harder to attack.

  • Experimental setup: The feature-importance experiments use ImageNet with pretrained SqueezeNet and CIFAR-10 with a trained convolutional network.The evaluated methods are simple gradients, integrated gradients, and DeepLIFT, with a smaller Deep Taylor experiment on ImageNet.
  • Experimental setup: The influence-function experiment trains the final layer of an InceptionNet v3 on 1,000 rose-versus-sunflower images while freezing earlier layers.Robustness is evaluated on a 200-image test set.
  • Feature importance results: Mass-center and top-k attacks have similar effects by rank correlation and top-1000 intersection, while mass-center attacks produce greater perceptual change.Integrated gradients is the most difficult of the three feature-importance methods to attack, with similar results reported for CIFAR-10.
  • Influence-function results: A representative influence-function attack changes the most influential training examples entirely while leaving the test image’s prediction unchanged.This demonstrates label-preserving fragility for sample-importance interpretations.
  • Influence-function results: At ϵ = 8, only 2 of the top 5 influential training images remain in the top 5 after gradient-sign attacks, and the influences become essentially uncorrelated.Random attacks also reduce influence rank correlation to 0.8 at approximately ϵ = 10.

5 Hessian Analysis

The analysis explains interpretation fragility through high dimensionality and nonlinear score functions: small input perturbations can substantially alter feature importance. The same analysis motivates attack directions that change interpretations while preserving predictions.

  • For a linear score S = w⊤x, feature importance is robust because the gradient is independent of the input.Nonlinearity is therefore required for adversarial attacks against interpretation in this analysis.
  • The first-order change in a gradient-based interpretation is approximated by the Hessian applied to the input perturbation, Hδ.The Hessian is defined through the second derivatives of the score function.
  • The simplest vulnerable nonlinear model is a weight vector followed by a nonlinearity, S = g(w⊤x), whose interpretation sensitivity depends on derivatives of g.The analysis uses g′ and g′′ to characterize the relative feature-importance change.
  • High dimensionality can amplify small perturbations, making relative changes in feature importance substantial.For perturbations δ = ϵsign(w), the relative change grows with the input dimension because it is proportional to the ℓ1-norm of w.
  • Across 512 ImageNet images, top-k and center attacks perform similarly and outperform random sign perturbations on top-1000 intersection and rank correlation.These measures quantify how much feature-importance rankings change under attack.

6 Discussion

The paper shows that neural-network interpretations can change substantially for similar inputs with unchanged predictions, creating robustness and security concerns. It argues that interpretation methods should be used and evaluated cautiously while motivating defenses and broader study.

  • 6 Discussion: Similar inputs with the same prediction can receive substantially different interpretations, making interpretation fragility partly independent of prediction fragility.The paper links both forms of fragility partly to high dimensionality.
  • 6 Discussion: Interpretation robustness is especially important in biomedical and financial settings where users may care about explanations as much as predictions.
  • 6 Discussion: Small random perturbations can change importance scores, while targeted perturbations can produce more dramatic manipulations across several interpretation methods.The paper specifically highlights simple gradients, DeepLIFT, influence functions, and integrated gradients.
  • 6 Discussion: Different saliency maps after perturbation do not necessarily mean the interpretation method is broken; each map may correctly measure local sensitivity at its own input.The paper attributes the difference to fragility in the network itself.
  • 6 Discussion: Input discretization and constraints on network nonlinearity are proposed as potentially useful directions for defending against interpretation attacks.
  • 6 Discussion: The authors studied standard image benchmarks, while noting that interpretation fragility may extend to biomedical, economic, and other neural-network applications.They identify robust methods and application-specific understanding as research priorities.

Appendices

The appendix reports a controlled training setup and shows that replacing ReLU with Softplus preserved the reported test accuracy.

  • Appendices: ADAM with default parameters was used to train the network, which achieved 73% test accuracy with ReLU activation.
  • Appendices: Replacing ReLU with Softplus and initializing from the ReLU weights also yielded 73% accuracy.

B Additional examples of feature importance perturbations

Additional ImageNet examples attack all three feature-importance methods with random sign noise and two adversarial algorithms while preserving the displayed predicted classes.

  • B Additional examples of feature importance perturbations: All three feature-importance methods are attacked using random sign noise and two adversarial algorithms in each additional ImageNet example.
  • B Additional examples of feature importance perturbations: The images in Figure 7 are all classified as an airedale.
  • B Additional examples of feature importance perturbations: The images in Figure 8 are all classified as a damselfly.
  • B Additional examples of feature importance perturbations: Figure 9 shows images classified as a lighter.

C Objective metrics and subjective change in feature importance maps

The appendix compares objective metrics with subjective changes in feature-importance maps and identifies center-shift as the most correlated measure among those evaluated.

  • C Objective metrics and subjective change in feature importance maps: Snapshots of mass-center attack iterations illustrate how rank-order correlation and top-1000 intersection relate to subjective changes in feature-importance maps.
  • C Objective metrics and subjective change in feature importance maps: Targeted attacks can change the most salient object while leaving the predicted label unchanged.
  • C Objective metrics and subjective change in feature importance maps: Center-shift was the most correlated of the three measurements with subjective changes in feature-importance maps.The center attack with the largest average center-shift also produced the most significant subjective change.

F Results for adversarial attacks against CIFAR-10 feature importance methods

On CIFAR-10, mass-center and top-k attacks with k=100 perform similarly on rank correlation and top-100 intersection, both outperforming random perturbations; integrated gradients is the most robust method.

  • Mass-center and top-k attacks with k=100 achieve similar rank-correlation and top-100-intersection results, and both are stronger than random perturbations.The mass-center attack moves the center of mass more than the other two perturbations.
  • Integrated gradients is more robust than the two other feature-importance methods.

G Additional examples of adversarial attacks on influence functions

Influence-function interpretations can change substantially after small input perturbations, with high-dimensionality and network nonlinearity contributing to fragility. The proposed Lipschitz-based analysis suggests operator-norm regularization as a possible defense, but its bounds are not generally tight.

  • Additional examples of adversarial attacks on influence functions: Perturbing a test image can make previously influential training images among the least influential, while pre- and post-attack influences become uncorrelated or nearly reversed.
  • Dimensionality-based explanation for fragility of influence functions: The influence vector’s test-image-dependent term is J = ∇θL, representing each parameter’s infinitesimal effect on the test-image loss.
  • Dimensionality-based explanation for fragility of influence functions: A first-order input perturbation changes J by ∇θ∇xL·δ, with relative parameter-wise change given by (∇θ∇xL · δ)i/(∇θL)i.
  • Dimensionality-based explanation for fragility of influence functions: For δ = ϵsign(w), relative feature-importance change grows with input dimension because it is proportional to the L1-norm of w.The derivatives do not scale with input dimension under the stated normalization assumptions.
  • Designing interpretability-robust networks: The analysis proposes bounding interpretability change through the Lipschitz constant of the input gradient, whose layer-wise terms depend on weight operator norms and derivatives of nonlinearities.
  • Designing interpretability-robust networks: Operator-norm regularization may improve robustness to feature-importance attacks, but the appendix calculations are suggestive rather than conclusive because practical Lipschitz bounds are rarely tight.
Loading 1710.10547v2…