PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpretable Deep Learning under Fire

Xinyang Zhang, Ningfei Wang, Hua Shen, Shouling Ji, Xiapu Luo, Ting Wang

arXiv:1812.00891v3cs.CRcs.LG

TL;DR

Interpretable deep learning systems are intended to make DNN decisions understandable, but the security of their explanations has been insufficiently studied. This paper systematically studies that vulnerability through ADV^2 attacks and countermeasures. It finds that attacks can manipulate both predictions and interpretations, with a prediction-interpretation gap providing one possible root cause.

  • Problem

    The security vulnerability of interpretable deep learning systems is insufficiently understood despite their intended role in providing trustworthy explanations for DNN decisions.

  • Method

    The paper develops ADV^2 attacks against coupled classifiers and interpreters, evaluates them across interpreter types, analyzes the prediction-interpretation gap, and explores ensemble and adversarial-training defenses.

  • Results

    ADV^2 can generate inputs with adversarially chosen predictions and interpretations, while the prediction-interpretation gap is identified as one possible cause of this vulnerability.

  • Takeaways & Limitations

    Interpretability in existing IDLSes may provide limited security assurance, motivating more careful assessment and defenses such as interpreter ensembles and AID.

  • Takeaways & Limitations

    The proposed ensemble defense faces disparate interpreter outputs and potentially adaptive attacks, while black-box attacks remain future work.

Abstract

from arXiv · show

Providing explanations for deep neural network (DNN) models is crucial for their use in security-sensitive domains. A plethora of interpretation models have been proposed to help users understand the inner workings of DNNs: how does a DNN arrive at a specific decision for a given input? The improved interpretability is believed to offer a sense of security by involving human in the decision-making process. Yet, due to its data-driven nature, the interpretability itself is potentially susceptible to malicious manipulations, about which little is known thus far. Here we bridge this gap by conducting the first systematic study on the security of interpretable deep learning systems (IDLSes). We show that existing \imlses are highly vulnerable to adversarial manipulations. Specifically, we present ADV^2, a new class of attacks that generate adversarial inputs not only misleading target DNNs but also deceiving their coupled interpretation models. Through empirical evaluation against four major types of IDLSes on benchmark datasets and in security-critical applications (e.g., skin cancer diagnosis), we demonstrate that with ADV^2 the adversary is able to arbitrarily designate an input's prediction and interpretation. Further, with both analytical and empirical evidence, we identify the prediction-interpretation gap as one root cause of this vulnerability -- a DNN and its interpretation model are often misaligned, resulting in the possibility of exploiting both models simultaneously. Finally, we explore potential countermeasures against ADV^2, including leveraging its low transferability and incorporating it in an adversarial training framework. Our findings shed light on designing and operating IDLSes in a more secure and informative fashion, leading to several promising research directions.

1 Introduction

Interpretable deep learning systems couple DNN classifiers with interpreters to make decisions more understandable, but their explanations may themselves be vulnerable to manipulation. This study introduces ADV^2, identifies a prediction-interpretation gap underlying the vulnerability, and explores countermeasures.

  • Motivation: DNNs are difficult to understand because of high non-linearity and nested architectures, while adversarial inputs can cause unpredictable behavior in security-sensitive domains.These limitations motivate explanations that help users inspect model decisions and detect adversarial inputs.
  • IDLSes: An interpretable deep learning system couples a DNN classifier with an interpretation model that explains feature importance for a given prediction.The paper mainly considers post-hoc, instance-level interpretations represented as attribution maps.
  • Research gap: Interpretability is believed to provide security benefits, yet its data-driven nature leaves the security vulnerability of interpretable deep learning systems insufficiently studied.The paper frames this as a gap in understanding and mitigating malicious manipulation of interpretations.
  • ADV^2: ADV^2 generates inputs that both mislead target DNNs and deceive their coupled interpreters, enabling adversaries to choose predictions and interpretations across four major interpreter types.The study evaluates ADV^2 on benchmark datasets and security-critical applications, including skin cancer diagnosis.
  • Causes and countermeasures: The prediction-interpretation gap arises when an interpreter is misaligned with the classifier and only partially explains its behavior, allowing simultaneous exploitation of both models.The paper also investigates low transferability across interpreters and proposes adversarial interpretation distillation to reduce the gap and potentially improve robustness.
  • Scope: The paper presents a systematic study of IDLS security and argues that its findings can inform more secure and informative design and operation.The work covers attack construction, evaluation, root-cause analysis, and possible countermeasures.

3 ADV2 Attack

ADV2 is an attack framework that jointly steers a DNN’s prediction and its interpreter’s attribution map while keeping input changes small. Its construction is adapted to four interpreter families because their internal mechanisms create different optimization challenges.

  • 3.1 Attack Formulation: ADV2 seeks adversarial inputs that reach a target class, produce a target attribution map, and remain imperceptibly close to the benign input.The attack formulates prediction and interpretation constraints jointly, with the distance measure depending on the perturbation framework.
  • 3.1 Attack Formulation: ADV2 combines prediction and interpretation losses, balancing them with a hyper-parameter and solving the resulting optimization problem through adversarial-attack frameworks such as PGD.The formulation is redesigned into an optimization-friendly loss because the classifier and interpreter constraints are highly nonlinear.
  • 3.2 Back-Propagation-Guided Interpretation: For gradient-based interpreters, direct optimization can fail because ReLU networks yield zero Hessians, leaving the interpretation-loss gradient uninformative.The attack smooths ReLU’s gradient with h(z), whose gradient is non-zero everywhere, and this substantially outperforms sigmoid and the original ReLU derivative in the reported GRAD attack.
  • 3.3 Representation-Guided Interpretation: For representation-guided interpreters such as CAM, ADV2 differentiates through the classifier’s convolutional features and linear layer used to construct attribution maps.CAM projects classifier weights back onto last-layer feature maps, and the attack extends to related methods such as GRADCAM.
  • 3.4 Model-Guided Interpretation: For model-guided RTS, the attack adds an encoder-output loss because the encoder materially influences the attribution map beyond the masking model’s output.RTS normally predicts maps with a separately trained DNN, so relying only on the masking model provides insufficient attack guidance.

4 Attack Evaluation

The evaluation tests ADV2 across classifiers, interpreters, datasets, and attack frameworks, asking whether it can deceive predictions and interpretations effectively. Against the reported settings, ADV2 remains highly effective at fooling target DNNs despite its dual objectives.

  • Evaluation Setup: The evaluation examines ADV2’s effectiveness against classifiers and interpreters, its evasiveness, real-world applicability, and flexibility across attack frameworks.Experiments use qualitative and quantitative tests spanning five stated research questions.
  • Experimental Setting: ImageNet evaluation uses 1,000 correctly classified validation images sampled from a 1.2-million-image, 1,000-class dataset, with images center-cropped to 224×224 pixels.Pixels are normalized to [0,1].
  • Experimental Setting: ADV2 is implemented with PGD and spatial transformation attacks and compared with regular PGD under targeted misclassification settings.Both attacks use a maximum of 1,000 iterations for the reported comparison.
  • Classifier Attacks: Above 95% attack success rate and above 0.98 misclassification confidence are achieved by ADV2 across all reported classifier–interpreter cases.These results are described as comparable with regular PGD despite ADV2’s additional interpretation objective.

Q2. Attack Effectiveness (Interpretation)

ADV2 produces adversarial inputs whose interpretations closely resemble benign cases across four major interpreters, unlike conventional PGD examples. Quantitative measures confirm this similarity across interpreters and underlying DNNs.

  • The average L1 measure for ADV2 is more than 60% lower than PGD across all interpreters.L1 distances are normalized to [0,1] by dividing by the number of pixels.
  • ADV2 effectiveness varies by interpreter, with the PGD–ADV2 difference relatively marginal on GRAD.This suggests interpreters may have differing inherent robustness against ADV2.
  • ADV2 achieves similar L1 measures on both ResNet and DenseNet, indicating limited sensitivity to the underlying DNN.
  • ADV2 achieves IoU scores above 0.5 across all interpreters relative to benign attribution maps.The paper treats an attribution map as plausible when its IoU with the benign map exceeds 0.5.
  • ADV2 generates attribution maps highly similar to benign cases across GRAD, CAM, MASK, and RTS.Qualitative comparisons show ADV2 maps are perceptually indistinguishable from benign maps, whereas PGD maps are easily identifiable.

Q3. Attack Evasiveness

Basic ADV2 is about as detectable as PGD under feature squeezing, while adaptive ADV2 can be modified to evade that detection method. The adaptive attack also preserves the study’s interpretation-focused evaluation.

  • Feature squeezing does not significantly distinguish ADV2 from PGD in overall detectability.Local smoothing detects both attack types above 97%, with a difference below 2%.
  • Adaptive ADV2 augments the adversarial objective with a squeezing loss to align predictions on original and squeezed inputs.For smoothing squeezers, the added term is the cross entropy between predictions of original and squeezed inputs.
  • The adaptive attack is implemented through a two-stage search spanning squeezed and original input spaces.It first attacks the squeezed input, then searches the benign input’s neighborhood for an input with similar classification.
  • Adaptive ADV2 can generate adversarial inputs that evade feature squeezing detection.The paper reports a significant drop in detection rate relative to basic ADV2, while noting that an in-depth study remains ongoing.

Q4. Real Application

In a skin cancer screening application, ADV2 produces adversarial interpretations visually indistinguishable from benign ones. Quantitative results show IoU scores above 0.62 across all four interpreters.

  • ADV2 generates interpretations visually indiscernible from benign counterparts in the skin cancer screening task.The case study uses a seven-disease taxonomy and a competition-winning ResNet-backboned classifier.
  • IoU scores of ADV2 attribution maps exceed 0.62 across all interpreters in the skin cancer application.The similarity is quantitatively evaluated against benign attribution maps using L1 measures and IoU scores.

Q5. Alternative Attack Framework

ADV2 remains effective when built on STADV and when targeting arbitrary interpretations, including random patches or interpretations associated with another class. These results support partial classifier–interpreter alignment and motivate ensemble-based defenses.

  • Alternative Attack Framework: STADV-based ADV2 produces attribution maps much more similar to benign cases than STADV.L1 and IoU results confirm the qualitative comparison, although MASK appears more resilient under this attack variant.
  • Alternative Attack Framework: Interpreter robustness varies with the concrete attack framework, as shown by MASK’s relative resilience to STADV-based ADV2.The comparison with PGD-based ADV2 indicates that robustness rankings may change across attacks.
  • Arbitrary Interpretations: Targeting random patch interpretations has little impact on ADV2’s effectiveness at deceiving classifiers.Random patches are generated with random shape, angle, and position, and the target maps typically differ substantially from benign maps.
  • Arbitrary Interpretations: ADV2 can produce inputs with highly similar interpretations even when target and adversarial inputs are fairly distinct.This behavior is supported by L1 and IoU comparisons between ADV2 maps and target maps.
  • Discussion: The experiments show that ADV2 can target arbitrary predictions and interpretations.The paper attributes this possibility to incomplete alignment between the DNN and its interpreter, allowing simultaneous exploitation of both models.

Q2. Root of Prediction-Interpretation Gap

The prediction-interpretation gap arises because interpreters capture distinct, incomplete aspects of DNN behavior, creating exploitable misalignment between prediction and interpretation. Evidence from transferability, robustness, and analytical bounds indicates that this misalignment can leave attack constraints loose enough for ADV2 to remain effective.

  • Attack transferability: Low transferability across interpreters indicates that attacks crafted for one interpreter generally produce low-quality interpretations on another.Cross-interpreter attribution quality is comparable to interpretation-agnostic PGD attacks.
  • Attack robustness: GRAD attains the lowest IoU scores among interpreters, suggesting greater robustness because ADV2 relies heavily on gradient information.The paper validates this explanation by analyzing integrated gradients and related back-propagation interpreters.
  • Analytical evidence: For integrated gradients, the prediction loss is upper bounded by the interpretation loss: ℓprd(x∗) ≤ ℓint(x∗).The bound follows from expressing prediction loss as an inner product and interpretation loss as the L1 norm of the elementwise product.
  • Analytical evidence: ADV2 maximizes prediction loss while minimizing interpretation loss, creating a fundamental conflict between its two objectives.The conflict does not preclude effective attacks when the thresholds for successful prediction and interpretation objectives are loose or when attacks rely less on gradients.
  • Conclusion: The combined evidence of low transferability and disparate robustness supports the conclusion that interpreters’ distinct behavioral perspectives contribute to the gap.These distinct perspectives can impose loose constraints because the adversary may need to preserve behavior from only one perspective.
  • Root cause: Existing interpreters focus on distinct DNN aspects, producing a prediction-interpretation gap that can leave model behavior only partially explained.GRAD uses gradients, MASK input-prediction correspondence, CAM intermediate representations, and RTS learned representations and training correspondences.

Q3. Potential Countermeasures

The paper considers complementary interpreter ensembles and adversarial training as countermeasures against ADV2. AID trains interpreters using ADV2-derived losses and makes RTSA more sensitive to DNN behavior changes while preserving benign-case behavior.

  • Defense 1: Ensemble Interpretation: Deploying multiple complementary interpreters could provide a more holistic view of DNN behavior against ADV2.The proposal combines perspectives such as CAM’s deep representations and MASK’s input-prediction correspondence.
  • Defense 1: Ensemble Interpretation: Ensemble defenses must address disparate interpretations and adversaries that adapt ADV2 to optimize against all interpreters.The paper identifies both ensemble aggregation and adaptive attack strategies as open research challenges.
  • Defense 2: Adversarial Interpretation: AID incorporates ADV2 into interpreter training to reduce the prediction-interpretation gap.Its additional loss encourages similar attribution maps for benign inputs and their adversarial counterparts.
  • Defense 2: Adversarial Interpretation: RTSA becomes more sensitive than RTS to random-noise-induced DNN behavior changes at noise levels causing 3% and 30% test-set misclassification.The sensitivity appears in contrastive attribution maps and is quantitatively confirmed using L1 distances.
  • Defense 2: Adversarial Interpretation: ADV2 fails to keep RTSA’s adversarial interpretations similar to benign ones, while RTSA behaves almost identically to RTS on benign inputs.This indicates that AID improves adversarial sensitivity with little impact on benign cases in the reported RTS study.
  • Conclusion: ADV2 can also be used during interpreter training to reduce the prediction-interpretation gap.This training-based use is presented as a potential countermeasure direction.

6 Related Work

The related work spans adversarial attacks and defenses, transferability, and interpretability. This paper positions itself among early studies of attacks that target both DNN predictions and coupled interpretations, including transferability across interpreters.

  • Attacks and Defenses: Prior adversarial-learning research studies poisoning and evasion attacks, while defenses seek greater DNN resilience through training and inference strategies.The paper notes that stronger attacks can circumvent existing defenses.
  • Transferability: Adversarial-input transferability enables black-box attacks by allowing inputs crafted on surrogate DNNs to target other models.Ensemble adversarial training is cited as a defense against this transferability.
  • Interpretability: Interpretation models use back-propagation, intermediate representations, input perturbation, and meta-model techniques to explain black-box DNNs.These approaches form the main interpretability categories reviewed by the paper.
  • Interpretability: Interpretability has been used for debugging DNNs, digesting security-analysis results, and detecting adversarial inputs.The expected difference between benign and adversarial interpretations motivates examining interpretation reliability under attack.
  • Interpretability: Some prior findings report that interpretation models can be insensitive to DNNs or data-generation processes, while DNN-neutral transformations can substantially alter interpretations.These observations provide additional motivation for studying interpretation reliability.
  • This work: This paper contributes a systematic security study of IDLSes, showing that DNNs and coupled interpreters can be deceived simultaneously.It introduces ADV2, identifies the prediction-interpretation gap as one possible cause, and discusses countermeasures.

Appendix

The appendix details Grad-CAM and MASK formulations, ADV2 optimization procedures, analytical assumptions, and AID training. It also describes implementation choices for spatial attacks and adversarial interpreter training.

  • Grad-CAM: Grad-CAM weights last-convolutional-layer feature maps using global averaged gradients to form attribution maps.The map uses model predictions, feature-map activations, and a normalization constant.
  • Grad-CAM: ADV2 against Grad-CAM can be optimized with stochastic gradient descent because feature-map activations retain nonzero input gradients.The gradient of the channel weights may be zero almost everywhere, but the activations provide usable gradients.
  • MASK: MASK optimizes an objective combining total variation, sparsity, and perturbation terms controlled by regularization coefficients and noise parameters.Total variation reduces artifacts, sparsity encourages compact masks, and the perturbation operator blends inputs with Gaussian noise.
  • MASK analysis: The MASK analysis assumes a unique local minimizer and a nondegenerate Hessian, enabling an implicit-function argument for the interpreter’s behavior.The proof uses the implicit function theorem and differentiates the resulting local solution function.
  • MASK analysis: For MASK optimization, the method estimates the optimal attribution map and takes an update toward a target map using a Hessian-based step.The implementation uses an estimated optimum and a chosen positive step size.
  • Spatial ADV2: Spatial ADV2 represents each adversarial pixel through a per-pixel flow vector and computes values using bilinear interpolation.The attack objective includes a spatial-transformation magnitude term weighted by a hyper-parameter and is solved with Adam.
  • AID: AID trains an RTS interpreter by alternating minimization of the regular interpretation loss and an adversarial distillation loss.The adversarial loss measures the attribution-map difference between benign and adversarial inputs under the current interpreter.

B. Parameter Setting

The paper specifies default parameter settings for PGD-based ADV2 and STADV-based ADV2 attacks, including optimizer and attack hyperparameters.

  • B. Parameter Setting: PGD-based ADV2 uses the regular PGD learning rate α = 1./255 and perturbation threshold ε = 0.031.
  • B. Parameter Setting: Table 11 reports the parameter setting for PGD-based ADV2, with distinct Q4 settings shown when applicable.

B2. StAdv-based ADV2

The supplementary experiments visualize and quantify how PGD-, ADV2-, and STADV-based adversarial inputs affect attribution maps across benchmark and skin-cancer settings.

  • § 4 Q2 Attack Effectiveness (Interpretation): Figures 17 and 19 visualize benign and adversarial attribution maps produced by GRAD, CAM, MASK, and RTS on DenseNet.
  • § 4 Q2 Attack Effectiveness (Interpretation): For the benchmark attribution-map comparisons, L2 measures are normalized by dividing by the square root of the number of pixels.
  • § 4 Q4 Real Application: The skin-cancer case study uses the ISIC 2018 dataset and a competition-winning ResNet-backbone model that placed second in the challenge.
  • § 4 Q4 Real Application: Table 14 reports average Lp distances, for p = 1,2, between benign and PGD- or ADV2-generated attribution maps in skin-cancer diagnosis.
  • § 4 Q5 Alternative Attack Framework: Figure 20 compares L1 measures and IoU scores of STADV-based adversarial inputs against benign attribution maps on DenseNet.
  • § 5 Q1 Random Class Interpretation: Figures 21 and 22 compare target and ADV2 inputs and their attribution maps using L1 measures and IoU scores relative to benign and target cases.
Loading 1812.00891v3…