Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients

Andrew Slavin Ross; Finale Doshi-Velez

Source-linked AI summary

Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients

Andrew Slavin Ross, Finale Doshi-Velez

arXiv:1711.09404v1Nov 26, 2017cs.LGcs.CRcs.CV

TL;DR

Deep neural networks are vulnerable to small adversarial perturbations and often provide uninterpretable reasons for predictions, limiting use where security or transparency matters. The paper evaluates input-gradient regularization, which penalizes sensitivity of predictions to small input changes, and finds improved transferred-attack robustness alongside more interpretable adversarial misclassifications and rationales.

Problem
DNNs can be fooled by small adversarial perturbations and their prediction reasons are often uninterpretable, creating obstacles in security- and transparency-sensitive domains.
Method
The paper trains differentiable models with input-gradient regularization that penalizes how infinitesimal input changes affect predictions.
Results
Across architectures, datasets, and attacks, gradient regularization improves robustness to transferred adversarial examples and makes adversarial misclassifications more interpretable to human subjects.
Takeaways & Limitations
Robustness and interpretability can improve together when models are trained to make predictions less sensitive to small input perturbations.
Takeaways & Limitations
Gradient regularization requires second derivatives, increased training time per batch by slightly more than 2× in the experiments, and may not support all autodifferentiation operations.

Abstract

from arXiv · show

Deep neural networks have proven remarkably effective at solving many classification problems, but have been criticized recently for two major weaknesses: the reasons behind their predictions are uninterpretable, and the predictions themselves can often be fooled by small adversarial perturbations. These problems pose major obstacles for the adoption of neural networks in domains that require security or transparency. In this work, we evaluate the effectiveness of defenses that differentiably penalize the degree to which small changes in inputs can alter model predictions. Across multiple attacks, architectures, defenses, and datasets, we find that neural networks trained with this input gradient regularization exhibit robustness to transferred adversarial examples generated to fool all of the other models. We also find that adversarial examples generated to fool gradient-regularized models fool all other models equally well, and actually lead to more "legitimate," interpretable misclassifications as rated by people (which we confirm in a human subject experiment). Finally, we demonstrate that regularizing input gradients makes them more naturally interpretable as rationales for model predictions. We conclude by discussing this relationship between interpretability and robustness in deep neural networks.

Introduction

Deep neural networks face linked problems of adversarial vulnerability and limited interpretability. The paper tests whether smoothing input gradients through gradient regularization can improve both properties.

DNNs can be manipulated by carefully doctored inputs that look normal to humans but cause incorrect predictions.
Adversarial examples can transfer across models, making black-box attacks practical and threatening safety-sensitive applications.
Existing defenses often struggle against transferred attacks, while adversarial training can be circumvented by transferred or randomized perturbations.
DNN predictions are also difficult to interpret, complicating bias-sensitive and medical applications where explanations can reveal wrong reasons.
Raw input gradients provide local explanations but are often noisy, motivating smoothing methods such as integrated gradients and SmoothGrad.
The authors hypothesize that training models with smooth, less extreme input gradients will improve both interpretability and resistance to adversarial examples.
Gradient regularization directly optimizes prediction gradients and yields robustness to transferred attacks across multiple architectures and datasets, while its generated adversarial examples appear more interpretable to humans.

Background

The paper introduces notation for differentiable classifiers and reviews the attacks and defenses used for comparison. These include gradient-based attacks, JSMA, defensive distillation, and adversarial training.

The models map N inputs in D dimensions to K-class probability predictions and are trained by minimizing information distance from one-hot labels.
Attacks: FGSM perturbs inputs in the direction that increases the loss’s local linear approximation; small perturbations can remain visually indistinguishable while degrading accuracy.
Attacks: Iterative gradient-sign attacks use smaller steps to induce misclassification with a smaller total perturbation by following the nonlinear loss surface.
Attacks: TGSM modifies gradient-sign attacks to decrease a loss toward specified target labels, with targets often formed by incrementing digit labels modulo 10.
Attacks: JSMA searches for one or two pixels whose changes increase the target probability and decrease all others, producing sparse perturbations that can be hard to detect.
Defenses: The study compares architecture-agnostic defensive distillation and adversarial training with other defenses excluded when they detect attacks or depend on specific architectures.
Defenses: Distillation trains a model first on one-hot labels and then on softened probability outputs using a temperature-scaled softmax.
Defenses: Adversarial training augments batches with FGSM examples and averages losses on normal and adversarial inputs without backpropagating through attack generation.

Input Gradient Regularization

Input gradient regularization extends double backpropagation by penalizing prediction sensitivity to input changes, with λ controlling penalty strength. The method aims to keep prediction-label divergence stable under small input changes.

Input Gradient Regularization: Input gradient regularization minimizes cross-entropy together with a penalty on its input gradients.The formulation adapts double backpropagation’s quadratic-loss objective to cross-entropy.
Input Gradient Regularization: The paper frames this analysis as examining the previously unassessed defensive effectiveness of double backpropagation.
Input Gradient Regularization: λ specifies the strength of the input-gradient penalty.
Input Gradient Regularization: The method seeks to keep the divergence between predictions and labels from changing significantly when inputs change slightly.
Input Gradient Regularization: The authors evaluate gradient penalties on cross-entropies involving the labels and uniform uncertainty, including a penalty called certainty sensitivity.Certainty sensitivity penalizes sensitivity of the divergence between predictions and uniform uncertainty.

Experiments

The experiments compare distillation, adversarial training, and gradient regularization across three image datasets and several gradient-based attacks. Models are evaluated by testing adversarial examples generated against each model, while a human study assesses legitimacy when accuracy is insufficient.

Experiments: The study evaluates three defenses against FGSM, TGSM, and JSMA on MNIST, SVHN, and notMNIST.
Experiments: All datasets use a convolutional network with two convolutional layers, max pooling, and a 1024-unit fully connected layer.The architecture also uses batch normalization after convolutions and in the fully connected stage.
Experiments: Gradient-regularized models use double backpropagation across a range of λ values, with the best validation-accuracy setting selected.Adversarial training uses FGSM perturbations at ϵ = 0.3, while distillation uses temperature T = 50.
Experiments: Testing every model on examples generated for each model measures both white-box and black-box robustness.
Experiments: For JSMA and iterated TGSM, a human experiment evaluates adversarial misclassification legitimacy because accuracy is inadequate for gradient-regularized models.

Accuracy Evaluations (FGSM and TGSM)

Across FGSM and TGSM evaluations, gradient-regularized models resist transferred adversarial examples, especially at larger perturbations, while examples generated against them transfer effectively. Combining gradient regularization with adversarial training further improves robustness in selected SVHN settings.

Accuracy Evaluations (FGSM and TGSM): Across MNIST, SVHN, and notMNIST, gradient regularization generally surpasses adversarial training against transferred FGSM at higher ϵ.Adversarial training sometimes performs slightly better at ϵ ≤0.3.
Accuracy Evaluations (FGSM and TGSM): Examples generated against gradient-regularized models fool other models equally well despite those models’ apparent white-box vulnerability.
Accuracy Evaluations (FGSM and TGSM): Distilled models’ apparent FGSM robustness results from vanishing gradients, and disappears when a nonzero perturbation is generated from another model.
Accuracy Evaluations (FGSM and TGSM): Gradient regularization and adversarial training can be combined because they appear to provide robustness through different mechanisms.
Accuracy Evaluations (FGSM and TGSM): Under TGSM, gradient-regularized models retain robustness at large ϵ, while their examples again fool other models equally well.
Accuracy Evaluations (FGSM and TGSM): Gradient-regularized models have controlled input-gradient means and variances, unlike the much larger non-predicted-class gradients of distilled models.The paper connects less variable, non-vanishing gradients with robustness to small input perturbations.

Human Subject Study (JSMA and Iterated TGSM)

The human-subject study compared adversarial examples from distilled, gradient-regularized, and undefended models using quantitative and qualitative judgments. Participants generally found gradient-regularized examples more convincing and reasonable as target-class instances.

Study Results: Iterated TGSM produced intuitive digit transformations for gradient-regularized models, a distinction not captured by accuracy against the original labels.This shows why attack success rates alone can misrepresent the visual plausibility of targeted adversarial examples.
Study Protocol: The pilot study used 11 subjects to assess whether MNIST JSMA and SVHN iterated TGSM examples appeared plausible as their adversarial targets.The quantitative portion presented 30 randomly selected images spanning original digits and the three model types.
Study Protocol: Figure 5 applies JSMA with maximum distortion γ = 0.25 to MNIST 0 and 1 images across target digits for distilled and gradient-regularized models.Each row begins with a highlighted digit and is modified until the target prediction is reached or the distortion limit is exhausted.
Study Results: Subjects most often judged gradient-regularized adversarial examples as their targets rather than their original digits, especially on MNIST and SVHN.They also more often rated gradient-regularized mispredictions as reasonable, whereas distilled-model mispredictions were more often unreasonable.
Study Results: Qualitative responses described gradient-regularized examples as fairly convincing, while normal and distilled examples were often characterized as being fooled by spurious noise.Few commentators reported a stark difference between normal and distilled examples.

Connections to Interpretability

The paper links adversarial robustness with interpretability by examining raw input gradients and gradient-generated perturbations. Gradient-regularized models show qualitatively more interpretable gradients and transformations than normal or distilled models, though the visual assessment is not quantitative.

Connections to Interpretability: Input gradients are commonly used as explanations, but smoothing methods are often needed because raw gradients can be noisy and difficult to interpret.The paper questions whether this need for smoothing reflects adversarially vulnerable models rather than merely an explanation-method problem.
Connections to Interpretability: The user study found that gradient-regularized adversarial examples more clearly transformed images away from their original classes and toward another class.Because these attacks are generated directly from input gradients, their interpretability provides evidence about the gradients themselves.
Connections to Interpretability: Visualizations showed qualitatively more interpretable input gradients for gradient-regularized models than for normal and distilled models, while adversarially trained models showed a smaller improvement.The paper explicitly makes no quantitative claim about this comparison.
Connections to Interpretability: Repeated iterated TGSM perturbations magnified interpretability differences and sometimes revealed what gradient-regularized models had learned.The authors suggest that interpretable gradients can make adversarial attacks function as explanations.

Discussion

The paper finds that gradient regularization improves both adversarial robustness and interpretability, while changing model decision boundaries in ways that may support qualitatively different predictions. The approach remains computationally costly and requires further testing on larger, more modern networks.

Gradient-regularized models produce smoother, more intuitive input gradients than normal, distilled, and adversarially trained models.These gradients function as more interpretable local linear approximations of model behavior.
Gradient regularization penalizes how infinitesimal input changes affect predictions, increasing robustness and interpretability together.The method directly shapes input sensitivity during training.
The method remains open to alternative penalty formulations, different hyperparameters, and evaluation on larger, more state-of-the-art networks.The paper presents these as directions for future work.
Gradient regularization increases training cost because parameter updates require second derivatives; training time per batch rose by slightly more than 2× in the experiments.The computational graph also becomes larger, and some autodifferentiation operations may not support second derivatives.
Gradient regularization changes decision-boundary geometry, suggesting that models may make predictions for qualitatively different reasons.The authors qualify this interpretation as suggestive rather than definitive.

Appendix

The appendix examines how gradient-regularization strength and defense choice affect gradients, accuracy, interpretability, and adversarial transfer. It identifies improved robustness and interpretability at appropriate regularization levels, but also shows risks from over-regularization and instability.

More regularization generally produces smaller input gradients and better FGSM validation accuracy, but delays high normal accuracy and can become excessive.Over-regularized models may have equal gradients across log probabilities and equal normal and adversarial accuracy.
Distillation can make true-label input gradients vanish while other examples’ gradients explode, whereas adversarially trained gradients remain similar in magnitude to normal-model gradients.The appendix contrasts these gradient statistics across defense methods.
Double backpropagation generally achieves slightly better FGSM accuracy than certainty insensitivity, though it can be more sensitive to λ.Certainty sensitivity sometimes destabilized training, while distilled probabilities were comparable to but no better than double backpropagation.
Gradient-regularized model FGSM examples fool normal, adversarially trained, and gradient-regularized models, whereas adversarially trained examples that fool defended models are closer to disjoint.The figure suggests that combining different defenses may be effective.
TGSM examples generated for gradient-regularized models resemble their target classes more than their original labels.These examples may provide insight into what the model has learned.