PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Interpretable Deep Neural Networks by Leveraging Adversarial Examples

Yinpeng Dong, Hang Su, Jun Zhu, Fan Bao

arXiv:1708.05493v1cs.CV

TL;DR

DNNs are often treated as black boxes, and existing interpretability approaches have limited real-world justification and rarely examine errors. This paper analyzes representations using adversarial images and proposes adversarial training with a consistent loss; it reports that neurons respond to recurrent discriminative patches and that training improves interpretability and representation consistency.

  • Problem

    DNNs are treated as black boxes, while existing interpretability methods use specific datasets and give limited attention to why models err.

    Source: §1 · p. 1

  • Method

    The paper re-examines DNN representations using adversarial images and proposes adversarial training with a consistent loss.

    Source: §2 · p. 3, §3 · p. 6, §5 · p. 8

  • Results

    Neurons respond to objects and parts as recurrent discriminative patches, while adversarial training yields more robust concepts and more consistent representations between real and adversarial images.

    Source: §2.2.2 · p. 5, §5 · p. 8

  • Takeaways & Limitations

    Interpretable representations let users trace predictions to influential neurons and examine when and why models make errors.

    Source: §5 · p. 8

    Existing interpretability methods are conducted on specific datasets and are not comprehensively justified in complex real-world settings.

    Source: §1 · p. 1

Abstract

from arXiv · show

Deep neural networks (DNNs) have demonstrated impressive performance on a wide array of tasks, but they are usually considered opaque since internal structure and learned parameters are not interpretable. In this paper, we re-examine the internal representations of DNNs using adversarial images, which are generated by an ensemble-optimization algorithm. We find that: (1) the neurons in DNNs do not truly detect semantic objects/parts, but respond to objects/parts only as recurrent discriminative patches; (2) deep visual representations are not robust distributed codes of visual concepts because the representations of adversarial images are largely not consistent with those of real images, although they have similar visual appearance, both of which are different from previous findings. To further improve the interpretability of DNNs, we propose an adversarial training scheme with a consistent loss such that the neurons are endowed with human-interpretable concepts. The induced interpretable representations enable us to trace eventual outcomes back to influential neurons. Therefore, human users can know how the models make predictions, as well as when and why they make errors.

1. Introduction

DNNs are difficult to interpret, limiting users’ ability to understand and address model errors. The paper examines representations using adversarial images, then proposes adversarial training to make those representations more interpretable and support tracing predictions to neurons.

1. Introduction

  • DNNs are treated as black boxes because their components and working mechanisms are difficult to understand, while users need rationales to validate, trust, and fix models.These needs are especially salient in mission-critical applications such as healthcare and autonomous driving.Source: §1 · p. 1
  • Prior interpretability methods provide textual or region-based explanations, integrate semantic attributes, or visualize neurons as object/part detectors.The cited approaches include natural-language justifications [10] [22], highlighted image regions [34], semantic attributes [4], and neuron visualization [27] [32] [33] [29] [16].Source: §1 · p. 1
  • These methods often use specific datasets and focus on correct predictions, leaving their real-world applicability and error explanations insufficiently examined.The paper notes examples including ImageNet [24] and Place.Source: §1 · p. 1

1.1. Our Contributions

  • The authors analyze DNN behavior on adversarial images alongside real images to investigate both accurate predictions and errors.They construct 500K adversarial images by generating 10 target-label images for each image in the ILSVRC 2012 validation set, using an ensemble-optimization attack.Source: §1.1 · p. 2
  • Neurons respond to objects and parts as recurrent discriminative patches rather than truly detecting them, and adversarial-image representations are inconsistent with real-image representations, contrary to earlier findings [33] [2] [7] and [1].The authors test AlexNet, VGG [28], and ResNet [9], manually inspecting neuron responses and quantitatively evaluating concept alignment.Source: §1.1 · p. 2
  • Adversarial training with a consistent loss improves representation interpretability, helping neurons activate consistently with preferred objects or parts and enabling users to trace predictions and errors to influential neurons.The authors describe tracing predictions to influential neurons and identifying neuron groups contributing to flawed predictions.Source: §1.1 · pp. 2–3, Figure 1 · p. 2

2. Re-examine the Internal Representations

The section tests whether DNN neurons detect semantic objects and whether adversarial-image representations align with real-image representations. It finds that neuron activations track discriminative patches and that adversarial representations are inconsistent with real-image representations.

2. Re-examine the Internal Representations

  • The ImageNet experiments test whether adversarial representations align with semantic objects and parts or fail to align with them, potentially contributing to inaccurate predictions.The authors frame these as competing hypotheses and use ensemble-generated adversarial images to investigate the models.Source: §2 · p. 3

2.1. Ensemble-Optimization Attack

  • The ensemble-optimization attack adds small, imperceptible noise to a real image so multiple classifiers predict a chosen target class different from its ground-truth class.The experiments attack an ensemble of AlexNet, VGG-16, and ResNet-18, generating ten target-class images per ILSVRC 2012 validation image.Source: §2.1 · p. 3

2.2. Dummy Object/Part Detectors

  • Neurons that activate for semantic concepts in real images often fire for adversarial images of their preferred target class, but not for adversarial images containing the named objects.The authors conclude that neurons respond to recurrent discriminative patches rather than truly detecting semantic objects or parts.Source: §2.2 · pp. 3–4, Figure 2 · p. 4

2.2.1 Quantification of Features

  • The proposed feature metric uses WordNet class relationships; higher scores indicate neurons consistently detecting higher-level concepts or concentrating on more specific categories.WordNet tree distance distinguishes semantically closer classes, such as tabby cat and kitty cat, from more distant classes, such as Persian cat and wolfhound.Source: §2.2.1 · pp. 4–5

2.2.2 Quantitative Results

  • As neuron feature level increases, CS1 decreases while CS2 increases, indicating alignment with target classes rather than image contents.The authors interpret this pattern as evidence that high-level neurons respond to discriminative patches, not objects or parts in the input.Source: Figure 4 · p. 5, §2.2.2 · p. 5

2.3. Inconsistent Deep Visual Representations

  • A large number of adversarial images have r1 > 1, indicating representations far from real images of their original class; a great majority have r2 much smaller than 1, closer to target-class representations than the average distance between the classes.Across AlexNet, VGG-16, and ResNet-18, average r1 values for all adversarial images are 1.07, 1.26, and 1.34, respectively; average r2 values are 0.80, 0.73, and 0.76.Source: §2.3 · pp. 5–6, Figure 5 · p. 5

2.4. When Do DNNs Make Errors?

  • The representation-based detector distinguishes adversarial images with ROC AUCs of 0.817 for AlexNet, 0.847 for VGG-16, and 0.926 for ResNet-18.The procedure can indicate when models make mistakes, but is insufficient to explain why.Source: Figure 6 · p. 6, §2.4 · p. 6

3. Adversarial Training

Adversarial training with a consistent feature-matching loss aims to make neurons respond consistently to preferred objects or parts despite adversarial noise. After training, real and adversarial images produce more similar visual concepts and more consistent overall representations, with a reported 1%–4% accuracy drop.

3. Adversarial Training

  • The method adds a consistent loss to adversarial training because output supervision can smooth input-output mappings without ensuring smooth input-representation mappings.The L2 feature-matching term compares representations of real and adversarial image pairs in the last convolutional layer, encouraging activation when preferred objects or parts appear and deactivation when they disappear.Source: §3 · p. 6
  • Real and adversarial images with the highest activations showed similar visual concepts, and interpretability improved with a reported 1%–4% accuracy drop.The authors interpret the similar concepts as indicating that neurons are more likely to respond to true objects or parts.Source: Figure 7 · p. 7, §3 · p. 7
  • As the LC score increases, CS1 remains high and CS2 remains low, while the reported high correlation between p and q indicates responses to true image contents rather than corresponding classes.The authors describe these findings as evidence of more interpretable and robust representations.Source: Figure 8 · p. 7, §3 · p. 7
  • Average r1 values were 0.98, 0.99, and 0.98, while average r2 values were 0.90, 0.92, and 0.87 for AlexNet, VGG-16, and ResNet-18, respectively; overall representations moved closer to original-class representations and farther from target-class representations.The authors conclude that adversarial training with a consistent loss makes representations more consistent between real and adversarial images.Source: Figure 9 · p. 7, §3 · p. 7

4. Towards Interpretable DNNs

The paper traces predictions to influential neurons using prediction differences, then interprets their learned features. Real-image explanations are coherent, while inconsistent neuron concepts in adversarial-image explanations can help expose errors.

4. Towards Interpretable DNNs

  • Prediction difference measures each neuron's relevance to the output by comparing the prediction with and without that feature; larger differences indicate more influential neurons.The method retrieves neurons whose prediction differences exceed a threshold.Source: §4 · p. 7
  • For real images, the retrieved neurons’ visual concepts can explain the prediction, as illustrated with VGG-16 pool5 neurons and their learned features.The paper shows the top two neurons with the highest prediction-difference values for each image.Source: §4 · p. 7, Figure 10 · p. 8
  • For adversarial images, retrieved neuron concepts can conflict: true-object detectors and target-class detectors may both have large prediction differences.In the example, an image of a bulbul is classified as “moving van,” but retrieved features reveal “bird,” which is unusual in real moving-van images; this inconsistency can help users detect an adversarial image and understand the error.Source: §4 · p. 8

5. Conclusions

The paper argues that DNN neurons respond to objects and parts as recurrent discriminative patches, and that adversarial-image representations are inconsistent with real-image representations. It proposes adversarial training with a consistent loss and reports more robust concepts and more consistent representations.

5. Conclusions

  • Neurons respond to objects and parts as recurrent discriminative patches rather than truly detecting semantic objects or parts.The paper presents this conclusion as contrary to previous findings.Source: §5 · p. 8
  • Adversarial-image representations are not robust distributed codes of visual concepts and are inconsistent with representations of real images.The paper states that this limits their effectiveness when transferred to other tasks.Source: §5 · p. 8
  • The proposed adversarial training algorithm uses a consistent loss; the reported results show more robust neuron concepts and more consistent representations between real and adversarial images.Source: §5 · p. 8
  • Interpretable representations let users trace outcomes to influential neurons and understand how predictions are made and when and why errors occur.Source: §5 · p. 8

Appendix

The appendix supplements the experiments with model performance and robustness results, an assessment of the learned-feature consistency metric, and further visualizations.

Appendix

  • Section A reports AlexNet, VGG-16, and ResNet-18 performance before and after adversarial training on real and generated adversarial ImageNet validation sets, including robustness results.Source: Appendix · p. 10
  • Section B evaluates the proposed LC metric for quantifying neuron-feature level and consistency through human judgment.Source: Appendix · p. 10
  • Section C provides additional AlexNet and ResNet-18 visualizations to supplement Figures 2 and 7 in the main paper.Source: Appendix · p. 10

A. Model Details

The appendix reports ImageNet accuracy after adversarial training and tests robustness against FGS attacks. It also gives model and training details for the evaluated networks.

A. Model Details

  • The evaluated networks are AlexNet, VGG-16, and ResNet-18; ResNet-18 is trained from scratch with SGD, momentum 0.9, weight decay 5 × 10−5, batch size 100, and 300K iterations.Its learning rate starts at 0.05 and is divided by 10 when the error plateaus; AlexNet and VGG-16 use pre-trained models.Source: A. Model Details · p. 10
  • 62.01% for AlexNet, 60.48% for VGG-16, and 51.13% for ResNet-18 are the reported adversarial-set accuracies after adversarial training.The paper reports these accuracies as largely increased; Table 1 covers ImageNet accuracy.Source: Table 1 · p. 10, A. Model Details · p. 10
  • Top-5 accuracy on the real validation set drops 3.89% for AlexNet, 3.72% for VGG-16, and 1.38% for ResNet-18 after adversarial training.The paper characterizes adversarial training as a trade-off between performance, interpretability, and robustness.Source: A. Model Details · p. 10
  • The reported ImageNet results use single-center-crop testing, and the adversarially trained variants are named AlexNet-Adv, VGG-16-Adv, and ResNet-18-Adv.The real and generated adversarial validation sets are denoted Valid and Valid-Adv.Source: A. Model Details · p. 10
  • The adversarially trained models improve accuracy against Fast Gradient Sign attacks at ϵ = 1 and ϵ = 5 on ImageNet validation images.Table 2 reports accuracy against FGS attacks; the supplied prose gives no attack-specific values.Source: A. Model Details · p. 10, Table 2 · p. 10

B. Evaluation of the LC Metric

A VGG-16 human-judgment study compares LC scores with volunteers’ assessments of concept level and consistency, finding strong agreement.

B. Evaluation of the LC Metric

C. Visualization Results

Supplementary visualizations show dummy object/part detectors in AlexNet and ResNet-18, while adversarial training leads neurons to resist perturbations and respond to true objects or parts.

C. Visualization Results

Loading 1708.05493v1…