PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpreting Adversarially Trained Convolutional Neural Networks

Tianyuan Zhang, Zhanxing Zhu

arXiv:1905.09797v1cs.LGcs.CVstat.ML

TL;DR

The paper asks what adversarially trained CNNs learn, given that standard CNNs often rely on local textures rather than object shape. It systematically compares adversarially and normally trained models using visual salience maps and transformed test datasets, finding that adversarial training produces more shape-focused, texture-invariant representations.

  • Problem

    Standard CNNs often rely on local textures rather than object shape, weakening generalization to images with distorted textures but preserved shape information.

  • Method

    The study systematically compares adversarially and normally trained CNNs using salience maps and classification tests on clean, stylized, saturated, and patch-shuffled images.

  • Results

    AT-CNNs capture more long-range shape information and are less texture-biased than normally trained CNNs across object-recognition datasets.

  • Takeaways & Limitations

    The findings help explain why AT-CNNs are more robust and suggest that capturing global features or increasing texture invariance may inform robustness-oriented model design.

  • Takeaways & Limitations

    AT-CNNs are not robust to all image distortions, and further investigation is needed because robustness depends on distortion type.

Abstract

from arXiv · show

We attempt to interpret how adversarially trained convolutional neural networks (AT-CNNs) recognize objects. We design systematic approaches to interpret AT-CNNs in both qualitative and quantitative ways and compare them with normally trained models. Surprisingly, we find that adversarial training alleviates the texture bias of standard CNNs when trained on object recognition tasks, and helps CNNs learn a more shape-biased representation. We validate our hypothesis from two aspects. First, we compare the salience maps of AT-CNNs and standard CNNs on clean images and images under different transformations. The comparison could visually show that the prediction of the two types of CNNs is sensitive to dramatically different types of features. Second, to achieve quantitative verification, we construct additional test datasets that destroy either textures or shapes, such as style-transferred version of clean data, saturated images and patch-shuffled ones, and then evaluate the classification accuracy of AT-CNNs and normal CNNs on these datasets. Our findings shed some light on why AT-CNNs are more robust than those normally trained ones and contribute to a better understanding of adversarial training over CNNs from an interpretation perspective.

1. Introduction

The paper asks what adversarially trained CNNs learn and whether they differ from standard CNNs, which often rely on local texture rather than object shape. It systematically compares the models visually and quantitatively, finding that AT-CNNs capture more shape-related, long-range features and are less texture-biased.

  • Motivation: Standard CNNs often rely on local textures rather than long-range shape information, limiting generalization when textures are distorted.This motivates investigating whether other trained CNNs develop stronger shape bias.
  • Adversarial training: Adversarial training minimizes loss on nearly worst-case perturbed examples and produces the adversarially trained networks studied here.The paper names these models AT-CNNs and relates them to improved robustness against adversarial examples.
  • Approach: The study systematically interprets AT-CNNs and compares them with normally trained CNNs using qualitative and quantitative experiments.The experiments are designed to examine what features each model uses for object recognition.
  • Main finding: AT-CNNs capture long-range correlations such as shapes and are less biased toward textures than normally trained CNNs.The paper reports that this finding partially explains their greater robustness.
  • Validation: The paper compares salience maps and evaluates accuracy on datasets that destroy textures or shapes, including stylized, saturated, and patch-shuffled images.These experiments provide visual and quantitative comparisons of the models’ feature biases.
  • Contribution: The work presents a systematic visual and quantitative investigation of interpreting adversarially trained CNNs.The authors characterize this as the first investigation of its kind to their knowledge.

2. Preliminary

This section introduces adversarial training and salience-map methods used to study CNN behavior. Adversarial training uses perturbed examples under norm constraints, while salience methods measure how class outputs respond to input pixels.

  • Adversarial training: Adversarial training formulates learning as robust optimization over adversarial perturbations constrained by a chosen norm.The network, data pair, perturbation, loss, and constraint set define the optimization components.
  • Adversarial training: The inner maximization is approximated with adversarial examples, including PGD-based training under bounded l∞ and l2 constraints.The paper also investigates FGSM-based adversarial training.
  • Salience maps: Salience maps assign sensitivity or attribution values to input pixels to indicate their influence on a network output.The paper discusses perturbation-based and gradient-based salience methods.
  • Salience maps: For an input image x, the Grad explanation is the gradient of a class activation with respect to x, producing a salience map E.The network maps image inputs to class outputs, and each class has an activation function.
  • SmoothGrad: SmoothGrad reduces visual noise by averaging gradients over Gaussian-noised copies of the input.The experiments use n = 100 noisy copies, normalized noise level 0.1, and class activation Sc(x) = log pc(x).

3. Methods

The method compares AT-CNNs and standard CNNs through salience-map visualization and accuracy under transformations that preferentially disrupt textures or shapes. The experiments use clean training data and transformed test data across three image datasets.

  • Experimental design: The study uses Tiny ImageNet, Caltech-256, and CIFAR-10 to compare adversarially and normally trained CNNs.Both salience maps and transformed-data performance are evaluated.
  • Salience visualization: Salience maps are compared on clean images and on texture-preserving or shape-preserving distortions to reveal feature sensitivity.SmoothGrad is mainly reported because it reduces the visual noise of Grad maps.
  • Quantitative evaluation: The quantitative evaluation measures performance degradation under distortions that preserve either shapes or textures.Perfect disentanglement of texture, shape, and other information is acknowledged as impossible.
  • Evaluation protocol: Training uses only the original datasets, while generalization is evaluated on transformed data that are withheld during training.This isolates performance on the constructed transformations as a test of generalization.
  • Stylizing: Style transfer destroys most texture information while preserving global shape structures, making stylized accuracy a test of shape sensitivity.Models capturing shapes better should perform better on stylized images under similar generalization error.
  • Saturation: Increasing saturation pushes pixels toward binarization and can destroy texture information while preserving much of the contour structure.The saturation level p = 2 leaves the image unchanged, while p = ∞ yields pure binarization.
  • Patch-shuffling: Patch-shuffling divides images into k × k patches and rearranges them, preserving much texture while destroying long-range shape information.The method uses k ∈ {2, 4, 8}; larger k loses more original information, especially at low resolution.
  • Model comparison: Table 1 reports clean-data accuracy and PGD robustness for the trained models, including underfitting CNNs with similar clean generalization to some AT-CNNs.The underfitting comparison addresses whether performance differences arise from adversarial-training generalization.

4. Experiments and analysis

Experiments compare adversarially trained and standard CNNs across visualizations and transformed test sets. AT-CNNs emphasize object shapes and contours, remain more invariant to texture loss, but are not robust to every distortion.

  • Experimental setup: Experiments span CIFAR-10, Tiny ImageNet, and Caltech-256, using standard and adversarially trained CNNs with FGSM or PGD attacks at varied strengths.An underfitting clean-data CNN with similar generalization performance is also included to separate adversarial training effects from poor generalization.
  • Visualization results: SmoothGrad maps show AT-CNNs focusing sparsely on object contours across clean, saturated, and stylized images, unlike noisier standard-CNN maps.The comparison includes standard, underfitting, and PGD-l∞ adversarially trained models.
  • Stylizing: On stylized images with dramatically changed textures, AT-CNNs retain attention to original object shapes, whereas standard CNNs fail.Style transfer is used to alter local textures while preserving global shape structure.
  • Saturation: As saturation removes texture information while preserving shape, adversarially trained models maintain higher accuracy and show less sensitivity to texture loss.The authors report the same pattern on CIFAR-10 and interpret it as greater reliance on shapes than textures.
  • Saturation: Within each adversarial-training approach, models more robust to stronger PGD attacks are generally more invariant to saturation changes, while clean-data generalization typically worsens.This supports the reported trade-off between robustness and clean accuracy.
  • Saturation: When saturation is decreased, all models degrade similarly, showing that AT-CNN robustness is specific to certain distortion types rather than universal.Further investigation of this limitation is left for future work.
  • Patch-shuffling: Patch shuffling sharply reduces AT-CNN confidence but leaves normal-CNN confidence high, indicating that AT-CNNs depend more on shapes and edges.The effect is stronger for models trained against stronger attacks in most experiments.

5. Related work and discussion

The paper interprets adversarially trained CNNs through salience maps and robustness comparisons, finding that their feature representations differ from standard classifiers. It also links robustness to capturing long-range features, while noting that conclusions may depend on the adversary type.

  • Insights for defensing adversarial examples: AT-CNN robustness toward adversarial examples is correlated with capturing long-range features such as shapes or contours.The paper raises whether models with more global features or texture invariance could also improve robustness without adversarial training.
  • Adversarial training with other types of attacks: The difference between normally trained and adversarially trained CNNs may depend strongly on the adversary type.The discussion contrasts norm-constrained perturbations with spatially transformed and GAN-based adversaries.
  • Adversarial training with other types of attacks: ST-AT-CNNs have robustness toward PGD attacks similar to standard models, while their salience maps remain different from PGD-AT-CNNs.Their average salience-map distance is close to that of standard CNNs and much higher than that of PGD-AT-CNNs.
  • Interpreting AT-CNNs: Salience maps are used to compare the features recognized by standard CNNs, PGD-l∞AT-CNNs, and ST-AT-CNNs.Figure 8 presents SmoothGrad sensitivity maps for the original image and the three models.

6. Conclusion

The paper systematically studies adversarially trained CNNs using qualitative and quantitative comparisons. Across constructed distorted datasets, AT-CNNs rely more on global shape information and are less sensitive to texture distortion than normally trained CNNs.

  • 6. Conclusion: The study compares AT-CNNs and normal CNNs qualitatively and quantitatively using distorted test sets that preserve shapes or local textures.It compares sensitivity maps on clean, stylized, and saturated images and evaluates generalization on stylized, saturated, and patch-shuffled datasets.
  • 6. Conclusion: AT-CNNs are less sensitive to texture distortion and focus more on shape information, whereas normally trained CNNs show the opposite preference.The conclusion reports this pattern across the constructed stylized, saturated, and patch-shuffled datasets.
  • 6. Conclusion: The proposed interpretation strategies can also be extended to models for object detection and semantic segmentation.The paper presents understanding learned representations as an essential topic in machine learning and computer vision.

A. Experiment Setup

The experiments use ResNet-18 models across CIFAR-10, Caltech-256, and Tiny ImageNet, with ImageNet initialization for the latter two datasets. Robustness is evaluated against an l∞ projected-gradient adversary with specified perturbation and iteration settings.

  • Experiment Setup: CIFAR-10 experiments use a standard ResNet-18 with four residual-layer groups of filter sizes 64, 128, 256, and 512.The architecture contains two residual units.
  • Experiment Setup: Caltech-256 and Tiny ImageNet experiments use ResNet-18 initialized with ImageNet-pretrained weights provided by PyTorch.The models use code from PyTorch.
  • Experiment Setup: Robustness is evaluated using an l∞ projected-gradient-descent adversary with ϵ = 8/255, step size = 2, and 40 iterations.These settings define the adversary used to evaluate all models.

A.2. Adversarial Training

The study trains models against multiple PGD and FGSM adversaries across datasets and norm constraints. It also constructs stylized test sets using AdaIN style transfer to evaluate sensitivity to texture changes.

  • Adversarial Training: Each dataset receives nine adversarial-training configurations: seven against PGD adversaries and two against FGSM adversaries.The configurations vary the adversarial-training attack type.
  • l∞-norm bounded adversary: For l∞-bounded training, models use ϵ ∈ {1/255, 2/255, 4/255, 8/255}, with corresponding model names PGD-inf:1, 2, 4, and 8.The step sizes are 1/255, 1/255, 2/255, and 4/255 respectively.
  • l2-norm bounded adversary: For l2-bounded training, Caltech-256 and Tiny ImageNet use ϵ ∈ {4, 8, 12}, while CIFAR-10 uses ϵ ∈ {4/10, 8/10, 12/10}.The models are denoted PGD-l2:4, 8, and 12 in both settings.
  • Adversarial Training: FGSM-trained models use ϵ ∈ {4, 8} and are denoted FGSM 4 and 8.These are the two FGSM adversarial-training configurations.
  • Stylized test set: Stylized Caltech-256 and Tiny ImageNet test sets apply AdaIN style transfer with α = 1.0 using randomly selected paintings as styles.The construction follows Geirhos et al. and uses paintings from Kaggle’s Painter by Numbers dataset.

C. Experiments on Fourier-filtered datasets

The experiments test CNN tendencies toward surface statistics or high-level structure using Fourier-filtered versions of the test set. Three filtering schemes isolate low-, high-, or randomly selected frequency information.

  • Experimental design: The study evaluates trained CNNs on high-pass and low-pass filtered datasets to probe tendencies toward surface regularities or high-level abstractions.This follows the approach of Jo and Bengio (2017).
  • Filtering schemes: Three Fourier-filtered test-set variants are constructed: low-frequency, high-frequency, and random filtering.The random mask is generated during testing, with each Fourier mode independently set to zero with probability p.
  • Filtering schemes: Low-pass filtering removes higher-frequency modes, whereas high-pass filtering preserves only higher-frequency modes.Both operations use radial masks in the Fourier domain.

C.2. Results

On Fourier-filtered Caltech-256 data, AT-CNNs perform better with low-frequency information but worse with high-frequency information than normal CNNs. The authors interpret this pattern as greater reliance on shapes and contours than textures.

  • Results: AT-CNNs perform better on the Low-pass filtered dataset and worse on the High-pass filtered dataset.Performance is measured as accuracy on correctly classified images across three filtered Caltech-256 test sets.
  • Results: The results indicate that AT-CNN predictions depend more on low-frequency information.The comparison uses generalization performance on Fourier-filtered test sets.
  • Interpretation: The authors associate textures with high-frequency information and shapes and contours with low-frequency information.This interpretation is presented as consistent with the reported filtering results.

D. Detailed results

The detailed-results section points to tabulated evaluations across saturation and patch-shuffling conditions, alongside additional salience-map comparisons. A table is identified for Fourier-filtered Caltech-256 accuracy.

  • Quantitative results: Tables 5, 4, and 6 report model results on test sets with different saturation levels.The passage introduces these as detailed quantitative experiment results.
  • Quantitative results: Table 3 reports accuracy on correctly classified images for different models across three Fourier-filtered Caltech-256 test sets.The table covers low-, high-, and randomly filtered test data as described in the surrounding experiment.
  • Quantitative results: Tables 8 and 7 list model results after different patch-shuffling operations.The passage refers to test-set evaluations following patch-shuffling transformations.
  • Qualitative results: Additional sensitive maps are shown in Figure 9, with Grad and SmoothGrad comparisons in Figure 10.These figures extend the salience-map analysis.
Loading 1905.09797v1…