Source-linked AI summary
Intriguing properties of neural networks
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, Rob Fergus
TL;DR
Deep neural networks achieve strong recognition performance but can learn difficult-to-interpret, counter-intuitive input representations and mappings. This paper examines their high-level units and sensitivity to perturbations, finding that semantic information resides in activation spaces rather than individual units and that imperceptible adversarial perturbations can transfer across networks.
Problem
The paper asks whether high-level units form meaningful semantic bases and whether well-performing neural networks are stable to small input perturbations.
Method
The paper compares random projections with individual activation coordinates and generates misclassifying perturbations by optimizing inputs to maximize prediction error.
Results
Individual units are semantically indistinguishable from random projections, while imperceptible adversarial examples transfer across networks with different architectures, hyperparameters, and training sets.
Takeaways & Limitations
The findings indicate that high-layer semantics reside in activation spaces rather than individual units and that learned networks contain intrinsic blind spots.
Takeaways & Limitations
The paper does not deeply explain how often adversarial examples occur or why they generalize across hyperparameters and training sets.
Abstract
from arXiv · showhide
Deep neural networks are highly expressive models that have recently achieved state of the art performance on speech and visual recognition tasks. While their expressiveness is the reason they succeed, it also causes them to learn uninterpretable solutions that could have counter-intuitive properties. In this paper we report two such properties. First, we find that there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis. It suggests that it is the space, rather than the individual units, that contains of the semantic information in the high layers of neural networks. Second, we find that deep neural networks learn input-output mappings that are fairly discontinuous to a significant extend. We can cause the network to misclassify an image by applying a certain imperceptible perturbation, which is found by maximizing the network's prediction error. In addition, the specific nature of these perturbations is not a random artifact of learning: the same perturbation can cause a different network, that was trained on a different subset of the dataset, to misclassify the same input.
1 Introduction
The paper identifies two counter-intuitive properties of deep neural networks: semantic information resides in representation spaces rather than individual high-level units, and small imperceptible perturbations can cause misclassification. These adversarial perturbations transfer across networks, revealing intrinsic blind spots connected non-obviously to the data distribution.
- Motivation: Deep neural networks achieve strong recognition performance through expressive, automatically discovered computations that can be difficult to interpret and counter-intuitive.Their expressiveness comes from arbitrary computations built from a modest number of massively parallel nonlinear steps.
- Semantic meaning of units: Random projections of high-level representations are not distinguished from individual units by unit-analysis methods, suggesting semantic information lies in the representation space.The vector representations are stable up to rotations, making individual coordinates unlikely to carry unique semantic meaning.
- Input stability: Imperceptible, non-random input perturbations can arbitrarily change a well-generalizing network’s prediction, despite leaving the image’s object category unchanged.This demonstrates that deep networks can be substantially unstable to small input changes.
- Adversarial examples: Adversarial examples are relatively robust and shared across networks with different architectures or training-data subsets, rather than being random artifacts of learning.The shared examples persist across varied numbers of layers and activations and can be generated by one network to fool another.
- Implications: Together, these findings indicate that backpropagation-learned networks possess nonintuitive characteristics and intrinsic blind spots linked to the data distribution in a non-obvious way.The two reported properties motivate studying how learned representations and input-output mappings can diverge from intuitive expectations.
2 Framework
The framework examines the geometry of layer activations and searches for network blind spots through experiments across several architectures and datasets. MNIST experiments additionally use weight decay and, in some cases, two disjoint 30,000-example training subsets.
- Notation: The framework represents an input image as x ∈ R^m and analyzes layer activations φ(x) before searching for blind spots.The analysis first examines properties of the image of φ(x), then searches for blind spots.
- Experimental setup: Experiments use several networks across MNIST, ImageNet, and approximately 10M YouTube image samples.The studied models include fully connected and autoencoder-based MNIST classifiers, AlexNet, and an unsupervised network with approximately 1 billion learnable parameters.
- Experimental setup: For MNIST experiments, regularization uses weight decay with parameter λ.The passage specifies regularization with a weight decay of λ.
- Experimental setup: 30,000 training cases per subset are used when the MNIST training data is split into two disjoint datasets, P1 and P2.The split is used only in some experiments.
3 Units of: φ(x)
The experiments show that random directions in the hidden representation produce semantic properties comparable to those found in individual-unit directions. This suggests that semantic information lies in the representation space rather than being disentangled across its natural coordinates.
- Random versus natural basis: Random directions v ∈ R^n produce images with similarly interpretable semantic properties, showing that semantic structure is not unique to individual units.The result holds for many maximizing images x′ whose members are semantically related.
- Random versus natural basis: The natural basis is not better than a random basis for inspecting φ(x), challenging the idea that neural networks disentangle variation factors across coordinates.This conclusion follows from the comparable semantic interpretability of random directions and individual-unit directions.
- Random versus natural basis: On MNIST, images maximizing individual-unit and random-direction activations share many high-level similarities.The comparison used a convolutional network and the MNIST test set as the held-out image set I.
- Random versus natural basis: In AlexNet, rows were semantically meaningful for both individual units and combinations of units.The experiment used the validation set as I and compared natural-basis with random-basis activations.
- Limitation: This analysis reveals invariance-generation capacity on a particular input subset but does not explain behavior across the rest of the domain.The paper next examines counterintuitive properties of φ near almost every point from the data distribution.
4 Blind Spots in Neural Networks
Deep neural networks admit imperceptibly perturbed inputs that cause confident misclassification, revealing discontinuous input-output behavior and blind spots. These adversarial examples can transfer across independently trained models, while layer-wise operator norms provide conservative measures of network instability.
- Adversarial examples: Imperceptibly small perturbations can turn correctly classified images into misclassified adversarial examples, contradicting the smoothness assumption underlying many kernel methods.A simple optimization procedure finds these perturbations in the input space.
- Adversarial examples: For MNIST, QuocNet, and AlexNet, every studied network yielded visually indistinguishable adversarial examples for each sample.These examples were misclassified by the original network.
- Cross-model generalization: A relatively large fraction of adversarial examples transfer to networks trained from scratch with different depths, regularization, or initial weights.This supports the view that adversarial examples are not merely artifacts of one model or training-set selection.
- Network instability: Small additive input perturbations can produce large perturbations at the last-layer output, independently of generalization across networks and training sets.The paper describes a procedure to measure and control this instability.
- Network instability: Instabilities can appear as early as the first convolutional layer, according to upper Lipschitz bounds computed for an ImageNet deep convolutional network.The bounds are conservative: large values do not ensure adversarial examples, whereas small values rule them out.
5 Discussion
The discussion highlights counter-intuitive properties of deep neural networks concerning both the semantic meaning of individual units and their discontinuities. It also frames adversarial negatives as apparently inconsistent with high generalization performance because they are indistinguishable from regular examples.
- Discussion: Deep neural networks exhibit counter-intuitive properties in both individual-unit semantics and network discontinuities.These two properties are presented as the central findings discussed in the paper.
- Discussion: Adversarial negatives appear to contradict the network’s ability to achieve high generalization performance.The discussion poses this tension as an unresolved question about how well-generalizing networks can be confused by such examples.
- Discussion: The apparent contradiction arises because adversarial negatives are indistinguishable from regular examples yet can confuse the network.The passage explicitly asks how a network can generalize well while being confused by these examples.