PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Manipulation of Deep Representations

Sara Sabour, Yanshuai Cao, Fartash Faghri, David J. Fleet

arXiv:1511.05122v9cs.CVcs.LGcs.NE

TL;DR

Existing adversarial-image work largely targets erroneous class labels, leaving internal representation manipulation as a distinct question. This paper optimizes small source-image perturbations to match a guide’s DNN representation and finds guide-like, natural-looking encodings, while identifying scope-dependent failures.

  • Problem

    Prior adversarial-image research focused mainly on disrupting classification, whereas this paper examines whether internal DNN representations can be manipulated toward another image.

  • Method

    The paper optimizes a new image that stays close to a source while minimizing the Euclidean distance between its layer-k representation and the guide image’s representation.

  • Results

    Feature-adversarial representations are much closer to guides than sources, often nearest neighbors of guides, and appear natural across deep-representation analyses; 54% transfer across networks.

  • Takeaways & Limitations

    The findings raise questions about DNN representations and the properties of natural images, including whether network structure contributes to these phenomena.

  • Takeaways & Limitations

    The method does not always work for low-resolution images or hand-drawn characters, and can require perceptible guide traces or fail after narrow-domain fine-tuning.

Abstract

from arXiv · show

We show that the representation of an image in a deep neural network (DNN) can be manipulated to mimic those of other natural images, with only minor, imperceptible perturbations to the original image. Previous methods for generating adversarial images focused on image perturbations designed to produce erroneous class labels, while we concentrate on the internal layers of DNN representations. In this way our new class of adversarial images differs qualitatively from others. While the adversary is perceptually similar to one image, its internal representation appears remarkably similar to a different image, one from a different class, bearing little if any apparent similarity to the input; they appear generic and consistent with the space of natural images. This phenomenon raises questions about DNN representations, as well as the properties of natural images themselves.

1 INTRODUCTION

The paper introduces feature adversaries, which manipulate a source image’s internal DNN representation toward a guide image while preserving perceptual similarity. This extends adversarial-image research beyond erroneous classification labels to questions about learned representations.

  • Adversarial images may help probe DNN structure and inform learning algorithms aimed at better generalization and robustness.
  • Existing adversarial-image methods primarily disrupt classification by producing labels inconsistent with human perception.
  • Small perturbations can make a source image’s representation remarkably similar to a guide’s and far from the source’s representation.
  • The resulting adversarial representations appear generic and indistinguishable from natural-image representations across multiple DNN layers.

2 RELATED WORK

Earlier adversarial-image work mainly targeted erroneous labels or visibly unnatural inputs. The paper distinguishes feature adversaries by showing that their representations remain similar to natural images while conflicting with human perception.

  • Some prior methods generated high-confidence object labels for 2D patterns that differ substantially from natural training images.
  • Szegedy et al. optimized image perturbations against classification loss while penalizing perturbation magnitude to preserve natural appearance.
  • Goodfellow et al. attributed adversarial examples partly to model linearity, while Fawzi et al. proposed a broader flexibility-based framework.
  • Feature adversaries differ qualitatively because their DNN representations resemble natural images even though they remain inconsistent with human perception.

3 ADVERSARIAL IMAGE GENERATION

The method optimizes a minimally changed source image to match a guide’s representation at a chosen Caffenet layer. Across layers and models, the resulting images can preserve appearance while often adopting the guide’s DNN label and encoding.

  • 3 ADVERSARIAL IMAGE GENERATION: The optimization minimizes Euclidean distance between the adversarial and guide representations while constraining the adversarial image to remain close to the source.
  • 3 ADVERSARIAL IMAGE GENERATION: The L∞ constraint limits each pixel color’s maximum deviation to δ, controlling perceptibility rather than directly optimizing a human visual-discriminability measure.
  • 3 ADVERSARIAL IMAGE GENERATION: δ = 10 out of 255 generally produced compelling images with negligible perceptual distortion across intermediate layers, networks, and most images.
  • 3 ADVERSARIAL IMAGE GENERATION: Figure 1 varies Caffenet layers FC7, P5, and C3 and perturbation bounds δ = 5, 10, and 15; larger δ makes perturbations more noticeable.
  • 3 ADVERSARIAL IMAGE GENERATION: The guide image was not perceptible in generated adversarial images, and difference images lacked significant salient structure.
  • 3 ADVERSARIAL IMAGE GENERATION: 95% of 100 Imagenet source-guide pairs optimized at Caffenet FC7 with δ = 10 received the guide class, while none received the source class.
  • 3 ADVERSARIAL IMAGE GENERATION: The guide-label pattern extended across AlexNet, GoogleNet, VGG CNN-S, Places205, and hybrid datasets using 100 random source-guide pairs per model.
  • 3 ADVERSARIAL IMAGE GENERATION: 54% of adversarial images generated on one network were misclassified by other networks, but usually did not receive the guide label consistently across networks.

4 EXPERIMENTAL EVALUATION

Experiments show that small source-preserving perturbations can move DNN representations toward guide images, making adversarial encodings close to and often indistinguishable from natural guide neighborhoods. Comparisons indicate this behavior differs from label optimization, is not explained by local linearity, and may reflect network architecture.

  • 4.1 SIMILARITY TO THE GUIDE REPRESENTATION: Adversarial representations are much closer to guide encodings than source encodings across a wide range of natural images.They are often nearest neighbors of their respective guides despite remaining perceptually indistinguishable from the sources.
  • 4.1 SIMILARITY TO THE GUIDE REPRESENTATION: 0.1% of FC7 source-guide pairs fail to reduce the adversarial-to-guide distance ratio below 0.8 when δ = 5.Relaxing the L∞ bound makes adversarial encodings still closer to guides, with proximity increasing from lower to higher layers.
  • 4.1 SIMILARITY TO THE GUIDE REPRESENTATION: At δ = 10, only 8% of adversarial images are closer to their source than the average within-source-class FC7 distance.The adversarial-guide distance is also much smaller than typical same-class nearest-neighbor distances, and α is often the guide’s 1-NN.
  • 4.1 SIMILARITY TO THE GUIDE REPRESENTATION: In most cases, adversarial and guide encodings share exactly the same 3NN, while α has a more similar rank than 90% of same-class data in at least 50% of cases.These statistics hold for training-set source-guide pairs and are also observed for test and validation data.
  • 4.2 SIMILARITY TO NATURAL REPRESENTATIONS: Adversarial encodings behave as natural-representation inliers under both local tangent-space likelihood and angular-consistency analyses.The angular-consistency distributions resemble the likelihood comparisons, supporting inlier status near the guide.
  • 4.3 COMPARISONS AND ANALYSIS: Feature-opt adversaries differ from label-opt adversaries, local feature-linear approximations, and trained-network-specific explanations.Feature-linear encodings reach no closer than 80% of the source-guide distance, versus 50% or less for feature-opt at layers down to C2; random-weight networks show similar distance ratios, with deviations up to 2% at FC7 and Norm2 and up to 10% at C2.

5 DISCUSSION

The paper introduces feature adversaries whose perceptually source-like images have deep representations that mimic natural guide images, unlike other adversarial categories. It also reports that network structure may contribute, while the underlying human–DNN representational discrepancy and observed failure cases remain unresolved.

  • 5 DISCUSSION: Feature adversaries appear perceptually similar to a source while their intermediate-layer representations mimic natural guide images.The paper reports that these imposters capture guide proximity and occupy high-density feature-space regions.
  • 5 DISCUSSION: These representation properties are not shared by other categories of adversarial images.
  • 5 DISCUSSION: The linearity hypothesis does not provide an obvious explanation for feature adversaries, and random-network results suggest network structure may matter.
  • 5 DISCUSSION: The underlying reasons for the discrepancy between human and DNN representations remain undetermined and require further experiments and analysis.
  • 5 DISCUSSION: Observed failures involve handwritten digits and narrowly fine-tuned domains, motivating future study of depth, receptive fields, and natural-image classes.The paper also leaves detection of feature adversaries for future work because it would require a diverse, relatively large adversarial-image dataset.

SUPPLEMENTARY MATERIAL

The supplementary illustration shows that a minimally changed source image can acquire an internal representation nearly identical to that of a very different guide image. Experiments use diverse source and guide sets across Caffenet layers.

  • S1 ILLUSTRATION OF THE IDEA: A car adversary remains visually close to the source car while its activation pattern becomes almost identical to the guide image of a dog.The mapping places a nearby image-space point near the representation of a different natural image.
  • S1 ILLUSTRATION OF THE IDEA: The supplementary figure summarizes the paper’s central idea of manipulating deep representations without substantially changing image appearance.
  • SUPPLEMENTARY MATERIAL: Experiments use two source-and-guide image sets, with the first targeting FC7 and the second supporting computationally cheaper experiments on other layers.
  • SUPPLEMENTARY MATERIAL: Each source is guided by all guide images, and correctly classified training-set guides simplify reporting classification behavior.
  • SUPPLEMENTARY MATERIAL: The first source set contains 20 images, while its guide set spans all 1000 ILSVRC classes plus additional validation and test images; the second guide set covers 100 classes.

S3 EXAMPLES OF ADVERSARIES

Examples show adversarial images generated at different Caffenet layers and perturbation settings, while the guide image remains imperceptible in the resulting adversary. The model’s layer structure is documented separately.

  • S3 EXAMPLES OF ADVERSARIES: The guide is not perceptible in any sampled adversary across source, guide, or layer choices; δ controls noise visibility.
  • S3 EXAMPLES OF ADVERSARIES: The study primarily uses the Caffenet model, whose layer names and representation dimensionalities are listed in Table S1.

S5 RESULTS FOR NETWORKS WITH RANDOM WEIGHTS

Feature adversaries can be generated in Caffenet architectures with random weights, but the relationship between representation dimensionality and achieved source-to-guide distance is not straightforward.

  • S5 RESULTS FOR NETWORKS WITH RANDOM WEIGHTS: Feature adversaries can be generated on networks with random weights using the distance ratio between the adversary, guide, and source representations.
  • S5 RESULTS FOR NETWORKS WITH RANDOM WEIGHTS: Convergence on Norm2 and Conv2 is nearly similar despite Norm2’s substantially smaller dimensionality.
  • S5 RESULTS FOR NETWORKS WITH RANDOM WEIGHTS: Norm2 has lower dimensionality than Conv3, yet optimization reaches a closer point on Conv3 than on Conv2 and therefore Norm2.
  • S5 RESULTS FOR NETWORKS WITH RANDOM WEIGHTS: Random, orthogonal-random, and trained Caffenet networks are evaluated by the ratio d(α,g)/d(s,g) as δ varies from 5 to 25 across Conv2, Norm2, and FC7.

S6 ADVERSARIES BY FAST GRADIENT

The supplementary analysis contrasts fast-gradient adversaries with optimization-based feature adversaries. It applies gradient perturbations to classification loss or internal representation distance and compares their geometric properties.

  • S6 ADVERSARIES BY FAST GRADIENT: The supplementary comparison tests whether the linearity hypothesis associated with label-adversary misclassification also explains feature adversaries.
  • S6 ADVERSARIES BY FAST GRADIENT: The fast gradient method creates label adversaries with δsign(∇_I loss(f(I),ℓ)), where ℓ is an erroneous label.
  • S6 ADVERSARIES BY FAST GRADIENT: Applying the same gradient strategy to internal representation distance produces feature adversaries via fast gradient, abbreviated feat-fgrad.
  • S6 ADVERSARIES BY FAST GRADIENT: Compared with feature-opt, fast-gradient adversaries are less well represented by a Gaussian around the adversary’s nearest neighbor.
  • S6 ADVERSARIES BY FAST GRADIENT: Figure S3 analyzes label-fgrad and feat-fgrad using manifold and nearest-neighbor rank analyses on FC7.

S7 FAILURE CASES

The optimization does not reliably produce good feature adversaries in several settings, especially for low-resolution or hand-drawn inputs and some fine-tuned network layers.

  • Failure cases: Low-resolution images and hand-drawn characters are settings where adversary generation does not always work well.On some MNIST or CIFAR10 images, success required relaxing the perturbation bound until guide-image traces became perceptible.
  • Failure cases: For Caffenet fine-tuned on Flickr Style, FC8 readily supported adversary generation, whereas FC7 often did not produce adversaries close to guide images.The passage suggests fine-tuning may distort FC7's generic image representation, making Euclidean distance there less useful for optimization.
  • Failure cases: Relaxing the perturbation bound can recover success in some difficult cases but may make guide-image traces perceptible.This limits the method's ability to maintain imperceptible perturbations in those settings.

S8 MORE EXAMPLES WITH ACTIVATION PATTERNS

The supplementary material presents additional source–guide pairs, adversaries, activation patterns, and inverted images using the same setup as the main example.

  • S8 MORE EXAMPLES WITH ACTIVATION PATTERNS: The remaining supplementary pages provide several source–guide pairs together with their adversaries, activation patterns, and inverted images.These examples complement the results shown in Fig. 2.
  • S8 MORE EXAMPLES WITH ACTIVATION PATTERNS: The supplementary examples are intended as a complement to the activation-pattern analysis discussed in Section 3.They extend the illustrated source–guide comparisons across multiple figure panels.
  • S8 MORE EXAMPLES WITH ACTIVATION PATTERNS: Figures S4 through S8 use the same setup as Fig. 2 for each displayed source and guide pair.Each figure includes inverted images and an activation plot.
Loading 1511.05122v9…