Source-linked AI summary
Adversarial Reprogramming of Neural Networks
Gamaleldin F. Elsayed, Ian Goodfellow, Jascha Sohl-Dickstein
TL;DR
Existing adversarial attacks mainly induce errors or predefined outputs, leaving the challenge of making a fixed model perform a new attacker-selected task without per-input output computation. The paper learns a single adversarial program that transforms inputs and outputs around the target model, demonstrating reprogramming of ImageNet classifiers for counting and MNIST and CIFAR-10 classification. The authors present this as a first demonstration of adversarial reprogramming, while identifying unresolved limitations and broader-domain questions.
Problem
The work addresses how to make a fixed model perform an attacker-chosen task without computing the desired output separately for each test-time input.
Method
The method learns input and output transformations around a target model, using a shared adversarial program that maps new-task inputs into the model and its labels back to the adversarial task.
Results
The authors demonstrate adversarial reprogramming on ImageNet models for square counting, MNIST classification, and CIFAR-10 classification, including a first demonstration of this attack class.
Takeaways & Limitations
Adversarial reprogramming illustrates both flexibility and vulnerability in deep neural networks and motivates investigation of its properties, limitations, and defenses.
Takeaways & Limitations
The study demonstrates reprogramming in image-domain classification tasks, while success in audio, video, text, or other domains remains a future research question.
Abstract
from arXiv · showhide
Deep neural networks are susceptible to \emph{adversarial} attacks. In computer vision, well-crafted perturbations to images can cause neural networks to make mistakes such as confusing a cat with a computer. Previous adversarial attacks have been designed to degrade performance of models or cause machine learning models to produce specific outputs chosen ahead of time by the attacker. We introduce attacks that instead {\em reprogram} the target model to perform a task chosen by the attacker---without the attacker needing to specify or compute the desired output for each test-time input. This attack finds a single adversarial perturbation, that can be added to all test-time inputs to a machine learning model in order to cause the model to perform a task chosen by the adversary---even if the model was not trained to do this task. These perturbations can thus be considered a program for the new task. We demonstrate adversarial reprogramming on six ImageNet classification models, repurposing these models to perform a counting task, as well as classification tasks: classification of MNIST and CIFAR-10 examples presented as inputs to the ImageNet model.
1 INTRODUCTION
The paper introduces adversarial reprogramming: using a crafted program applied to inputs to make a pretrained model perform an attacker-chosen task without computing each test-time output. It frames this as a new adversarial goal and demonstrates it across several tasks and ImageNet models.
- Adversarial reprogramming repurposes a model to perform an attacker-chosen task without computing the desired output for each test-time input.
- The attack learns input- and output-side transformations so the original model's computation implements the adversarial task.hf maps new-task inputs into the original input domain, while hg maps model outputs back to the new task.
- For the image setting, small task images are placed at the center of a large image while learned parameters occupy the borders, with output classes hard-coded into a mapping.
- The paper defines adversarial reprogramming as attacks that repurpose a model for a new task, and notes that imperceptible versions are possible although perturbation magnitude need not be constrained.
- Experiments target ImageNet classifiers and repurpose them for square counting, MNIST classification, and CIFAR-10 classification.
2 BACKGROUND AND RELATED WORK
The related work places adversarial reprogramming alongside adversarial examples, universal modifications, computational exploits, and transfer learning. Its distinctive challenge is repurposing a fixed model through input manipulation, potentially across substantially different datasets.
- Adversarial examples: Adversarial examples intentionally modify inputs to cause model mistakes, including untargeted errors and targeted predictions.
- Universal modifications: A single modification can be applied to many inputs, but prior adversarial patches typically force one specific class rather than implement a new task per input.
- Computational exploits: Adversarial reprogramming resembles parasitic computing and weird machines because crafted inputs make a target system perform computations beyond its original purpose.
- Transfer learning: Unlike transfer learning, adversarial reprogramming changes only the input while leaving the model unaltered and may operate across tasks with very different datasets.
3 METHODS
The method learns an additive adversarial program that embeds new-task inputs into the target model's input format and maps its original labels to new-task labels. Training maximizes mapped-label probability while regularizing the program, and successful reprogramming depends on nonlinear input-program interactions.
- The attacker assumes access to a neural network's parameters and seeks to insert a learned adversarial program into its input to perform a new task.
- The same additive adversarial program is applied to all images rather than optimized separately for each image.
- A new-task image is resized into the target ImageNet dimensions and placed in the area selected by the mask before the program is added.
- Training maximizes the target probability P(hg(yadv)|Xadv), with a weight-norm penalty used to reduce overfitting.
- Successful reprogramming requires nonlinear interactions between the new-task input and the adversarial program, which linear models cannot provide.
4 RESULTS
Across six ImageNet models, adversarial programs repurposed networks for counting, MNIST, CIFAR-10, and shuffled-MNIST classification. The programs remained effective under several model, data, and visibility variations, including adversarially trained models and concealed inputs.
- Experimental setup: Six ImageNet architectures were reprogrammed for counting squares, MNIST classification, and CIFAR-10 classification.The evaluation also included shuffled-MNIST classification and comparisons involving adversarially trained and randomly initialized networks.
- Counting squares: All networks mastered counting squares despite the mismatch between ImageNet and adversarial labels, using only additive input contributions.The program was equivalent to a first-layer bias in this task.
- MNIST classification: ImageNet networks successfully classified MNIST digits with additive programs that generalized from training to test data.This indicates the reprogramming was not purely memorization-based or brittle to small input changes.
- CIFAR-10 classification: CIFAR-10 accuracy increased from chance to moderate accuracy when ImageNet models were reprogrammed, with minimal adversary-side computation at inference.The reported accuracy was near that expected from typical fully connected networks.
- Model details: Adversarially trained Inception V3 remained vulnerable to MNIST reprogramming, with only a slight reduction in attack success.The authors report that standard adversarial defenses had little efficacy against adversarial reprogramming.
- Model details: Randomly initialized networks were much harder to reprogram than ImageNet-pretrained networks, and their learned programs looked qualitatively different.The result demonstrates that the original task performed by the network is important for reprogramming.
- Original data: Pretrained ImageNet networks could classify shuffled MNIST with almost the same accuracy as standard MNIST, despite the absence of shared spatial structure.In some cases, shuffled MNIST achieved higher accuracy, suggesting reprogramming can cross unrelated datasets and domains.
- Concealing adversarial programs: Adversarial data and programs could be shuffled, scaled, and hidden inside a normal ImageNet image, though the resulting MNIST accuracy was lower.The concealed adversarial images were reported to resemble normal ImageNet images.
5 DISCUSSION
The results indicate that trained networks can be repurposed flexibly, including when adversarial data differ substantially from the original task. The study also identifies unresolved limits and broader risks, while restricting demonstrated attacks to image-domain classification.
- 5.1 FLEXIBILITY OF TRAINED NEURAL NETWORKS: Trained neural networks were more susceptible to adversarial reprogramming than random networks.
- 5.1 FLEXIBILITY OF TRAINED NEURAL NETWORKS: Reprogramming remained successful when adversarial data had a very different structure from data in the original task.
- 5.1 FLEXIBILITY OF TRAINED NEURAL NETWORKS: The authors suggest that dynamically reusing neural circuits could make modern neural networks easier to repurpose, more flexible, and more efficient through shared compute.
- 5.1 FLEXIBILITY OF TRAINED NEURAL NETWORKS: It remains unclear whether reduced performance on random networks and CIFAR-10 reflects limited perturbation expressivity or harder optimization.
- 5.2 ADVERSARIAL GOALS BEYOND THE IMAGE DOMAIN: The demonstrated attacks were limited to image-domain classification, while success on audio, video, text, or other tasks remains future work.
- 5.2 ADVERSARIAL GOALS BEYOND THE IMAGE DOMAIN: The study raises risks including computational-resource theft and repurposing AI assistants into spies or spam bots.
6 CONCLUSION
The paper proposes adversarial reprogramming as a new class of attacks that repurposes neural networks for attacker-chosen tasks. The results provide an initial demonstration of this possibility and expose both flexibility and vulnerability in deep networks.
- 6 CONCLUSION: The paper proposes adversarial attacks that reprogram neural networks to perform novel adversarial tasks.
- 6 CONCLUSION: The results demonstrate for the first time that such adversarial reprogramming attacks are possible.
- 6 CONCLUSION: The authors identify future work on the properties, limitations, and defenses of adversarial programming.
A SUPPLEMENTARY TABLES
The supplementary material reports ImageNet-model results, training configurations, and qualitative evidence that adversarial programs can support counting and cross-dataset classification, including shuffled MNIST.
- A SUPPLEMENTARY TABLES: Top-1 precision on ImageNet was 0.78 for Inception V3, 0.802 for Inception V4, and 0.804 for Inception ResNet V2.
- A SUPPLEMENTARY TABLES: Training used Adam with default parameters, exponential learning-rate decay, distributed batches across GPUs, and synchronized adversarial-program updates.
- A SUPPLEMENTARY TABLES: The supplementary figures compare adversarial programs across networks and tasks including square counting, MNIST classification, and CIFAR-10 classification.
- A SUPPLEMENTARY TABLES: Shuffling MNIST pixels produced data with no resemblance to ordinary images, yet the combined reprogramming image successfully enabled Inception V3 to classify the shuffled digits.