Source-linked AI summary
Adversarial Attacks on Neural Network Policies
Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, Pieter Abbeel
TL;DR
Adversarial examples had been extensively studied for classifiers, but their effect on neural network policies in reinforcement learning was less established. This paper evaluates small raw-input perturbations against trained policies across algorithms and white-box or black-box settings, finding significant performance degradation across tasks, including with human-indistinguishable perturbations.
Problem
The paper asks whether adversarial examples can affect fully trained neural network policies in reinforcement learning when an adversary makes limited changes to raw test-time inputs.
Method
The paper evaluates white-box and black-box adversarial attacks across policies trained with different deep RL algorithms, using FGSM and transferability assumptions.
Results
Across games and training algorithms, small input perturbations significantly reduce policy performance; FGSM with ϵ = 0.001 decreases performance by 50% or more in many cases.
Takeaways & Limitations
Adversarial examples can confuse neural network policies even in black-box scenarios, creating implications for online and real-world deployment.
Takeaways & Limitations
The study assumes black-box adversaries may access the training environment and training algorithm and know the target architecture, but not its random initialization.
Abstract
from arXiv · showhide
Machine learning classifiers are known to be vulnerable to inputs maliciously constructed by adversaries to force misclassification. Such adversarial examples have been extensively studied in the context of computer vision applications. In this work, we show adversarial attacks are also effective when targeting neural network policies in reinforcement learning. Specifically, we show existing adversarial example crafting techniques can be used to significantly degrade test-time performance of trained policies. Our threat model considers adversaries capable of introducing small perturbations to the raw input of the policy. We characterize the degree of vulnerability across tasks and training algorithms, for a subclass of adversarial-example attacks in white-box and black-box settings. Regardless of the learned task or training algorithm, we observe a significant drop in performance, even with small adversarial perturbations that do not interfere with human perception. Videos are available at http://rll.berkeley.edu/adversarial.
1 Introduction
Neural network policies in deep reinforcement learning can be attacked at test time through small perturbations to raw inputs. The paper characterizes vulnerability across learning algorithms and white-box or black-box access, finding broad susceptibility and transferable attacks.
- Deep RL policies map raw inputs to actions and have achieved strong performance across games, robotics, locomotion, and driving.
- The paper tests whether limited test-time changes to raw inputs can affect fully trained neural network policies.
- Training algorithms and random initialization can produce substantially different policies for the same task, potentially affecting adversarial resistance.
- In Pong, FGSM perturbations make a DQN policy miss the ball and lose the point despite choosing a good action on the original input.
- Three white-box attack types are evaluated across four Atari games and policies trained with DQN, TRPO, and A3C; all policies are vulnerable, while TRPO and A3C seem more resistant.
- Black-box adversarial examples transfer between policies trained with different trajectory rollouts and between policies trained with different algorithms on the same task.
2 Related Work
Prior adversarial-example research mainly studied supervised learning, whereas this work examines interference with an RL agent during test-time operation.
- Adversarial machine learning studies model and learning-algorithm behavior under attacks during training or inference.
- Most prior adversarial-example work focused on supervised learning algorithms.
- This work studies adversaries interfering with an RL agent’s operation by presenting adversarial examples at test time.
3 Preliminaries
The paper introduces adversarial-example crafting with FGSM and reviews deep RL policies, including deterministic DQN and stochastic TRPO and A3C algorithms.
- 3.1 Adversarial Example Crafting with the Fast Gradient Sign Method: Adversarial perturbations maximize harm subject to a size constraint intended to limit detectability, with methods trading computational cost against attack success.
- 3.1 Adversarial Example Crafting with the Fast Gradient Sign Method: FGSM efficiently generates perturbations by linearizing a deep model and solving the resulting maximization problem analytically.
- 3.1 Adversarial Example Crafting with the Fast Gradient Sign Method: Under an ℓ∞ constraint, FGSM changes each input pixel by at most ϵ and uses η = ϵ sign(w) for a linear function g(x) = w⊤x.
- 3.1 Adversarial Example Crafting with the Fast Gradient Sign Method: For image classification, linearizing the loss J(θ, x, y) around x yields an input perturbation based on the loss gradient.
- 3.2 Deep Reinforcement Learning: Deep RL policies optimize expected cumulative reward and may map states to either individual actions or distributions over actions.
- 3.2 Deep Reinforcement Learning: The paper compares adversarial examples on feed-forward policies trained with DQN, TRPO, and A3C.
- 3.2.1 Deep Q-Networks: DQN estimates action Q-values with a neural network and selects the maximum-Q action, producing a deterministic policy.
- 3.2.2 Trust Region Policy Optimization: TRPO updates a stochastic policy from whole-trajectory rollouts while controlling policy change using KL divergence.
4 Adversarial Attacks
The paper applies FGSM to trained neural-network policies in white-box and black-box settings, adapting the loss and perturbation constraints to reinforcement-learning policies.
- Applying FGSM to Policies: The experiments use FGSM in white-box attacks on policies with known architectures and parameters, and in black-box attacks by computing gradients for separately trained policies.The black-box procedure relies on transferability from a surrogate policy.
- Applying FGSM to Policies: FGSM computes input perturbations from the gradient of the policy’s cost function with respect to the current input.The attack uses a trained policy’s action distribution and assumes its highest-weighted action is optimal.
- Applying FGSM to Policies: For DQN, the attack replaces the deterministic policy output with a temperature-1 softmax over Q-values when computing gradients.This avoids the zero gradient produced by the deterministic action-selection rule; test-time DQN execution remains deterministic.
- Choosing a Norm Constraint: The study compares ℓ∞-, ℓ2-, and ℓ1-norm constraints, corresponding respectively to small changes across all features, broader distributed changes, and sparse feature changes.The ℓ1 constraint can concentrate the perturbation budget on a few high-impact input dimensions.
5 Experimental Evaluation
Across four Atari games and policies trained with A3C, TRPO, or DQN, small white-box perturbations substantially reduce performance, while black-box transfer remains effective but weakens as target-policy information decreases.
- Experimental Setup: The evaluation covers Chopper Command, Pong, Seaquest, and Space Invaders using A3C, TRPO, and DQN policies.Each game and algorithm combination uses the specified Atari environments and three deep reinforcement-learning algorithms.
- Experimental Setup: The study trains five policies per game and algorithm, then focuses on up to three top-performing policies meeting the 80% performance criterion.Some game-algorithm combinations have fewer qualifying policies.
- Experimental Setup: Performance is measured as average undiscounted cumulative reward across ten rollouts of the target policy.This averaging is used to reduce experimental variance.
- Vulnerability to White-Box Attacks: Small input perturbations significantly decrease performance across games and training algorithms, with ℓ∞ FGSM at ϵ = 0.001 often reducing performance by 50% or more.The corresponding 8-bit image inputs are indistinguishable from the originals.
- Vulnerability to White-Box Attacks: ℓ1-norm adversaries produce the largest performance decreases at equal ϵ by changing a few pixels by large amounts.This option is preferable when imperceptibility is not essential.
- Vulnerability to White-Box Attacks: Policies trained with A3C, TRPO, and DQN are all susceptible, with DQN especially vulnerable to ℓ∞ perturbations on Pong, Seaquest, and Space Invaders.The result compares susceptibility across both training algorithms and tasks.
- Vulnerability to Black-Box Attacks: Black-box transfer is less effective as the adversary knows less about the target policy, but transfer across algorithms still significantly reduces performance for most games at larger ϵ.Transfer across algorithms is weaker than transfer across policies, which is weaker than direct access to the target network.
- Vulnerability to Black-Box Attacks: For ℓ1-norm attacks, transfer across algorithms is nearly as effective as no transferability for most game and algorithm combinations.This shows that sparse perturbations retain substantial effectiveness without matching the target policy’s training algorithm.
6 Discussion and Future Work
The discussion highlights deployment risks from adversarial inputs, including transferability across policies and algorithms, and calls for defenses against these attacks.
- Adversarial perturbations can confuse neural network policies using computationally efficient attacks, including in black-box scenarios.The authors connect this vulnerability to online and real-world deployment, including possible physical perturbations.
- Figure 4 examines transferability of adversarial inputs for policies trained with TRPO.
- Figure 5 examines transferability of adversarial inputs for policies trained with DQN.
- Strategically placed paint on a road is offered as a possible real-world perturbation that could confuse an autonomous car’s lane-following policy.
- The authors identify defenses based on adversarial training or test-time detection as an important direction for future work.
A Experimental Setup
Experiments use rllab with Gym interfaces to Atari environments and a convolutional policy architecture based on prior work.
- Experiments use rllab, integrating TRPO, DQN, and A3C implementations with OpenAI Gym interfaces to the Arcade Learning Environment.
- The policy network has two convolutional layers followed by a 256-unit fully connected layer, with rectified nonlinearities in hidden layers.The convolutional layers use 16 8 × 8 filters with stride 4, then 32 4 × 4 filters with stride 2.
- All games use a frame skip of 4, repeating the agent’s chosen action four times.
A.1 Training
The training setup uses separate compute configurations and algorithm-specific schedules for TRPO, A3C, and DQN policies.
- TRPO policies are trained for 2,000 iterations of 100,000 steps each with a KL-divergence bound of 0.01.Training takes 1.5 to 2 days on Amazon EC2 c4.8xlarge machines.
- A3C uses 18 actor-learner threads, a learning rate of 0.0004, and 200 iterations of 1,000,000 steps each.The setup also uses entropy regularization, RMSProp, five-step updates, and shared policy/value-network weights except at the output layer.
- DQN policies are trained for two days using 100,000 steps per epoch on Amazon EC2 p2.xlarge machines.
- The DQN implementation is identified by a GitHub repository, as is the A3C implementation.