Source-linked AI summary
Delving into adversarial attacks on deep policies
Jernej Kos, Dawn Song
TL;DR
Deep reinforcement learning policies can be vulnerable to small adversarial perturbations, but evidence on attack effectiveness, efficient injection, and resilience remains limited. The paper compares FGSM attacks with random noise, introduces value-function-guided injection, and evaluates retraining, finding stronger attacks from adversarial examples and improved FGSM resilience after retraining.
Problem
The paper addresses limited understanding of adversarial attacks and resilience in deep reinforcement learning policies operating on raw inputs.
Method
The paper compares FGSM adversarial perturbations with random noise, tests periodic and value-function-guided injection, and evaluates retraining under both perturbation types.
Results
Adversarial examples are an order of magnitude more effective than random noise, while value-guided injection succeeds with fewer targeted frames and retraining improves FGSM resilience.
Takeaways & Limitations
Deep reinforcement learning attacks depend on when perturbations are injected, and retraining can increase resilience to FGSM attacks.
Takeaways & Limitations
The evaluation is limited to Atari Pong agents trained with A3C and FGSM perturbations.
Abstract
from arXiv · showhide
Adversarial examples have been shown to exist for a variety of deep learning architectures. Deep reinforcement learning has shown promising results on training agent policies directly on raw inputs such as image pixels. In this paper we present a novel study into adversarial attacks on deep reinforcement learning polices. We compare the effectiveness of the attacks using adversarial examples vs. random noise. We present a novel method for reducing the number of times adversarial examples need to be injected for a successful attack, based on the value function. We further explore how re-training on random noise and FGSM perturbations affects the resilience against adversarial examples.
1 INTRODUCTION
The paper investigates adversarial attacks on deep reinforcement learning policies, showing that small perturbations can fool them and that adversarial examples outperform random noise. It also studies value-guided injection, retraining-based resilience, and the limited scope of its evaluation.
- The study is among the first to investigate adversarial examples against deep reinforcement learning policies and finds that small perturbations can easily fool them.
- Adversarial examples are an order of magnitude more effective than random noise for attacking deep reinforcement learning policies.
- Value-guided injection can make attacks successful while perturbations are injected in only a fraction of frames.
- Retraining with random noise or FGSM perturbations improves resilience to FGSM attacks, with FGSM retraining potentially more effective.Retrained agents may remain vulnerable to slower optimization-based attacks.
- The evaluation focuses on Atari Pong agents trained with A3C and FGSM perturbations.The authors describe this as a first step toward understanding challenges and limitations of deep reinforcement learning under adversarial inputs.
2 STUDY OBJECTIVES
The study examines how adversarial attacks on deep reinforcement learning policies compare with random noise, how injection frequency can be reduced, and whether retraining improves resilience and transfers across perturbations.
- The study compares injecting random noise with injecting FGSM adversarial perturbations into the environment.
- Three injection strategies are tested: periodic injection, periodic recomputation with reuse, and value-function-guided injection above a threshold.
- The retraining experiments test whether random noise or adversarial perturbations improve resilience to later attacks and whether resilience transfers across magnitudes and perturbation types.
3 EXPERIMENTAL EVALUATION
Experiments on A3C agents for Atari Pong evaluate FGSM attacks, injection schedules, value-guided targeting, and retraining with noise or adversarial perturbations.
- The experiments use A3C on Atari Pong with cropped, scaled 42x42 grayscale image inputs.The resulting frame dimensions are 42x42x1.
- FGSM perturbations are generated from the gradient of a cross-entropy loss designed to move the policy output away from the optimal action.
- FGSM perturbations are orders of magnitude more effective than random noise, which severely degrades performance only at greater magnitudes such as β ≥0.05.Low random-noise levels, β ≤0.02, have little performance impact.
- Recomputing FGSM perturbations every tenth frame and reusing them between recomputations is as effective as the original attack, whereas injecting only every tenth frame is ineffective.These experiments use ϵ=0.001.
- Value-function-guided injection is much more effective than blind injection at similar frequencies, averaging 120 versus 125 injections per episode.The value-function threshold in this experiment is 1.4.
- Retraining in a noisy environment can produce resilience to certain FGSM perturbation levels and to perturbations with greater or smaller magnitudes than those used during retraining.
A APPENDIX
Figure 3 compares attack effectiveness for random noise and FGSM adversarial perturbations across specified perturbation magnitudes.
- Top panels show random-noise attacks with β values 0.02 and 0.05, while bottom panels show FGSM attacks with ϵ values 0.001 and 0.005.
A.1 VISUALIZING THE POLICY NETWORK ACTION BOUNDARY
The appendix visualizes policy actions around adversarial directions and compares baseline policies with policies re-trained using random noise or FGSM perturbations. These visualizations show fragmented decision spaces and compare action-boundary behavior before and after re-training.
- Policy network action boundary: The visualization samples each policy 7 times over image-space inputs x+ud1+vd2, with d1 aligned to the adversarial perturbation and d2 orthogonal to it.
- Policy network action boundary: Small input perturbations can cause the policy’s selected optimal action to change drastically, indicating a fragmented decision space.
- Policy network action boundary: Figure 4 evaluates agents re-trained with random noise or FGSM perturbations against FGSM perturbations at multiple magnitudes.
- Policy network action boundary: Figure 5 compares baseline, random-noise-retrained, and FGSM-retrained policies using colored regions for their most commonly predicted discrete actions.
- Policy network action boundary: Re-training with random noise or FGSM perturbations does not smooth the decision boundary; the space appears even more fragmented after re-training.
- Policy network action boundary: Figure 6 maps duplicated actions to three semantic actions—noop, move up, and move down—and still finds a fragmented action space.