Source-linked AI summary
Tactics of Adversarial Attack on Deep Reinforcement Learning Agents
Yen-Chen Lin, Zhang-Wei Hong, Yuan-Hong Liao, Meng-Li Shih, Ming-Yu Liu, Min Sun
TL;DR
Deep RL agents may inherit DNN vulnerability to adversarial examples, raising concerns for sequential control. The paper introduces strategically timed reward-reduction and enchanting state-luring attacks, evaluated on DQN and A3C agents. In five Atari games, strategically timed attacks matched uniform attacks while using 25% of episode steps, and enchanting attacks exceeded 70% success in three games.
Problem
Adversarial-example vulnerability in deep RL agents remained insufficiently understood despite known DNN vulnerability, especially for sequential control.
Method
The paper develops strategically timed and enchanting attacks, using selective timing, planning, and generative modeling to manipulate deep RL agents.
Results
In five Atari games, strategically timed attacks matched uniform-attack reward reduction at 25% of episode steps, while enchanting attacks exceeded 70% success in three games.
Takeaways & Limitations
Deep RL agents trained with DQN and A3C exhibit vulnerability to adversarial attacks that can reduce rewards or lure agents toward target states.
Takeaways & Limitations
The state-transition and future-frame prediction models are learned without assuming information from the RL agent.
Abstract
from arXiv · showhide
We introduce two tactics to attack agents trained by deep reinforcement learning algorithms using adversarial examples, namely the strategically-timed attack and the enchanting attack. In the strategically-timed attack, the adversary aims at minimizing the agent's reward by only attacking the agent at a small subset of time steps in an episode. Limiting the attack activity to this subset helps prevent detection of the attack by the agent. We propose a novel method to determine when an adversarial example should be crafted and applied. In the enchanting attack, the adversary aims at luring the agent to a designated target state. This is achieved by combining a generative model and a planning algorithm: while the generative model predicts the future states, the planning algorithm generates a preferred sequence of actions for luring the agent. A sequence of adversarial examples is then crafted to lure the agent to take the preferred sequence of actions. We apply the two tactics to the agents trained by the state-of-the-art deep reinforcement learning algorithm including DQN and A3C. In 5 Atari games, our strategically timed attack reduces as much reward as the uniform attack (i.e., attacking at every time step) does by attacking the agent 4 times less often. Our enchanting attack lures the agent toward designated target states with a more than 70% success rate. Videos are available at http://yenchenlin.me/adversarial_attack_RL/
1 Introduction
The paper investigates adversarial-example vulnerability in deep RL and introduces two attack tactics: strategically timed reward reduction and enchanting state luring.
- Deep RL agents may inherit DNN vulnerability to adversarial examples, motivating empirical study of attacks specific to sequential decision-making.RL attacks involve correlated observations, sequential actions, and goals beyond classification accuracy.
- The strategically-timed attack targets critical moments rather than perturbing every observation, aiming to reduce reward with fewer adversarial examples.The attack selects time steps when an adversarial example is expected to be effective.
- The strategically-timed attack matches uniform-attack reward reduction while attacking four times less often, whereas enchanting succeeds at more than 70%.These are reported outcomes for the two attack tactics.
- The enchanting attack combines planning with a generative model to lure an agent toward a designated state.It generates preferred action sequences while predicting future states.
- The attacks are evaluated on DQN and A3C agents across 5 Atari games.The experiments assess vulnerability and compare robustness across the two algorithms.
2 Related Work
Related work established adversarial examples in supervised DNN tasks, while adversarial attacks on deep RL remained less explored and initially used uniform attacks.
- Earlier DNN attacks commonly seek minimal image perturbations that change a classifier’s predicted class.
- Adversarial examples had been demonstrated across supervised learning tasks, but their existence in deep RL agents remained largely unexplored.
- The uniform attack perturbs a deep RL agent at every time step to reduce its reward.
- This paper differs by introducing strategically timed and planning-based enchanting attacks for deep RL agents.
- Prior defenses included adversarial training, stability regularization, and defensive distillation for DNNs.
3 Adversarial Attacks
The paper reviews adversarial-example attacks on DNN classifiers, generalizes them to deep RL agents, and introduces strategically timed and enchanting attacks.
- The section first reviews adversarial-example attacks on DNN-based classification systems before generalizing them to deep RL.
- It then introduces the strategically-timed and enchanting attack tactics.
- The section’s organization connects classifier attack methods with attacks designed for sequential deep RL interaction.
3.1 Preliminaries
The preliminaries define adversarial examples as minimal observation perturbations that alter DNN decisions, then describe RL policies and their vulnerability to reward-reducing actions.
- An adversarial example seeks a minimal perturbation δ that changes a DNN’s class assignment for an image.The optimization uses an image similarity metric DI.
- Figure 1 uses the relative action-preference function c(st) and threshold β to illustrate when strategically timed attacks are launched.At s84, perturbation changes the action and the agent misses the ball; at s25, it has no reward impact.
- An RL agent chooses actions from observations through a policy while maximizing accumulated future rewards over an episode.
- In deep RL, perturbing observations can make the policy select nonpreferred actions that reduce accumulated future rewards.
3.2 Adversarial Attacks on RL
Adversarial attacks on deep RL differ from classification attacks because agents act through correlated observations and are evaluated by reward consequences. These characteristics motivate selective attacks and action sequences that manipulate future states.
- RL agents receive correlated observations because each action changes the environment and affects later states.
- Unlike classification attacks, RL attacks can target reward reduction rather than merely inducing any non-optimal action.
- Sparse rewards mean an adversary need not attack at every time step.
- Because actions influence future observations, an adversary can plan sequential adversarial examples to lure an agent toward a chosen state.
3.3 Strategically-Timed Attack
The strategically-timed attack selects a small set of attack times and crafts perturbations that replace the agent’s preferred action with its least preferred action. A heuristic uses action preference to decide when to attack and handles the resulting optimization difficulty.
- The attack limits adversarial perturbations to fewer than Γ << L time steps while minimizing expected accumulated reward.
- Binary variables b_t indicate whether perturbation δ_t is applied at each time step.
- The original optimization is difficult because it is mixed-integer and observations depend on all previous observations.
- The relative action preference function c measures how strongly the policy prefers its most preferred action over its least preferred action.
- The adversary attacks when c(s_t) ≥ β, with β controlling attack frequency and relating to Γ.
- For DQN, action Q-values are converted into a probability distribution using softmax with temperature T.
- Each perturbation targets the originally least-preferred action to reduce accumulated reward.
3.4 Enchanting Attack
The enchanting attack plans actions toward a designated target state, then crafts sequential adversarial examples to make the RL agent follow that plan. A learned video-prediction model evaluates candidate action sequences by their predicted final states.
- The attack aims to move the agent from s_t to target state s_g after H steps using a sequence of adversarial examples.
- The method separates action planning under arbitrary control from crafting perturbations that induce each planned action.
- The video prediction model M predicts a future state from the current state and a sequence of future actions.
- Candidate action sequences are evaluated by the distance between their predicted final state and the target state, using the L2-norm in the experiments.
- A sampling-based cross-entropy method samples N action sequences, retains the best K, and refits categorical distributions; experiments use N = 2000, K = 400, and J = 5.
- The closest predicted sequence becomes the plan, whose first action is targeted by a Carlini–Wagner adversarial example.
- The attack replans from s_t+1 rather than directly crafting the next example, improving robustness to potential previous-attack failure.
- The transition and future-frame prediction models are learned without assuming information from the RL agent.
4 Experiments
The experiments evaluate strategically-timed and enchanting attacks against A3C and DQN agents across five Atari games. Results show that selective attacks can match uniform attacks, while planning-based attacks can lure agents toward target states.
- The evaluation used A3C and DQN agents across five Atari 2600 games.The games were MsPacman, Pong, Seaquest, Qbert, and ChopperCommand.
- Strategically-Timed Attack: 25% of episode time steps was sufficient for the strategically-timed attack to reach the same effect as the uniform attack, on average.The attack rate is the average portion of episode time steps at which the adversary attacks.
- Strategically-Timed Attack: DQN agents were more vulnerable than A3C agents in most games except Pong.The comparison concerns robustness to the strategically-timed attack.
- Enchanting Attack: Enchanting-attack success was measured by whether the final state reached the synthesized target within normalized Euclidean distance tolerance 1.Experiments varied H across [1, 5, 10, 20, 40, 60, 80, 100, 120] and evaluated ten initial time points.
- Enchanting Attack: Both A3C and DQN agents were enchanted, with success rates above 70% when H < 40 in several games except Seaquest and ChopperCommand.The lower effectiveness in those two games was attributed to multiple random enemies reducing video-prediction accuracy.
5 Conclusion
The paper introduces strategically-timed and enchanting attacks for deep RL agents. Across five Atari games, selective attacks match uniform-attack effects at 25% of time steps, while planning and video prediction lure agents to target states with over 70% success in three games.
- The strategically-timed attack significantly reduced accumulated rewards while attacking only 25% of episode time steps.
- The enchanting attack combined video prediction and planning to lure agents toward maliciously defined target states with over 70% success in 3 of 5 games.
- Future work includes improving strategically-timed attacks, video-prediction accuracy, and defenses against adversarial attacks.