Source-linked AI summary
Execute Order 66: Targeted Data Poisoning for Reinforcement Learning
Harrison Foley, Liam Fowl, Tom Goldstein, Gavin Taylor
TL;DR
Targeted reinforcement-learning poisoning has been limited by prior attacks that modify states, actions, or rewards, while general degradation has been a common focus. This paper adapts gradient alignment to create minimally modified training observations that induce misbehavior at chosen states or watermark cues, and reports successful attacks in two Atari games. The approach currently requires access to the victim agent’s parameters, and initial poisons did not transfer successfully to a black-box agent.
Problem
Method
The paper adapts gradient alignment to perturb a small fraction of training observations so training steers an RL agent toward a chosen misbehavior without directly manipulating policy or reward.
Source: §4 · p. 3, §4.1 · p. 3Results
The attacks succeed in two Atari games, inducing misbehavior at a chosen target state or watermark cue while agents otherwise play normally, though some attacks reduce game performance.
Source: §1 · p. 1, §5.1 · p. 6, §5.2 · p. 6Takeaways & Limitations
The findings raise security concerns for RL neural networks because successful poisoning used tiny, hard-to-detect observation changes and could have life-threatening implications in safety-critical agents.
Source: §7 · p. 7The approach requires access to the victim agent’s parameters, and initial poisons did not successfully transfer to a black-box agent.
Source: §6 · p. 7
Abstract
from arXiv · showhide
Data poisoning for reinforcement learning has historically focused on general performance degradation, and targeted attacks have been successful via perturbations that involve control of the victim's policy and rewards. We introduce an insidious poisoning attack for reinforcement learning which causes agent misbehavior only at specific target states - all while minimally modifying a small fraction of training observations without assuming any control over policy or reward. We accomplish this by adapting a recent technique, gradient alignment, to reinforcement learning. We test our method and demonstrate success in two Atari games of varying difficulty.
1 Introduction
As reinforcement learning enters higher-risk applications, security vulnerabilities become more pressing. This paper adapts gradient alignment to poison a small fraction of training observations so an agent behaves normally until a chosen state or cue triggers misbehavior.
1 Introduction
- Security vulnerabilities in reinforcement learning are increasingly pressing as the technology enters high-risk applications.The paper identifies autonomous driving and cyberdefense among RL applications.Source: §1 · p. 1
- Gradient alignment enables an attack that minimally alters training data without changing the policy or reward.The trained agent can perform normally until it encounters a target state, where it misbehaves.Source: §1 · p. 1
- The attack can target either an attacker-chosen state or a translucent watermark appearing on part of any state.The paper demonstrates these cues in Atari games.Source: §1 · p. 1
2 Reinforcement Learning
The paper frames reinforcement learning as an agent choosing actions to accomplish a task. It explains the Markov decision process and how agents use sampled experience and policy gradients to learn.
2 Reinforcement Learning
- An RL agent selects actions based on its situation and understanding of which actions may produce positive outcomes.The goal is to accomplish a task.Source: §2 · p. 1
- An MDP represents a scenario with states, actions, transition probabilities, rewards, and a discount factor.The discount factor describes the significance of future rewards relative to immediate rewards.Source: §2 · pp. 1–2
- Agents learn from sampled transitions and can use neural-network policy gradients to optimize a policy objective.Samples contain the starting state, action, reward, and resulting state; learning improves the approximate value function and policy.Source: §2 · p. 2
3 Data Poisoning: Prior Work
Prior poisoning research distinguishes general performance degradation from targeted integrity attacks. In reinforcement learning, preserving ordinary behavior while causing targeted misbehavior is challenging because excessive data modification can disrupt learning.
3 Data Poisoning: Prior Work
- Supervised-learning poisoning includes availability attacks that degrade general performance and integrity attacks that target selected examples.Integrity attacks may leave overall performance unchanged until a specific target is encountered.Source: §3 · p. 2
- RL poisoning has often targeted availability because the trade-off between attack success and general ability makes targeted attacks challenging.Over-modifying training data can harm learning and undermine otherwise good behavior.Source: §3 · p. 2
- Prior RL integrity attacks altered states, actions, and rewards, making attacked data visibly different and assuming control over policy and reward.Policy-gradient training also requires new poisons for the current policy because older samples are discarded.Source: §3 · p. 2
4 Our Approach
The approach adapts gradient alignment to reinforcement learning, using minimally modified observations to steer training toward a chosen adversarial behavior. It assumes access to the victim’s parameters and evaluates target-state and watermark attacks.
4 Our Approach
- Gradient alignment perturbs training inputs so the training gradient aligns with a target-objective gradient, which can decrease that objective during training.The paper applies this idea to RL without directly manipulating the policy or reward function.Source: §4 · p. 3
4.1 Gradient Alignment Poisoning for RL
- The attack aligns gradients of the training loss and adversarial loss, where the latter measures whether the agent takes a desired action at a predetermined state.The alignment objective optimizes cosine similarity between the gradients.Source: §4.1 · p. 3
- Perturbations are constrained by a per-pixel maximum change and by a limit on the fraction of data modified, then optimized with projected gradient descent.The maximum change is represented by ε; constraints are specified for individual experiments.Source: §4.1 · p. 3
4.2 Attack Scenarios and Implementation Details
- The attacker assumes white-box access to victim parameters but can only minimally modify a small portion of observations, with image changes constrained in ℓ∞ norm.The experiments consider an arbitrary target state not encountered during training and a watermark appearing on any state.Source: §4.2 · pp. 3–4
- The implementation uses PPO and defines the adversarial loss with cross-entropy against a distribution assigning the desired action 100% probability.Signed PGD searches for poisoned observations with low alignment loss; Seaquest illustrations compare clean observations with ε=1 and ε=8 perturbations.Source: §4.2 · p. 4, Figure 1 · p. 4
5 Experimental Results
The experiments test targeted poisoning at arbitrary future states and at watermark-bearing states in Breakout and Seaquest. Both scenarios induce the desired actions, with larger attacks generally carrying a performance cost in the more difficult Seaquest setting.
5.1 First Attack Scenario: Arbitrary Target State
- At arbitrary target states, poisoning decisively induced all four Breakout actions and desired actions in both games, though Seaquest usually needed stronger attacks and lost performance.Breakout’s target state drew all four actions fairly evenly before poisoning; Seaquest’s chosen state was more consequential, and stronger attacks generally reduced performance.Source: §5.1 · pp. 4, 6, Figure 2 · p. 5, Figure 4 · p. 5
- Seaquest presented a harder target: its 18-action setting favored up-right-fire at the chosen state, and inducing different actions required stronger attacks in most cases.The authors describe the chosen action as important for successful gameplay; up-right was a possible exception where performance remained similar to clean agents.Source: §5.1 · pp. 4, 6
5.2 Second Attack Scenario: Target Watermark
- Watermark-triggered poisoning succeeded in both games; Breakout saw minor performance drops, while larger Seaquest attacks reduced performance, yet agents played reasonably without the watermark.The larger the induced action’s difference from Seaquest’s preferred up-right-fire action, the greater the performance impact.Source: §5.2 · p. 6, Figure 5 · p. 6, Figure 6 · p. 6
6 Ongoing Work
The authors identify black-box transfer as a priority for further work: current attacks require direct access to the victim and have not transferred successfully. They also propose testing more games, domains, and learning algorithms.
6 Ongoing Work
- The current approach requires direct access to the victim, and initial attacks generated using a separate black-box agent have not transferred successfully in Breakout.Figure 7 reports no desired misbehavior despite poisoning all training data with ϵ = 8.Source: §6 · p. 7, Figure 7 · p. 7
- The authors suggest mismatched training stages or random initialization may explain failed transfer, and propose generating poisons from an ensemble of black-box agents.They frame these explanations as hypotheses and the ensemble as a possible way to make attacks work across agents.Source: §6 · p. 7
- Further work could test whether the vulnerability extends to more Atari games, other domains, and learning algorithms, including a proposed MuJoCo attack.The suggested MuJoCo example would make an agent fall at a particular state while walking normally otherwise.Source: §6 · p. 7
7 Conclusion
The tested attacks succeeded with tiny perturbations to a very small fraction of training data, without altering policy or reward. Their potential consequences in safety-critical agents underscore the need for greater RL robustness.
- Successful online attacks in two tested domains required minuscule perturbations to a very small fraction of training data.Source: §7 · p. 7
- The perturbations modified states without changing the policy or reward function, making the attack difficult for an onlooker to detect.Source: §7 · p. 7
- A successful poisoning attack in a safety-critical agent such as an autonomous vehicle could be life-threatening, motivating more work on RL robustness.Source: §7 · p. 7