Adversarial Policies: Attacking Deep Reinforcement Learning

Adam Gleave; Michael Dennis; Cody Wild; Neel Kant; Sergey Levine; Stuart Russell

Source-linked AI summary

Adversarial Policies: Attacking Deep Reinforcement Learning

Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, Stuart Russell

arXiv:1905.10615v3May 25, 2019cs.LGcs.AIcs.CRstat.ML

TL;DR

The paper asks whether an RL agent can be attacked by another policy acting naturally in a shared environment rather than by directly modifying observations. It models the victim as fixed and black-box, trains an adversary against it, and finds reliable attacks against self-play victims, including observation-confusing behavior, greater vulnerability in high-dimensional environments, and distinct victim activations.

Problem
The paper addresses whether an attacker can induce adversarial natural observations through actions in a shared environment when direct observation modification is unavailable.
Method
The authors model a fixed black-box victim in a two-player Markov game and train an adversarial policy with model-free RL against it.
Results
Adversarial policies reliably beat self-play-trained victims, winning by confusing them with natural observations rather than by learning generally strong behavior.
Takeaways & Limitations
Adversarial policies expose previously unrecognized failure modes in apparently robust self-play agents and provide a constructive way to test victim exploitability.
Takeaways & Limitations
Fine-tuning protects against a particular adversary, but repeating the attack finds a new adversary, indicating that adversarial policies are difficult to eliminate.

Abstract

from arXiv · show

Deep reinforcement learning (RL) policies are known to be vulnerable to adversarial perturbations to their observations, similar to adversarial examples for classifiers. However, an attacker is not usually able to directly modify another agent's observations. This might lead one to wonder: is it possible to attack an RL agent simply by choosing an adversarial policy acting in a multi-agent environment so as to create natural observations that are adversarial? We demonstrate the existence of adversarial policies in zero-sum games between simulated humanoid robots with proprioceptive observations, against state-of-the-art victims trained via self-play to be robust to opponents. The adversarial policies reliably win against the victims but generate seemingly random and uncoordinated behavior. We find that these policies are more successful in high-dimensional environments, and induce substantially different activations in the victim policy network than when the victim plays against a normal opponent. Videos are available at https://adversarialpolicies.github.io/.

1 INTRODUCTION

The paper asks whether an adversarial policy can attack an RL victim by taking physically realistic actions that induce adversarial natural observations. In simulated zero-sum robotics games, such policies reliably defeat self-play victims through observation manipulation, with greater vulnerability in higher-dimensional settings.

Threat model: The threat model lets an attacker affect a victim’s observations only through physically realistic actions in a shared environment.The attacker cannot directly modify arbitrary pixels or another agent’s market-data observations.
Adversarial policies: Adversarial policies reliably beat self-play-trained victims despite using less than 3% of the victims’ original training timesteps.The victims were state-of-the-art policies trained via self-play to be robust to opponents.
Attack mechanism: The adversaries win by creating natural observations that confuse the victim, rather than by becoming generally strong opponents.They often fall into contorted positions instead of learning to run, kick, or block like normal opponents.
Attack mechanism: 86% of episodes were won by an adversarial opponent that never stood up, compared with 47% for a normal opponent.Figure 1 illustrates the victim and opponent behavior in the finish-line game.
Analysis: Victim policies in higher-dimensional Humanoid environments are substantially more vulnerable than those in lower-dimensional Ant environments.Adversarial policies also induce significantly different, typically more widely dispersed victim-network activations than normal opponents.
Defense: Fine-tuning against one adversary protects against that adversary, but repeating the attack finds a new adversary with a different physical-interference strategy.The authors suggest repeated fine-tuning might protect against a range of adversaries.
Implications: The paper presents adversarial policies as a previously unrecognized threat model and as a tool for uncovering unexpected policy failure modes.It also identifies adversarial training using such policies as a potential robustness direction.

2 RELATED WORK

Prior RL attacks mainly perturb image observations under an ℓp threat model, whereas this work studies physically realistic attacks through actions in shared environments. The paper relates this setting to adversarial agents, opponent coupling, worst-case RL analysis, and adversarial training.

Adversarial examples: Most prior RL adversarial-example work assumes an ℓp-norm threat model that perturbs image observations directly.This work instead disallows direct modification of the victim’s observations.
Adversarial agents: Related work studies adversarial cars, but this paper focuses on zero-sum games because even optimal policies may be exploitable in collaborative games.The paper also distinguishes its victims from agents tightly coupled to their training opponents.
Defenses: Adversarial training has improved robustness in image classification and has been applied to RL through force perturbations or altered dynamics parameters.The paper’s fine-tuning defense is inspired by these approaches.
Worst-case analysis: The work follows worst-case RL analyses that choose adversarial transition functions or verify controllers and find counterexamples.These approaches frame robustness through adversarial dynamics or formal policy analysis.

3 FRAMEWORK

The framework models a victim and attacker as players in a two-player Markov game, then treats the fixed black-box victim as part of an attacker-controlled MDP. Because the resulting dynamics remain unknown, the attacker must solve an RL problem.

Markov game: The two-player Markov game contains states, both players’ action sets, joint transitions, and rewards depending on states and both actions.Each player maximizes its discounted reward sum.
Threat model: The victim is a fixed stochastic policy, while the adversary has unlimited black-box access to sampled victim actions but no weights or activations.This reflects deployment settings in which validated models are frozen.
Attacker MDP: Holding the victim policy fixed reduces the game to a single-player adversary MDP with the victim policy embedded in its transition and reward functions.The adversary retains the original state and action spaces.
Optimization: The attacker’s objective is to learn an adversarial policy that maximizes its discounted rewards while the victim samples actions from its stochastic policy.The victim’s action is sampled conditional on the current state.
Optimization: Even when the original game dynamics are known, the attacker’s embedded dynamics are unknown because the victim policy is a black box.Consequently, the attacker must solve an RL problem.

4 FINDING ADVERSARIAL POLICIES

The paper evaluates adversarial policies in zero-sum simulated robotics games against self-play-trained victims, finding that they often outperform standard opponents through observation manipulation rather than task performance.

4.1 ENVIRONMENTS AND VICTIM POLICIES: The evaluation uses two-player MuJoCo games with proprioceptive observations, including Kick and Defend, You Shall Not Pass, Sumo Humans, and Sumo Ants.Agents observe their own and their opponent’s joint positions, velocities, and contact forces.
4.3 RESULTS: Adversarial policies reliably win against most victims and outperform the pre-trained Zoo baseline for a majority of environments and victims.Win-rate curves compare Adv against Zoo, random, and zero-action baselines across victims.
4.3 RESULTS: In Kick and Defend and You Shall Not Pass, adversaries win by remaining upright only rarely or never and positioning their bodies to induce observations that trigger poor victim actions.Their behavior exploits victim weaknesses rather than executing the intended task, such as blocking a goal.
4.3 RESULTS: In Sumo Humans, falling immediately loses, so the adversary instead kneels centrally; it achieves 88% wins against victim 1 but only 62% and 45% against victims 2 and 3.These results are below the Zoo baselines for victims 2 and 3.
4.3 RESULTS: Random and zero-action off-distribution baselines win 30%–50% of Kick and Defend games but under 1% in Sumo and You Shall Not Pass, below adversarial-policy performance.This comparison tests whether adversarial success is explained only by being outside the victims’ training distribution.

5 UNDERSTANDING ADVERSARIAL POLICIES

The paper explains that adversarial policies manipulate victims through body-position observations, with vulnerability increasing in higher-dimensional settings and activations shifting off distribution. Masking the adversary’s position reverses the outcome against adversaries and reveals non-transitive policy relationships.

5.1 MASKED POLICIES: Adversarial policies manipulate victims through body position, creating natural observations that are adversarial rather than physically interfering with them.
5.1 MASKED POLICIES: Against Adv1, normal victim ZooV1 loses 86% of games, whereas masked victim ZooMV1 wins 99%; against normal ZooO1, the masked victim performs worse.In You Shall Not Pass, ZooO1 wins 78% against ZooMV1 but 47% against ZooV1.
5.1 MASKED POLICIES: These results imply highly non-transitive relationships among adversarial policies, victims, and masked victims, challenging self-play methods that assume transitivity.
5.3 VICTIM ACTIVATIONS: The paper identifies a trade-off: greater observation can improve benign-environment performance while increasing vulnerability to adversarial policies.
5.2 DIMENSIONALITY: The adversary wins at a much higher rate in Sumo Humans, where dim P = 24, than in low-dimensional Sumo Ants, where dim P = 15.P is the position of the adversary’s joints.
5.3 VICTIM ACTIVATIONS: Victim activations under adversarial policies differ substantially from normal-opponent activations and are typically more widely dispersed between timesteps.The analysis uses a Gaussian Mixture Model and t-SNE.

6 DEFENDING AGAINST ADVERSARIAL POLICIES

The paper evaluates fine-tuning victims against adversaries as a defense. Single and dual fine-tuning reduce vulnerability to the original adversary, but single training harms normal-opponent performance and retraining exposes new attacks.

6 DEFENDING AGAINST ADVERSARIAL POLICIES: Single and dual fine-tuned victims reduce Adv1’s win rate from 87% to around 10%.
6 DEFENDING AGAINST ADVERSARIAL POLICIES: Single fine-tuning catastrophically forgets how to play against ZooO1, while dual fine-tuning limits but does not eliminate this degradation.ZooO1 wins 57% against ZooDV1, up from 48% against original ZooV1.
6 DEFENDING AGAINST ADVERSARIAL POLICIES: Retraining the attack against fine-tuned victims produces new adversaries that win at equal or greater rates and transfer successfully to the original victim.These adversaries trip victims by lying prone, unlike Adv1, which causes ZooV1 to fall without touching it.

7 DISCUSSION

The paper frames adversarial policies as a natural-observation threat model and shows that self-play-trained deep RL policies can harbor difficult-to-find failure modes. It also evaluates exploitability testing and fine-tuning defenses, finding that attacks can be reapplied although confusion-based attacks may be reduced.

The paper proposes a threat model in which an adversarial policy creates natural adversarial observations through actions in a shared environment.
Adversarial policies exist against state-of-the-art victims trained via self-play to be robust to adversaries.
Self-play-trained policies can harbor serious, hard-to-find failure modes, making careful testing practically necessary.
Training an adversary constructively lower-bounds a victim’s exploitability, while a declining victim win rate confirms the pair was not in a global Nash.
Fine-tuning against an adversary can be successfully attacked again, although the resulting adversary is forced to trip the victim rather than confuse it.
The authors present adversarial policies as a previously unrecognized deep-RL problem with implications for robustness, security, and policy understanding.

A TRAINING: HYPERPARAMETERS AND COMPUTATIONAL INFRASTRUCTURE

Appendix training details specify PPO hyperparameters, their tuning procedure, and the computational infrastructure used for adversarial-policy experiments.

Table A.1 lists the hyperparameters used for Proximal Policy Optimization training.
The number of environments and total timesteps were chosen after observing diminishing returns from increasing them.
Batch size, mini-batches, epochs per update, entropy coefficient, and learning rate were tuned by a 100-sample random search on two environments.
Fine-tuning experiments reused the same hyperparameters after a repeated sweep produced similar results.
Adversary training took around 8 hours per victim using 4 cores of an Intel Xeon Platinum 8000 processor.

B ACTIVATION ANALYSIS: T-SNE AND GMM

The activation analysis concatenates outputs from all victim-policy feed-forward layers into 128-dimensional vectors for Gaussian-mixture modeling and t-SNE visualization.

Activations from all feed-forward layers form two 64-length vectors concatenated into a single 128-dimensional analysis vector.
The resulting vectors are analyzed with a Gaussian Mixture Model and represented using t-SNE.

B.1 T-SNE HYPERPARAMETER SELECTION

The appendix selects t-SNE and GMM hyperparameters by comparing visualization clarity, Bayesian Information Criterion, and held-out validation likelihood.

GMM configurations varied from 5 to 80 components and used either full or diagonal covariance matrices.
Twenty components with a full covariance matrix achieved the lowest BIC and highest validation log-likelihood for most environment-victim pairs.

C FIGURES

The supplementary figures report adversary win rates, episode outcomes, and victim-policy activations across several robot-game settings. They compare adversarial, normal, random, and zero-action opponents, including decomposed activation views.

Policy labels: The supplementary material labels the compared policies as adversary Adv, normal opponent Zoo, random Rand, and Zero.
Environments: The figure set includes Kick and Defend cases numbered 1, 2, and 3.
Win rates: Figure C.1 plots Adv’s smoothed win rate during training against horizontal baseline rates.The solid curve is the median across 5 seeds, with shading showing the minimum and maximum; smoothing uses 100 000 timesteps.
Episode outcomes: Figure C.2 organizes episode outcomes by victim rows and opponent columns, showing opponent wins, victim wins, and ties across 1000 episodes.The maximal opponent win rate in each row is marked in red.
Victim activations: The activation figures compare victim responses to adversarial and normal opponents, with one view aggregating opponents and another decomposing them individually.Figure C.4 focuses on victim Zoo1 in Sumo or ZooV1 in other environments and provides individual-opponent views for clarity.