PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Policies: Attacking Deep Reinforcement Learning

Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, Stuart Russell

arXiv:1905.10615v3cs.LGcs.AIcs.CRstat.ML

TL;DR

Because an attacker generally cannot directly modify an RL agent’s observations, the paper asks whether an adversarial policy can use actions in a shared environment to create natural observations that affect the victim. It trains adversarial policies against fixed, black-box victims in simulated robotics games. These policies reliably beat self-play-trained victims despite using less than 3% as many training timesteps.

  • Problem

    Because real-world agents generally cannot directly alter another agent’s observations, whether actions in a shared environment can induce natural observations that adversarially affect an RL policy remains an open question.

    Source: §1 · p. 1

  • Method

    The paper formulates a physically realistic shared-environment threat model and trains adversarial policies with model-free RL against fixed, black-box victims.

    Source: §1 · p. 1

  • Results

    Less than 3% as many training timesteps, adversarial policies reliably beat self-play-trained victims while exhibiting seemingly random behavior.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Training an adversary provides a constructive lower bound on a victim policy’s exploitability.

    Source: §7 · p. 8

    After fine-tuning the victim against an adversary, the attack can be reapplied successfully, though the defense appears to protect against attacks that confuse the victim.

    Source: §7 · p. 8

Abstract

from arXiv · show

Deep reinforcement learning (RL) policies are known to be vulnerable to adversarial perturbations to their observations, similar to adversarial examples for classifiers. However, an attacker is not usually able to directly modify another agent's observations. This might lead one to wonder: is it possible to attack an RL agent simply by choosing an adversarial policy acting in a multi-agent environment so as to create natural observations that are adversarial? We demonstrate the existence of adversarial policies in zero-sum games between simulated humanoid robots with proprioceptive observations, against state-of-the-art victims trained via self-play to be robust to opponents. The adversarial policies reliably win against the victims but generate seemingly random and uncoordinated behavior. We find that these policies are more successful in high-dimensional environments, and induce substantially different activations in the victim policy network than when the victim plays against a normal opponent. Videos are available at https://adversarialpolicies.github.io/.

1 INTRODUCTION

The paper proposes a physically realistic threat model in which an adversarial policy attacks an RL victim through actions that induce natural observations. It demonstrates that such policies can exploit self-play-trained victims and analyzes how they succeed.

1 INTRODUCTION

  • The threat model has an adversary act in a shared environment to induce natural, physically realistic observations, rather than directly altering the victim’s inputs.This reflects settings such as driving and trading, where others can affect observations through actions but cannot arbitrarily modify them.Source: §1 · pp. 1–2
  • Adversaries succeed by creating adversarial observations rather than becoming generally strong opponents; the strategy fails when victims cannot see the adversary’s position.They may fall into contorted positions instead of learning to run, kick, or block.Source: §1 · pp. 1–2
  • Adversarial opponents win 86% of episodes, versus 47% for normal opponents; adversarial policies reliably beat self-play-trained victims with under 3% of the victims’ training timesteps.Figure 1 illustrates an adversary winning despite never standing up.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Victims in higher-dimensional Humanoid environments are more vulnerable than those in lower-dimensional Ant environments, and adversaries induce different, more widely dispersed victim-network activations.The paper characterizes the activation differences using a Gaussian Mixture Model and t-SNE, and describes the adversarial activations as off-distribution.Source: §1 · p. 2
  • Fine-tuning a victim against one adversary protects against that adversary, but repeating the attack finds a new vulnerability involving physical interference.The authors suggest repeated fine-tuning may protect against a range of adversaries.Source: §1 · p. 2
  • The authors propose adversarial policies as tools for uncovering unexpected policy failures and as a possible complement to conventional self-play for improving robustness.Their rationale is that adversaries can exploit weaknesses undiscovered by the similar opponents encountered during self-play.Source: §1 · p. 2

2 RELATED WORK

This work places adversarial policies in a broader view of adversarial inputs while distinguishing its physically realistic, multi-agent threat model from prior observation-perturbation approaches. It also builds on prior work in adversarial training and worst-case RL while targeting complex environments beyond the practical reach of formal methods.

2 RELATED WORK

  • Adversarial examples include intentionally designed inputs that cause mistakes, not only visually imperceptible image perturbations, following Goodfellow et al. (2017).Gilmer et al. (2018a) and Uesato et al. (2018) argue that attackers can go beyond small perturbations and that the ℓp model approximates worst-case risk locally.Source: §2 · p. 2
  • Prior RL adversarial-example work assumes ℓp-norm threats that directly perturb image observations, whereas this work disallows direct modification of the victim’s observations.Huang et al. (2017), Kos and Song (2017), and Lin et al. (2017) demonstrated vulnerability to small image-observation perturbations.Source: §2 · p. 2
  • Unlike autonomous-car adversarial settings, this work focuses on zero-sum games; its victims beat a range of opponents and are not tightly coupled to training partners.Behzadan and Munir (2019) used a similar threat model, while Lanctot et al. (2017) found agents can fail against new opponents after becoming tightly coupled to those used in training.Source: §2 · p. 3
  • Prior adversarial training improves deep RL robustness against force-vector or dynamics-parameter perturbations, and this work’s victim fine-tuning against the adversary is inspired by it.The cited dynamics parameters include friction.Source: §2 · p. 3
  • Worst-case RL and verification methods address adversarial transitions, controller properties, and disturbances, but are practical only in simple, low-dimensional settings; this method tests complex multi-agent tasks without formal guarantees.The authors state their method naturally scales with improvements in RL algorithms.Source: §2 · p. 3

3 FRAMEWORK

The framework models an attacker controlling an opponent in a two-player Markov game against a fixed stochastic victim. Because the victim is a black box, the attacker’s problem reduces to an MDP with unknown dynamics that must be solved through reinforcement learning.

3 FRAMEWORK

  • The game is a two-player Markov game with joint state transitions and rewards depending on both players’ actions, and each player maximizes discounted reward.The players are denoted adversary α and victim ν.Source: §3 · p. 3
  • The attacker controls the opponent and has unlimited black-box access to victim actions, but no white-box access; the victim follows a fixed stochastic policy.The paper describes a fixed victim as realistic for pre-trained models deployed with static weights.Source: §3 · p. 3
  • Fixing πν reduces the game to attacker MDP Mα, which embeds victim actions sampled from πν; its unknown dynamics mean the attacker must use RL to maximize discounted reward.The attacker must learn because the victim policy is a black box, even if the Markov game’s dynamics are known.Source: §3 · p. 3

4 FINDING ADVERSARIAL POLICIES

This section describes adversarial policies in zero-sum simulated robotics games, their training and baselines, and quantitative and qualitative evaluations against self-play-trained victims.

4.2 METHODS EVALUATED

  • Adversaries optimize the win/loss/tie objective with PPO (Schulman et al., 2017) for 20 million timesteps, compared with random, zero-control, and pretrained Zoo policies.They receive sparse end-of-episode rewards, positive for a win and negative for a loss or tie.Source: §4.2 · p. 4

4.3 RESULTS

  • Adversarial policies reliably beat most victims and outperform the Zoo baseline for a majority of environments and victims; against median victims, they outperform Zoo in Kick and Defend and You Shall Not Pass.They are competitive with Zoo in Sumo Humans against the median victim.Source: §4.3 · pp. 4–5, Figure 3 · p. 5
  • In Kick and Defend and You Shall Not Pass, adversaries never stand up; they position their bodies to induce observations that make victims take poor actions.The paper notes that a robust victim could easily win, as demonstrated in Section 5.1.Source: §4.3 · p. 5
  • In Sumo Humans, the kneeling adversary wins 88% against victim 1, but only 62% against victim 2 and 45% against victim 3, below Zoo baselines.It kneels in the center in a stable position; 88% of victim 1’s attempts to tackle it end with that victim falling.Source: §4.3 · p. 5
  • Random and zero-action baselines win 30%–50% in Kick and Defend but under 1% in Sumo and You Shall Not Pass, below adversarial policies; victims are robust to non-adversarial off-distribution observations.Source: §4.3 · p. 5

5 UNDERSTANDING ADVERSARIAL POLICIES

The section explains how adversarial policies exploit victims, how observation dimensionality relates to vulnerability, and how adversarial opponents alter victim-network activations. Its analyses examine masked observations, Sumo environments, and activation distributions.

5.1 MASKED POLICIES

  • Against Adv1, masked ZooMV1 won 99% while normal ZooV1 lost 86%; this supports adversarial observations, rather than physical interference, as the attack mechanism.Against normal ZooO1, ZooMV1 lost 78% while ZooV1 lost 47%, showing that masking harms performance against normal opponents but helps against an adversary.Source: Figure 4 · p. 6, §5.1 · p. 7

5.3 VICTIM ACTIVATIONS

  • In t-SNE, Adv, Rand, and ZooO2 activations separated clearly, with Adv more dispersed than Rand and Rand more dispersed than ZooO2.This visualization analyzed Kick and Defend victim ZooV2 activations.Source: Figure 5 · p. 6, §5.3 · p. 7

5.1 MASKED POLICIES

  • Masked victims and adversaries exhibit non-transitive relationships, motivating caution with self-play methods that assume transitivity and favoring more general methods where practical (Balduzzi et al., 2019; Brown et al., 2019).The findings also suggest a trade-off: broader observations improve performance in benign environments but increase vulnerability to adversaries.Source: §5.1 · p. 7

5.2 DIMENSIONALITY

  • In Sumo, the adversary had a much lower win rate against low-dimensional Ants (dim P = 15) than against higher-dimensional Humans (dim P = 24), supporting greater vulnerability with higher-dimensional adversary-controlled observations.P denotes the adversary’s joint positions.Source: §5.2 · p. 7

5.3 VICTIM ACTIVATIONS

  • GMM analysis found that Adv induced the lowest-likelihood victim activations, with Rand only slightly more probable; normal opponents were nearly as likely as validation activations except in Sumo Humans.The model was fit on activations from play against a normal opponent and evaluated on held-out validation activations.Source: §5.3 · p. 7

6 DEFENDING AGAINST ADVERSARIAL POLICIES

Fine-tuning victim policies against adversaries improves robustness to the original attack, but leaves a trade-off against normal opponents and does not prevent newly trained adversaries from attacking successfully.

6 DEFENDING AGAINST ADVERSARIAL POLICIES

  • Single training fine-tunes the victim zoo policy against an adversary, while dual training randomly selects an adversary or zoo policy at each episode’s start.The rest of the training procedure is the same as for adversaries in Section 4.2.Source: §6 · p. 8
  • Fine-tuning cut Adv1’s win rate from 87% to around 10%, but ZooO1 won 57% against dual ZooDV1 versus 48% against original ZooV1.Single-trained ZooSV1 catastrophically forgot how to play against ZooO1.Source: §6 · p. 8
  • New adversaries AdvS1 and AdvD1 trained against fine-tuned victims won at equal or greater rates than before and transferred successfully to original ZooV1.AdvS1 and AdvD1 trip victims up by lying prone, unlike Adv1, which caused ZooV1 to fall without touching it.Source: §6 · p. 8

7 DISCUSSION

The discussion argues that adversarial policies expose hard-to-find failures in self-play-trained RL agents by confusing victims with natural observations. It frames adversarial training as a way to test exploitability and reports that fine-tuning can change, but not eliminate, the attack.

Contributions

  • Adversarial policies create natural adversarial observations, exist against robust self-play victims, and win by inducing off-distribution activations that improve when victims cannot see them.These are the paper’s three stated contributions.Source: §7 · p. 8

Self-play

  • Even apparently strong self-play policies can harbor serious, hard-to-find failure modes, making self-play’s known theoretical limitations practically relevant and motivating careful testing.The authors note that deep-RL self-play may fail to converge or converge only to a local rather than global Nash.Source: §7 · p. 8

Exploitability testing

  • Training an adversary constructively lower-bounds victim exploitability; declining win rates show the victim and its self-play opponent were not at a global Nash, and attacks may also succeed from outside a local Nash basin.The authors expect this because adversary training starts from a random point likely outside the victim’s attractive basin.Source: §7 · p. 8

Defense

  • Fine-tuning the victim against an adversary does not eliminate attacks: a reapplied attack can beat the defense, though it must trip up rather than confuse the victim.The authors propose scaling the defense with population-based training that continually adds diverse agents and has them train against fixed opponents for prolonged periods to avoid local equilibria.Source: §7 · p. 8

Conclusion

  • The authors characterize adversarial policies as a previously unrecognized deep-RL problem and call for further research into policy robustness, security, and understanding.Videos and supplementary material are available online, and the source code is available on GitHub.Source: §7 · p. 8

A TRAINING: HYPERPARAMETERS AND COMPUTATIONAL INFRASTRUCTURE

Training hyperparameters were selected through performance checks and a random search, with the same settings retained for defense fine-tuning. Training an adversary for one victim took around 8 hours on four CPU cores.

A TRAINING: HYPERPARAMETERS AND COMPUTATIONAL INFRASTRUCTURE

  • Hyperparameters were chosen using diminishing-return checks and a 100-sample random search on Kick and Defend and Sumo Humans; remaining settings used Stable Baselines PPO2 defaults.The random search tuned batch size, mini-batches, epochs per update, entropy coefficient, and learning rate; Table A.1 specifies the training hyperparameters.Source: A TRAINING: HYPERPARAMETERS AND… · p. 12, Table A.1 · p. 12
  • The defense experiments reused the same hyperparameters because a repeated sweep for fine-tuning victim policies produced similar results.Source: A TRAINING: HYPERPARAMETERS AND… · p. 12
  • Training an adversary for a single victim took around 8 hours using 4 cores of an Intel Xeon Platinum 8000 (Skylake) processor.Experiments used a mixture of in-house and cloud infrastructure.Source: A TRAINING: HYPERPARAMETERS AND… · p. 12

B ACTIVATION ANALYSIS: T-SNE AND GMM

The analysis represents the victim’s policy-network activations as a concatenated vector for Gaussian Mixture Model and t-SNE analysis.

B ACTIVATION ANALYSIS: T-SNE AND GMM

B.1 T-SNE HYPERPARAMETER SELECTION · B.2 GAUSSIAN MIXTURE MODEL HYPERPARAMETER SELECTION

The appendix chose t-SNE perplexity 250 for qualitative visualization clarity and compared GMM settings using BIC and heldout validation log-likelihood; 20 full-covariance components performed best in most environment-victim pairs.

B.1 T-SNE HYPERPARAMETER SELECTION

  • Perplexity 250 was chosen because it qualitatively produced the clearest visualization with a moderate number of distinct clusters.Source: §B.1 · p. 12

B.2 GAUSSIAN MIXTURE MODEL HYPERPARAMETER SELECTION

  • Across most environment-victim pairs, 20 full-covariance components achieved the lowest BIC and highest heldout validation log-likelihood, and ranked second in the remainder.The criteria for selecting hyperparameters were BIC and average log-likelihood on a heldout validation set.Source: §B.2 · p. 12

C FIGURES

The supplementary figures show adversary training win-rate patterns across environments and a clear separation between victim activations under adversarial versus normal opponents.

C FIGURES

  • Adv exceeds baseline win rates against most victims in Kick and Defend and You Shall Not Pass, is competitive in Sumo Humans, but performs poorly in low-dimensional Sumo Ants.The plotted Adv line summarizes five random seeds with a rolling average over 100,000 timesteps; dashed lines show baselines.Source: Figure C.1 · p. 13
  • The figure reports episode outcomes across victim-opponent pairs over 1,000 episodes, highlighting each row’s maximal opponent win rate in red.It includes baseline opponents Rand and Zero and defended victims fine-tuned against single opponents, dual opponents, or masked observations.Source: Figure C.2 · p. 14
  • Victim activations induced by Adv clearly separate from those induced by normal Zoo opponents; a companion figure decomposes the visualization by individual opponents.The t-SNE plots use activations from 5,000 timesteps per opponent and perplexity 250.Source: Figure C.3 · p. 15, Figure C.4 · p. 16

Related papers

Loading 1905.10615v3…