PaperlayerView exact revision on arXiv

Source-linked AI summary

BACKDOORL: Backdoor Attack against Competitive Reinforcement Learning

Lun Wang, Zaynah Javed, Xian Wu, Wenbo Guo, Xinyu Xing, Dawn Song

arXiv:2105.00579v3cs.CRcs.LG

TL;DR

Existing RL backdoors require direct observation manipulation, limiting them to simpler environments. BACKDOORL uses another agent’s action as a trigger in two-player competitive RL; across four environments, triggering it lowers the victim’s winning rate by 17% to 37%.

  • Problem

    Existing RL backdoor attacks require direct manipulation of an agent’s observation, limiting their application to simple tasks such as Atari games.

    Source: §1 · p. 1

  • Method

    BACKDOORL trains a fast-failing policy and uses imitation learning to embed normal and backdoor behavior in a victim, with another agent’s environment action as the trigger.

    Source: §1 · pp. 1–2

  • Results

    17% to 37%: the victim’s winning rate drops by this amount when the backdoor is activated across four competitive environments.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The paper concludes that adversarial input actions can insert a backdoor into deep RL and force an opponent to fail, including in safety-critical machine-learning and robotic systems.

    Source: §7 · p. 6

    Fine-tuning can only weaken, not completely remove, the backdoor, and may require information about the trigger behavior.

    Source: §6 · p. 5

Abstract

from arXiv · show

Recent research has confirmed the feasibility of backdoor attacks in deep reinforcement learning (RL) systems. However, the existing attacks require the ability to arbitrarily modify an agent's observation, constraining the application scope to simple RL systems such as Atari games. In this paper, we migrate backdoor attacks to more complex RL systems involving multiple agents and explore the possibility of triggering the backdoor without directly manipulating the agent's observation. As a proof of concept, we demonstrate that an adversary agent can trigger the backdoor of the victim agent with its own action in two-player competitive RL systems. We prototype and evaluate BACKDOORL in four competitive environments. The results show that when the backdoor is activated, the winning rate of the victim drops by 17% to 37% compared to when not activated.

1 Introduction

BACKDOORL extends RL backdoors to two-player competitive settings, where an adversary triggers the victim through interaction rather than direct observation manipulation. The paper reports a 17% to 37% drop in victim winning rate when triggered and finds that fine-tuning does not fully remove the backdoor.

1 Introduction

  • The method trains a fast-failing agent and uses adversarial policy training, reward manipulation, and imitation learning to embed normal and backdoor behavior in the victim.The imitation trajectories mix episodes of an expert against a trigger-less agent with episodes of a fast-failing agent against the trigger agent.Source: §1 · pp. 1–2
  • 17% to 37%: the victim’s winning rate drops by this amount when the backdoor is triggered.Without activation, victim performance is comparable to policies without backdoors.Source: §1 · p. 1
  • Fine-tuning can weaken BACKDOORL but cannot completely remove its backdoor functionality.The authors also report that further fine-tuning does not improve winning rate and may require information about the trigger behavior.Source: §1 · pp. 1–2
  • BACKDOORL targets two-player competitive RL and uses another agent’s in-environment action, rather than direct observation modification, as the trigger.Existing RL backdoor attacks require direct observation manipulation and are limited to tasks with tractable environments such as Atari games.Source: §1 · pp. 1–2
  • The authors evaluate BACKDOORL in four competitive environments and report that the attack validates the feasibility of backdoors in competitive RL.The systems motivating the work include safety-critical applications such as autonomous driving and automated trading.Source: §1 · pp. 1–2

2 Related Work

Prior work established backdoors across several machine-learning settings, including RL, while existing RL attacks often rely on observation or environment manipulation. Most surveyed detection and defense approaches target image classifiers, leaving competitive RL defenses as future work.

2 Related Work

  • Backdoor research spans image classifiers, NLP, GANs, distributed and federated learning, and variants such as clean-label and hidden-trigger attacks.The cited work includes [Gu et al., 2017], [Chen et al., 2020], and [Salem et al., 2020b].Source: §2 · p. 2
  • Prior RL backdoors manipulate the environment or observation, with attacks covering mazes, Atari, and traffic congestion control.The traffic-control attack uses acceleration or deceleration and depends on established traffic-physics principles; its victim does not interact with vehicles.Source: §2 · p. 2
  • Prior defenses include activation clustering, fine-pruning, optimization-based detection, norm clipping, weak differential privacy, and GAN-based backdoor removal.The surveyed defenses were designed mostly for image classifiers; migrating them to competitive RL is presented as future work.Source: §2 · p. 2
  • Adversarial RL methods in [Huang et al., 2017; Gleave et al., 2020] provide related approaches, and the paper adapts [Gleave et al., 2020] to train its fast-failing agent.The text also points to [Lin et al., 2017] for a survey of adversarial attacks in RL.Source: §2 · p. 2

3 Background

The background formalizes competitive RL as a two-player Markov process and describes imitation learning as a way to train a student from expert behavior. It introduces behavioral cloning as supervised imitation and situates the BACKDOORL workflow across fast-failing-agent training and imitation learning.

3.1 Competitive Reinforcement Learning

  • A two-player Markov decision process specifies a state space, two agents’ action spaces, a transition function, and reward functions for both agents.When one agent’s policy is fixed, the process reduces to a single-agent MDP whose transition and reward functions incorporate that policy.Source: §3.1 · p. 2

3.2 Imitation Learning

  • Imitation learning trains a student by having it imitate behavior demonstrated by an expert.Behavioral cloning uses supervised learning to directly clone the expert’s behavior.Source: §3.2 · p. 2
  • Behavioral cloning learns a policy that mimics the expert’s decisions by minimizing a loss between predicted and expert actions.The loss may use a distance function such as MSE or cross-entropy.Source: §3.2 · pp. 2–3
  • Figure 1 connects fast-failing-agent training to imitation learning through trajectory generation.The figure labels these processes x and y, respectively.Source: Figure 1 · p. 3

4 Methodology

BACKDOORL trains a fast-failing policy and embeds its behavior in a victim policy that normally seeks to win but switches after a trigger. The method uses reward minimization and behavioral cloning to create the attack.

4 Methodology

  • The methodology trains a fast-failing policy and inserts its behavior into a victim policy.The section introduces the overview and notation before describing fast-failing-agent training and backdoor insertion.Source: §4 · p. 3

4.1 Overview & Threat Model

  • BACKDOORL combines a winning policy with a fast-failing policy, activated by a series of trigger actions.The victim acts normally while the backdoor is dormant and tries to fail as soon as possible after activation.Source: §4.1 · p. 3
  • The user can detect anomalous program structures but cannot find a backdoor hidden in neural-network parameters.The developer can deviate from normal training to implant the backdoor in the output policy.Source: §4.1 · p. 3

4.2 Train the Fast-failing Agent

  • Minimizing accumulated reward, with a constant per-step penalty, trains a fast-failing agent without hand-designed failure behavior.The paper notes that simply staying still can slow failure by blocking an opponent’s path, as in Run to Goal (Ants).Source: §4.2 · p. 3

4.3 Mimic the Hardcoded Agent

  • Behavioral cloning teaches an LSTM victim from normal episodes and triggered episodes that encode activation followed by failure.The trigger indicator is excluded from trajectory states, so the victim must recognize the trigger’s effect on its observations and retain the triggered state after the trigger disappears.Source: §4.3 · p. 4

5 Evaluation

The evaluation tests BACKDOORL across four competitive environments, examining fast-failing agent training, backdoor activation, untriggered performance, and hyperparameter effects. With the trigger, the victim’s winning rate drops; without it, performance remains comparable to a normal agent.

5 Evaluation

  • The evaluation covers four competitive environments and tests whether fast-failing agents fail faster than passive dummies, whether the trigger activates the backdoor, and how performance changes without the trigger.It also examines hyperparameter effects.Source: §5 · p. 4, §5.1 · p. 4

5.1 Experiment Setup

  • The tested games include Run to Goal, You Shall Not Pass, and Sumo, with distinct race, runner-blocker, and contact-based win conditions.The described agents include a 17-joint Humanoid and an 8-joint Ant; the victim policy uses a two-layer LSTM with MLP feature extraction.Source: §5.1 · p. 4

5.2 Evaluation Results

  • The fast-failing agent saves 12.2% to 51.3% of the steps to failure compared with dummy agents, relieving pressure to memorize the trigger.Adversarial training typically converges in 40 to 60 epochs; sampled victim episodes generally last 40–100 timesteps, most roughly 60–70.Source: §5.2 · pp. 4–5
  • The triggered victim’s failing rate rises by 4% to 33% and its winning rate drops by 17% to 37%; without the trigger, its performance is comparable to a normal agent.The victim learns the backdoor functionality after around 150 epochs.Source: §5.2 · p. 5
  • Staying still serves as the trigger in all four environments; it is chosen to balance visibility against similarity to normal actions.In Sumo (Humans), the trigger is delayed because the game requires contact to avoid a tie; longer triggers can raise win rate but are less stealthy, while higher poisoning rates are more harmful but less stealthy.Source: §5.2 · p. 5

6 Defenses

The authors test fine-tuning as a defense by retraining victim agents on normal episodes. It improves winning rates in three environments but does not fully remove the backdoor, and may reduce performance.

6 Defenses

  • Winning rates improve by 6.8% to 15.2% in 3 out of 4 environments after fine-tuning.The victim agents were retrained on 1000 normal episodes for 200 epochs and tested for 1000 games per environment.Source: §6 · p. 5
  • Run to Goal (Humans) instead has a 4.9% drop in winning rate after fine-tuning.The authors conjecture that over-fitting during fine-tuning degrades performance.Source: §6 · p. 5
  • Another 500 fine-tuning epochs do not further improve winning rate, which stays roughly the same within 1%.The authors conjecture that normal trajectories seldom include the trigger action, so fine-tuning may enhance the winning policy without making the agent forget the failing policy; it requires information about trigger behavior and may reduce performance.Source: §6 · p. 5
  • Migrating image-classifier defenses to BACKDOORL and validating their effectiveness remains future work.Source: §6 · pp. 5–6

7 Conclusion

The authors conclude that an adversary can use malicious input actions to trigger a backdoor in deep reinforcement learning and force an opponent to fail.

7 Conclusion

  • Malicious input actions can trigger a backdoor in deep reinforcement learning and force an opponent to fail.Source: §7 · p. 6

Related papers

Loading 2105.00579v3…