PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Deep Reinforcement Learning through Adversarial Loss

Tuomas Oikarinen, Wang Zhang, Alexandre Megretski, Luca Daniel, Tsui-Wei Weng

arXiv:2008.01976v2cs.LGcs.AIcs.CRstat.ML

TL;DR

Deep RL agents are vulnerable to small adversarial input perturbations, creating concerns for deployment in safety-critical settings. RADIAL-RL trains agents with adversarial losses based on robustness verification bounds. The authors report strong performance against larger perturbations with greater training efficiency, and propose GWC as a surrogate for worst-case reward.

  • Problem

    Deep RL agents can suffer arbitrarily bad performance under adversarial perturbations, a concern given their use in safety-critical applications.

    Source: §1 · p. 1

  • Method

    RADIAL-RL trains agents with adversarial losses designed using neural-network robustness verification bounds and is demonstrated with DQN, A3C, and PPO.

    Source: §1 · p. 2, §3 · pp. 4–5

  • Results

    RADIAL agents achieved very good performance against 5× larger perturbations than prior state of the art and were more computationally efficient to train.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Greedy Worst-Case Reward is presented as an efficient surrogate for evaluating deep RL agents under adversarial input perturbations.

    Source: §5 · p. 10

    The method addresses ℓp-norm-bounded perturbations, not robustness to changing environment conditions.

    Source: §6 · p. 10

Abstract

from arXiv · show

Recent studies have shown that deep reinforcement learning agents are vulnerable to small adversarial perturbations on the agent's inputs, which raises concerns about deploying such agents in the real world. To address this issue, we propose RADIAL-RL, a principled framework to train reinforcement learning agents with improved robustness against $l_p$-norm bounded adversarial attacks. Our framework is compatible with popular deep reinforcement learning algorithms and we demonstrate its performance with deep Q-learning, A3C and PPO. We experiment on three deep RL benchmarks (Atari, MuJoCo and ProcGen) to show the effectiveness of our robust training algorithm. Our RADIAL-RL agents consistently outperform prior methods when tested against attacks of varying strength and are more computationally efficient to train. In addition, we propose a new evaluation method called Greedy Worst-Case Reward (GWC) to measure attack agnostic robustness of deep RL agents. We show that GWC can be evaluated efficiently and is a good estimate of the reward under the worst possible sequence of adversarial attacks. All code used for our experiments is available at https://github.com/tuomaso/radial_rl_v2.

1 Introduction

Deep RL agents can suffer severe performance losses from adversarial perturbations, yet most defenses target classification rather than RL. RADIAL-RL uses adversarial losses based on verification bounds, and its reported contributions span multiple RL algorithms, benchmarks, and efficient worst-case evaluation.

Motivation

  • Small adversarial perturbations can cause deep RL agents to perform arbitrarily badly, raising concerns for safety-critical uses such as self-driving cars and robotics.The attacks described affect observations and/or actions.Source: §1 · p. 1

Prior defenses

  • Prior heuristic defenses for image classification can fail against stronger attacks; one cited study broke 13 such methods.Verification-based approaches instead use bounds in the loss to improve certified accuracy, even when the verifier produces loose bounds for nominal models.Source: §1 · pp. 1–2

Research gap

  • Most defense algorithms target classification, while deep RL poses additional challenges including credit assignment and a non-stationary training set.The paper presents RADIAL-RL to address this gap using adversarial loss functions based on robustness verification bounds.Source: §1 · p. 2

Contributions

  • RADIAL-RL applies to different deep RL algorithms; the authors report better robustness to up to 5× stronger perturbations and 2−10× greater training efficiency than prior methods.The reported demonstrations include DQN, A3C, and PPO, with Atari, MuJoCo, and ProcGen experiments.Source: §1 · p. 2

2 Related work and background

The background distinguishes observation-space perturbations from other RL threat models and reviews robust-training approaches that use verification bounds. It then introduces MDPs and the DQN, A3C, and PPO algorithm basics relevant to the paper.

2.1 Adversarial attacks in Deep RL

  • The paper focuses on ℓp-norm perturbations to agents’ observations, while noting that other RL threats include action perturbations, environment mis-specification, and adversarial agents.The framework is described as extensible to semantic changes such as rotations and color or brightness changes.Source: §2.1 · p. 3

2.2 Formal verification and robust training for Deep RL

  • Prior RL work includes fixed-network certification and adversarial training, while RADIAL uses verification bounds to discourage potentially overlapping actions.The paper reports that prior adversarial-training approaches can cost more than standard training and that RADIAL outperforms cited methods while training 2–10× more efficiently than.Source: §2.2 · p. 3

2.3 Basics of Deep Reinforcement Learning

  • An MDP specifies states, actions, transition probabilities, rewards, an initial-state distribution, and a discount factor for characterizing an environment.The passages also identify timestep reward, action, and state as components of an episode.Source: §2.3 · p. 3
  • DQN uses neural networks to approximate an action-value function, with Double-DQN and Dueling-DQN described as advanced variants used by robust-training baselines.Double-DQN uses two Q-networks, while Dueling-DQN separates state value and action advantage.Source: §2.3 · pp. 3–4
  • A3C learns policy and state-value functions, whereas PPO constrains policy updates by clipping the ratio of new to old action probabilities.The A3C objective includes value, policy, and entropy terms; PPO also includes value-loss and policy-entropy terms.Source: §2.3 · p. 4

3 A Robust Deep RL framework with adversarial loss

RADIAL-RL combines nominal and adversarial losses, using neural-network robustness bounds to train deep RL agents against bounded input perturbations. It adapts the framework to DQN, A3C, and PPO, and proposes GWC as an efficient estimate of worst-case reward.

3 A Robust Deep RL framework with adversarial loss

  • RADIAL-RL blends nominal and adversarial losses and uses verified network output bounds to construct adversarial losses for deep RL agents.The trade-off parameter κ controls standard versus robust performance; the two approaches bound perturbed loss or reduce overlap between action-output bounds.Source: §3 · pp. 4–5

3.1 RADIAL-DQN

  • DQN Approach #2 penalizes weighted overlap between action Q-value bounds, giving greater weight to overlap between actions with larger Q-value differences.A margin η = 0.5 · Qdiff encourages separation beyond merely eliminating overlap.Source: §3.1 · p. 5

3.2 RADIAL-A3C

  • A3C adapts Approach #2 to the policy network, replacing DQN’s Q-values with policy outputs and the penultimate-layer activations.The value network and entropy remain unperturbed because they are used to aid training.Source: §3.2 · pp. 5–6

3.3 RADIAL-PPO

  • PPO uses Approach #1 because Approach #2 does not work for continuous actions, bounding perturbed loss; Gaussian policies bound distance, while discrete policies bound action probabilities from logits.Optimizing the loss increases lower bounds on probabilities of good actions and decreases upper bounds on probabilities of bad actions.Source: §3.3 · p. 6

3.4 New efficient evaluation metric: Greedy Worst-Case Reward

  • GWC greedily selects the worst plausible action at each state to approximate worst-case reward in linear time, and is often close to AWC.AWC is a lower bound on worst-case reward when both policy and environment are deterministic.Source: §3.4 · pp. 6–7

4 Experimental results

The experiments compare RADIAL agents with baselines across Atari, ProcGen, and MuJoCo, and assess generalization and GWC alongside performance under attack.

4 Experimental results

  • Experiments cover pixel-input Atari and ProcGen tasks with discrete actions and MuJoCo continuous control, measuring attacked reward, GWC, and ACR.A3C and PPO policies were evaluated deterministically by choosing the highest-probability action.Source: §4 · p. 7

4.1 Atari results

  • The A3C baseline performed well against small PGD attacks without robust training, which the authors attribute to Max-pooling and stochastic training.RADIAL-A3C beat baselines on 2/4 tasks and outperformed RADIAL-DQN against ε = 5/255 PGD attacks on RoadRunner.Source: §4.1 · pp. 7–8
  • RADIAL-DQN outperformed or matched all baselines on all four Atari games against ε = 1/255 PGD attacks.RADIAL-DQN was roughly 6× faster to train than.Source: §4.1 · p. 7, Table 1 · p. 8
  • Untargeted PGD sometimes increased reward because changing an agent’s actions can produce a higher-reward trajectory.The authors identify this as a weakness of untargeted PGD attacks on RL policies.Source: §4.1 · p. 8

4.2 ProcGen Results

  • RADIAL training increased robustness on both ProcGen training and evaluation sets, often with a smaller train/evaluation gap for RADIAL-PPO.Agents trained on 200 levels and were evaluated on the full distribution.Source: §4.2 · pp. 8–9, Table 2 · p. 9

4.3 MuJoCo Results

  • RADIAL-PPO outperformed SA-PPO on Walker2D and Hopper in natural and robust testing; Half-Cheetah was naturally robust against attacks up to ε = 0.075.Agents were trained for 4096k steps and evaluated for 50 episodes.Source: Table 3 · p. 9, §4.3 · p. 9

4.4 Evaluating GWC

  • GWC matched AWC perfectly at ε ∈ {1.0, 1.2, 1.5} and matched 37/40 episodes at 1.3 and 1.4, overestimating on three.GWC was a better indicator of robust performance than ACR on Freeway and Pong.Source: §4.4 · p. 10

5 Conclusions and Future works

RADIAL improves deep RL robustness against adversarial perturbations while using less training computation, and the paper introduces GWC as a surrogate for worst-case reward.

5 Conclusions and Future works

  • RADIAL-trained agents perform well against perturbations up to 5× larger than prior state of the art while requiring less training computation.The authors identify extending the framework to semantic perturbations as future work.Source: §5 · p. 10
  • Greedy Worst-Case Reward (GWC) is presented as a good surrogate for evaluating deep RL agents’ reward under adversarial input perturbations.The paper describes GWC as an evaluation method for worst-case reward.Source: §5 · p. 10
  • Future work includes extending RADIAL to semantic perturbations such as contrast and brightness changes.The paper names vision-based self-driving cars and robots as examples of relevant settings.Source: §5 · p. 10

6 Limitations and Potential negative impact

The paper limits its robustness scope to ℓp-norm-bounded perturbations; its appendix describes adversarial-loss constructions for DQN and A3C.

6 Limitations and Potential negative impact

  • RADIAL addresses robustness to ℓp-norm-bounded perturbations, not other axes such as changing environment conditions.The authors identify semantic perturbations as a possible extension.Source: §6 · p. 10

A Appendix: Radial-DQN

  • The DQN appendix constructs an upper bound on perturbed nominal loss, with B terms bounding DQN loss and C terms tightening bounds for actions not taken.The construction reduces to the original loss when ϵ = 0.Source: A Appendix: Radial-DQN · p. 13

B Appendix: Radial-A3C

  • The A3C appendix defines Ladv as an upper bound on the original loss under worst-case input perturbations, reducing to Lstandard when ϵ = 0.The policy-output bound π(at) is computed using an upper bound on the selected logit and lower bounds on other logits.Source: B Appendix: Radial-A3C · p. 13

C On Approach #2 for continuous actions

Approach #2 is designed around discrete actions; extending it to continuous actions raises integration and overlap-definition problems.

C On Approach #2 for continuous actions

  • Approach #2 relies on a discrete action space because it minimizes overlap between the chosen action’s lower bound and other actions’ upper bounds.Replacing the sum with an integral over other actions is problematic because the chosen action is a single point and would overlap with itself.Source: C On Approach #2 for continuous actions · p. 13
  • For continuous actions, the proposed integral is generally infeasible and would require approximation by uniform Monte Carlo sampling.Source: C On Approach #2 for continuous actions · p. 13
  • Continuous-control outputs lack action-specific bounds for defining overlap, and the authors favor a loss designed explicitly for continuous control.They describe a possible overlap expression based on action probabilities but say the overlap term remains challenging to define.Source: C On Approach #2 for continuous actions · p. 13

D Appendix: Discussion on GWC and the metrics used in [17, 18]

The appendix explains why action-stability metrics may not capture reward and discusses comparisons among GWC, AWC, and ACR.

D Appendix: Discussion on GWC and the metrics used in [17, 18]

  • High action-certification rates may not imply high reward because changing one early action can redirect an agent onto a different trajectory.Prior methods discussed here measure action certification or the size of a provable region rather than reward directly.Source: D Appendix: Discussion on GWC and the… · p. 13
  • Figure 3 compares GWC, AWC, and ACR for RADIAL-DQN on Freeway across perturbation sizes, using means over 20 runs and the first 80 steps.The caption defines GWC as the percentage of episodes reaching +1 reward within 80 steps under Greedy Worst-Case Reward, and AWC as the percentage under absolute worst-case calculation.Source: Figure 3 · p. 14, D Appendix: Discussion on GWC and the… · p. 14
  • The absolute worst-case reward procedure expands possible actions from states and accumulated rewards, then returns the minimum reward found.The procedure selects actions whose values meet the maximum among the considered action values, advances to the resulting states, and continues until no states remain open.Source: D Appendix: Discussion on GWC and the… · p. 14

E.1 Atari results

Figure 4 shows SA-DQN overestimating Q-values on Freeway, with greater Q-value bias than RADIAL-DQN. Table 4 covers experiments changing the margin coefficient or using a more symmetric loss.

E.1 Atari results

  • SA-DQN has higher Q-value bias than RADIAL-DQN on Freeway, with error defined as predicted minus ideal Q-value.Ideal Q-value is the cumulative time-discounted reward remaining in the episode.Source: §E.1 · p. 14, Figure 4 · p. 15
  • Figure 4 depicts SA-DQN consistently overestimating its Q-values during a Freeway episode.Source: Figure 4 · p. 15
  • Table 4 reports RADIAL-DQN experiments changing the margin coefficient or using a more symmetric loss.Source: Table 4 · p. 15

F Appendix: Alternative Approach 2 loss formulations

The appendix examines margin choices and a symmetric loss for RADIAL-DQN, then reports a preliminary test of RADIAL-PPO against compounding attacks on MuJoCo.

F.1 Margin

  • The margin is set to c · Qdiff with 0 < c < 1; c must be positive, and the authors used c = 0.5 without tuning because it worked well.A margin equal to Qdiff is impossible when bounds are tight; the authors report c = 0 did not achieve desired robustness.Source: §F.1 · p. 15
  • Margin choice affects robustness and standard performance: c = 0.75 hurt standard performance, while c = 0.25 was less robust than c = 0.5 in BankHeist and much worse in RoadRunner.The authors attribute the c = 0.75 result to a potentially overstrict requirement that collapses the policy to a lower-reward simple policy.Source: §F.1 · p. 15

F.2 Symmetric Loss Function

  • The original adversarial loss can become zero when the lowest-Q action is selected by chance, motivating experiments with a more symmetric loss.Source: §F.2 · pp. 15–16
  • The symmetric loss adds a flipped regularization term to enforce the same regularization when the alternative action has the lower Q-value.Source: §F.2 · p. 16
  • On BankHeist, the symmetric loss slightly reduced nominal performance but slightly improved performance against 5/255 PGD.The experiments also covered RoadRunner; the supplied results passage does not state its outcome.Source: §F.2 · p. 16

G Compounding attack on MuJoCo

  • The compounding attack was tested on RADIAL-PPO models trained for MuJoCo, using a learned dynamics model and simulated forward steps from the policy.Source: G Compounding attack on MuJoCo · p. 16
  • At each test step, the attack perturbs the observation to maximize deviation between the perturbed trajectory and the policy’s simulated target trajectory.Source: G Compounding attack on MuJoCo · p. 16
  • The three-step compounding-attack experiment gives the authors increased confidence that RADIAL-PPO training helps defend against these attacks, though results are preliminary.Attack strength was similar to MAD; the authors note the target function, learned dynamics model, and attack hyperparameters could be improved.Source: G Compounding attack on MuJoCo · p. 16

H.1 Atari training details

The appendix describes Atari training and implementation settings, then gives separate training configurations for Procgen and MuJoCo. It also compares bound-calculation methods on Atari.

H.1 Atari training details

  • Atari DQN and A3C were first trained without robust training and then fine-tuned with RADIAL; DQN used a released standard model for comparison, while A3C used a model trained by the authors.DQN training used 6M standard steps followed by 4.5M RADIAL steps; A3C used 20M standard steps followed by 10M RADIAL steps.Source: §H.1 · p. 17
  • Atari agents used 84x84 grayscale inputs without frame stacking, acted every four frames, and had rewards clipped to [-1, 1].The appendix also specifies DQN and A3C network architectures, optimizers, and hyperparameters.Source: §H.1 · p. 17
  • RADIAL-DQN performed similarly with cheaper IBP bounds and computationally expensive CROWN-IBP, while SA-DQN using IBP worked on Pong but failed on BankHeist.Source: Table 6 · p. 18

H.2 Procgen training details

  • Procgen RADIAL-PPO training followed 2.5M standard steps with 22.5M robust steps, and used κ = 0.5 without tuning.Training from scratch was reported as lower compute cost than fine-tuning; total Procgen compute was estimated at 200–300 GPU hours.Source: §H.2 · p. 18

H.3 MuJoCo training details

  • MuJoCo models used 1,024k standard steps and 3,072k adversarial steps, with κ = 0.5; training took about 1.5 hours per model on an AMD 3700X CPU.RADIAL-PPO compute time was empirically two-thirds that of the CROWN-IBP-based SA-PPO method in this environment.Source: §H.3 · p. 18

I Additional results

Additional results compare bound-calculation algorithms, include RADIAL-DQN Approach #1, and examine Procgen performance across policy types and training-level counts.

I Additional results

  • RADIAL-DQN performed similarly with IBP and CROWN-IBP bounds, while SA-DQN with IBP worked on Pong but failed on BankHeist.Source: I Additional results · p. 18
  • Approach #1 performed well on some Atari games but poorly on others and was outperformed by Approach #2 in every tested game.Table 7 reports mean reward and standard deviation over 20 runs; robust models used ϵ = 1/255.Source: Table 7 · p. 19
  • With 50 Fruitbot training levels, training performance was best; 200 levels maximized evaluation performance, and RADIAL-PPO evaluation performance was competitive with PPO except at 10 levels.Source: I Additional results · p. 19, Figure 5 · p. 20
  • Procgen stochastic-policy results were similar to deterministic-policy results, and Figure 6 presents the stochastic Fruitbot comparison.The table caption states that stochastic evaluation improved CoinRun and Jumper results and worsened Fruitbot results for both agents.Source: Table 8 · p. 20, Figure 6 · p. 20

Related papers

Loading 2008.01976v2…