PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations

Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, Cho-Jui Hsieh

arXiv:2003.08938v7cs.LGstat.ML

TL;DR

DRL agents can make unsafe or suboptimal decisions when sensor errors or adversarial perturbations distort state observations, while naive adversarial training is often ineffective. The paper formulates this problem as an SA-MDP and derives a policy regularizer applicable across DRL algorithms. Across 10 environments, the method significantly improves robustness under strong white-box attacks, with natural performance also improving in some settings.

  • Problem

    DRL state observations may contain natural or adversarial errors that mislead agents, while existing robustness methods have limited success and lack theoretical principles.

  • Method

    The paper introduces the state-adversarial Markov decision process and a policy regularizer based on divergences between policies under perturbed observations.

  • Results

    The proposed method significantly improves PPO, DDPG, and DQN robustness under strong white-box attacks across 10 environments.

  • Takeaways & Limitations

    A robust policy can also achieve natural performance similar to or better than vanilla PPO in several environments.

  • Takeaways & Limitations

    Experiments were conducted in simulated environments, so benefits in real-world settings remain a supported expectation rather than an evaluated result.

Abstract

from arXiv · show

A deep reinforcement learning (DRL) agent observes its states through observations, which may contain natural measurement errors or adversarial noises. Since the observations deviate from the true states, they can mislead the agent into making suboptimal actions. Several works have shown this vulnerability via adversarial attacks, but existing approaches on improving the robustness of DRL under this setting have limited success and lack for theoretical principles. We show that naively applying existing techniques on improving robustness for classification tasks, like adversarial training, is ineffective for many RL tasks. We propose the state-adversarial Markov decision process (SA-MDP) to study the fundamental properties of this problem, and develop a theoretically principled policy regularization which can be applied to a large family of DRL algorithms, including proximal policy optimization (PPO), deep deterministic policy gradient (DDPG) and deep Q networks (DQN), for both discrete and continuous action control problems. We significantly improve the robustness of PPO, DDPG and DQN agents under a suite of strong white box adversarial attacks, including new attacks of our own. Additionally, we find that a robust policy noticeably improves DRL performance even without an adversary in a number of environments. Our code is available at https://github.com/chenhongge/StateAdvDRL.

1 Introduction

The paper studies DRL policies that must act from uncertain or adversarially perturbed state observations, where ignoring observation uncertainty can cause unsafe actions. It argues that naive adversarial training is often ineffective and introduces SA-MDP theory and a principled robust policy regularizer.

  • Motivation: DRL agents can be misled by sensor errors or adversarial perturbations because observed states differ from unchanged true environment states.The paper motivates this setting with safety-critical failures such as autonomous navigation crashes.
  • Limitations of Existing Defenses: Naive adversarial training can destabilize training, reduce performance, or fail to improve robustness under strong attacks in many environments.The approach generates adversarial trajectories during training and inserts adversarial states into the replay buffer.
  • Contributions: SA-MDP formulates state-observation perturbations as a modified Markov decision process and studies their fundamental properties.The paper reports that an optimal adversary may prevent a stationary and Markovian optimal policy from existing.
  • Contributions: The proposed policy regularizer is related to total variation distance or KL-divergence between policies under perturbed observations.It is designed for application to PPO, DDPG, and DQN across discrete and continuous action problems.
  • Contributions: Experiments cover 10 environments and report significantly improved robustness for PPO, DDPG, and DQN under strong white-box state-observation attacks.The evaluation includes the proposed robust Sarsa and maximal action difference attacks.

2 Related Work

Related work addresses robustness through transition uncertainty, adversarial multi-agent interaction, state-observation attacks, and direct adversarial training. The paper distinguishes its approach by manipulating only state observations, developing principles first, and introducing critic-independent attacks.

  • Robust Reinforcement Learning: Robust MDPs model worst-case perturbations in transition probabilities, whereas this paper perturbs state observations while leaving the true environment state unchanged.The related settings also include distributional and partially observed MDP extensions.
  • Adversarial Reinforcement Learning: Adversarial multi-agent RL models opponents or environmental interaction, while this work changes only the observation presented to the agent.Prior work extends Q-learning and policy-gradient methods to settings involving opponent actions or action-space perturbations.
  • Adversarial Attacks: Prior state-observation attacks include FGSM-based, value-guided, restricted-time-step, and generative attack strategies.The paper notes that critic-dependent attacks can be sub-optimal or impractical in many cases.
  • Improving Robustness: Earlier adversarial-training studies used weak or direct extensions and found adaptation challenging or achieved worse results than the proposed approach.The cited work spans discrete Atari tasks and continuous-control environments, with some evaluations targeting environmental changes rather than state perturbations.
  • Other Approaches: Other approaches use policy switching, path-finding-specific adversarial training, or worst-case rollouts without providing a general single-agent training procedure.These methods differ in scope and training assumptions from the paper’s approach.

3 Methodology

The paper models adversarial observation perturbations as an SA-MDP, establishes that optimal policies may require stochasticity or may not exist, and derives policy regularization and critic-independent attacks from this analysis.

  • 3.1 State-Adversarial Markov Decision Process (SA-MDP): SA-MDPs let an adversary replace the observed state ν(s) while the environment transitions from the unchanged true state s.The action is selected as π(a|ν(s)), so observation perturbations can induce suboptimal actions without altering the underlying dynamics.
  • 3.1 State-Adversarial Markov Decision Process (SA-MDP): The analysis assumes a stationary, deterministic, Markovian adversary whose perturbation lies in a bounded set B(s).The paper notes that non-Markovian and non-stationary adversaries are left for future work.
  • 3.1 State-Adversarial Markov Decision Process (SA-MDP): For a fixed policy, a Bellman contraction evaluates performance under the strongest adversary, while bounded action-distribution differences yield a performance-gap bound.This bound motivates regularizing DTV(π(·|s), π(·|ŝ)) between policies at true and perturbed states.
  • 3.1 State-Adversarial Markov Decision Process (SA-MDP): Under an optimal adversary, stochastic policies can outperform every deterministic policy, and an optimal policy may not exist.The paper attributes nonexistence to trade-offs that prevent one policy from maximizing values across all states.
  • 3.2 State-Adversarial DRL for Stochastic Policies: A Case Study on PPO: The proposed policy regularizer uses total variation or KL divergence across allowed perturbed states and is applied to PPO, DDPG, and DQN.For PPO, KL regularization over ŝ ∈ B(s) reduces an upper bound related to performance under the optimal adversary; the approach is grounded in SA-MDP theory rather than supervised-learning smoothness alone.
  • 3.5 Robust Sarsa (RS) and Maximal Action Difference (MAD) Attacks: The paper introduces critic-independent RS and MAD attacks because critic-based attacks can depend on critic quality or fail to represent observation perturbations correctly.MAD finds perturbed states by maximizing the KL divergence between action distributions at true and perturbed states.

4 Experiments

Experiments evaluate SA-based robustness for PPO, DDPG, and DQN under bounded observation perturbations and strong attacks. Across tasks, the robust methods improve attack robustness, while sometimes preserving or improving natural performance and enabling action-certification guarantees.

  • Evaluation setup: The experiments use ℓ∞ observation-perturbation balls, with normalized state representations in MuJoCo environments.The perturbation budget is denoted ϵ.
  • SA-PPO: Naive adversarial training deteriorates PPO performance and does not reliably improve robustness across Walker2d, Hopper, and Humanoid.Strong RS and MAD attacks achieve lower rewards than critic and random attacks, motivating stronger evaluation attacks.
  • SA-PPO: SA-PPO significantly improves robustness against strong attacks while matching vanilla PPO’s natural performance in Walker2d and Hopper.In Humanoid, SA-PPO significantly improves natural reward; the environment has a 376-dimensional state space and is difficult to train.
  • Evaluation protocol: Table 2 reports average episode rewards over 50 episodes for DDPG, adversarial training, and SA-DDPG, selecting agents by median best-attack reward across 11 runs.The table includes standard deviations and summarizes additional attack results elsewhere.
  • SA-DDPG: SA-DDPG improves natural agent performance in Ant and Walker2d, which have state dimensions 111 and 17, respectively.The comparison is reported as similar to the observations for SA-PPO.
  • Robustness certificates: Convex-relaxation training yields robustness certificates: SA-DQN can certify unchanged actions, while SA-DDPG reduces action-change upper bounds versus baselines.For Pong, unchanged actions across all rollout frames certify unchanged cumulative rewards under perturbation; certified rewards are generally unavailable for complex RL tasks.

Broader Impact

The paper frames robust reinforcement learning as important for trustworthy deployment beyond games, especially when sensing errors create a reality gap. It also acknowledges that the experiments use simulated environments and points to robustness as relevant to future real-world applications.

  • Motivation: Robustness receives less attention in reinforcement learning than in supervised learning, despite reinforcement learning’s continuing development.The paper presents this as a motivation for studying adversarial robustness.
  • Motivation: Sensor noise, measurement errors, and man-in-the-middle attacks are identified as realistic robustness settings for DRL systems.The paper connects robustness with trustworthiness before possible use in mission-critical tasks such as autonomous driving.
  • Limitations and broader impact: The reality gap can cause a simulation-trained reinforcement learning agent to fail in real-world experiments.Potential sensing errors are identified as one reason for this failure.
  • Limitations and broader impact: The experiments were conducted in simulated environments, although the authors believe the smoothness regularizer may also benefit real-world settings such as robot hand manipulation.This is presented as a potential extension rather than an experimentally established result.
  • Supporting material: The paper provides additional material on SA-MDP theory, adversarial attacks, optimization, algorithm details, and repeated-run empirical results.The empirical appendix reports at least 15 repetitions per experiment and improvements over baseline performance in some environments such as Humanoid.

A An example of SA-MDP

A three-state, two-action toy environment illustrates how state-observation adversaries can radically change policy values. The example contrasts an MDP-optimal deterministic policy, a state-invariant stochastic policy, and other deterministic policies under SA-MDP evaluation.

  • Environment: The toy environment has three states, two actions, specified transition probabilities and rewards, and an adversary that can perturb one state to either neighboring state.The example is used to compare ordinary MDP and SA-MDP behavior.
  • Policy representation: For γ = 0.99, stationary Markovian policies are parameterized by p11, p21, and p31, while V and Ṽ denote MDP and SA-MDP value functions.Each pij is the probability of taking action Aj in state Si.
  • MDP versus SA-MDP: The MDP-optimal policy p11 = 0, p21 = 1, p31 = 1 achieves V(S1) = V(S2) = V(S3) = 100.This policy takes A2 in S1 to receive reward and leave S1, then repeatedly takes A1 in S2 and S3.
  • MDP versus SA-MDP: Under the optimal adversary, that same policy has Ṽ(S1) = Ṽ(S2) = Ṽ(S3) = 0 because perturbations can consistently induce the wrong actions.The adversary maps observations among states so the policy receives no reward.
  • Stochastic policy: The uniform stochastic policy p11 = p21 = p31 = 0.5 gives V = Ṽ = 50 for every state because its action distribution is identical across states.Since the policy does not change under state perturbations, the adversary has no power in this example.
  • Value-function analysis: The value functions for SA-MDP are graphed for p11 = 0, 0.5, and 1.0 while p21 and p31 vary over [0, 1].Figures 6–8 are generated using Algorithm 1.

B Proofs for State-Adversarial Markov Decision Process

The SA-MDP analysis characterizes optimal adversaries through MDP-style Bellman methods, but reveals that standard optimal-policy properties can fail under adversarial observations.

  • Optimal adversaries: For a fixed policy, finding an optimal SA-MDP adversary is equivalent to finding an optimal policy in a constructed regular MDP.The constructed MDP uses perturbed observations as adversarial actions.
  • Optimal adversaries: An optimal adversary exists and can be chosen deterministic under the stated regularity conditions.Bellman contraction yields a unique fixed point for the adversarial value function when 0 < γ < 1.
  • Policy evaluation: Policy evaluation computes values under the optimal adversary for a fixed policy using an SA-MDP Bellman update.Algorithm 1 iterates minimization over admissible perturbations until convergence.
  • Optimal policies: A stochastic policy can outperform every deterministic policy in some SA-MDPs, so deterministic-policy dominance does not hold.The counterexample reports eight deterministic policies no better than a stochastic policy assigning 0.5 probabilities.
  • Optimal policies: Under an optimal adversary, an optimal policy need not exist, making classic optimality unsuitable for some SA-MDPs.The paper points to alternative optimality definitions or more complex policy classes as possible directions.
  • Regularizer optimization: SGLD-based inner maximization can provide only a lower bound because the policy objective is non-convex, and the bound may have a large gap from the true maximum.This limitation means minimizing the lower bound does not guarantee minimizing the original robust objective.

D.3 More details on the Robust Sarsa attack

The Robust Sarsa attack learns a critic with an explicit robustness objective, while MAD and hybrid attacks target action differences or critic values during state perturbation.

  • Robust Sarsa attack: Robust Sarsa jointly optimizes the ordinary temporal-difference loss and a robust objective that constrains value stability under perturbed actions.The robust critic is trained with convex relaxations for the inner maximization.
  • Robust Sarsa attack: The learned robust critic can support black-box attacks by minimizing its value over norm-constrained action perturbations.The critic can be learned from environment and agent observations without internal agent information.
  • Empirical comparison: The robust objective generally makes attacks more damaging than a non-robust Sarsa critic on PPO, SA-PPO, DDPG, and SA-DDPG comparisons.Tables 4 and 5 report mean reward and standard deviation over 50 attack episodes.
  • MAD and hybrid attacks: MAD directly maximizes KL-divergence with SGLD and usually outperforms random and critic attacks across many environments.The paper identifies MAD and Robust Sarsa as achieving the lowest attack rewards in many cases.
  • MAD and hybrid attacks: The hybrid attack combines robust-critic minimization with action-difference maximization, balancing the two losses through α_RS-MAD.Because critic magnitudes depend on environment rewards, α_RS-MAD is typically close to 1.
  • DQN attack: For DQN, untargeted PGD iteratively perturbs states to induce an action different from the original state’s maximizing action.Projection keeps the adversarial state within the specified norm ball.

E Robustness Certificates for Deep Reinforcement Learning

The paper interprets robustness certificates differently across action spaces: DQN can certify unchanged discrete actions, whereas PPO and DDPG certify bounded action changes.

  • Certificate interpretation: Convex relaxation can produce robustness certificates for DRL, but their interpretation differs from certificates in classification.The paper develops action-level and action-change guarantees rather than directly equating them with classification robustness.
  • DQN certificates: For DQN, a nonpositive upper bound on every competing Q-value difference certifies that the selected action remains unchanged throughout the perturbation set.Unchanged actions also preserve the current-step reward and transition in the DQN setting.
  • DQN certificates: 100% action certification can certify accumulated reward for specific initial conditions, but uncertified states require exploring all possible action branches.With n uncertified states and m possible actions each, worst-case evaluation requires n^m trajectories.
  • Certificate scope: Even a 100% certificate rate may not establish robustness for arbitrary starting conditions because random initialization makes all possible trajectories impractical to enumerate.The paper compares this scope boundary with certified classification evaluated on a specific test set.
  • Continuous-action certificates: For PPO and DDPG, certificates upper-bound action changes under norm-bounded state perturbations rather than proving unchanged actions.Small perturbation sets and small action-change bounds are desirable, but cumulative-reward certification remains difficult.
  • SA-PPO: SA-PPO adds a state-adversarial KL regularizer and gradually increases the perturbation budget during training.The regularizer can be solved using SGLD or neural-network convex relaxations.
  • SA-DDPG: SA-DDPG adds a loss term that upper-bounds the maximum policy-action change within the perturbation set.Its perturbation budget is likewise increased gradually during training.

H Additional Details for SA-DQN

SA-DQN augments standard DQN training with a state-adversarial regularizer that encourages action stability under observation perturbations, optimized with PGD or convex relaxations.

  • Algorithm: SA-DQN adds a regularizer encouraging the Q-network’s output to remain stable under state-observation perturbations.The regularizer may use hinge loss or alternative losses such as cross-entropy.
  • Algorithm: The algorithm computes standard DQN targets and TD loss before optimizing the additional state-adversarial regularizer.Target-network updates and replay-buffer training remain part of the procedure.
  • Regularizer optimization: PGD solves the regularizer by finding perturbed states where a competing action challenges the original maximizing action.The resulting hinge losses are included in the parameter update.
  • Training setup: The Atari network uses three convolutional layers followed by two fully connected layers, with prioritized replay and Huber TD loss.The reported architecture and training choices follow the cited DQN setup.
  • Training setup: SA-DQN uses the same network structure and general hyperparameters as DQN, with six million training steps in all environments.RoadRunner uses a longer target-network update interval for convergence under the short training schedule.
  • Training cost: Vanilla DQN, SA-DQN with SGLD, and SA-DQN with convex relaxation require roughly 15, 40, and 50 hours respectively on one 1080 Ti GPU.The paper notes that these runs use six million frames, versus 20 million frames in many published DQN results.
  • Training cost: Convex-relaxation training is slower than SGLD, while tighter relaxations improve accuracy but can increase computational demands.The paper identifies IBP+Backward as tighter than IBP and notes possible speedups through loss fusion.

I.1 More results on SA-PPO

SA-PPO improves reward distributions under strong attacks and maintains or improves training performance across repeated runs and perturbation budgets.

  • Rewards under the strongest attacks significantly improve for SA-PPO agents.The comparison includes median, 25th-percentile, and 75th-percentile cumulative rewards.
  • SA-PPO’s lowest attack rewards exceed PPO’s across all perturbation budget ϵ values.RS and RS+MAD are typically the strongest attacks in this evaluation.
  • SA-PPO consistently outperforms vanilla PPO in Humanoid across at least 15 training runs.The percentile ranges are reported to address variability across runs and avoid cherry-picking concerns.
  • SA-PPO has almost no performance drop compared to vanilla PPO in Hopper and Walker.

I.2 More results on SA-DDPG

Across five environments and repeated training runs, SA-DDPG consistently improves robustness under attack compared with vanilla DDPG.

  • SA-DDPG’s median, 25th-percentile, and 75th-percentile attack rewards are significantly and consistently better than vanilla DDPG’s across all five environments.The comparison aggregates results from 11 independently trained agents per method.
  • Attack evaluation reports the strongest attack as the lowest reward over all attacks.This selection is used because the lowest reward determines the agent’s true robustness.

I.3 Robustness Certificates

The paper evaluates robustness certificates for DQN, DDPG, and PPO under bounded state perturbations, showing unchanged actions or bounded policy changes.

  • SA-DQN certifies whether actions remain unchanged under bounded adversarial noise.The Action Cert. Rate is the ratio of actions guaranteed unchanged under any ℓ∞-bounded noise; some settings reach 100%.
  • SA-DDPG bounds every action output over perturbed states using convex relaxations of the neural network.The certificates measure average output range and ℓ2 distance; smaller ranges indicate tighter bounds.
  • SA-DDPG provides non-vacuous robustness certificates, whereas vanilla DDPG usually does not.The action space is normalized to [−1, 1], making 2 the worst-case output range.
  • SA-PPO upper-bounds KL-divergence between policies under state perturbations.Increasing regularization parameter κ can tighten the certificate at the cost of model performance.
  • The certificate results use interval bound propagation for SA-DDPG and SA-PPO, while vanilla agents use CROWN yet often remain vacuous.
Loading 2003.08938v7…