Robust Reinforcement Learning on State Observations with Learned Optimal Adversary

Huan Zhang; Hongge Chen; Duane Boning; Cho-Jui Hsieh

Source-linked AI summary

Robust Reinforcement Learning on State Observations with Learned Optimal Adversary

Huan Zhang, Hongge Chen, Duane Boning, Cho-Jui Hsieh

arXiv:2101.08452v1Jan 21, 2021cs.LGcs.AIstat.ML

TL;DR

The paper studies robustness when adversaries or noise perturb state observations without changing the true environment state. It learns optimal observation adversaries and trains agents against them with ATLA, finding strong adversarial attacks and state-of-the-art robustness in continuous-control evaluations.

Problem
RL agents can remain vulnerable to perturbed state observations even without function-approximation errors, motivating robustness methods for adversarial attacks and sensing noise.
Method
ATLA alternates online policy-gradient training of an agent and a learned adversary within the SA-MDP framework, using history-dependent LSTM policies when useful.
Results
The learned adversary is significantly stronger than previous attacks, while ATLA achieves state-of-the-art robustness under strong adversarial attacks.
Takeaways & Limitations
History-dependent LSTM policies can improve robustness, and ATLA can be combined with state-adversarial regularization.
Takeaways & Limitations
Increasing SA-PPO’s function-approximator regularization does not always match ATLA and can substantially reduce natural reward at large regularization.

Abstract

from arXiv · show

We study the robustness of reinforcement learning (RL) with adversarially perturbed state observations, which aligns with the setting of many adversarial attacks to deep reinforcement learning (DRL) and is also important for rolling out real-world RL agent under unpredictable sensing noise. With a fixed agent policy, we demonstrate that an optimal adversary to perturb state observations can be found, which is guaranteed to obtain the worst case agent reward. For DRL settings, this leads to a novel empirical adversarial attack to RL agents via a learned adversary that is much stronger than previous ones. To enhance the robustness of an agent, we propose a framework of alternating training with learned adversaries (ATLA), which trains an adversary online together with the agent using policy gradient following the optimal adversarial attack framework. Additionally, inspired by the analysis of state-adversarial Markov decision process (SA-MDP), we show that past states and actions (history) can be useful for learning a robust agent, and we empirically find a LSTM based policy can be more robust under adversaries. Empirical evaluations on a few continuous control environments show that ATLA achieves state-of-the-art performance under strong adversaries. Our code is available at https://github.com/huanzhang12/ATLA_robust_RL.

1 INTRODUCTION

The introduction distinguishes neural-network approximation errors from intrinsic policy weakness as sources of vulnerability to perturbed state observations. It motivates learning stronger adversaries and training robust policies with ATLA, alongside history-dependent policies and existing regularization.

State-observation attacks threaten DRL agents in safety-critical applications by adding small observation noises that make the observed environment differ from the true environment.
Policy vulnerability can persist without function approximation errors, because perturbing a state observation to a neighboring state may prevent a tabular-policy agent from reaching its goal.
Robust RL therefore requires both more robust function approximators and policies that account for observation perturbations.
Existing defenses improve function-approximator robustness through certified defenses or policy-smoothness regularization around observation neighborhoods.
ATLA instead learns an adversary online with the agent to address intrinsic policy weakness, while its learned adversary is reported as stronger than previous attacks.
The paper evaluates ATLA on four continuous control environments and reports that combining it with explicit regularization achieves state-of-the-art results.

2 RELATED WORK

Related work covers robustness to state-observation attacks, online adversary learning, adversarial training, and other perturbation or reward-manipulation settings. These studies differ in whether they perturb observations, environment dynamics, actions, or training signals.

SA-MDP models adversarial or natural noise in state observations while leaving the true environment state unchanged, unlike RMDP models that perturb transition probabilities.
Online adversary-learning methods such as RARL alter environment states through adversary actions and target robustness to parameter changes including mass, length, or friction.
Prior adversarial-training approaches place gradient-basedly perturbed observations into replay buffers, but are reported as unsuccessful in some Atari settings where function-approximator weakness dominates.
Other robustness research studies action perturbations, while policy teaching and poisoning manipulate rewards or costs during training to induce desired policies.

3 METHODOLOGY

The methodology formulates observation attacks as SA-MDPs, derives optimal adversaries for fixed policies, and extends this framework to learned adversaries and robust policy training. It also uses history-dependent policies to address the difficulty of solving robust POMDP policies directly.

SA-MDP formulation: SA-MDP keeps the true environment state unchanged while an adversary perturbs the state observation within a bounded set before the agent acts.The adversary samples a perturbed state ˆs from ν(·|s), with its support constrained by B(s).
Finding the optimal adversary: For a fixed stationary policy, the agent and environment can be merged into a redefined MDP whose optimal policy is the SA-MDP’s optimal adversary.The adversary receives the negative of the agent’s reward, while a large negative constant prevents actions outside B(s).
Finding the optimal adversary: In GridWorld, the learned adversary repeatedly drives the agent into traps, producing −∞ reward for the agent and +∞ reward for the adversary.Without an adversary, policy iteration reaches the target and earns +1; the adversary perturbs observations only to adjacent cells.
Learned adversaries: In deep RL, a neural adversary is trained with standard DRL algorithms to approximate the optimal attack, without requiring gradient access to the victim policy.The attack learns perturbation vectors and projects the resulting observations into B(s).
Learned adversaries: The learned attack can induce opposite-direction motion and large negative rewards, whereas Robust Sarsa can make agents stop with small positive rewards.The learned adversary is optimized online against the agent and can further reduce rewards of robustly trained SA-PPO agents.
Alternating training with learned adversaries: ATLA alternates policy and adversary optimization, requiring the policy to defeat a strong learned adversary rather than relying on explicit function-approximator regularization.Because optimal POMDP solutions are difficult in modern DRL environments, the policy uses recurrent policy gradients with LSTM function approximators over state-action history.

4 EXPERIMENTS

Experiments evaluate learned optimal attacks and ATLA across MuJoCo continuous-control environments, comparing them with PPO, SA-PPO, and other adversarial baselines. ATLA improves robustness under strong attacks, while combining ATLA with state-adversarial regularization performs best broadly, especially in high-dimensional settings.

Optimal attack evaluation: PPO trains adversaries with model-free RL in four OpenAI Gym MuJoCo continuous-control environments.The implementation uses PPO to approximate the optimal attack when environment dynamics are unknown.
Optimal attack evaluation: The optimal attack achieves lower attack rewards than the other evaluated attacks across all environments, sometimes by a large margin.Table 1 compares random noise, critic, MAD, RS, Snooping, and optimal attacks on PPO and SA-PPO agents.
ATLA evaluation: ATLA-PPO outperforms SA-PPO on Hopper and Walker2d and remains competitive on HalfCheetah, while SA-PPO is more effective on high-dimensional Ant for MLP policies.The comparison reports natural rewards and worst-case attack rewards under six attack types.
ATLA evaluation: LSTM policies improve robustness under ATLA, whereas LSTM policies trained without robust training do not improve robustness over vanilla training.The results support using historical observations together with learned adversarial training.
ATLA evaluation: ATLA-PPO with state-adversarial regularization outperforms all other baselines in all settings, particularly high-dimensional Ant.The combined method targets both intrinsic policy robustness and function-approximator robustness.
Regularization analysis: Increasing SA-PPO regularization cannot match ATLA performance, and at κ = 1.0 its natural reward falls to 1436 ± 96.The figure and discussion indicate that stronger function-approximator regularization can substantially reduce natural performance.

5 CONCLUSION

The paper proposes optimal learned adversarial attacks and ATLA to improve robustness against strong state-observation attacks, while highlighting LSTM history dependence and compatibility with existing regularization.

The optimal adversarial attack on state observations is significantly stronger than many existing adversarial attacks.
ATLA trains an agent together with a learned optimal adversary to improve robustness under attacks.
A history-dependent LSTM policy can be helpful for robustness.
ATLA is orthogonal to existing regularization techniques and can be combined with state-adversarial regularization for robustness under strong attacks.

A.1 FULL RESULTS OF ALL ENVIRONMENTS UNDER DIFFERENT TYPES OF ATTACKS

The full-results evaluation compares multiple agents and attack methods using episode rewards, with reported attack results selected to emphasize the strongest attacks.

Table 2 includes the best attack rewards, defined as the lowest rewards over all attacks.
The RS attack result for each agent is the strongest result among 30 trained robust value functions.
Table 3 reports average episode rewards and standard deviations over 50 episodes for five baselines and SA-PPO.
The evaluation includes natural rewards and rewards under six adversarial attacks, including random noise, critic, MAD, RS, Snooping, and optimal attacks.

A.2 AGENT PERFORMANCE DURING TRAINING

Training-time evaluation examines agent performance at multiple checkpoints and reports increasing robustness under Robust Sarsa attacks as training progresses.

Agent performance is evaluated at 20%, 40%, 60%, and 80% of total training epochs using Robust Sarsa attacks.
The overall trend is increasing RS attack rewards, indicating better robustness at later training checkpoints.
Table 4 reports natural and RS attack rewards during training.

A.3 NETWORK STRUCTURE

The experiments use compact fully connected or LSTM architectures for both agents and adversaries, with LSTM adversaries used for optimal attacks against LSTM agents.

Fully connected policy and value networks use two hidden layers with 64 hidden neurons per layer for both the agent and adversary.
LSTM agents use one 64-neuron LSTM layer, a 64-dimensional input embedding, and an output layer mapping 64 dimensions to the output dimension.
For optimal attacks against LSTM agents, the adversary also uses an LSTM network to remain sufficiently powerful.

A.4 HYPERPARAMETER FOR THE LEARNING-BASED “OPTIMAL” ATTACK

The learning-based “optimal” attack requires tuning policy-gradient adversary hyperparameters during agent testing. These include optimizer rates, entropy regularization, and learning-rate annealing.

Testing the “optimal” attack learns an adversary with policy-gradient methods while selecting attack hyperparameters.The search includes the adversary’s policy and value networks, entropy coefficient, and learning-rate annealing.

A.5 HYPERPARAMETERS FOR ATLA PERFORMANCE EVALUATION

The evaluation uses tuned or inherited hyperparameters across vanilla PPO, LSTM PPO, SA-PPO, and ATLA-PPO. Agents are repeatedly trained and selected by median robustness before final optimal-attack evaluation.

Hyperparameters for PPO (vanilla): Vanilla PPO reuses published hyperparameters for Hopper and Walker2d, while other environments use grid searches over learning and entropy parameters.The searched values include policy-network learning rate, value-network learning rate, and entropy bonus coefficient.
Hyperparameters for PPO (LSTM): PPO (LSTM) uses a smaller search near the vanilla PPO settings and trains for the same number of steps.
Hyperparameters for SA-PPO: SA-PPO retains vanilla hyperparameters except κ, selecting κ from 1×10^-6 to 1 by median worst-case reward across 21 training runs.The selected value is the one whose median agent achieves the highest worst-case reward under all attacks.
Hyperparameters for ATLA-PPO: ATLA-PPO retains vanilla agent settings except entropy bonus, which may be increased to support exploration during adversarial training.Adversary settings are separately searched over policy and value learning rates and adversary entropy bonus, using values near the agent’s settings.
Agent selection: Each setup is repeated 21 times, agents are ranked by their lowest reward across four attacks, and the median-robustness agent receives the final “optimal” attack.The ranking attacks are random, critic, MAD, and RS.