Source-linked AI summary
Robust Deep Reinforcement Learning with Adversarial Attacks
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, Girish Chowdhary
TL;DR
The paper addresses the vulnerability of deep reinforcement learning to adversarial attacks and parameter uncertainties. It engineers RL-specific attacks and uses them in adversarial training, finding degraded DRL performance under attack and improved robustness across parameter variations. The approach is evaluated with DDQN and DDPG in continuous-domain benchmarks.
Problem
DRL algorithms can be fooled by adversarial attacks, while robustness to real-world parameter variations is important for robotics and safety-critical systems.
Method
The paper designs an RL-specific objective and gradient-based adversarial attacks, then trains DDQN and DDPG agents in their presence within a robust-control framework.
Results
Adversarial attacks degrade DRL performance, while adversarial training produces robust performance across parameter variations for DDPG and DDQN.
Takeaways & Limitations
The proposed attacks expose robustness issues in DRL and can be leveraged to train policies that remain high-performing over uncertain dynamics.
Takeaways & Limitations
The attack formulation assumes an agent is sufficiently trained for its Q values to be close to optimal, an assumption not generally proved for nonlinear deep-network approximators.
Abstract
from arXiv · showhide
This paper proposes adversarial attacks for Reinforcement Learning (RL) and then improves the robustness of Deep Reinforcement Learning algorithms (DRL) to parameter uncertainties with the help of these attacks. We show that even a naively engineered attack successfully degrades the performance of DRL algorithm. We further improve the attack using gradient information of an engineered loss function which leads to further degradation in performance. These attacks are then leveraged during training to improve the robustness of RL within robust control framework. We show that this adversarial training of DRL algorithms like Deep Double Q learning and Deep Deterministic Policy Gradients leads to significant increase in robustness to parameter variations for RL benchmarks such as Cart-pole, Mountain Car, Hopper and Half Cheetah environment.
1 Introduction
The paper examines robustness problems in deep reinforcement learning under adversarial attacks and real-world parameter variations. It proposes adversarial training to improve performance across uncertain dynamics while avoiding the conservatism associated with robust control.
- Deep neural networks address high-dimensional reinforcement learning tasks but can be fooled by adversarial input perturbations.The paper motivates robustness particularly for robotics and safety-critical systems, where adversarial noise may create hazardous situations.
- Adversarial training of DRL agents improves robustness and policy performance across parameter and dynamical uncertainties such as friction and mass.The approach trains agents with engineered attacks and seeks high-performing policies over uncertain parameters rather than conservative robust-control policies.
- The paper proposes an RL-specific objective whose optimization degrades RL performance and uses its minima to induce worst-action behavior.The attack is designed to make the agent believe it occupies a state whose optimal action is worst for the actual state.
- The proposed work extends adversarial attacks beyond image-based RL environments to reinforcement-learning settings that are not restricted to high-dimensional pixel inputs.The paper contrasts its objective with a prior heuristic minimizing the control-input norm, which it says does not ensure an optimal attack.
- Prior robust RL approaches sample model variations or seek worst-performing parameter settings, but trajectory sampling may be risky or difficult with many parameters.The related work also discusses risk-sensitive RL, whose methods are reported not to scale well with state and action complexity.
2 Background
The background reviews adversarial attacks on image classifiers and their extensions to image-based DRL, then introduces DQN, RBF-based Q learning, and DDPG. These methods provide the algorithmic context for the paper’s attacks and robustness experiments.
- Adversarial attacks: Fast signed gradient method constructs a loss-based perturbation intended to increase the probability of incorrect classification.The method uses a linear approximation of the deep-learning model and is extended to nonlinear multilayer networks.
- Adversarial attacks: The adversarial perturbation is constrained by an l∞ norm with bound ϵ.
- Adversarial attacks: The classifier loss J(θ, x, y) is used during training or testing; for images, it is the cross-entropy loss between true labels and predicted label distributions.
- Deep RL algorithms: DQN uses a deep neural network to approximate the Q function and stabilizes learning with experience replay and a target network.Q values represent the quality of contemplated actions, while temporal-difference updates support long-run return maximization.
- Deep RL algorithms: RBF-based Q learning replaces the deep network with a Gaussian-kernel radial basis function network and typically omits experience replay and target networks.Learning proceeds through stochastic gradient descent using temporal-difference error.
- Deep RL algorithms: DDPG combines an actor that generates policies with a critic that evaluates them, updating both networks by gradient descent.The critic uses experience replay and a target network for its updates.
3 Method
The method defines observation perturbations that steer value-based RL agents toward poor actions, then uses adversarial training to improve robustness to model uncertainty. It includes naive sampling attacks, gradient-based objectives, and a robust-control interpretation linking attack magnitude to worst-return percentiles.
- Adversarial attack: Adversarial attacks corrupt the agent’s current-state observation so it believes it is in a state whose policy produces a sub-optimal action for the actual state.The paper defines the worst action as the action with the least Q value in that state.
- Adversarial attack: The attack definition applies to value-function-based algorithms and assumes the agent is sufficiently trained for Q values to be close to optimal.The paper notes that this assumption is established for tabular Q learning asymptotically but not generally for nonlinear DNN approximators.
- Adversarial training: The attack noise is bounded by an l2 norm constraint, while states and attack magnitudes are normalized.The paper states that other norm constraints are possible.
- Naive attack: Naive attacks sample bounded noise, add it to the current state, select the action induced by each perturbed state, and retain a perturbation whose action has lower value than the best action.For DDQN, the Q network selects the perturbed-state action and the target network evaluates that action at the current state; DDPG analogously uses actor and critic networks.
- Gradient-based attack: The gradient-based attack minimizes a reinforcement-learning-specific objective designed to find the worst possible action, rather than merely reducing the probability of the best action.The objective is described as the cross-entropy loss between an adversarial probability distribution and the optimal policy; Q values can be converted to probabilities with softmax.
- Adversarial training: Adversarial training uses the gradient-based attack, which performed best among the attacks, to train DDQN and DDPG agents against model uncertainty.The approach directly fools the agent into sampling worst trajectories, and larger attack magnitudes correspond to optimization over higher worst-return percentiles.
4 Results
The proposed attacks degrade DRL performance, while gradient-based attacks are strongest among the tested methods. Adversarial training improves DDQN and DDPG performance across parameter variations in several OpenAI Gym environments.
- Adversarial Attack: Gradient Based (GB) attacks outperform Naive Sampling (NS), Stochastic Gradient Descent (SGD), and HFSGM attacks in the tested environments.
- Adversarial Attack: Adversarial attacks degrade deep learning-based algorithms, while RBF-based Q learning is relatively more resilient than DDQN.The paper attributes this difference to smoother RBF interpolation versus piecewise-linear deep-network approximations.
- Robust Training: Adversarial training evaluates agents across wide parameter ranges after vanilla DDQN and DDPG agents are trained on default parameters.The robust agents are then trained using gradient-based attacks on the same parameters.
- Robust Training: Robust DDQN and Robust DDPG achieve higher returns across parameter variations in Cart-pole, Mountain Car, Hopper, and Half-Cheetah.The results use average returns over 100 episodes for each parameter set and four random seeds.
- Robust Training: The robust-training comparison uses the same number of training steps for vanilla and adversarially trained algorithms.Thus, the reported improvement is not attributed to additional training.
5 Conclusion
The paper concludes that adversarial attacks expose vulnerabilities in DRL and can be used to train policies that remain robust across parameter variations. It identifies a theoretical relationship between attacks and robustness as future work.
- The proposed attacks can fool DRL more easily than RBF-network-based reinforcement learning algorithms.
- Adversarially trained DDQN and DDPG achieve robust performance across parameter variations.
- Future work includes establishing a theoretical relationship between adversarial attacks and robustness to parameter variation.
A Experimental setup
The experimental setup specifies separate DDQN network architectures and training settings for Cart-pole and Mountain Car.
- Cart-pole DDQN uses three hidden layers of 16 ReLU units, while Mountain Car uses two hidden layers of 100 ReLU units.
- Both DDQN environments use a discount factor of 0.99 and target-network update rate of 10^-2.
- The networks use Adam optimization with a learning rate of 10^-3.
- Cart-pole training runs for 50,000 timesteps, while the Mountain Car training duration is only partially specified in the passage.
A.2 RBF
The RBF Q-learning setup discretizes state dimensions into uniformly distributed bins and uses environment-specific training settings for Cart-pole and Mountain Car.
- RBF: Cart-pole divides each state dimension into three bins with uniformly distributed centroids.
- RBF: RBF Q learning uses a discount factor of 0.99 for both Cart-pole and Mountain Car.
- RBF: Cart-pole uses a 0.001 learning rate and 40,000 training steps, whereas Mountain Car uses 0.01 and 60,000 steps.
A.3 DDPG
The DDPG setup uses specified actor–critic architectures, training horizons, discounting, and learning rates for Hopper and Half Cheetah.
- Hopper and Half Cheetah use actor and critic networks with two hidden layers of 400 and 300 ReLU units.
- The Hopper agent is trained for 1 million time steps, while Half Cheetah is trained for 2 million time steps.
- Both environments use a discount factor of 0.99, critic learning rate of 10^-3, and actor learning rate of 10^-4.
A.4 Adversarial Training
Adversarial training retrains vanilla networks with sampled attacks while matching the vanilla and adversarial models on total training timesteps.
- Adversarial training samples attacks 200 times before adversarially retraining the vanilla network.
- The adversarial magnitude is 0.05 for Half Cheetah and 0.03 for Hopper.
- The sampling frequency is 100, and vanilla and adversarially trained networks are compared after exactly the same number of timesteps.
B Robust Training Colormap for Cartpole
Figure 4 compares average episode returns for vanilla and adversarially trained DDQN agents across cart-pole parameter variations.
- Subfigure (a) shows average return per episode for DDQN across variations in cart mass and pole length.
- Subfigure (b) shows the corresponding average returns for an adversarially trained DDQN agent.
- The caption reports significant improvement in return across different parameter settings for the adversarially trained agent.