Source-linked AI summary
Robust Adversarial Reinforcement Learning
Lerrel Pinto, James Davidson, Rahul Sukthankar, Abhinav Gupta
TL;DR
RL policies often struggle to transfer from simulation or limited training scenarios because modeling errors and changed conditions create disturbances. RARL jointly trains a task-solving protagonist against a reinforced destabilizing adversary in a zero-sum game, and experiments report improved robustness and generalization across environments. The paper also identifies computational difficulty in exact Nash-equilibrium solutions and omits Swimmer evaluation for changed mass or friction because that task is not significantly affected.
Problem
RL policy learning can fail to generalize because simulation-to-reality gaps, scarce real-world data, and changed training-test conditions expose modeling and transfer weaknesses.
Method
RARL jointly trains a protagonist and a reinforced adversary that applies destabilizing forces, framing their interaction as a two-player zero-sum Markov game.
Results
Across multiple OpenAI Gym environments, RARL is reported to improve robustness to training initializations, environmental changes, and disturbances that are difficult to model during training.
Takeaways & Limitations
The adversary samples hard, low-reward trajectories, leading to a more robust control strategy that transfers across changed conditions and can remain effective without the adversary.
Takeaways & Limitations
Exact Nash-equilibrium solutions can be computationally prohibitive because greedy minimax solving has exponential complexity in action-space cardinality.
Abstract
from arXiv · showhide
Deep neural networks coupled with fast simulation and improved computation have led to recent successes in the field of reinforcement learning (RL). However, most current RL-based approaches fail to generalize since: (a) the gap between simulation and real world is so large that policy-learning approaches fail to transfer; (b) even if policy learning is done in real world, the data scarcity leads to failed generalization from training to test scenarios (e.g., due to different friction or object masses). Inspired from H-infinity control methods, we note that both modeling errors and differences in training and test scenarios can be viewed as extra forces/disturbances in the system. This paper proposes the idea of robust adversarial reinforcement learning (RARL), where we train an agent to operate in the presence of a destabilizing adversary that applies disturbance forces to the system. The jointly trained adversary is reinforced -- that is, it learns an optimal destabilization policy. We formulate the policy learning as a zero-sum, minimax objective function. Extensive experiments in multiple environments (InvertedPendulum, HalfCheetah, Swimmer, Hopper and Walker2d) conclusively demonstrate that our method (a) improves training stability; (b) is robust to differences in training/test conditions; and c) outperform the baseline even in the absence of the adversary.
1. Introduction
RARL addresses poor transfer, limited real-world data, and unstable policy learning by modeling uncertainty as adversarial disturbances. It jointly trains a protagonist for task performance and an adversary that generates destabilizing forces, improving robustness across conditions.
- Real-world policy learning is expensive, dangerous, and data-scarce, so policies trained on limited scenarios can overfit and fail under changed friction or other conditions.
- Simulation-trained policies can fail to transfer because modeling errors create a reality gap, while stochastic algorithms can produce unstable learning across runs.
- RARL models uncertainty and training-test differences as extra forces or disturbances, such as forces representing high friction at contact points.
- RARL jointly trains a protagonist to achieve task goals and a reinforced adversary to impede it with destabilizing forces, sampling hard failure-inducing examples.
- Experiments across InvertedPendulum, HalfCheetah, Swimmer, Hopper, and Walker2d report greater robustness to initializations, modeling errors, and changed mass or friction settings.
- The adversary can exploit domain knowledge by targeting the protagonist’s weak points and using simulated-environment capabilities unavailable through symmetric action spaces.
2. Background
The paper frames standard reinforcement learning with continuous-state MDPs and discounted stochastic policies. RARL extends this setting to a two-player zero-sum Markov game in which one player maximizes reward and the other minimizes it.
- A continuous-space MDP is represented by states, actions, transition probabilities, rewards, a discount factor, and an initial-state distribution.
- Batch policy algorithms learn a stochastic policy that maps states to actions while maximizing cumulative discounted reward.
- RARL formulates its adversarial setting as a two-player discounted zero-sum Markov game with separate continuous action spaces and shared reward structure.
- In the zero-sum formulation, the protagonist maximizes discounted reward while the adversary minimizes it.
3. Robust Adversarial RL
RARL learns a protagonist policy that maximizes reward while remaining robust across uncertain transition functions and adverse trajectories. It uses an adversarial zero-sum game with alternating policy optimization to approximate robust control without repeatedly solving costly equilibrium problems.
- Robust objective: RARL optimizes protagonist parameters for reward while accounting for modeling errors and differences between training and test transition functions.The formulation treats the transition function as uncertain rather than fixed, targeting policies that perform well across possible dynamics.
- Robust objective: CVaR replaces mean-performance optimization to focus learning on low-reward, worst-percentile trajectories.The method uses the α-quantile of return values to target adverse outcomes rather than assuming a known distribution over model parameters.
- Adversarial formulation: An adversary applies forces that minimize the protagonist’s reward, thereby generating difficult trajectories for robust protagonist learning.The adversarial agent is trained rather than kept fixed, because a fixed adversary could encourage the protagonist to overfit to its actions.
- Adversarial formulation: The protagonist and adversary form a zero-sum Markov game in which the adversary maximizes the negative of the protagonist’s reward.Both players observe the state and act at each timestep; the protagonist maximizes its reward while the adversary optimizes the opposing objective.
- Optimization procedure: Exact Nash-equilibrium computation can be prohibitive because greedy solutions may require exponential complexity in the action-space cardinality.RARL instead approximates the advantage function while learning stationary policies, avoiding costly minimax optimization at each iteration.
- Optimization procedure: RARL alternates optimization: it updates the protagonist with the adversary fixed, then updates the adversary with the protagonist fixed.This sequence repeats for a prescribed number of iterations and is intended to learn stationary policies while avoiding equilibrium computation at every iteration.
4. Experimental Evaluation
RARL is evaluated across multiple MuJoCo control tasks for training stability, adversarial robustness, and generalization to changed physical conditions. Across these evaluations, RARL generally outperforms TRPO and remains more robust under disturbances and parameter changes.
- Learned-policy evaluation: Across four reported tasks, RARL learns higher-mean-reward policies with lower variance than TRPO, including tests without disturbances or changed settings.InvertedPendulum is omitted from the corresponding graph because both methods show similar performance there.
- Training robustness: Percentile curves show significant training-robustness gains for RARL on HalfCheetah, Swimmer, and Hopper across multiple initializations.The percentile plots sort cumulative final rewards from policies trained with different initializations.
- Adversarial disturbances: Under a learned adversarial disturbance applied at test time, RARL performs better than the baseline because its policy was trained against similar adversaries.The adversary is learned after policies are trained and then used to disturb the fixed protagonist policy during testing.
- Changing mass: When test-time mass differs from training, RARL policies generalize significantly better across the evaluated tasks.Mass is varied for the pendulum in InvertedPendulum and for the torso in the other tasks; Swimmer is omitted because its policy is not significantly affected by mass or friction changes.
- Changing friction and combined conditions: With changed friction, baseline performance falls significantly, whereas RARL remains more resilient; RARL also yields higher rewards for most joint mass-and-friction settings.These evaluations use standard training parameters and vary the physical conditions only at test time.
5. Related Research
Related work addresses robustness in reinforcement learning through worst-case, risk-based, adaptive, and adversarial formulations. RARL is positioned as a continuous-space, model-free worst-case approach that uses an adversarial agent to represent disturbances.
- Motivation: Deep RL policies often require substantial data and can produce unstable policies, motivating robust and sample-efficient learning.These concerns are especially relevant for practical applications such as robotics.
- Worst-case formulations: Worst-case RL treats uncertainty as an adversary in a continuous dynamic zero-sum game and seeks a minimax reward solution.The paper connects this formulation to H-infinity optimal control and prior robust RL work.
- Related approaches: Prior approaches include minimax dynamic programming, Markov games, probabilistic guarantees for unknown parameters, and percentile-risk formulations.These methods span tabular, probabilistic, and risk-based treatments of uncertainty.
- Adversarial learning: Adversarial learning has also been applied to robust classifiers, generative models, and self-supervised robot grasping.These examples use adversarial objectives or agents outside the continuous-control formulation emphasized here.
- Adaptive and model-based robustness: Recent deep RL methods address uncertainty through recurrent adaptive control, online parameter identification, or worst-case trajectories sampled from parameterized models.RARL instead focuses on continuous-space, model-free control without an explicit parametric formulation of model uncertainty.
- Robustness evaluations: RARL evaluates robustness to changing friction and jointly changing friction and mass, excluding tasks where friction is not relevant.The friction evaluation omits InvertedPendulum and Swimmer, while the joint variation evaluation reports increased robustness for Hopper and HalfCheetah.
6. Conclusion
The paper concludes that RARL improves robustness across training initializations, environmental changes, and difficult-to-model test disturbances. Its central mechanism is a reinforced adversary that applies disturbances and samples hard trajectories for robust control learning.
- Conclusion: RARL is reported to be robust to training initializations and to environmental changes between training and test conditions.The conclusion also states robustness to test disturbances that are hard to model during training.
- Conclusion: Visualizations show adversarial forces applied to the InvertedPendulum and Hopper in distinct states and contact conditions.The InvertedPendulum visualization contrasts stationary and moving-cart states; the Hopper visualization contrasts an airborne foot with ground interaction.
- Conclusion: RARL models uncertainty as system disturbances generated by an adversary rather than relying on a fixed adversarial policy.The adversary learns an optimal policy intended to thwart the protagonist.
- Conclusion: The adversary effectively samples hard examples with the worst rewards, leading to a more robust control strategy for the protagonist.This connects adversarial training to the generation of difficult trajectories for the protagonist.